Guidelines for Automation framework design

Introduction
A Framework defines a set of guidelines for all phases of test automation: Requirement Analysis, Script Design, Execution, Reporting and maintenance. A framework can be a wrapper around some complex internal architecture which makes it easy to use for the end user. It also enforces a set of standards for implementation

Problem Statement
There is no standard set of guidelines available on developing a framework and what all considerations need to be taken during the development of the same. There are different white papers which go over types of framework and how they work. But none of them defines what all factors go in to the design of the same
Design guidelines
This paper covers different aspect of a framework and key features it needs to have based on the requirements.

1. Selection of a framework – Different types of frameworks that exist are:
   • Data Driven framework – Used when flow of the application remains constant, only the data changes. The data is provided by external medium e.g. – excel sheet, XML etc…
   • Keyword driven framework – This framework provides generic keywords that can be used with any type of application. It also provides abstraction from the type of automation tool used and type of being application tested, e.g. – it can test a similar Web and Windows application with the same test case
   • Hybrid framework – A hybrid framework is the one which takes advantages from both Data Driven and keyword driven frameworks. These frameworks do not implement generic keywords but implement business logic keywords based on the application being tested. For ex – Login, Logout could be application specific keyword that can be used.
2. Don’t reinvent the wheel – A framework should try and use the power of the automation tool rather than re-defining the whole language by implementing new keywords. Developing a keyword driven framework is time consuming and costly. A Hybrid framework can be developed in a shorter time period and with less cost.
3. Reusability -The framework should allow highest possible reusability. Combining individual actions into business logic provides re-usability. E.g. – Combing actions like “Enter username”, “Enter password” and “Click Login” into one re-usable component “Login”
4. Support of different application versions -A framework should allow re-use of baselines scripts in case different versions/flavors of an applications are to be tested. There are two different ways to support different applications
   • Copy and Modify – This method involves creating copies of the baseline scripts and modifying them for a specific application version
   • Re-use and Upgrade – This method involves re-using baseline script and providing a upgrade code for specific version of application. This ensures maximum re-usability and should be preferred.
5. Support of script versioning – Scripts should be stored in a version control system like CVS, Microsoft® VSS etc…This ensures recovery from any disaster.
6. Different environment for development and production – Automation should be considered as any other development project. Test scripts should be created and debugged in Test environment. Once tested then only should be deployed to the production environment. This holds true for any emergency releases also
7. Externally Configurable – Configurable items of a script should be kept in an external file. This would contain configuration like Application URL, version, path etc…This allows running the same script against different environment. Ensure that location of the configuration file is not hard coded. Hard coded files would allow running on any environment but only one at a time. Keeping the configuration relative to current test path allows overcoming this limitation
8. Self configurable – Ideally a framework should be self configurable. Once deployed to a system, no manual configuration changes should be required and scripts should automatically configure the required settings
9. Minimal changes required for any object changes -Most common issues faced during automation are object identification changes. Framework should be able to patch such changes easily. This can be achieved by storing all object identification settings at a shared location. This could be an external XML file, excel file, database or automation proprietary format. There are two possible way to load this object identification configuration
   • Static – In this all the object definitions are loaded into the memory at the start of the test. Any changes made to object definition can only be loaded by stopping and re-running the test
   • Dynamic –Object definition is pulled as per request. This approach is a bit slow as compared to the static one. But in case of huge scripts where the fix needs to be made at run-time this is suitable.
10. Execution – Framework might need to cater to below requirements (on need bases)
    • Execution of a individual test case
    • Execution of a test batch (combination of tests)
    • Re-execution of only failed test cases
    • Execution of a test case/test batch based on result of another test case/test batch
There could be many other needs based on the project requirement. A framework might not implement all of them, but should be flexible enough to accommodate such requirements in future
11. Status monitoring – A framework should allow monitoring the execution status in real time and should be capable of sending alerts in case of failure. This ensures quick turnaround time in event of a failure
12. Reporting – Different applications have different reporting needs. Some require combined results for a test batch and some require individual level test report for each test case in test batch. The framework should be flexible enough to generate required reports
13. Minimum dependency on Automation tool for changes – Some fixes can only be made by opening the script in the automation tool and then saving it. Scripts should be developed in such a way that modification is possible even without the unavailability of the automation tool. This deflates company cost by reducing the number of licenses required. It also allows anyone to make changes to the script without having the need to setup the tool
14. Easy debugging -Debugging takes a lot of time during automation and hence special care needs to be taken for this part. Keyword driven frameworks which use external data source (like a excel spread sheet) to read scripts keywords and process the same are difficult to debug.
15. Logging – Log generation is important part of execution. It is very important to generate debug information at various points in a test case. This information can help find problem area quickly and reduce the time to make a fix at the same time
16. Easy to Use – The framework should be easy to learn and use. It is time consuming and costly to train a resource on a framework. A well documented framework is easier to understand and implement
17. Flexible – Framework should be flexible enough to accommodate any enhancements without impacting existing test cases
18. Performance impacts – A framework should also consider the performance impacts of the implementation. A complex framework which increases the load time or execution time of scripts is never desirable. Techniques like caching, compiling all code into single library while execution etc… should be used to improve performance whenever possible
19. Framework Support Tools – External Tools can be developed to perform tasks that help in framework design. Some example tasks would be
    • Uploading scripts from local folder to HP Quality Center
    • Associating library files to currently open scripts
    • Synchronizing local files with HP Quality Center.
20. Coding Standards – Coding standards ensures scripts that are consistent, readable and easily maintainable. Coding standard should define all the below listed things
    • Naming convention for variables, subs, functions, file names, script names etc… Ex – i_VarName for interger, fn_i_FuncName for function returning interger
    • Library, subs, functions comment header. This should include information like version history, created by, last modified by, last modified date, description, parameters, example
    • Object naming conventions. Ex – txt_FieldName for a text box

Summary
Automation should be considered as a development project and not just record and playback of events. Starting automated testing with a good framework ensures low maintenance. Guidelines discussed in this paper can be used as input for developing requirements for a framework.

Selenium Web driver Architecture.

Architectural Themes

Before we start looking at the individual pieces to understand how they're wired together, it's useful to understand the the overarching themes of the architecture and development of the project. Succinctly put, these are:

·         Keep the costs down.

·         Emulate the user.

·         Prove the drivers work…

·         …but you shouldn't need to understand how everything works.

·         Lower the bus factor.

·         Have sympathy for a JavaScript implementation.

·         Every method call is an RPC call.

·         We are an Open Source project.

Before starting the automation using any automation tool, it is very important to know how that tool works and how it is architecture. This will helps to take the good advantage of the tool at the same time it will helps to make right automation framework. In my further posts I will start explain how to use selenium and how to create selenium framework in details but before that let’s get an overview of Selenium web driver architecture. Selenium can be a little bit confusing. As a beginner you will find how simply you can record and play the selenium scripts but it is not straight forward to how it’s doing that. At first glance it might appear that Selenium is actually driving the browser directly from your code but there’s actually a little bit more going on here and it’s going to help us understand how we can remote execute our test by looking at this basic architecture. So here’s a picture of the architecture for Selenium Web Driver, which is the current version of Selenium. 

Selenium web driver architecture mainly divided into three parts

1.    Language level bindings

2.    Selenium Web driver API

3.    Drivers

1) Language Level Bindings:
You can see at the Left hand side here we’ve got some bindings and these are language level bindings and with which you can implement the Selenium webdriver code. In simple words these the languages in which are making an framework, will interact with the Selenium Webdriver and work on various browsers and other devices. So we have a common API that we use for Selenium that has a common set of commands and we have various bindings for the different languages. So you can see there’s Java, Java, Python, Ruby, there’s also some other bindings and new bindings can be added very easily.
2) Selenium Web driver API:
Now these bindings communicate with Selenium Web driver API and This API send the commands taken from language level bindings interpret it and sent it to Respective driver. Right now don’t worry about how it works. I will explain them in upcoming posts. In basic term it contains set of common library which allow sending command to respective drivers.
3) Drivers:
Drivers here at the right hand side, you see we have various internet browser specific drivers such as IE driver, a Firefox, Chrome, and other drivers such as HTML unit which is an interesting one. It works in headless mode which makes text execution faster. It also contains mobile specific drivers as well. But the basic idea here is that each one of these drivers knows how to drive the browser that it corresponds to. So the Chrome driver knows how to handle the low level details of Chrome browser and drive it to do things like clicking button, going into pages, getting data from the browser itself, the same thing for Firefox, IE, and so on.

How all blocks work together?
So what’s happening here is you’re going to write your test in let’s say in Java and you’re going to be using common Selenium API and that Java binding is going to be sending command across this common Web Driver API. Now on the other end is going to be listening a driver, It’s going to interpret those commands and it’s going to execute them on the actual browser and then it’s going to return the result backup using the Web Driver API to your code where you can look at that result.

Let’s take more closure look that how exactly that works
Let say you have written test using java (binding code) against Selenium API and that binding code is going to issue commands across Web Driver wire protocol this is a rest-based web service that is able to interpret those commands. The driver server is just a little executable that runs each one of the drivers has this driver server that basically listens on a port on your local machine when you run your tests and it’s waiting for these commands to come in. And when these commands come in it interprets those commands and then automates the browser and then returns those results back.

I hope this will give some clear idea about how Selenium Web driver being architect.