Automation Testing

1 Introduction

After a certain point of time, the repetitive manual testing of test scripts becomes very monotonous and mundane. Also, for certain types of testing like regression testing, it always helps if somebody or something could go on testing the same thing again and again. With the current trend of software release cycles becoming shorter and the list of tests in the regression test suite becoming larger, it becomes improbable to do complete manual regression testing with every release. Incomplete and inefficient testing may cause new bugs (defects introduced while fixing a bug) going undetected before the release. By automating the process (regression testing in this case), the time needed to test run the full suite reduces, thereby providing the tester more time to focus on manual testing of new code and on areas of more risk.

The concept of automation finds its use in all such cases. In short, it’s mainly about

• Using software tools to execute tests without manual intervention
• Use of other software tools to control the execution of tests, the comparison of actual with the expected outcome and other test control & test reporting functions.

Automation Testing as such refers to automating the manual testing process. Automation can be applied to

• Web applications
• Client -server environments.

The process basically consists of test cases that contain expected results based on business requirements and also an exclusive test environment having a test database in which the test cases can be repeated whenever the application undergoes a change.

2 Why Test Automation

The following points elucidate the reasons behind the need for automation (selective and not exhaustive):

      Running the existing tests on a new version(s) of a program/application

      More number of tests can be run in a lesser period of time

      Bugs that can be uncovered only with long runs

      Better use of resources

      Testers can do better jobs, machines can be run all the time

      Consistency, Reliability and Repeatability of tests

      The same tests can be repeated exactly in the same manner, every time when the automated scripts are executed thus eliminating manual testing errors            

3 Automation Life Cycle

Stages involved in automation process are (in order)

1. Automation Feasibility Analysis
2. Planning and Design
3. Test Environment Setup
4. Automation Script Generation
5. Test Execution
6. Defect Analysis & Fixing

Each stage has specific Entry Criteria, Activities, Measurements and Exit Criteria defined for them. The details are as follows:

3.1 Automation Feasibility Analysis

Overview

Requirements & expectations have to be defined clearly & explicitly before starting off any project and so is the case involving automating a project. Things like

ü      How and what is to be automated

ü      Which all modules can be automated

should be decided in the beginning itself to prevent any confusion after the automation process has been flagged off. This stage hence requires a great deal of attention.

Entry-criteria

      Clearly defined Requirements

      Sound understanding of client expectations

Activities

      Requirement analysis for a thorough understanding

      Identify specific requirement tests that can be automated

      Identify test cases that cannot be automated due to tool limitations and / or complexity of the functionality

      Understanding the System and the environment in which the application has to be installed

      Analyse applicability of a suitable tool to carry out the automation

      Rough estimate of the size, effort and cost involved in this automation

      Analyze if automation is actually possible and benefit in the long run

      Prepare a feasibility analysis report

Validation

      Review of Feasibility Analysis Report

Exit-criteria

      Approved Feasibility Analysis report

3.2 Planning and Design

Overview

Once the requirements & expectations have been highlighted clearly and the feasibility studies yield a positive result, methodical planning has to be done. This would include defining the approach, decide on which framework is to be used and subsequently, estimating the time and finance needed to complete the same.

Entry-criteria

      Automation Feasibility analysis report

Activities

      Identify applicable approaches

      Analyse the suitability of each approach and select the best among them

      Identify the frameworks that can be used

      Analyze every framework for their pros and cons

      Finalizing the framework to be used

      Form a test plan based on the above.

      Analyse manual test scripts from an automation standpoint.

      Define automation scripting standards (Common language functionality)

      Identify common functionality across test cases and design compiled modules

      Categorize test scripts based on functionality

      Identify functionality which need to be parameterised

      Prepare the detailed cost, size and effort estimate

Validation

      Review of Test Plan by the client

Exit-criteria

      Approved automation Framework

      Approved automation test plan

3.3 Test Environment Setup

Overview

Environment set-up is one of the critical aspects of testing process. Test team may not be involved in this activity if the customer/development team provides the test environment in which case the test team is required to do a readiness check of the given environment. Test environment decides the software and hardware conditions under which a work product is tested.

Entry-criteria

      System Design and architecture documents are available

      Environment set-up plan is available

Activities

      Understand the required architecture, environment set-up

      Prepare environment setup checklist

      Prepare hardware and software requirements

      Finalize connectivity requirements

      Setup Test Environment

Validation

      Effort for setup

      Defects in setup

Exit-criteria

      Environment setup is working as per the plan and checklist 

3.4 Automation Script Generation

Overview

The project team will go through the test plan & obtains clarifications, if needed. It will then decide upon the type of automation to be used, it could be capture/playback, scripting etc. The project team will share the knowledge and experience through presentations and/or discussions at appropriate intervals during the progress of the project.

Entry-criteria

      Test plan is available

      Test tool available

      Manual test case if any

Activities

      Create test scenarios

      Identify reusable test scenarios

      Group the common test cases together

      Finalize Test Scripts

Validation

      Review of scripts

Exit-criteria

      Scripts are tested and reviewed

      Defects are fixed

3.5 Test Execution

Overview

The project team member(s) will carry out testing based on the test requirements.

Entry-criteria

      Baselined Test plan is available

      Test environment is ready

      Test data set up is done

      Test Scripts are available

Activities

      Perform the testing as per the test plan

      Perform the batch test wherever necessary

      Update the status in the status matrix

      Log the defects

Validation

      Testing

Exit-criteria

      All test cases are executed.

      Defects logged in the defect tracking tool

3.6 Defect Analysis & Fixing

Overview

The project team members will analyze the defects logged and will fix the defects. The project team members will also prepare a document on the defects found, their priority and also the solution for the same.

Entry-criteria

      Testing has been done

      Test records are available

      Defect log is available

Activities

      Analyze the defects for severity and priority

      Fix the defects

      Document the defect along with the solution

Validation

      Testing

Exit-criteria

      Defects are analysed and fixed

      Application is ready to be retested for bug fixes

4 Cost Involved in automation

Fixed Cost

Y      Automation feasibility analysis cost

Y      Tool selection and Acquisition cost

Y      Hiring skilled resources OR training existing team members

Y      Time in learning the application/business processes

Y      Pilot project identification effort and Proof of Concept

Y      First time automation of the identified parts of application/product

Y      Test suite Documentation effort

Variable cost

Y      Test script and documentation maintenance cost

Y      Automated Test infrastructure maintenance cost

Y      Execution cost

5 Myths & Realities

Automated testing is just an add-on to the testing process to make the testers’ life easier. Unlike prevalent myths, it doesn’t necessitate ramp-down of resources in the testing department. Also, automation is not as inexpensive as we think. As a matter of fact, it may take much longer to develop, verify, and document an automated test case than to create and execute a manual test case! This is even truer if the record/playback feature is selected as the primary testing technique. In fact, record/playback is the least cost-effective method of automating test cases.
Having said that, it doesn’t mean that automation testing can’t be made cost effective at all. Depending on the type of testing requirements of the company, the tool could be decided upon. If there be a case wherein the same functionality is achieved by two different tools wherein one is significantly costlier than the other, the less expensive one could be easily preferred over the other. Again, the tool chosen should be serving the interests of the company well. Otherwise, even a small amount of money spent on a “less expensive tool” won’t do any good. Simply put, it’s the requirements that come first while choosing a tool and then all other monetary and other considerations.

Apart from the monetary aspect, there are other issues relating to time management as well. Trying to automate very complex test cases wouldn’t serve much purpose as they would consume a lot of time and at the end of the day, no one can be sure if the automation would actually be successful time and again. So rather than breaking heads over some very complex test cases, the tester would do well to automate a significant number of relatively easier test cases in the mean time. This method is the most costly (in terms of time consumption) among all methods. The record/playback feature of a test tool is useful for determining how the tool interacts with the application under test, and can give ideas regarding developing test scripts, but its usefulness ends there. It can’t handle any unexpected pop-up that might suddenly appear or any such unexpected behavior for that matter that was not a part of the recording.

6 Feasible Strategies

Other than the record/playback strategy (which has more disadvantages than advantages), there are a few more methodologies which are very effective for automating functional or system testing. A few of such strategies are:

6.1 Functional Decomposition Method

Essentially, it involves reducing all test cases to their most fundamental tasks, and write user-defined functions, business function scripts, and sub-routine or utility scripts which perform the tasks independently of one another. In general, these fundamental areas include navigation, specific business function, and data verification & return navigation. For achieving this, it is mandatory to separate data & function. This allows an automated test script to be written for a business function, using data-files to provide both the input and the expected-results verification. The hierarchical architecture employed, has a structured or modular design. The engine of the test is the driver script which is at the highest level. This driver script has a number of calls to one or more test scripts. These test scripts contain the actual logic, calling the business function scripts necessary to perform the application testing. Functions & utility scripts are called as needed by Drivers, Main, and Business Function scripts.
While the driver scripts do the initialization and call the scripts in the correct order, the test case scripts perform the application test case logic using business function scripts which in turn perform specific business functions within the application. The subroutine scripts perform specific tasks required by two or more business scripts while user-defined functions are general, application-specific and screen-access functions. The business function and subroutine scripts invoke user defined functions to perform navigation. The Test Case script would call these two scripts, and the Driver Script would call this Test Case script some number of times required to perform all the required Test Cases of this kind. In each case, the only thing that changes are the data contained in the files that are read and processed by the business function & subroutine scripts.
This method, however, requires only that we add the data-files required for each test, and these can easily be updated/maintained using Notepad or some such text-editor. Updating these files does not require any knowledge of the automated tool, scripting, programming, etc. meaning that the non-technical testers can perform this function, while one technical tester can create and maintain the automated scripts.
Advantages:

Y      The best part is that scripts can be created while application development is still in progress. If the functionality were to change, only the specific business function script would have to be modified..

Y      Being text records, the data involved (both input and output) & the expected results are easily maintainable.

Y      Decreases duplicity in creating automated test scripts as it uses a modular design & uses files to input and verify data.

Y      Better error handling as the functions return values to the calling script, rather than aborting, thereby increasing the robustness of the test scripts.

Disadvantages:

Y      Prior knowledge needed in the scripting language used by the tool

Y      Multiple data-files are needed for every test case.

Y      Maintenance of the Detail Test Plan with specific data becomes increasingly important.

6.2 Test Plan Driven Method

It is also called the key-word driven approach. This method uses the actual test case document developed by the tester using a spreadsheet containing special "Key-Words".  This method preserves most of the advantages of the "Functional Decomposition" method, while eliminating most of the disadvantages.  In this method, the entire process is data-driven, including functionality. The Key Words control the processing here.

Architecture

The architecture of the Test Plan Driven method appears similar to that of the Functional Decomposition method, but in fact, they are substantially different. The driver script performs the initialization if needed (just as in functional decomposition method) and calls the application specific controller script, which in turn processes the file name received from driver and matches on key words contained in the input file, builds a parameter-list from the records & calls utility scripts associated with the key words. The utility scripts in turn processes input parameter-list received from the controller script, perform specific tasks, raise errors that might have occurred and then return to the controller script. User defined functions, both general and application-specific ones can be called by any of the scripts from above.

Advantages

Apart from the advantages that the functional decomposition method has, it has a few more plus points too and they are:

Y      Ease of working as the detail test plan need be written only once in the spreadsheet format containing all input and verification data.

Y      The best part is that even if the tester doesn’t have prior knowledge on the tool, the work can be started! I.e. before the plan is written, if the utility scripts are created by someone proficient in the tool’s scripting language, then the tester can use the test tool immediately via the spreadsheet-input method, without having to learn the scripting language. All he needs to know are the specific key-words and the format to use within the test plan. This saves time as there is no dire need of spending time on training the tester about the tool first and then starting off with coding.

Y       Re-usability is another desirable thing in this approach. After a number of generic Utility scripts have already been created for testing an application, most of these can be re-used to test another application.

Disadvantages

¨       Prior knowledge of the tool specific scripting language is needed for creating customized functions and utilities. So if someone technically sound is present to do it, then fine otherwise that would become a bottleneck as in the advantage of the tester not having to know anything about the tool would no more hold good.

¨       In the event of application requiring more than a few customized utilities, the tester is required to learn a number of key words and special formats which can be pretty time-consuming. Once the testers get used to this, however, the time required to produce a test case is greatly improved.