googletesting: Partial Automation: Keeping humans in the loop

Partial Automation: Keeping humans in the loop

Tuesday, November 4, 2008

Posted by Patricia Legaspi, Test Engineering Manager

One of the challenges of automation is achieving complete automation. Ideally, complete or total automation would not require any human intervention or verification yet this is a difficult level to achieve. Investing Engineering time to completely automate tests is expensive and, many times, has diminishing returns. Rather than trying to achieve complete automation, investing in ways to make the most out of the automated test and the human time is time better spent.

Effective test report
Consider an automated UI test... Routinely, automated UI tests result in false negatives due to timing issues, the complexity of the steps taken, and other factors. These have to be investigated which can be time consuming. A thorough report can be created for automated UI tests to present log information, error reports, screen shots of the state of the application when the test failed, and an easy way to re-run the test in question. A human can make use of the information provided and effectively investigate the possible issue. If we can reduce the amount of work that someone has to do to investigate a test failure, the UI tests become more valuable and the human plays a much more important role as a "verifier." Had the report not provided any information for the failed test, the human would have spent, in some cases, hours investigating the issue rather than continuing to automate or run exploratory tests which would be of more value. Is there a way to maximize what people do well, while also maximizing the use of automation?

Applying human intelligence
What about tests that require a human eye? There are many arguments about tests that require a human make a judgment about the appearance of rendered UI objects. Machines are great at running tests that return in a firm pass or fail but for tests that require opinion or judgment, the human has an advantage. We've been experimenting with image comparison tests. Screen shots of a golden version of the application under test are compared to screen shots of the current release candidate to verify that the application continues to render properly and that the UI "looks good". Although image comparison techniques can determine if the screen shots are different, they cannot help determine if the difference is "important". The human comes back into the picture to complement the automated test. To make effective use of a person's time, the test results should be organized and well presented. For image comparison tests, a report which shows the golden and release candidate screen shots side-by-side with clearly highlighted differences or allows you to replace the golden screen shot is key. A tester can quickly navigate the reported differences and determine if they are actual failures or acceptable differences. Frequently, an application's UI will change which will result in expected image differences. If the differences are expected changes, the tester should be able to replace the golden with the new image with the click of a button for future comparison.

Aside from test automation, efficiency can also be improved by streamlining the various components of the process. This includes test environment setup, test data retrieval, and test execution. Release cycles tighten and so should the testing processes. Automating the environment setup can be a major time saver and allow for added time to run in depth tests. Creating continuous builds to run automated tests ahead of time and preparing environments overnight results in added testing time.

Automated tests are rarely bullet proof, they are prone to errors and false failures. In some cases, creating an infrastructure to make test analysis easy for humans, is much more effective than trying to engineer tests to automatically "understand" the UI. We coined this, "Partial Automation." We found that by having a person in the loop dramatically reduces the time spent trying to over engineer the automated tests. Obviously, there are trade-offs to this approach and one size doesn't fit all. We believe that automation does not always need to mean complete automation; we automate as much as you can and consider how human judgment can be used efficiently.

Testing Blog

Partial Automation: Keeping humans in the loop

No comments :

Post a Comment

Labels

Archive

Feed

Company-wide

Products

Developers