Testing Blog
GUI Testing: Don't Sleep Without Synchronization
Tuesday, October 28, 2008
Posted by Philip Zembrod, Software Engineer in Test, Sweden
So you're working on TheFinalApp - the ultimate end-user application, with lots of good features and a really neat GUI. You have a team that's keen on testing and a level of unit test coverage that others only dream of. The star of the show is your suite of automatic GUI end-to-end tests — your team doesn't have to manually test every release candidate.
Life would be good if only the GUI tests weren't so flaky.
Every once and again, your test case clicks a menu item too early, while the menu is still opening. Or it double-clicks to open a tree node, tries to verify the open too early, then retries, which closes the node (oops). You have tried adding sleep statements, which has helped somewhat, but has also slowed down your tests.
Why all this pain? Because
GUIs are not designed to synchronize with other computer programs. They are designed to synchronize with human beings
, which are not like computers:
Humans act much more slowly. Well-honed GUI test robots drive GUIs at near theoretical maximum speed.
Humans are much better at observing the GUI, and they react intelligently to what they see.
Humans extract more meaningful information from a GUI.
In contrast to testing a server, where you usually find enough methods or messages in the server API to synchronize the testing with the server,
a GUI application usually lacks these means of synchronization
. As a result, a running automated GUI test often consists of one long sequence of race conditions between the automated test and the application under test.
GUI test synchronization boils down to the question:
Is the app under test finished with what it's doing?
"What it's doing" may be small, like displaying a combo box, or big, like a business transaction. Whatever "it" is, the test must be able to tell whether "it" is finished.
Maybe you want to test something while "it" is underway
, like verify that the browser icon is rotating while a page is loading. Maybe you want to deliberately click the "Submit" button again in the middle of a transaction to verify that nothing bad happens.
But usually, you want to wait until "it" is done
.
How to find out whether "it" is done? Ask!
Let your test case ask your GUI app. In other words:
provide one or several test hooks suitable for your synchronization needs
.
The questions to ask depend on the type, platform, and architecture of your application. Here are three questions that worked for me when dealing with a single-threaded Win32 MFC database app:
The first is a question for the OS. The Win32 API provides a function to
wait while a process has pending input events
:
DWORD WaitForInputIdle(HANDLE hProcess, DWORD dwMilliseconds). Choosing the shortest possible timeout (dwMilliseconds = 1) effectively turns this from a wait-for to a check-if function, so you can explicitly control the waiting loop; for example, to combine several different check functions. Reasoning:
If the GUI app has pending input, it's surely not ready for new input.
The second question is:
Is the GUI app's message queue empty?
I did this with a test hook, in this case a WM_USER message; it could perhaps also be done by calling PeekMessage() in the GUI app's process context via CreateRemoteThread(). Reasoning:
If the GUI app still has messages in its queue, it's not yet ready for new input.
The third is
more like sending a probe than a question
, but again using a test hook. The test framework resets a certain flag in the GUI app (synchronously) and then (asynchronously) posts a WM_USER message into the app's message queue that, upon being processed, sets this flag. Now
the test framework checks periodically
(and synchronously again)
to see whether the flag has been set.
Once it has, you know the posted message has been processed. Reasoning:
When the posted message (the probe) has been processed, then surely messages and events sent earlier to the GUI app have been processed.
Of course, for multi-threaded applications this might be more complex.
These three synchronization techniques resulted in
fast and stable test execution, without any test flakiness due to timing issues. All without sleeps,
except in the synchronization loop.
Applying this idea to different platforms requires finding the right questions to ask and the right way to ask them.
I'd be interested to hear if someone has done something similar
, e.g. for an Ajax application. A query into the server to check if any XML responses are pending, perhaps?
No comments :
Post a Comment
Labels
Aaron Jacobs
1
Adam Porter
1
Alan Faulkner
1
Alan Myrvold
1
Alberto Savoia
4
Alek Icev
2
Alex Eagle
1
Allen Hutchison
6
Andrew Trenk
8
Android
1
Anthony Vallone
25
Antoine Picard
1
APIs
2
App Engine
1
April Fools
2
Arif Sukoco
1
Bruce Leban
1
C++
11
Chaitali Narla
2
Christopher Semturs
1
Chrome
3
Chrome OS
2
Dave Chen
1
Diego Salas
2
Dmitry Vyukov
1
Dori Reuveni
1
Eduardo Bravo Ortiz
1
Ekaterina Kamenskaya
1
Erik Kuefler
3
Espresso
1
George Pirocanac
2
Google+
1
Goranka Bjedov
1
GTAC
54
Hank Duan
1
Harry Robinson
5
Havard Rast Blok
1
Hongfei Ding
1
James Whittaker
42
Jason Arbon
2
Jason Elbaum
1
Jason Huggins
1
Java
5
JavaScript
7
Jay Han
1
Jessica Tomechak
1
Jim Reardon
1
Jobs
14
Joe Allan Muharsky
1
Joel Hynoski
1
John Penix
1
John Thomas
3
Jonathan Rockway
1
Jonathan Velasquez
1
Julian Harty
5
Julie Ralph
1
Karin Lundberg
1
Kaue Silveira
1
Kevin Graney
1
Kirkland
1
Kurt Alfred Kluever
1
Lesley Katzen
1
Marc Kaplan
3
Mark Ivey
1
Mark Striebeck
1
Marko Ivanković
1
Markus Clermont
3
Michael Bachman
1
Michael Klepikov
1
Mike Wacker
1
Misko Hevery
32
Mobile
2
Mona El Mahdy
1
Noel Yap
1
Patricia Legaspi
1
Patrick Copeland
23
Patrik Höglund
5
Peter Arrenbrecht
1
Phil Rollet
1
Philip Zembrod
4
Pooja Gupta
1
Radoslav Vasilev
1
Rajat Dewan
1
Rajat Jain
1
Rich Martin
1
Richard Bustamante
1
Roshan Sembacuttiaratchy
1
Ruslan Khamitov
1
Sean Jordan
1
Sharon Zhou
1
Shyam Seshadri
4
Simon Stewart
2
Stephen Ng
1
Tejas Shah
1
Test Analytics
1
Tony Voellm
2
TotT
54
Vojta Jína
1
WebRTC
2
Yvette Nameth
2
Zhanyong Wan
6
Zuri Kemp
2
Archive
2015
December
November
October
August
June
May
April
March
February
January
2014
December
November
October
September
August
July
June
May
April
March
February
January
2013
December
November
October
August
July
June
May
April
March
January
2012
December
November
October
September
August
2011
November
October
September
August
July
June
May
April
March
February
January
2010
December
November
October
September
August
July
June
May
April
March
February
January
2009
December
November
October
September
August
July
June
May
April
February
January
2008
December
November
October
TotT: Contain Your Environment
GUI Testing: Don't Sleep Without Synchronization
Testability Explorer: Measuring Testability
Dependency Injection Myth: Reference Passing
Test Engineering at Google
TotT: Floating-Point Comparison
To "new" or not to "new"...
TotT: Simulating Time in jsUnit Tests
Six Hats of Software Testing
September
August
July
June
May
April
March
February
January
2007
October
September
August
July
June
May
April
March
February
January
Feed
Follow @googletesting
No comments :
Post a Comment