googletesting: Watching movies to find localization bugs

Watching movies to find localization bugs

Wednesday, March 12, 2008

By Sharon Zhou, Kirkland Client Test Lead

In December, Google Pack shipped 10 new languages in 10 new countries/regions including China Pack. This was in addition to the 30 languages Pack was all ready available in. Localization testing for these 10 languages is not trivial. The testing needs to be done very quickly by experts in the language who may not have seen the application before. Localization testing (LQA) can also be costly since it requires multiple external vendors, and the LQA schedule is highly sensitive to changes in the product schedule.

The process that has been followed so far has been to have each product documented by an engineer detailing the workflow to navigate to each area of the UI, the appropriate inputs to be entered at each step, and what should be expected. The documentation time is considerable, and changes with product changes. The vendors must each consume this documentation and become functional at using the product in order to navigate through the product. There are also challenges to get vendors the appropriate permissions to access our unreleased products, and to download them at the site where they work.

To minimize the test cost, the Pack test team has implemented a significant amount of automation across the entire product driving the UI. One feature of the automation harness is the ability to record movies. For the new 10 languages, the Pack team tried a new process of using the automation to drive the UI, recording movies of the product UI, and sending these movies to our vendors along with a top level test plan. To evaluate the new approach, we also asked them to fill out a survey to have a quantitative concept of how much time we can save, and hence how much cost we can reduce.

The survey results come back very positive and encouraging. We received valuable feedback on what vendors need to conduct a fast and efficient test pass. Overall, this experiment saved the vendors an estimated 25% of their time overall. It was just as effective as the previous process, but was much simpler for them to complete. Our next steps will be to drive more of the UI. If the automation can touch every page, link and dialog, it can replace the traditional LQA testing method of installing and running build as people perform functional testing.

Testing Blog

Watching movies to find localization bugs

No comments :

Post a Comment

Labels

Archive

Feed

Company-wide

Products

Developers