Testing Blog
GTAC Speakers and Attendees Finalized
Thursday, August 18, 2011
We've completed the agenda for GTAC 2011 and are in the process of notifying accepted speakers and attendees. Once we have firm accepts we'll be publicizing the agenda.
Pretotyping: A Different Type of Testing
Tuesday, August 16, 2011
Have you ever poured your heart and soul and blood, sweat and tears to help test and perfect a product that, after launch, flopped miserably? Not because it was not working right (you tested
the snot out of it
), but because it was not the right product.
Are you currently wasting your time testing a new product or feature that, in the end, nobody will use?
Testing typically revolves around making sure that we have built something right. Testing activities can be roughly described as “verifying that something works as intended, or as specified.” This is critical. However, before we take steps and invest time and effort to make sure that something built right, we should make sure that the thing we are testing, whether its a new feature or a whole new product, is the right thing to build in the first place.
Spending time, money and effort to test something that nobody ends up using is a waste of time.
For the past couple of years, I’ve been thinking about, and working on, a concept called pretotyping.
What is pretotyping? Here’s a somewhat formal definition – the dry and boring kind you’d find in a dictionary:
Pretotyping [pree-tuh-tahy-ping], verb: Testing the initial appeal and actual usage of a potential new product by simulating its core experience with the smallest possible investment of time and money.
Here’s a less formal definition:
Pretotyping is a way to test an idea quickly and inexpensively by creating extremely simplified, mocked or virtual versions of that product to help validate the premise that "If we build it, they will use it."
My favorite definition of pretotyping, however, is this:
Make sure – as quickly and as cheaply as you can – that you are building the right it before you build it right.
My thinking on pretotyping evolved from my positive experiences with Agile and Test Driven Development. Pretotyping applies some of the core ideas from these two models and applies them further
upstream
in the development cycle.
I’ve just finished writing the first draft of a booklet on pretotyping called “Pretotype It”.
You can download a PDF of the booklet from
Google Docs
or
Scribd
.
The "Pretotype It" booklet is itself a pretotype and test. I wrote this first-draft to test my (possibly optimistic) assumption that people would be interested in it, so please let me know what you think of it.
You can follow my pretotyping work on
my pretotyping blog
.
Post content
Posted by Alberto Savoia
Keynote Lineup for GTAC 2011
Monday, August 1, 2011
By James Whittaker
The call for proposals and participation is now closed. Over the next few weeks we will be announcing the full agenda and notifying accepted participants. In the meantime, the keynote lineup is now locked. It consists of two famous Googlers and two famous external speakers that I am very pleased to have join us.
Opening Keynote:
Test is Dead
by
Alberto Savoia
The way most software is designed, developed and launched has changed dramatically over the last decade – but what about testing? Alberto Savoia believes that software testing as we knew it is dead – or at least moribund – in which case we should stick a fork in it and proactively take it out of its misery for good. In this opening keynote of biblical scope, Alberto will cast stones at the old test-mentality and will try his darnedest to agitate you and convince you that these days most testers should follow a new test-mentality, one which includes shifting their focus and priority from “Are we building it right?” to “Are we building the right it?” The subtitle of this year’s GTAC is “cloudy with a chance of tests,” and if anyone can gather the clouds into a hurricane, it's Alberto – it might be wise to bring your umbrella.
Alberto Savoia is Director of Engineering and Innovation Agitator at Google. In addition to leading several major product development efforts (including the launch of Google AdWords), Alberto has been a lifelong believer, champion, innovator and entrepreneur in the area of developer testing and test automation tools. He is a frequent keynote speaker and the author of many articles on testing, including the classic booklet “The Way of Testivus” and “Beautiful Tests” in O’Reilly’s Beautiful Code. His work in software development tools has won him several awards including the 2005 Wall Street Journal Technical Innovator Award, InfoWorld’s Technology of the Year award, and no less than four Software Development Magazine Jolt Awards.
Day 1 Closer:
Redefining Security Vulnerabilities: How Attackers See Bugs
by
Herbert H. Thompson
Developers see features, testers see bugs, and attackers see “opportunities.” Those opportunities are expanding beyond buffer overflows, cross site scripting, etc. into logical bugs (and features) that allow attackers to use the information they find to exploit trusting users. For example, attackers can leverage a small information disclosure issue in an elaborate phishing attempt. When you add people in the mix, we need to reevaluate which “bugs” are actual security vulnerabilities. This talk is loaded with real world examples of how attackers are using software “features” and information tidbits (many of which come from bugs) to exploit the biggest weakness of all: trusting users.
Dr. Herbert H. Thompson is Chief Security Strategist at People Security and a world-renown expert in application security. He has co-authored four books on the topic including, How to Break Software Security: Effective Techniques for Security Testing (with Dr. James Whittaker) and The Software Vulnerability Guide (with Scott Chase). In 2006 he was named one of the “Top 5 Most Influential Thinkers in IT Security” by SC Magazine. Thompson continually lends his perspective and expertise on secure software development and has been interviewed by top news organizations including CNN, MSNBC, BusinessWeek, Forbes, Associated Press, and the Washington Post. He is also Program Committee Chair for RSA Conference, the world’s leading information security gathering. He holds a Ph.D. in Applied Mathematics from Florida Institute of Technology, and is an adjunct professor in the Computer Science department at Columbia University in New York.
Day 2 Opener:
Engineering Productivity: Accelerating Google Since 2006
by
Patrick Copeland
Patrick Copeland is the founder and architect of Google's testing and productivity strategy and in this "mini keynote" he tells the story and relates the pain of taking a company from ad hoc testing practices to the pinnacle of what can be accomplished with a well oiled test engineering discipline.
Conference Closer:
Secrets of World-Class Software Organizations
by
Steve McConnell
Construx consultants work with literally hundreds of software organizations each year. Among these organizations a few stand out as being truly world class. They are exceptional in their ability to meet their software development goals and exceptional in the contribution they make to their companies' overall business success. Do world class software organizations operate differently than average organizations? In Construx's experience, the answer is a resounding "YES." In this talk, award-winning author Steve McConnell reveals the technical, management, business, and cultural secrets that make a software organization world class.
Steve McConnell is CEO and Chief Software Engineer at Construx Software where he consults to a broad range of industries, teaches seminars, and oversees Construx’s software engineering practices. Steve is the author of Software Estimation: Demystifying the Black Art (2006), Code Complete (1993, 2004), Rapid Development (1996), Software Project Survival Guide (1998), and Professional Software Development (2004), as well as numerous technical articles. His books have won numerous awards for "Best Book of the Year," and readers of Software Development magazine named him one of the three most influential people in the software industry along with Bill Gates and Linus Torvalds.
How We Tested Google Instant Pages
Wednesday, July 27, 2011
By Jason Arbon and Tejas Shah
Google Instant Pages
are a cool new way that Google speeds up your search experience. When Google thinks it knows which result you are likely to click, it preloads that page in the background, so when you click the page it renders instantly, saving the user about 5 seconds. 5 seconds is significant when you think of how many searches are performed each day--and especially when you consider that the rest of the search experience is optimized for sub-second performance.
The testing problem here is interesting. This feature requires client and server coordination, and since we are pre-loading and rendering the pages in an invisible background page, we wanted to make sure that nothing major was broken with the page rendering.
The original idea was for developers to test out a few pages as they went.But, this doesn’t scale to a large number of sites and is very expensive to repeat. Also, how do you know what the pages should look like? To write Selenium tests to functionally validate thousands of sites would take forever--the product would ship first. The solution was to perform automated test runs that load these pages from search results with Instant Pages turned on, and another run with Instant Pages turned off. The page renderings from each run were then compared.
How did we compare the two runs? How to compare pages when content and ads on web pages are constantly changing and we don't know what the expected behavior is? We could have used cached versions of these pages, but that wouldn’t be the realworld experience we were testing and would take time setting up, and the timing would have been different. We opted to leverage some other work that compares pages using the Document Object Model (DOM). We automatically scan each page, pixel by pixel, but look at what element is visible at the point on the page, not the color/RGB values. We then do a simple measure of how closely these pixel measurements match. These so-called "quality bots" generate a score of 0-100%, where 100% means all measurements were identical.
When we performed the runs, the vast majority (~95%) of all comparisons were almost identical, like we hoped. Where the pages where different we built a web page that showed the differences between the two pages by rendering both images and highlighting the difference. It was quick and easy for the developers to visually verify that the differences were only due to content or other non-structural differences in the rendering. Anytime test automation scales, is repeatable, quantified, and developers can validate the results without us is a good thing!
How did this testing get organized? As with many things in testing at Google, it came down to people chatting and realizing their work can be helpful for other engineers. This was bottom up, not top down. Tejas Shah was working on a general quality bot solution for compatibility (more on that in later posts) between Chrome and other browsers. He chatted with the Instant Pages developers when he was visiting their building and they agreed his bot might be able to help. He then spend the next couple of weeks pulling it all together and sharing the results with the team.
And now more applications of the quality bot are surfacing. What if we kept the browser version fixed, and only varied the version of the application? Could this help validate web applications independent of a functional spec and without custom validation script development and maintenance? Stay tuned...
GTAC: Call for Team Attendance
Thursday, July 7, 2011
Attending conferences can be a great way to network and learn new concepts. However, taking those concepts back to your office and trying to convince your team apply them can be daunting. In order to make GTAC attendees more successful at implementing what they learn at this conference we are going to give preference to teammates from the same company applying for attendance. Bring another developer or tester (or two or three) and attend as a team so you can discuss what you learn and experience, hopefully increasing your chances of putting it into practice when you return to work.
We're extending the deadline for attendees until the end of July to give you a chance to round up some teammates.
Google at STAR West 2011
Tuesday, June 28, 2011
By James Whittaker
STAR West will feature something unprecedented this year: back-to-back tutorials by Googlers plus a keynote and track session.
The tutorials will be Monday October 3. I have the morning session on "How Google Tests Software" and my colleague Ankit Mehta has the afternoon session on "Testing Rich Internet AJAX-based Applications." You can spend the whole day in Google Test Land.
I highly recommend Ankit's tutorial. He is one of our top test managers and has spent years minding Gmail as it grew up from a simple cloud-based email system into the mass-scale, ubiquitous rich web app that it is today. Ankit now leads all testing efforts around our social offerings (which are already starting to appear). Anyone struggling to automate the testing of rich web apps will have plenty to absorb in his session. He's not spouting conjecture and generalities; he's speaking from the position of actual accomplishment. Bring a laptop.
Jason Arbon and Sebastian Schiavone are presenting a track talk on "Google's New Methodology for Risk Driven Testing" and will be demonstrating some of the latest tools coming out of Google Test Labs. Tools that were born of real need built to serve that need. I am expecting free samples! Jason was test lead for Chrome and Chrome OS before taking over Google Test Labs where incredibly clever code is woven into useful test tools. Sebastian is none other than my TPM (technical program manager) who is well known for taking my vague ideas about how things should be done and making them real.
Oh and the keynote, well that's me again, something about testing getting in the way of quality. I wrote this talk while I was in an especially melancholy mood about my place in the universe. It's a wake-up call to testers: the world is changing and your relevance is calling ... will you answer the call or ignore it and pretend that yesterday is still today?
Lessons in a 21st Century Tech Career: Failing Fast, 20% Time and Project Mobility
Thursday, June 23, 2011
By James Whittaker
If your name is Larry Page, stop reading this now.
Let me first admit that as I write this I am sitting in a company lounge reminiscent of a gathering room in a luxury hotel with my belly full of free gourmet food waiting for a meeting with the lighthearted title "Beer and Demos" to start.
Let me secondly admit that none of this matters. It's all very nice, and I hope it continues in perpetuity, but it doesn't matter. Engineers don't need to be spoiled rotten to be happy. The spoiling of engineers has little to do with the essence of a 21st century tech career.
Now, what exactly does matter? What is the essence of a 21st century tech career that keeps employees loyal and engaged with productivity that would shame the most seasoned agile-ist? I don't yet have the complete story, but here are three important ingredients:
Failing Fast
. Nothing destroys morale more than a death march. Projects going nowhere
should do so with the utmost haste
. The ability of a company to implode pet projects quickly correlates directly to a great place to work. Engineers working on these project gain not only valuable engineering experience, they experience first-hand the company's perception of what is important (and, in the case of their project, what is not important). It's a built-in lesson on company priorities and it ensures good engineers don't get monopolized by purposeless projects. You gotta like a company willing to experiment. You have to love a company willing to laugh at itself when the experiments don't pan out.
20% Time
. Any company worth working for has any number of projects that are worth working on. It's frustrating for many super-sharding engineers to see cool work going on down the hall or in the next building and not being part of it. A day job that takes all day is tiresome. Enter 20% time, a concept meant to send a strong message to all engineers:
you always have a spare day
. Use it wisely.
Project Mobility
. Staying fresh by changing projects is part of mobility. Continuous cycling of fresh ideas from new project members to existing projects is another part. The downside here is obviously projects with a steep learning curve but I scoff in the general direction of this idea. Whose fault is it when a wicked smart engineer can't learn the system fast enough to be useful in some (even a small) context? Only the weakest organization with the poorest documentation can use that excuse. The only good reason for keeping people on a project is because they have no desire to leave.
These three concepts are better than all the lounges and free food any company can provide. Here's an example, a real example, of how it worked recently for an employee I'll call Paul (because that happens to be his name!).
Paul joined Google a little over a year ago and spent two months on a project that was then cancelled. He learned enough to be useful anywhere but was new enough that he really didn't have great context on what project he wanted next. Solution: I assigned him to a project that was a good skill set match.
Less than a year later, his new project ships. He played an important role in making this happen but in that time he also realized that the role was leaning toward feature development and he was more interested in a pure test development role. However, he was steeped in post-ship duties and working on the next release. A cycle that, happily, can be broken pretty easily here.
Another project had a test developer opening that suited Paul perfectly. He immediately signed up for 20% on this new project and spent his 80% ramping down in his old project. At some point these percentages will trade places and he'll spend 20% of his time training his replacement on the old project. This is a friction-less process. His manager cannot deny him
his day
to do as he pleases and now he can spend his time getting off the critical path of his old project and onto the critical path of his new project.
Mobility means a constant stream of openings on projects inside Google. It also creates a population of engineering talent with an array of project experiences and a breadth of expertise to fill those positions. 20% time is a mechanism for moving onto and off of projects without formal permissions, interviews and other make-work processes engineers deplore.
Let's face it, most benefits are transient. I enjoy a good meal for the time it is in front of me. I enjoy great medical when I am sick. I appreciate luxury when I have time for it. Even my paycheck comes with such monotonous regularity that it is an expectation that brings little joy apart from the brief moment my bank balance takes that joyful upward tick. But if I am unhappy the rest of the day, none of those islands of pampering mean squat. Empower me as an engineer during the much larger blocks of my time when I am doing engineering. Feed my creativity. Remove the barriers that prevent me from working on the things I want to work on.
Do these things and you have me. Do these things and you make my entire work day better. This is the essence of a 21st century tech career:
make the hours I spend working better
. Anything more is so dot com.
Ok, Larry you can start reading again.
Labels
Aaron Jacobs
1
Adam Porter
1
Alan Faulkner
1
Alan Myrvold
1
Alberto Savoia
4
Alek Icev
2
Alex Eagle
1
Allen Hutchison
6
Andrew Trenk
8
Android
1
Anthony Vallone
25
Antoine Picard
1
APIs
2
App Engine
1
April Fools
2
Arif Sukoco
1
Bruce Leban
1
C++
11
Chaitali Narla
2
Christopher Semturs
1
Chrome
3
Chrome OS
2
Dave Chen
1
Diego Salas
2
Dmitry Vyukov
1
Dori Reuveni
1
Eduardo Bravo Ortiz
1
Ekaterina Kamenskaya
1
Erik Kuefler
3
Espresso
1
George Pirocanac
2
Google+
1
Goranka Bjedov
1
GTAC
54
Hank Duan
1
Harry Robinson
5
Havard Rast Blok
1
Hongfei Ding
1
James Whittaker
42
Jason Arbon
2
Jason Elbaum
1
Jason Huggins
1
Java
5
JavaScript
7
Jay Han
1
Jessica Tomechak
1
Jim Reardon
1
Jobs
14
Joe Allan Muharsky
1
Joel Hynoski
1
John Penix
1
John Thomas
3
Jonathan Rockway
1
Jonathan Velasquez
1
Julian Harty
5
Julie Ralph
1
Karin Lundberg
1
Kaue Silveira
1
Kevin Graney
1
Kirkland
1
Kurt Alfred Kluever
1
Lesley Katzen
1
Marc Kaplan
3
Mark Ivey
1
Mark Striebeck
1
Marko Ivanković
1
Markus Clermont
3
Michael Bachman
1
Michael Klepikov
1
Mike Wacker
1
Misko Hevery
32
Mobile
2
Mona El Mahdy
1
Noel Yap
1
Patricia Legaspi
1
Patrick Copeland
23
Patrik Höglund
5
Peter Arrenbrecht
1
Phil Rollet
1
Philip Zembrod
4
Pooja Gupta
1
Radoslav Vasilev
1
Rajat Dewan
1
Rajat Jain
1
Rich Martin
1
Richard Bustamante
1
Roshan Sembacuttiaratchy
1
Ruslan Khamitov
1
Sean Jordan
1
Sharon Zhou
1
Shyam Seshadri
4
Simon Stewart
2
Stephen Ng
1
Tejas Shah
1
Test Analytics
1
Tony Voellm
2
TotT
54
Vojta Jína
1
WebRTC
2
Yvette Nameth
2
Zhanyong Wan
6
Zuri Kemp
2
Archive
2015
December
GTAC 2015 Wrap Up
November
October
August
June
May
April
March
February
January
2014
December
November
October
September
August
July
June
May
April
March
February
January
2013
December
November
October
August
July
June
May
April
March
January
2012
December
November
October
September
August
2011
November
October
September
August
July
June
May
April
March
February
January
2010
December
November
October
September
August
July
June
May
April
March
February
January
2009
December
November
October
September
August
July
June
May
April
February
January
2008
December
November
October
September
August
July
June
May
April
March
February
January
2007
October
September
August
July
June
May
April
March
February
January
Feed
Follow @googletesting
