Testing Blog
How Google Tests Software - A Break for Q&A
Wednesday, May 4, 2011
By James Whittaker
New material for the this series is coming more slowly. I am beginning to get into areas where I want to start posting screen shots of internal Google tools and describe how our infrastructure works. This is material that takes longer to develop and also requires some scrutiny before being published externally. So in the meantime, I am pausing to answer some of the questions you've posted in the comments.
I am going to start with
Lilia
(because she likes Neil Young mainly, but also because she can run further than me and those two things combine to impress me to no small end) who asks about SET-SWE conversion and vice-versa and which I have seen the most. There is also the broader question of whether there is a ceiling on the SET career path.
SETs and SWEs are on the same pay scale and virtually the same job ladder. Both roles are essentially 100% coding roles with the former writing test code and the latter doing feature development. From a coding perspective the skill set is a dead match. From a testing perspective we expect a lot more from SETs. But the overlap on coding makes SETs a great fit for SWE positions and vice versa. Personally I think it is a very healthy situation to have conversions. Since I have both roles reporting to me I can speak from first hand experience that many of my best coders are former SETs and some of my best testers are former SWEs. Each is excellent training ground for the other. On my specific team I am even on the conversions from one role to the other. But I suspect that Google-wide there are more SETs who become SWEs.
Why convert in the first place? Well at Google it isn't for the money. It also isn't for the prestige as we have a lot more SWEs than SETs and it is a lot harder to standout. The scarcity of our SETs creates somewhat of a mystique about these folk. Who are these rare creatures who keep our code bases healthy and make our development process run so smoothly? Actually, most SWEs care more about making the SETs happy so they continue doing what they do. Why would any dev team force a conversion of a great developer from SET to SWE when finding a suitable SET replacement is so much harder than adding another feature developer? SWEs ain't that stupid.
Now pausing before I take another hit of the corp kool-aid, let me be honest and say that there are far more senior SWEs than SETs. Percentage wise we test folk are more outnumbered at the top of the org than at the middle and bottom. But keep in mind that developers have had a large head start on us. We have developers who have been at Google since our founding and testers ... well ... less time than that.
Where do TEs fit into this mix? TE is an even newer role than SET but already we have a number climbing to the Staff ranks and pushing on the senior most positions in the company. There is no ceiling, but the journey to the top takes some time.
Raghev
among others has asked about the career path and whether remaining an IC (individual contributor) is an option over becoming a manager. I have mixed feelings about answering this. As a manager myself, I see the role as one with much honor and yet I hear in your collective voices a hint of
why do I have to become a manager?
Ok, I admit, Dilbert is funny.
For me, being a manager is a chance to impart some of my experience and old-guy judgement on less experienced but more technically gifted ICs. The combination of an experienced manager's vision and an ICs technical skill can be a fighting force of incredible power. And yet, why should someone who does not want to manage be forced to do so in order to continue their career advancement?
Well, fortunately, Google does not make us choose. Our managers are expected to have IC tasks they perform. They are expected to be engaged technically and lead as opposed to just manage. And our ICs are expected to have influence beyond their personal work area. When you get to the senior/staff positions here you are a leader, period. Some leaders lead more than they manage and some leaders manage more than they lead.
But either way, the view from the top means that a lot of people are looking to you for direction ... whether you manage them or not.
How Google Tests Software - Part Six
Monday, May 2, 2011
By James Whittaker
The Life of an SET
SETs are Software Engineers in Test. They are software engineers who happen to write testing functionality. First and foremost, SETs are developers and the role is touted as a 100% coding role in our recruiting literature and internal job promotion ladders. When SET candidates are interviewed, the “coding bar” is nearly identical to the SWE role with more emphasis that SETs know how to test the code they create. In other words, both SWEs and SETs answer coding questions. SETs are expected to nail a set of testing questions as well.
As you might imagine, it is a difficult role to fill and it is entirely possible that the low numbers of SETs isn’t because Google has created a magic formula for productivity but more of a result of adapting our engineering practice around the reality that the SET skill set is really hard to find. We optimize on this very important task and build processes around the people who do it.
It is usually the case that SETs are not involved early in the design phase. Their exclusion is not so much purposeful as it is a by-product of how a lot of Google projects are born. A common scenario for new project creation is that some informal 20% effort takes a life of its own as an actual Google branded product. Gmail and Chrome OS are both projects that started out as ideas that were not formally mandated by Google but over time grew into shipping products with teams of developers and testers working on them. In such cases early development is not about quality, it is about proving out a concept and working on things like scale and performance that must be right before quality could even be an issue. If you can't build a web service that scales, testing is not your biggest problem!
Once it is clear that a product can and will be built and shipped, that's when the development team seeks out test involvement.
You can imagine a process like this: someone has an idea, they think about it, write experimental code, seek out opinions of others, write some more code, get others involved, write even more code, realize they are onto something important, write more code to mold the idea into something that they can present to others to get feedback ... somewhere in all this an actual project is created in Google's project database and the project becomes real. Testers don't get involved until it becomes real.
Do all real projects get testers? Not by default. Smaller projects and those meant for limited users often get tested exclusively by the people who build it. Others that are riskier to our users or the enterprise (much more about risk later) get testing attention.
The onus is on the development teams to solicit help from testers and convince them that their project is exciting and full of potential. Dev Directors explain their project, progress and ship schedule to Test Directors who then discuss how the testing burden is to be shared and agree on things like SWE involvement in testing, expected unit testing levels and how the duties of the release process are going to be shared. SETs may not be involved at project inception, but once the project becomes real we have vast influence over how it is to be executed.
And when I say "testing" I don't just mean exercising code paths. Testers might not be involved from the beginning ... but
testing
is. In fact, an SET's impact is felt even before a developer manages to check code into the build. Stay tuned to understand what I am talking about.
GTAC 2011: Cloudy with a Chance of Tests
Tuesday, April 26, 2011
By James Whittaker
Has it only been 179 days since the last GTAC? My how time flies when you have lots of testing to do!
It is my pleasure to announce that not only is our collective attention being drawn back to this most intriguing test conference but yours truly, along with "Shoeless" Brad Green, have been asked to plan it. Given that Brad and I both spend a lot of time with our heads in the Cloud, this year's theme is
cloudy with a chance of tests
.
The composite meaning of this theme is purposeful: Testing apps that reside in the Cloud is itself a cloudy, as in opaque, process. Clouds on the horizon often signal change and testing in the Cloud certainly changes things. The Cloud breaks old testing paradigms and tools requiring that even the tried-and-true be rethought and recast. And, yes, the future of testing itself is cloudy and the need for testers and testing as it exists today is unclear. There are clouds gathering on the horizons of the discipline and this conference will be dedicated to interpreting their meaning and planning for their arrival. As always, GTAC will attempt to bring together people who have thought deeply about these subjects and are responsible for actual progress, technology and insights that will benefit others in the community.
We are still in the very earliest stages of planning, but here are the details as we know them:
GTAC 2011
will be in
Mountain View, CA
the week of October 25th and will be held at the
Computer History Museum
. GTAC or no GTAC, this place is well worth a visit and as an actual conference venue just might be the coolest location for GTAC ever.
As in the past, GTAC 2011 will feature a single track, all-keynote format with both internal and external speakers. We will be soliciting feedback from potential attendees about what topics and speakers are the most interesting. Our opening keynote has been determined already and it is none other than our most famous Alberto Savoia, translator of the ancient tome
The Way of Testivus
, and agitator extraordinaire with quotes like "
Building the right 'it' is more important than building 'it' right.
" If anyone can gather the clouds into a hurricane, it's Alberto. You can also expect updates on our open source test tools strategy.
One important addition we seek to make this year is to have a
Test Executive Session
sometime during the event. The idea is to gather top decision makers and budget owners at the biggest/best/most influential web companies on the planet. It will be a discussion about testing culture, organizational structure, technology deployment, innovation and so forth by the very people who can make change happen within their company. Our readers will be asked to nominate their Directors, VPs, SVPs and so forth who own their companies' testing charter.
Stay tuned!
Entrepreneurial Innovation at Google
Tuesday, April 5, 2011
The April 2011 issue of Computer,
the flagship publication of the IEEE Computer Society,
features a cover article written by
Alberto Savoia and
Patrick Copeland (
@copelandpatrick
). The article,
Entrepreneurial Innovation at Google, explores some of the ideas we use to encourage engineers to well, innovate. For a limited time you can see a digital version of the article on
Computing Now – see
http://www.computer.org/
portal/web/computingnow
.
Enjoy,
Alberto & Pat
The SET Career Path
Tuesday, April 5, 2011
By James Whittaker
I've had a number of questions about the SET role and it seems I have confused folks when I say that the SWE is a tester and the SET is a tester and at the same time the SWE is a developer and the SET is a developer. What could possibly be confusing about that?
Oh, yeah. Right.
My next series of posts are going to detail the role of the SET and all will eventually be clear but some clarification on career path seems worthwhile.
SETs are developers who write test code and automation as their primary task. They are in every sense of the word a developer. When we interview SETs, SWEs are on the interview loop and SWE questions are asked. They are not all of the interview, but they are part of it.
This means that the skill set that our SETs possess makes them perfect candidates for switching to the SWE role. There is neither incentive nor deterrent to do so. SETs and SWEs are on the same pay scale and bonus structure (I have both roles reporting to me so I have real visibility into salary data) and their promotion velocity (again based on actual data) is roughly equivalent. This means that SETs have no outside influences to prompt them one way or the other.
The key factor is really the type of work you are doing. SETs who find themselves involved in SWE work usually convert to SWE. SWEs are also drawn in the opposite direction. Much of this happens through our 20% time work. Any SET interested in SWE work can take on a 20% task doing feature development. Any SWE interested in automation can find a group and sign up for a 20%. Right now I have both SWEs and SETs involved in such cross pollination.
The ideal situation is that the title reflects the actual work that you are involved in. So if an SET starts doing more feature dev work than automation, he or she should convert, same for SWEs doing automation work. In my time here, conversions in both directions have happened, but it is not all that common. The work of both roles is engaging, interesting and intense. Few Googlers are walking around bored.
Bottom line: do the work you are passionate about and capable of and the right job title will find you.
We're back live on twitter
Thursday, March 31, 2011
For our followers on twitter, we didn't realize our feed we broken until today! Shame, lots of good posts recently. Come check them out.
@googletesting
How Google Tests Software - Part Five
Wednesday, March 23, 2011
By James Whittaker
Instead of distinguishing between code, integration and system testing, Google uses the language of small, medium and large tests emphasizing scope over form. Small tests cover small amounts of code and so on. Each of the three engineering roles may execute any of these types of tests and they may be performed as automated or manual tests.
Small Tests
are mostly (but not always) automated and exercise the code within a single function or module. They are most likely written by a SWE or an SET and may require mocks and faked environments to run but TEs often pick these tests up when they are trying to diagnose a particular failure. For small tests the focus is on typical functional issues such as data corruption, error conditions and off by one errors. The question a small test attempts to answer is does this code do what it is supposed to do?
Medium Tests
can be automated or manual and involve two or more features and specifically cover the interaction between those features. I've heard any number of SETs describe this as "testing a function and its nearest neighbors." SETs drive the development of these tests early in the product cycle as individual features are completed and SWEs are heavily involved in writing, debugging and maintaining the actual tests. If a test fails or breaks, the developer takes care of it autonomously. Later in the development cycle TEs may perform medium tests either manually (in the event the test is difficult or prohibitively expensive to automate) or with automation. The question a medium test attempts to answer is does a set of near neighbor functions interoperate with each other the way they are supposed to?
Large Tests
cover three or more (usually more) features and represent real user scenarios to the extent possible. There is some concern with overall integration of the features but large tests tend to be more results driven, i.e., did the software do what the user expects? All three roles are involved in writing large tests and everything from automation to exploratory testing can be the vehicle to accomplish accomplish it. The question a large test attempts to answer is does the product operate the way a user would expect?
The actual language of small, medium and large isn’t important. Call them whatever you want. The important thing is that Google testers share a common language to talk about what is getting tested and how those tests are scoped. When some enterprising testers began talking about a fourth class they dubbed
enormous
every other tester in the company could imagine a system-wide test covering nearly every feature and that ran for a very long time. No additional explanation was necessary.
The primary driver of what gets tested and how much is a very dynamic process and varies wildly from product to product. Google prefers to release often and leans toward getting a product out to users so we can get feedback and iterate. The general idea is that if we have developed some product or a new feature of an existing product we want to get it out to users as early as possible so they may benefit from it. This requires that we involve users and external developers early in the process so we have a good handle on whether what we are delivering is hitting the mark.
Finally, the mix between automated and manual testing definitely favors the former for all three sizes of tests. If it can be automated and the problem doesn’t require human cleverness and intuition, then it should be automated. Only those problems, in any of the above categories, which specifically require human judgment, such as the beauty of a user interface or whether exposing some piece of data constitutes a privacy concern, should remain in the realm of manual testing.
Having said that, it is important to note that Google performs a great deal of manual testing, both scripted and exploratory, but even this testing is done under the watchful eye of automation. Industry leading recording technology converts manual tests to automated tests to be re-executed build after build to ensure minimal regressions and to keep manual testers always focusing on new issues. We also automate the submission of bug reports and the routing of manual testing tasks. For example, if an automated test breaks, the system determines the last code change that is the most likely culprit, sends email to its authors and files a bug. The ongoing effort to automate to within the “last inch of the human mind” is currently the design spec for the next generation of test engineering tools Google is building.
Those tools will be highlighted in future posts. However, my next target is going to revolve around The Life of an SET. I hope you keep reading.
Labels
Aaron Jacobs
1
Adam Porter
1
Alan Faulkner
1
Alan Myrvold
1
Alberto Savoia
4
Alek Icev
2
Alex Eagle
1
Allen Hutchison
6
Andrew Trenk
8
Android
1
Anthony Vallone
25
Antoine Picard
1
APIs
2
App Engine
1
April Fools
2
Arif Sukoco
1
Bruce Leban
1
C++
11
Chaitali Narla
2
Christopher Semturs
1
Chrome
3
Chrome OS
2
Dave Chen
1
Diego Salas
2
Dmitry Vyukov
1
Dori Reuveni
1
Eduardo Bravo Ortiz
1
Ekaterina Kamenskaya
1
Erik Kuefler
3
Espresso
1
George Pirocanac
2
Google+
1
Goranka Bjedov
1
GTAC
54
Hank Duan
1
Harry Robinson
5
Havard Rast Blok
1
Hongfei Ding
1
James Whittaker
42
Jason Arbon
2
Jason Elbaum
1
Jason Huggins
1
Java
5
JavaScript
7
Jay Han
1
Jessica Tomechak
1
Jim Reardon
1
Jobs
14
Joe Allan Muharsky
1
Joel Hynoski
1
John Penix
1
John Thomas
3
Jonathan Rockway
1
Jonathan Velasquez
1
Julian Harty
5
Julie Ralph
1
Karin Lundberg
1
Kaue Silveira
1
Kevin Graney
1
Kirkland
1
Kurt Alfred Kluever
1
Lesley Katzen
1
Marc Kaplan
3
Mark Ivey
1
Mark Striebeck
1
Marko Ivanković
1
Markus Clermont
3
Michael Bachman
1
Michael Klepikov
1
Mike Wacker
1
Misko Hevery
32
Mobile
2
Mona El Mahdy
1
Noel Yap
1
Patricia Legaspi
1
Patrick Copeland
23
Patrik Höglund
5
Peter Arrenbrecht
1
Phil Rollet
1
Philip Zembrod
4
Pooja Gupta
1
Radoslav Vasilev
1
Rajat Dewan
1
Rajat Jain
1
Rich Martin
1
Richard Bustamante
1
Roshan Sembacuttiaratchy
1
Ruslan Khamitov
1
Sean Jordan
1
Sharon Zhou
1
Shyam Seshadri
4
Simon Stewart
2
Stephen Ng
1
Tejas Shah
1
Test Analytics
1
Tony Voellm
2
TotT
54
Vojta Jína
1
WebRTC
2
Yvette Nameth
2
Zhanyong Wan
6
Zuri Kemp
2
Archive
2015
December
GTAC 2015 Wrap Up
November
October
August
June
May
April
March
February
January
2014
December
November
October
September
August
July
June
May
April
March
February
January
2013
December
November
October
August
July
June
May
April
March
January
2012
December
November
October
September
August
2011
November
October
September
August
July
June
May
April
March
February
January
2010
December
November
October
September
August
July
June
May
April
March
February
January
2009
December
November
October
September
August
July
June
May
April
February
January
2008
December
November
October
September
August
July
June
May
April
March
February
January
2007
October
September
August
July
June
May
April
March
February
January
Feed
Follow @googletesting
