googletesting

Multi-Repository Development

Friday, May 15, 2015

Author: Patrik Höglund roughly a hundred libraries

Figure 1. Holy dependencies, Batman!libphonenumberUpdating Dependencies is Hard

Update the library to its latest release. If it’s been a long time since you did this, it can be quite risky and you may have to spend significant testing resources to ensure all the accumulated changes don’t break your application. You may have to catch up to interface changes in the library as well.

Cherry-pick the feature/bug fix you want into your copy of the library. This is even riskier because your cherry-picked patches may depend on other changes in the library in subtle ways. Also, you still are not up to date with the latest version.

Find some way to make do without the feature or bug fix.

How “Rolling” Works DEPS{
# ...
'src/third_party/webrtc':
'https://chromium.googlesource.com/' +
'external/webrtc/trunk/webrtc.git' +
'@' + '5727038f572c517204e1642b8bc69b25381c4e9f',
}
572703...DEPSrollinggetUserMedia has started crashing on Androidmany tests, on all platforms we ship on

Figure 2. Life of a Roll Patch.Bots That Can See the Future: “FYI Bots” DEPS

If all the FYI bots were green, we knew a roll most likely would go smoothly.

If the bots didn’t compile, we knew we would have to adapt Chrome to an interface change in the next roll patch.

If the bots were red, we knew we either had a bug in WebRTC or that Chrome would have to be adapted to some semantic change in WebRTC.

FYI “waterfall”main waterfallMaking Gradual Interface Changes red blindness[1]API prime directive[2]class WebRtcAmplifier {
...
int SetOutputVolume(float volume);
}class WebRtcAmplifier {
...
int SetOutputVolume(float volume, bool allow_eleven¹);
}class WebRtcAmplifier {
...
int SetOutputVolume(float volume);
int SetOutputVolume2(float volume, bool allow_eleven);
}

Roll into Chrome

Make Chrome use SetOutputVolume2

Update SetOutputVolume’s signature

Roll again and make Chrome use SetOutputVolume

Delete SetOutputVolume2

Resultsperformance testsFuture Work: Optimistic Auto-rollingARB (AutoRollBot)ReferencesFootnotes

We actually did have a hilarious bug in WebRTC where it was possible to set the volume to 1.1, but only 0.0-1.0 was supposed to be allowed. No, really. Thus, our WebRTC implementation must be louder than the others since everybody knows 1.1 must be louder than 1.0.

Just Say No to More End-to-End Tests

Wednesday, April 22, 2015

by Mike Wacker End-to-End Tests in Theory ten things we know to be true

Developers like it because it offloads most, if not all, of the testing to others.

Managers and decision-makers like it because tests that simulate real user scenarios can help them easily determine how a failing test would impact the user.

Testers like it because they often worry about missing a bug or writing a test that does not verify real-world behavior; writing tests from the user's perspective often avoids both problems and gives the tester a greater sense of accomplishment.

End-to-End Tests in Practice

The latest version of the service is built.

This version is then deployed to the team's testing environment.

All end-to-end tests then run against this testing environment.

An email report summarizing the test results is sent to the team.

Days Left

Pass %

Notes

Everything is broken! Signing in to the service is broken. Almost all tests sign in a user, so almost all tests failed.

A partner team we rely on deployed a bad build to their testing environment yesterday.

-1

54%

A dev broke the save scenario yesterday (or the day before?). Half the tests save a document at some point in time. Devs spent most of the day determining if it's a frontend bug or a backend bug.

-2

54%

It's a frontend bug, devs spent half of today figuring out where.

-3

54%

A bad fix was checked in yesterday. The mistake was pretty easy to spot, though, and a correct fix was checked in today.

-4

Hardware failures occurred in the lab for our testing environment.

-5

84%

Many small bugs hiding behind the big bugs (e.g., sign-in broken, save broken). Still working on the small bugs.

-6

87%

We should be above 90%, but are not for some reason.

-7

89.54%

(Rounds up to 90%, close enough.) No fixes were checked in yesterday, so the tests must have been flaky yesterday.

Analysis What Went Well

Customer-impacting bugs were identified and fixed before they reached the customer.

What Went Wrong

The team completed their coding milestone a week late (and worked a lot of overtime).

Finding the root cause for a failing end-to-end test is painful and can take a long time.

Partner failures and lab failures ruined the test results on multiple days.

Many smaller bugs were hidden behind bigger bugs.

End-to-end tests were flaky at times.

Developers had to wait until the following day to know if a fix worked or not.

The True Value of Tests A failing test does not directly benefit the user. A bug fix directly benefits the user.

Stage

Failing Test

Bug Opened

Bug Fixed

Value Added

Yes

Building the Right Feedback Loop

It's fast. No developer wants to wait hours or days to find out if their change works. Sometimes the change does not work - nobody is perfect - and the feedback loop needs to run multiple times. A faster feedback loop leads to faster fixes. If the loop is fast enough, developers may even run tests before checking in a change.

It's reliable. No developer wants to spend hours debugging a test, only to find out it was a flaky test. Flaky tests reduce the developer's trust in the test, and as a result flaky tests are often ignored, even when they find real product issues.

It isolates failures. To fix a bug, developers need to find the specific lines of code causing the bug. When a product contains millions of lines of codes, and the bug could be anywhere, it's like trying to find a needle in a haystack.

Think Smaller, Not LargerUnit Tests

Unit tests are fast. We only need to build a small unit to test it, and the tests also tend to be rather small. In fact, one tenth of a second is considered slow for unit tests.

Unit tests are reliable. Simple systems and small units in general tend to suffer much less from flakiness. Furthermore, best practices for unit testing - in particular practices related to hermetic tests - will remove flakiness entirely.

Unit tests isolate failures. Even if a product contains millions of lines of code, if a unit test fails, you only need to search that small unit under test to find the bug.

buildstestsUnit Tests vs. End-to-End Tests

Unit

End-toEnd

Fast

Reliable

Isolates Failures

Simulates a Real User

Integration TestsTesting Pyramidtesting pyramid2014 Google Test Automation Conference

Inverted pyramid/ice cream cone. The team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests.

Hourglass. The team starts with a lot of unit tests, then uses end-to-end tests where integration tests should be used. The hourglass has many unit tests at the bottom and many end-to-end tests at the top, but few integration tests in the middle.

Quantum Quality

Wednesday, April 1, 2015

UPDATE: Hey, this was an April fool's joke but in fact we wished we couldhave realized this idea and we are looking forward to the day this hasbeen worked out and becomes a reality.
by Kevin Graney Quantum AI LabSchrodinger's cat

Figure 1 Some qubits inside a Google quantum device. nVon Neuman architecturenO(n)

Figure 2 The application state graph for a demonstrative 3-bit application. If the start state is 001 then 000, 110, 111, and 011 are all unreachable states. States 010 and 100 both result in deadlock. Once we have the state transition graph for the application under test, testing it becomes almost trivial. Given the initial startup state of the application, i.e. the executable bits of the application stored on disk, we can find from the application's state transition graph all reachable states. Assertions that ensure proper behavior are then written against the reachable subset of the transition graph. This paradigm of test writing allows both Google's security engineers and software engineers to work more productively. A security engineer can write a test, for example, that asserts "no executable memory regions become mutated in any reachable state". This one test effectively eliminates the potential for security flaws that result from memory safety violations. A test engineer can write higher level assertions using graph traversal methods that ensure data integrity is maintained across a subset of application state transitions. Tests of this nature can detect data corruption bugs.

Android UI Automated Testing

Friday, March 20, 2015

by Mona El Mahdy Overview Robolectricgradle unit tests supportEspressotest support librariesStrategy 1: Using an End-To-End Test as a UI Test UI testend-to-end (E2E)

Very large and slow tests.

High flakiness rate due to timeouts and memory issues.

Hard to debug/investigate failures.

Authentication issues (ex: authentication from automated tests is very tricky).

Strategy 2: Hermetic UI Testing using Fake Serversserver hermetic testsblog

Strategy 3: Dependency Injection Design for Apps. Daggerinstrumentation testing

Strategy 4: Building Apps into Smaller Libraries

Conclusion

Don’t write E2E tests instead of UI tests. Instead write unit tests and integration tests beside the UI tests.

Hermetic tests are the way to go.

Use dependency injection while designing your app.

Build your application into small libraries/modules, and test each one in isolation. You can then have a few integration tests to verify integration between components is correct .

Componentized UI tests have proven to be much faster than E2E and 99%+ stable. Fast and stable tests have proven to drastically improve developer productivity.

The First Annual Testing on the Toilet Awards

Tuesday, February 3, 2015

By Andrew Trenk createdErik Kuefler:Test Behaviors, Not MethodsDon't Put Logic in TestsAlex Eagle:Change-Detector Tests Considered Harmful

TotT blog posts

Testing on the Toilet: Change-Detector Tests Considered Harmful

Tuesday, January 27, 2015

by Alex Eagle

This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office. While fixing the tests, you get a sense that you are wasting time by mechanically applying the same transformation to many tests.What does it look like to write tests mechanically?// Production code:
def abs(i: Int)
return (i < 0) ? i * -1 : i

// Test code:
for (line: String in File(prod_source).read_lines())
switch (line.number)
1: assert line.content equals def abs(i: Int)
2: assert line.content equals return (i < 0) ? i * -1 : iThat test is clearly not usefulA correct or incorrect program is equally likely to pass// Production code:
def process(w: Work)
firstPart.process(w)
secondPart.process(w)

// Test code:
part1 = mock(FirstPart)
part2 = mock(SecondPart)
w = Work()
Processor(part1, part2).process(w)
verify_in_order
was_called part1.process(w)
was_called part2.process(w)This is a change-detector testit breaks in response to any change to the production code, without verifying correct behaviorChange detectors provide negative value

Testing on the Toilet: Prefer Testing Public APIs Over Implementation-Detail Classes

Wednesday, January 14, 2015

by Andrew Trenk

This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office. Does this class need to have tests? class UserInfoValidator {
public void validate(UserInfo info) {
if (info.getDateOfBirth().isInFuture()) { throw new ValidationException()); }
}
}what if its only user looks like this? public class UserInfoService {
private UserInfoValidator validator;
public void save(UserInfo info) {
validator.validate(info); // Throw an exception if the value is invalid.
writeToDatabase(info);
}
}it probably doesn’t need testsUserInfoServicethe class is an implementation detail, not a public APIA public API can be called by any number of usersYou want to make sure these are well-testedAn implementation-detail class exists only to support public APIsThese classes can sometimes be tested indirectlyTesting implementation-detail classes is still useful in many casesthey often don’t need to be tested in as much depth as a public API,UserInfoServiceUserInfonullnullUserInfoValidator.validateYou should also try to restrict the visibility of implementation-detail classes,Testing implementation-detail classes too often leads to a couple problemsCode is harder to maintain since you need to update tests more oftenIf you test a behavior only through an implementation-detail class, you may get false confidence in your codeYou also have to be more careful when refactoring

Testing Blog

Multi-Repository Development

Just Say No to More End-to-End Tests

Quantum Quality

Android UI Automated Testing

The First Annual Testing on the Toilet Awards

Testing on the Toilet: Change-Detector Tests Considered Harmful

Testing on the Toilet: Prefer Testing Public APIs Over Implementation-Detail Classes

Labels

Archive

Feed

Company-wide

Products

Developers