googletesting

Testing on the Toilet: Risk-Driven Testing

Friday, May 30, 2014

by Peter ArrenbrechtThis article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office. We are all conditioned to write testswe no longer question why we write themTests are a means to an end:reduce the key risksget the biggest bang for the buck“We built a new debugging aid. We wrote unit, integration, and UI tests. We were ready to launch.”

Outstanding practice. Missing the mark.

Our key risks were that we'd corrupt our data or bring down our servers for the sake of a debugging aid. None of the tests addressed this, but they gave a false sense of safety and “being done”.
We stopped the launch.“We wanted to turn down a feature, so we needed to alert affected users. Again we had unit and integration tests, and even one expensive end-to-end test.”

Standard practice. Wasted effort.

The alert was so critical it actually needed end-to-end coverage for all scenarios. But it would be live for only three releases. The cheapest effective test? Manual testing before each release.A Better Approach: Risks First think about testingDo this at the startWrite them downstandard practice remains a good idea in most casesJust rememberThe bang is what countsmaximize it

Testing on the Toilet: Effective Testing

Wednesday, May 7, 2014

by Rich Martin, Zurich
This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office. how effective are our tests at detecting and reporting bugs in our codethree important qualitiesFidelity A high-fidelity test is one which is very sensitive to defects in the code under testResilience A resilient test is one that only fails when a breaking change is made to the code under test.Precision a high-precision test tells you exactly where the defect liesAs you design and write tests, use these qualities as a framework to guide your implementation

Testing on the Toilet: Test Behaviors, Not Methods

Monday, April 14, 2014

by Erik Kuefler This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office. it can be harmful to think that tests and public methods should have a 1:1 relationship@Test public void testProcessTransaction() {
User user = newUserWithBalance(LOW_BALANCE_THRESHOLD.plus(dollars(2));
transactionProcessor.processTransaction(
user,
new Transaction("Pile of Beanie Babies", dollars(3)));
assertContains("You bought a Pile of Beanie Babies", ui.getText());
assertEquals(1, user.getEmails().size());
assertEquals("Your balance is low", user.getEmails().get(0).getSubject());
}
Tests like this very often become massive and difficult to maintain over time as additional behaviors keep getting added inIt's a much better idea to use separate tests to verify separate behaviors@Test public void testProcessTransaction_displaysNotification() {
transactionProcessor.processTransaction(
new User(), new Transaction("Pile of Beanie Babies"));
assertContains("You bought a Pile of Beanie Babies", ui.getText());
}
@Test public void testProcessTransaction_sendsEmailWhenBalanceIsLow() {
User user = newUserWithBalance(LOW_BALANCE_THRESHOLD.plus(dollars(2));
transactionProcessor.processTransaction(
user,
new Transaction(dollars(3)));
assertEquals(1, user.getEmails().size());
assertEquals("Your balance is low", user.getEmails().get(0).getSubject());
}
This will make your tests more resilient since adding new behaviors is unlikely to break the existing tests, and clearer since each test contains code to exercise only one behavior

The Real Test Driven Development

Tuesday, April 1, 2014

Update: APRIL FOOLS!

by Kaue Silveira GoogleplexThe Concept RealProgram Synthesis

See it in action: class LinkGeneratorTest(googletest.TestCase):

def setUp(self):
self.generator = link_generator.LinkGenerator()

def testGetLinkFromIDs(self):
expected = ('https://frontend.google.com/advancedSearchResults?'
's.op=ALL&s.r0.field=ID&s.r0.val=1288585+1310696+1346270+')
actual = self.generator.GetLinkFromIDs(set((1346270, 1310696, 1288585)))
self.assertEqual(expected, actual)import urllib

class LinkGenerator(object):

_URL = (
'https://frontend.google.com/advancedSearchResults?'
's.op=ALL&s.r0.field=ID&s.r0.val=')

def GetLinkFromIDs(self, ids):
result = []
for id in sorted(ids):
result.append('%s ' % id)
return self._URL + urllib.quote_plus(''.join(result))
How does it work? Google BrainType InferenceGirard-Reynolds IsomorphismSpanner

How can I use it? GPL-licensedBSD-licensed

Testing on the Toilet: What Makes a Good Test?

Tuesday, March 18, 2014

by Erik Kuefler This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office. writing good tests is about much more than just verifying correctnessClarity means that a test should serve as readable documentation for humans, describing the code being tested in terms of its public APIs.A test is complete when its body contains all of the information you need to understand it, and concise when it doesn't contain any other distracting information.@Test public void shouldPerformAddition() {
Calculator calculator = new Calculator(new RoundingStrategy(),
"unused", ENABLE_COSIN_FEATURE, 0.01, calculusEngine, false);
int result = calculator.doComputation(makeTestComputation());
assertEquals(5, result); // Where did this number come from?
}@Test public void shouldPerformAddition() {
Calculator calculator = newCalculator();
int result = calculator.doComputation(makeAdditionComputation(2, 3));
assertEquals(5, result);
}resilient test doesn't have to change unless the purpose or behavior of the class being tested changes

When/how to use Mockito Answer

Monday, March 3, 2014

by Hongfei Ding, Software Engineer, Shanghai Mockitointerface Service {
Data get();
}ServiceServicewhen-returnwhen(service.get()).thenReturn(cannedData);when-returnAnswerAnswerstub asynchronous methods that have callbacksinterface Service {
void get(Callback callback);
}when-returnAnsweremulate a successonSuccessdoAnswer(new Answer<Void>() {
public Void answer(InvocationOnMock invocation) {
Callback callback = (Callback) invocation.getArguments()[0];
callback.onSuccess(cannedData);
return null;
}
}).when(service).get(any(Callback.class));Answermake smarter stubs for synchronous methodsTranslatorinterface Translator {
String translate(String msg);
}Translatortranslatortesting interaction rather than testing stateAnswerto reverse the input message as a fake translationtranslatemsgthenAnswerdoAnswerwhen(translator.translate(any(String.class))).thenAnswer(reverseMsg())
...
// extracted a method to put a descriptive name
private static Answer<String> reverseMsg() {
return new Answer<String>() {
public String answer(InvocationOnMock invocation) {
return reverseString((String) invocation.getArguments()[0]));
}
}
}Answerfake

Minimizing Unreproducible Bugs

Monday, February 3, 2014

by Anthony ValloneAvoid and test for race conditions, deadlocks, timing issues, memory corruption, uninitialized memory access, memory leaks, and resource issues Guidelines for development:

Simplify your synchronization logic. If it’s too hard to understand, it will be difficult to reproduce and debug complex concurrency problems.

Always obtain locks in the same order. This is a tried-and-true guideline to avoid deadlocks, but I still see code that breaks it periodically. Define an order for obtaining multiple locks and never change that order.

Don’t optimize by creating many fine-grained locks, unless you have verified that they are needed. Extra locks increase concurrency complexity.

Avoid shared memory, unless you truly need it. Shared memory access is very easy to get wrong, and the bugs may be quite difficult to reproduce.

Guidelines for testing:

Stress test your system regularly. You don't want to be surprised by unexpected failures when your system is under heavy load.

Test timeouts. Create tests that mock/fake dependencies to test timeout code. If your timeout code does something bad, it may cause a bug that only occurs under certain system conditions.

Test with debug and optimized builds. You may find that a well behaved debug build works fine, but the system fails in strange ways once optimized.

Test under constrained resources. Try reducing the number of data centers, machines, processes, threads, available disk space, or available memory. Also try simulating reduced network bandwidth.

Test for longevity. Some bugs require a long period of time to reveal themselves. For example, persistent data may become corrupt over time.

Use dynamic analysis tools like memory debuggers, ASan, TSan, and MSan regularly. They can help identify many categories of unreproducible memory/threading issues.

Enforce preconditions void ScheduleEvent(int timeDurationMilliseconds) {
if (timeDurationMilliseconds <= 0) {
timeDurationMilliseconds = 1;
}
...
}
Guidelines for development:

Enforce preconditions in your functions unless you have a good reason not to.

Use defensive programming Defensive programmingdouble GetMonthlyLoanPayment() {
double rate = GetTodaysInterestRateFromExternalSystem();
if (rate < 0.001 || rate > 0.5) {
throw BadInterestRate(rate);
}
...
}
Guidelines for development:

When possible, use defensive programming to verify the work of your dependencies with known risks of failure like user-provided data, I/O operations, and RPC calls.

Guidelines for testing:

Use fuzz testing to test your systems hardiness when enduring bad data.

Don’t hide all errors from the user Guidelines for development:

Only hide errors from the user when you are certain that there is no impact to system state or the user.

Any error with impact to the user should be reported to the user with instructions for how to proceed. The information shown to the user, combined with data available to an engineer, should be enough to determine what went wrong.

Test error handling Guidelines for testing:

Always test your error handling code. This is usually best accomplished by mocking or faking the component triggering the error.

It’s also a good practice to examine your log quality for all types of error handling.

Check for duplicate keys Guidelines for development:

Try to guarantee uniqueness of all keys.

When not possible to guarantee unique keys, check if the recently generated key is already in use before using it.

Watch out for potential race conditions here and avoid them with synchronization.

Test for concurrent data access Guidelines for testing:

Always test for concurrent data access if it’s a feature of the system. Actually, even if it’s not a feature, verify that the system rejects it. Testing concurrency can be challenging. An approach that usually works for me is to create many worker threads that simultaneously attempt access and a master thread that monitors and verifies that some number of attempts were indeed concurrent, blocked or allowed as expected, and all were successful. Programmatic post-analysis of all attempts and changing system state may also be necessary to ensure that the system behaved well.

Steer clear of undefined behavior and non-deterministic access to data undefined behaviorJava’s SetGuidelines for development:

Understand when the APIs and operations you use might have undefined behavior and prevent those conditions.

Do not depend on data structure iteration order unless it is guaranteed. It is a common mistake to depend on the ordering of sets or associative arrays.

Log the details for errors or test failures Guidelines for development:

Follow good logging practices, especially in your error handling code.

If logs are stored on a user’s machine, create an easy way for them to provide you the logs.

Guidelines for testing:

Save your test logs for potential analysis later.

Anything to add?

Testing Blog

Testing on the Toilet: Risk-Driven Testing

Testing on the Toilet: Effective Testing

Testing on the Toilet: Test Behaviors, Not Methods

The Real Test Driven Development

Testing on the Toilet: What Makes a Good Test?

When/how to use Mockito Answer

Minimizing Unreproducible Bugs

Labels

Archive

Feed

Company-wide

Products

Developers

Testing Blog

Testing on the Toilet: Risk-Driven Testing

Testing on the Toilet: Effective Testing

Testing on the Toilet: Test Behaviors, Not Methods

The *Real* Test Driven Development

Testing on the Toilet: What Makes a Good Test?

When/how to use Mockito Answer

Minimizing Unreproducible Bugs

Labels

Archive

Feed

Company-wide

Products

Developers

The Real Test Driven Development