Testing Blog
ThreadSanitizer: Slaughtering Data Races
Monday, June 30, 2014
by Dmitry Vyukov, Synchronization Lookout, Google, Moscow
Hello,
I work in the Dynamic Testing Tools team at Google. Our team develops tools like
AddressSanitizer
,
MemorySanitizer
and
ThreadSanitizer
which find various kinds of bugs. In this blog post I want to tell you about ThreadSanitizer, a fast data race detector for C++ and Go programs.
First of all, what is a
data race
? A data race occurs when two threads access the same variable concurrently, and at least one of the accesses attempts is a write. Most programming languages provide very weak guarantees, or no guarantees at all, for programs with data races. For example, in C++ absolutely any data race renders the behavior of the whole program as completely undefined (yes, it can suddenly format the hard drive). Data races are common in concurrent programs, and they are notoriously hard to debug and localize. A typical manifestation of a data race is when a program occasionally crashes with obscure symptoms, the symptoms are different each time and do not point to any particular place in the source code. Such bugs can take several months of debugging without particular success, since typical debugging techniques do not work. Fortunately, ThreadSanitizer can catch most data races in the blink of an eye. See Chromium
issue 15577
for an example of such a data race and
issue 18488
for the resolution.
Due to the complex nature of bugs caught by ThreadSanitizer, we don't suggest waiting until product release validation to use the tool. For example, in Google, we've made our tools easily accessible to programmers during development, so that anyone can use the tool for testing if they suspect that new code might introduce a race. For both Chromium and Google internal server codebase, we run unit tests that use the tool continuously. This catches many regressions instantly. The Chromium project has recently started using ThreadSanitizer on
ClusterFuzz
, a large scale fuzzing system. Finally, some teams also set up periodic end-to-end testing with ThreadSanitizer under a realistic workload, which proves to be extremely valuable. When races are found by the tool, our team has zero tolerance for races and does not consider any race to be benign, as even the most benign races can lead to
memory corruption
.
Our tools are
dynamic
(as opposed to
static tools
). This means that they do not merely "look" at the code and try to surmise where bugs can be; instead they they
instrument
the binary at build time and then analyze dynamic behavior of the program to catch it red-handed. This approach has its pros and cons. On one hand, the tool does not have any false positives, thus it does not bother a developer with something that is not a bug. On the other hand, in order to catch a bug, the test must expose a bug -- the racing data access attempts must be executed in different threads. This requires writing good multi-threaded tests and makes end-to-end testing especially effective.
As a bonus, ThreadSanitizer finds some other types of bugs: thread leaks, deadlocks, incorrect uses of mutexes, malloc calls in signal handlers, and
more
. It also natively understands atomic operations and thus can find bugs in
lock-free
algorithms (see e.g.
this bug
in the V8 concurrent garbage collector).
The tool is supported by both Clang and GCC compilers (only on Linux/Intel64). Using it is very simple: you just need to add a
-fsanitize=thread
flag during compilation and linking. For Go programs, you simply need to add a
-race flag
to the go tool (supported on Linux, Mac and Windows).
Interestingly, after integrating the tool into compilers, we've found some bugs in the compilers themselves. For example, LLVM was
illegally widening stores
, which can introduce very harmful data races into otherwise correct programs. And GCC was injecting
unsafe code
for initialization of function static variables. Among our other trophies are more than
a thousand bugs
in Chromium, Firefox, the Go standard library, WebRTC, OpenSSL, and of course in our internal projects.
So what are you waiting for? You know what to do!
GTAC 2014: Call for Proposals & Attendance
Monday, June 16, 2014
Posted by Anthony Vallone on behalf of the GTAC Committee
The application process is now open for presentation proposals and attendance for GTAC (Google Test Automation Conference) (
see initial announcement
) to be held at the
Google Kirkland office (near Seattle, WA)
on October 28 - 29th, 2014.
GTAC will be streamed live on YouTube again this year, so even if you can’t attend, you’ll be able to watch the conference from your computer.
Speakers
Presentations are targeted at student, academic, and experienced engineers working on test automation. Full presentations and lightning talks are 45 minutes and 15 minutes respectively. Speakers should be prepared for a question and answer session following their presentation.
Application
For presentation proposals and/or attendance,
complete this form
. We will be selecting about 300 applicants for the event.
Deadline
The due date for both presentation and attendance applications is July 28, 2014.
Fees
There are no registration fees, and we will send out detailed registration instructions to each invited applicant. Meals will be provided, but speakers and attendees must arrange and pay for their own travel and accommodations.
Update
:
Our
contact
email was bouncing - this is now fixed.
GTAC 2014 Coming to Seattle/Kirkland in October
Wednesday, June 4, 2014
Posted by Anthony Vallone on behalf of the GTAC Committee
If you're looking for a place to discuss the latest innovations in test automation, then charge your tablets and pack your
gumboots
- the eighth
GTAC
(Google Test Automation Conference) will be held on October 28-29, 2014 at Google Kirkland! The Kirkland office is part of the
Seattle/Kirkland campus
in beautiful Washington state. This campus forms our third largest engineering office in the USA.
GTAC is a periodic conference hosted by Google, bringing together engineers from industry and academia to discuss advances in test automation and the test engineering computer science field. It’s a great opportunity to present, learn, and challenge modern testing technologies and strategies.
You can browse the presentation abstracts, slides, and videos from last year on the
GTAC 2013 page
.
Stay tuned to this blog and the GTAC website for application information and opportunities to present at GTAC. Subscribing to this blog is the best way to get notified. We're looking forward to seeing you there!
Labels
Aaron Jacobs
1
Adam Porter
1
Alan Faulkner
1
Alan Myrvold
1
Alberto Savoia
4
Alek Icev
2
Alex Eagle
1
Allen Hutchison
6
Andrew Trenk
8
Android
1
Anthony Vallone
25
Antoine Picard
1
APIs
2
App Engine
1
April Fools
2
Arif Sukoco
1
Bruce Leban
1
C++
11
Chaitali Narla
2
Christopher Semturs
1
Chrome
3
Chrome OS
2
Dave Chen
1
Diego Salas
2
Dmitry Vyukov
1
Dori Reuveni
1
Eduardo Bravo Ortiz
1
Ekaterina Kamenskaya
1
Erik Kuefler
3
Espresso
1
George Pirocanac
2
Google+
1
Goranka Bjedov
1
GTAC
54
Hank Duan
1
Harry Robinson
5
Havard Rast Blok
1
Hongfei Ding
1
James Whittaker
42
Jason Arbon
2
Jason Elbaum
1
Jason Huggins
1
Java
5
JavaScript
7
Jay Han
1
Jessica Tomechak
1
Jim Reardon
1
Jobs
14
Joe Allan Muharsky
1
Joel Hynoski
1
John Penix
1
John Thomas
3
Jonathan Rockway
1
Jonathan Velasquez
1
Julian Harty
5
Julie Ralph
1
Karin Lundberg
1
Kaue Silveira
1
Kevin Graney
1
Kirkland
1
Kurt Alfred Kluever
1
Lesley Katzen
1
Marc Kaplan
3
Mark Ivey
1
Mark Striebeck
1
Marko Ivanković
1
Markus Clermont
3
Michael Bachman
1
Michael Klepikov
1
Mike Wacker
1
Misko Hevery
32
Mobile
2
Mona El Mahdy
1
Noel Yap
1
Patricia Legaspi
1
Patrick Copeland
23
Patrik Höglund
5
Peter Arrenbrecht
1
Phil Rollet
1
Philip Zembrod
4
Pooja Gupta
1
Radoslav Vasilev
1
Rajat Dewan
1
Rajat Jain
1
Rich Martin
1
Richard Bustamante
1
Roshan Sembacuttiaratchy
1
Ruslan Khamitov
1
Sean Jordan
1
Sharon Zhou
1
Shyam Seshadri
4
Simon Stewart
2
Stephen Ng
1
Tejas Shah
1
Test Analytics
1
Tony Voellm
2
TotT
54
Vojta Jína
1
WebRTC
2
Yvette Nameth
2
Zhanyong Wan
6
Zuri Kemp
2
Archive
2015
December
November
October
August
June
May
April
March
February
January
2014
December
November
October
September
August
July
June
ThreadSanitizer: Slaughtering Data Races
GTAC 2014: Call for Proposals & Attendance
GTAC 2014 Coming to Seattle/Kirkland in October
May
April
March
February
January
2013
December
November
October
August
July
June
May
April
March
January
2012
December
November
October
September
August
2011
November
October
September
August
July
June
May
April
March
February
January
2010
December
November
October
September
August
July
June
May
April
March
February
January
2009
December
November
October
September
August
July
June
May
April
February
January
2008
December
November
October
September
August
July
June
May
April
March
February
January
2007
October
September
August
July
June
May
April
March
February
January
Feed
Follow @googletesting
