Author: Patrik Höglund roughly a hundred libraries into its third_party folder when you build the browser. The most significant of these libraries is Blink, the rendering engine, but there’s also ffmpeg for image processing, skia for low-level 2D graphics, and WebRTC for real-time communication (to name a few). Figure 1. Holy dependencies, Batman!
libphonenumber , which is battle-tested by real use in Android and Chrome and available under a permissive license? Using such software frees you up to focus on the core of your software so you can deliver a unique experience to your users. On the other hand, you need to keep your application up to date with changes in the library (you want that latest bug fix, right?), and you also run a risk of such a change breaking your application. This article will examine that integration problem and how you can reduce the risks associated with it.Updating Dependencies is Hard The simplest solution is to check in a copy of the library, build with it, and avoid touching it as much as possible. This solution, however, can be problematic because you miss out on bug fixes and new features in the library. What if you need a new feature or bug fix that just made it in? You have a few options:Update the library to its latest release. If it’s been a long time since you did this, it can be quite risky and you may have to spend significant testing resources to ensure all the accumulated changes don’t break your application. You may have to catch up to interface changes in the library as well. Cherry-pick the feature/bug fix you want into your copy of the library. This is even riskier because your cherry-picked patches may depend on other changes in the library in subtle ways. Also, you still are not up to date with the latest version. Find some way to make do without the feature or bug fix. How “Rolling” Works The open-sourced WebRTC codebase is used by Chrome but also by a number of other companies working on WebRTC. Chrome uses a toolchain called depot_tools to manage dependencies, and there’s a checked-in text file called DEPS where dependencies are managed. It looks roughly like this:{# ... 'src/third_party/webrtc' :'https://chromium.googlesource.com/' +'external/webrtc/trunk/webrtc.git' + '@' + '5727038f572c517204e1642b8bc69b25381c4e9f' , 572703... hash, similar to other dependency-provisioning frameworks. To build Chrome with a new version, we change the hash and check in a new version of the DEPS file. If the library’s API has changed, we must update Chrome to use the new API in the same patch. This process is known as rolling WebRTC to a new version. getUserMedia has started crashing on Android , or maybe the browser no longer boots on Windows. We don’t know until we have built and run all the tests. Therefore a roll patch is subject to the same presubmit checks as any Chrome patch (i.e. many tests, on all platforms we ship on ). However, roll patches can be considerably more painful and risky than other patches. Figure 2. Life of a Roll Patch.
Bots That Can See the Future: “FYI Bots” We found a way to mitigate this which we called FYI (for your information) bots. A bot is Chrome lingo for a continuous build machine which builds Chrome and runs tests. DEPS file, which meant they would build the WebRTC version we had rolled to up to that point. FYI bots replace that pinned version with WebRTC HEAD, but otherwise build and run Chrome-level tests as usual. Therefore: If all the FYI bots were green, we knew a roll most likely would go smoothly. If the bots didn’t compile, we knew we would have to adapt Chrome to an interface change in the next roll patch. If the bots were red, we knew we either had a bug in WebRTC or that Chrome would have to be adapted to some semantic change in WebRTC. FYI “waterfall” (a set of bots that builds and runs tests) is a straight copy of the main waterfall , which is expensive in resources. We could have cheated and just set up FYI bots for one platform (say, Linux), but the most expensive regressions are platform-specific, so we reckoned the extra machines and maintenance were worth it.Making Gradual Interface Changes This solution helped but wasn’t quite satisfactory. We initially had the policy that it was fine to break the FYI bots since we could not update Chrome to use a new interface until the new interface had actually been rolled into Chrome. This, however, often caused the FYI bots to be compile-broken for days. We quickly started to suffer from red blindness [1] and had no idea if we would break tests on the roll, especially if an interface change was made early in the roll cycle. API prime directive ” [2] . Consider this example:class WebRtcAmplifier {int SetOutputVolume(float volume);class WebRtcAmplifier {int SetOutputVolume(float volume, bool allow_eleven1 class WebRtcAmplifier {int SetOutputVolume(float volume);int SetOutputVolume2(float volume, bool allow_eleven);Roll into Chrome Make Chrome use SetOutputVolume2 Update SetOutputVolume’s signature Roll again and make Chrome use SetOutputVolume Delete SetOutputVolume2 Results When we implemented the above, we could fix problems as they came up rather than in big batches on each roll. We could institute the policy that the FYI bots should always be green, and that changes breaking them should be immediately rolled back. This made a huge difference. The team could work smoother and roll more often. This reduced our risk quite a bit, particularly when Chrome was about to cut a new version branch. Instead of doing panicked and risky rolls around a release, we could work out issues in good time and stay in control.performance tests . Before the FYI bots, it would frequently happen that a bunch of metrics regressed. However, it’s not fun to find which of the 100 patches in the roll caused the regression! With the FYI bots, we can see precisely which WebRTC revision caused the problem.Future Work: Optimistic Auto-rolling The final step on this ladder (short of actually merging the repositories) is auto-rolling. The Blink team implemented this with their ARB (AutoRollBot) . The bot wakes up periodically and tries to do a roll patch. If it fails on the trybots, it waits and tries again later (perhaps the trybots failed because of a flake or other temporary error, or perhaps the error was real but has been fixed).References Footnotes 1.1 , but only 0.0-1.0 was supposed to be allowed. No, really. Thus, our WebRTC implementation must be louder than the others since everybody knows 1.1 must be louder than 1.0.
It's on the WebRTC side. We happened to control WebRTC in this case as well, so we could update the interface as we wished. If the library API is managed by someone else, you're going to have with to work with them (or submit patches if the library is open-source). It might not be worth it to set up FYI bots if interface changes are hard to do.
ReplyDeleteIf the third-party library is controlled by someone else and widely used, they should already be using the API prime directive. I think it's common to not remove or change methods in a published interface; if so, the method is deprecated and left remaining for about a year so all clients can update to use the new method. In that case your product should still build at HEAD of the third-party library and your FYI bots will work.