Continuous Testing Drives Innovation with Drake Robotics Software

February 21, 2024

Testing Drake Since 2014

Kitware has been working since 2014 to develop and maintain a cloud-based continuous integration (CI) system for Drake, an open-source robotics toolbox started by the Robot Locomotion Group at MIT CSAIL with core development now led by Toyota Research Institute. Modern software development involves teams of developers working together on large code bases. For those teams and projects to be successful and create high-quality software, new code should be tested before it is accepted into the larger code base. This is done with continuous integration testing where new code is put through a series of automated tests before it can be integrated. Drake’s CI system is built using open-source tools running in a cloud (AWS, MacStadium) infrastructure. The testing is driven by Jenkins with structured results displayed in CDash. This allows many developers to work collaboratively and continuously to add features to Drake. Over the past 12 months, there have been an average of 130 merged pull requests per month, with contributions coming from 23 unique contributors each month.

What is Drake, and What Makes it Special?

As detailed in Model-based design in the age of robotics and machine learning, Drake aims to provide mature, battle-hardened, open-source tools in a roboticist’s toolkit, complementary to deep learning toolboxes, to further buttress advanced research in the field. It is a software library (in C++ with Python bindings) that encompasses a multibody dynamics engine, a framework for organizing and combining libraries of system models, an optimization framework for mathematical programming, and algorithms to combine those ingredients for robotics applications such as motion planning or optimal control.

Drake provides a meeting ground for a unique collaboration between industry and academia. It is informed by (and implements) important ideas from modern robotics research from the labs of MIT, Toyota Research Institute, and others, while also being heavily used in leading-edge commercial robotics programs (like Amazon, Dexai, and others).

The Drake team has committed to closing the “Sim2Real gap” for dexterous robotic manipulation. Today, you’ll find a state-of-the-art implementation of physics for both rigid and compliant bodies, with a particular emphasis on robust numerics for contact mechanics to fight the numerical instabilities that plague most physics engines. Importantly, this physics engine is not only for simulation but is also built for optimization and control. The exact same code used for simulation can be used to compute forward or inverse kinematics and Jacobians, or for more complex queries like the gradient of an object’s center of mass. Drake provides smooth gradients for optimization whenever they are available (even through contact).

Why is CI Important to Drake

Drake weighs in at over 500,000 source lines of code (not counting its many dependencies), yet even with its wide scope of features, Drake is a more reliable robotics toolkit than its peers in the ecosystem. That’s in large part because of the team’s focus on automated testing with continuous integration. The Drake developers have a philosophy of rigorous test-driven development. The governing equations for multibody physics are well known, but as with any complex mathematical software, subtle bugs can be easily injected into the software. To avoid that, comprehensive unit tests contain comparisons with closed-form solutions for nontrivial mechanics problems like a tumbling satellite, countless checks on energy conservation, and many other checks that help the rest of the team focus on manipulation with the confidence that the multibody models are implemented correctly.

What Does Drake CI Test

Aggregate coverage results from all coverage builds.

The CI system is designed to leverage Drake’s extensive test suite. Through its use of Bazel, a hermetic build system, the Drake CI system is able to produce builds that are self-contained and reproducible. All CI jobs provide the exact recipe needed to reproduce the build locally.

There are over 350 Jenkins jobs that have been defined to ensure coverage across a support matrix which includes the following configurations:

Operating systems (the latest two versions of Ubuntu and macOS x86 and arm)
Compilers (gcc, clang)
Build systems (Bazel, CMake)
Build modes (Debug, Release)
Optional external dependencies (e.g., open or closed-source solvers)

In addition to the above matrix, a subset of the jobs run on unprovisioned images, where the same build and test steps are followed except unprovisioned jobs always start with a clean image and run setup scripts to install dependencies. Other tools that are run on the codebase at least nightly include coverage (kcov), asan, tsan, lsan, ubsan and valgrind. Finally, there are jobs that produce documentation, docker images, wheels and other release artifacts.

Communicating the Results with CDash

A series of CMake scripts act as the bridge between Jenkins and the Drake build. They contain the logic to set any relevant build or test flags and run the appropriate Bazel or CMake commands. The CMake scripts also upload the build results to CDash, a dashboard that summarizes results and provides visibility to warnings and errors without the need to search through console output files. A link to each specific job result in CDash is provided in Jenkins, this helps quickly find the cause of a build error or test failure. The Drake team has invested in improving integration with CDash. Most notably, thanks to work done as part of this project, CDash is able to parse output from Bazel builds.

Navigating to the specific job in Jenkins shows details such as the changes since the last build with links to specific commits. Select “View in CDash” to be brought directly to the job results in CDash.

In CDash, a summary of the build and any warnings or errors is provided. Here, we can see one test failure.

CDash displays the output specifically for this test.

Developer Workflow

Before any changes are allowed to be merged into Drake’s master branch, a set of carefully selected pre-merge jobs must pass. These jobs provide broad coverage over the set of supported configuration options with the goal of preventing any breaking changes from being integrated into master. Importantly, to reduce overhead on developers, all of the pre-merge jobs finish and report results back to the Github Pull Request within minutes. This immediate feedback is made possible by leveraging Bazel’s remote caching support which significantly speeds up build and test times.

After a change has been merged, a set of continuous jobs are automatically launched to detect and report any compilation or test failures that were not seen by the pre-merge jobs. To help ensure a timely resolution to issues, Drake uses a build cop process where the on-call build cops (an actual person) are responsible for monitoring and triaging errors. Using this system, defects not caught in pre-merge are quickly fixed or reverted.

Build cops receive an email that mac-arm-ventura-clang-bazel-nightly-everything-release has failed. The Jenkins Nightly Production dashboard provides a list of all nightly jobs with any failures separated into the “Unstable” grid.

Finally, a series of jobs that are too slow and resource-intensive to run multiple times a day are run nightly or weekly. When preparing a release, Drake maintainers can be confident that all supported configurations have been well-tested without the need for further manual testing.

The Drake team, along with industry partners like Kitware, continue to develop a mature software library, battle-hardened against a multitude of real-world experiments and extensive Monte-Carlo (and adversarial) testing. Continuous integration, supported by open-source tools including Jenkins and CDash, helps ensure consistently high-quality, reliable software. Drake provides regular releases and has committed to deprecation timelines that support use by serious companies in development and production. If you have the need for an industrial-strength robotics toolbox that can enable your research and engineering applications, then take a serious look at Drake. If you are interested in adopting a continuous integration system for your complex C++ application, contact Kitware.

Summary

Tags:

CDash CMake Software Process