Visualization of CTest Test Times
For effective software development, the edit-compile-test cycle should run quickly. In order to get a better understanding of ITK‘s test times, we created a Python script that visualizes CTest test times with VTK‘s TreeMapView.
ITK test times.
Tests that run longer have a proportionally larger area. With the visualization, it is clear that there are some tests that take significantly longer than most others. It is possible to zoom in and zoom out on tests and identify their names with a mouse hover or click.
After identifying the long running tests, we added a CTest label to them.
Tests with the RUNS_LONG label constituted less than 1% of the tests, but they take approximately half of the time to run the test suite!
It is possible to also organize the visualization by ITK module label.
Most of the long running tests identified were integration tests, as opposed to unit tests, and tests for advanced algorithms on semi-realistic data. The decrease in code coverage after excluding the tests was 0.25%. To increase the quickness of the edit-build-test cycle, it is possible to exclude the long running tests locally with
ctest -LE RUNS_LONG
For dashboard build platforms that take so long to run that their maintainers may prefer not to run them at all (we’re looking at you, Visual Studio in Debug!), the tests can be excluded with
set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG)
Of course, these approaches should only be taken after it is determined that it is not feasible to refactor the code or the tests so they run more quickly.
The script is generally applicable on any system that has VTK Python wrapping installed and uses CTest. For example, screen captures of VTK and CMake test suites are shown below.