CDash Subprojects
A large project typically consists of several interdependent pieces such as libraries, executables, test suites, documentation, web pages and installers. Organizing your project into well-defined subprojects and presenting the results of nightly builds on a CDash dashboard can help you see where the problem spots are at whatever level of granularity you define. The new subproject feature of CDash (version 1.4 and later) saves you time, making it easy to identify and correct problems.

For the purposes of this article we assume you are using CDash 1.4 or later and CTest 2.7.20090520 or later (CVS HEAD of CMake from May 20, 2009, or later).

A project with subprojects has a different view for its top level CDash page than a project without any subprojects. It contains a summary row for the project as a whole and then one summary row for each subproject.

The Trilinos Project is an effort to develop algorithms and enabling technologies within an object-oriented software framework for the solution of large-scale, complex multiphysics engineering and scientific problems. A unique design feature of Trilinos is its focus on packages which are represented as subprojects on the Trilinos CDash dashboard (shown in the figure above). Explore the Trilinos dashboard to see how CDash displays projects and subprojects at trilinos-dev. sandia.gov/cdash.

The EpetraExt subproject page in the Trilinos project is an example of a subproject page with summary rows at the top (in the “SubProject Dependencies” section), as shown in the following figure.

The Teuchos subproject page in the Trilinos project is an example of an independent subproject page. Since it has no subprojects that it depends on, there is no “SubProject Dependencies” section.

Organizing and Defining Subprojects
To add subproject organization into your project, you must do two things: (1) define the subprojects for CDash, so that it knows how to display them properly, and (2) use build scripts with CTest to submit subproject builds of your project. Some (re-)organizational work in your project’s CMakeLists.txt files may also be necessary to allow building your project by subprojects.

There are two ways to define subprojects and their dependencies: interactively in the CDash GUI when logged in as a project administrator, or by submitting a Project.xml file describing the subprojects and dependencies.

Adding Subprojects Interactively
If you’re a project administrator, you will have a “Manage subprojects” button for each of your projects on the My CDash page, as highlighted in the figure below. For the purpose of experimenting with the subprojects feature of CDash, create a “Tutorial” project on your CDash server and make yourself a project administrator.

Clicking the Manage subprojects button takes you to the manage subproject page where you may add new subprojects or establish dependencies between existing subprojects. There are two tabs on this page. One for viewing the current subprojects and their dependencies and one for creating new subprojects.

When you first visit this page for a given project, it will be empty.

To add two subprojects, called Exes and Libs, and to make Exes depend on Libs:

• Click the “Add a subproject” tab
• Type “Exes” in the “Add a subproject” edit field
• Click the “Add subproject” button
• Click the “Add a subproject” tab
• Type “Libs” in the “Add a subproject” edit field
• Click the “Add Subproject” button
• In the “Exes” row of the “Current Subprojects” tab, choose “Libs” from the “Add dependency” drop down list and click the “Add dependency” button.

Now the “Current Subprojects” tab looks something like this:

To remove a dependency or a subproject, click on the “X” next to the one you want to delete.

After adding subprojects to the Tutorial project, the main dashboard page for Tutorial looks like this:

Adding Subprojects Automatically
The other way to define CDash subprojects and their dependencies is to submit a “Project.xml” file along with the usual submission files that CTest sends when it submits a build to CDash. To define the same two subprojects as in the interactive example above (Exes and Libs) with the same dependency (Exes depend on Libs) the Project.xml file would have contents as follows:

Once you write or generate the Project.xml file, you can submit it to CDash from a CTest -S script using the new FILES argument to the ctest_submit command, or directly from the CTest command line in a build tree configured for dashboard submission.

From inside a CTest -S script:

From the command line:

CDash will automatically add subprojects and dependencies according to how you define them in the Project.xml file. It will also remove any subprojects or dependencies not defined in the Project.xml file. Or, if you submit the exact same file multiple times, the second and subsequent submissions will have no observable effect: the first submission adds/modifies the data; the second and later submissions send the same data, so no change is necessary. CDash tracks changes to the subproject definitions over time to allow for project evolution. If you view dashboards from a past date, CDash will present the project/subproject views according to the subproject definitions in effect as of that date.

Whenever a client submits a Trilinos dashboard via Trilinos/cmake/ctest/TrilinosCTestDriverCore.cmake, it automatically submits a (possibly updated) subproject definition file with these lines of the script:

Other parts of the build process make sure that the CDashSubprojectDependencies.xml file is always up-to-date according to how the Trilinos packages are defined.

ctest_submit PARTS and FILES
CTest submits results to CDash with the ctest_submit command in CTest -S scripts. The ctest_submit command sends xml files describing the results of the stages of the dashboard (update, configure, build, test, …) and any note files attached by the script author.

In CTest 2.7.20090520 and later, the ctest_submit command supports new PARTS and FILES arguments.

With PARTS, you can send any subset of the xml files with each ctest_submit call. Previously, all PARTS would be sent with any call to ctest_submit. Typically, the script would wait until all dashboard stages were complete and then call ctest_submit once to send the results of all stages at the end of the run. Now, a script may call ctest_submit with PARTS to do partial submissions of subsets of the results. For example, you can submit configure results after ctest_configure, build results after ctest_build and test results after ctest_test. This allows for quicker posting of information as builds progress.

With FILES, you can send arbitrary xml files to CDash. In addition to the standard build result xml files that ctest sends, CDash also handles the new Project.xml file that describes subprojects and dependencies. (…described in the section “Adding Subprojects Automatically”…)

The valid PARTS values are listed in the “ctest --helpcommand ctest_submit” documentation:

The FILES option explicitly lists specific files to be submitted. Each individual file must exist at the time of the call.

Prior to the addition of the ctest_submit PARTS handling, a typical dashboard script would contain a single ctest_ submit() call as its last line. Now, submissions may occur incrementally with each part of the information sent piecemeal as it becomes available:

Submitting incrementally by PARTS means you can inspect the results of the configure stage live on the CDash dashboard while the build is still in progress. Likewise, you can inspect the results of the build stage live while the tests are still running.

Spliting Your Build into Multiple Subproject Builds
In a CTest -S dashboard driver script, a ctest_build call delegates to the native build tool to build the target named “${CTEST_BUILD_TARGET}”. If you have not defined the CTEST_BUILD_TARGET variable in your driver script, then the build tool will build everything, using the “all” or “ALL_BUILD” target.

One ctest_build() invocation that builds everything followed by one ctest_test() invocation that tests everything is sufficient for a project that has no subprojects. In order to submit results on a per-subproject basis to CDash, you will have to make ctest_build and ctest_test calls for each subproject. To split your one large, monolithic build into many smaller subproject builds, you can use a FOREACH loop in your CTest driver script. To help you iterate over your subprojects, CDash provides a variable named CTEST_ PROJECT_SUBPROJECTS in CTestConfig.cmake.

Given the above Tutorial example, CDash produces a variable like this:

CDash orders the elements in this list such that the independent subprojects (that do not depend on any other subprojects) occur first. Then subprojects that depend only on the independent subprojects, and then subprojects that depend on those, and so on. That logic is continued until all subprojects are listed exactly once in an order that makes sense for building them incrementally.

If you organize your CMakeLists.txt files such that you have a target to build for each subproject, and you can derive the name of that target based on the subproject name, then revising your script to separate it into multiple smaller configure/build/test chunks should be relatively painless.

To facilitate building just the targets associated with a subproject, use the variable named CTEST_BUILD_TARGET to tell ctest_build what to build. To facilitate running just the tests associated with a subproject, assign the LABELS test property to your tests and use the new INCLUDE_LABEL argument to ctest_test.

To organize your monolithic build into a sequence of smaller subproject builds, make the following changes:

CMakeLists.txt modifications

• Name targets same as subprojects, base target names on subproject names, or provide a look up mechanism to map from subproject name to target name
• Possibly add custom targets to aggregate existing targets into subprojects, using add_dependencies to say which existing targets the custom target depends on
• Add LABELS target property to targets with a value of subproject name
• Add LABELS test property to tests with a value of subproject name

CTest driver script modifications

• Iterate over the subprojects in dependency order (from independent to most dependent)
• Set the “SubProject” and “Label” global properties – CTest uses these properties to submit the results to the correct subproject on the CDash server
• Build the target(s) for this subproject: compute name of target to build from subproject name, set CTEST_BUILD_TARGET, call ctest_build()
• Run the tests for this subproject using INCLUDE or INCLUDE_LABEL arguments to ctest_test()
• Use ctest_submit() with PARTS argument to submit partial results as they are done

To simplify the following example which demonstrates the changes required to split a build into smaller pieces, assume the subproject name is exactly the same as the target name required to build the subproject’s components.

Here is a snippet from CMakeLists.txt, in the hypothetical Tutorial project. The only additions necessary (since the target and subproject names are the same) are the calls to set_property for each target and each test.

Here’s what the CTest driver script might look like before and after organizing this project into subprojects.

Before:

After (new chunks emphasized):

In some projects, more than one ctest_build step may be required to build all the pieces of the subproject. For example, in Trilinos, each subproject builds the “${subproject}_ libs” target and then builds the “all” target to build all the configured executables in the test suite. They also configure things such that only the executables that need to build for the currently configured packages build when the “all” target is built.

Normally, if you submit multiple Build.xml files to CDash with the same exact build stamp, it will delete the existing entry and add the new entry in its place. In the case where multiple ctest_build steps are required, each with their own ctest_submit(PARTS Build) call, use the “APPEND” keyword argument in all of the ctest_build calls that belong together. That APPEND flag tells CDash to accumulate the results from multiple Build.xml submissions and display the aggregation of all of them in one row on the dashboard. From CDash’s perspective, multiple ctest_build calls (with the same build stamp and subproject and APPEND turned on) result in a single CDash buildid.

Go Forth and Code!
Go ahead and adopt some of these tips and techniques in your favorite CMake-based project.

• LABELS is a new CMake/CTest property that applies to source files, targets and tests. Labels are sent to CDash inside the result xml files.
• Use ctest_submit(PARTS …) to do incremental submissions. Results are available for viewing on the dashboards sooner.
• Use INCLUDE_LABEL with ctest_test to run only the tests with labels that match the regular expression.
• Use CTEST_BUILD_TARGET to build your subprojects one at a time, submitting subproject dashboards along the way.

If you follow the guidelines in this article and split your favorite CMake-based project into subprojects, let us know about it on the CMake and CDash mailing lists. If you run into problems along the way, post questions to the mailing lists which are the principal means of communication among developers and users.

  David Cole is an R&D Engineer in Kitware’s Clifton Park, NY office. David has contributed code to the VTK, CMake, ITK, ParaView, KWWidgets and gccxml open-source projects. He has also contributed to Kitware’s proprietary products, including ActiViz and VolView.