Kitware Source Feature Article: October 2009

Building External Projects with CMake 2.8

The ExternalProject module in CMake 2.8 makes it easier to build projects dependent upon external software components. An “external project” is one that you can get the source code for, but does not readily build with a simple ADD_SUBDIRECTORY call in your CMakeLists.txt file. The ExternalProject_Add function makes it possible to say “download this project from the internet, run its configure step, build it and install it” with just a few lines of code added to your CMakeLists.txt file. The time intensive processing for each step is deferred until build time, making the CMake configuration of an ExternalProject_Add lightning fast.

The basic concept of ExternalProject is simple: given an external source of software (url to a .tar.gz file, cvs repository, svn repository, local directory), execute the sequence of commands necessary to build and install that software so that you can refer to it (include, link, run) from your project.

ExternalProject_Add is presently implemented in terms of a CMake custom target and several CMake custom commands. The custom target represents the external project itself, and each custom command is an “external project build step”. Some of the custom commands execute the important steps and some are just for housekeeping details. The following image illustrates custom command dependencies and the order of execution for one ExternalProject_Add call:

Custom Command Dependencies

The grayed out nodes are the housekeeping steps, the bold ones are the important steps that actually execute commands. An arrow between steps points to a “step that I depend on” or a “step that runs before I run.” All steps are shown in the above diagram: some steps are optional; all have reasonable default commands; each may be omitted or overridden.

After each step is successfully executed, a stamp file is produced indicating that the step is up-to-date with respect to its input. If an upstream step re-executes because its input changed, then all downstream steps will also re-execute. Steps that are always out of date will always re-execute when built. For example, the update step of a cvs or svn checkout is always considered out of date.

Whenever you build, the update step will execute and retrieve the latest copy of the source from the repository.

Since an external project is a CMake custom target, you can set up dependencies among multiple external projects and between your own CMake targets and the external projects. When a step in an external project re-executes, the downstream targets that depend on that project will also rebuild.

A call to ExternalProject_Add that downloads, configures, builds and installs CMake 2.6.4 looks like this:

include(ExternalProject)
ExternalProject_Add(
   CMake-2-6-4
   CVS_REPOSITORY :pserver:anonymous:cmake@
       www.cmake.org:/cvsroot/CMake
   CVS_MODULE CMake
   CVS_TAG -r CMake-2-6-4
   CMAKE_ARGS -DCMAKE_INSTALL_PREFIX:
       PATH=<INSTALL_DIR>
   UPDATE_COMMAND “”
)

Network connectivity is assumed if external project sources are downloaded via a URL to a remote computer or from a remote cvs or svn repository.

ExternalProject Build Steps
The first step to execute at build time is the download step. The download step could be a file download from an internet site, a checkout from a cvs or svn repository, or a reference to a pre-existing local file or directory. CMake supports local or downloaded *.tar/*.tar.gz/*.tgz files and cvs and svn repositories. For other download types, override the built-in download step with a custom DOWNLOAD_COMMAND.

The next step is either update or patch. By default, if the download step is a checkout from cvs or svn, then the update step is the appropriate cvs or svn update call. Otherwise, the update step is omitted. There is no default patch command. To patch a source tree, provide a custom PATCH_COMMAND.

After that, the configure step executes. By default, the configure command is used to run the same CMake command used to configure the calling project. To use a specific version of CMake, provide a CMAKE_COMMAND value. To use a configure script or some other custom configuring program, provide a custom CONFIGURE_COMMAND.

After configure, the build step executes. If CMake was used to configure the project, then ‘cmake--build’ is used to build the project. If a configure script was used, then ‘make’ is used to build the project. If a custom configure command was given, provide a custom BUILD_COMMAND, too.

The test step is omitted based on the assumption that ExternalProject_Add calls will be made to build stable snapshots of the target software. If TEST_BEFORE_INSTALL 1 or TEST_AFTER_INSTALL 1 is specified, the test step is executed before or after the install step. CMake configured projects run CTest. Configure script projects are tested with ‘make test’. Override by providing a custom TEST_COMMAND.

The final step in an external project build sequence is the install step. CMake configured projects use cmake--build with the install target to execute the install step. Configure script projects use ‘make install’. Override by providing a custom INSTALL_COMMAND.

ExternalProject_Add Arguments
Relative paths are interpreted with respect to the build directory corresponding to the source directory in which ExternalProject_Add is invoked in the calling project.

The first argument to ExternalProject_Add is the name for the custom target representing the external project. It should be a valid CMake target name and it must be unique across the set of all target names in your CMakeLists.txt files. This follows the same rule as other CMake targets: library, executable and custom target names must be globally unique.

The DEPENDS argument takes the names of other CMake targets that must build first before the current external project will build successfully. These could be library, executable target, external project, or CMake custom target names. If you need to make your own CMake target depend on an external project use the CMake ADD_DEPENDENCIES command. The target name will be the first argument and the external project <name> will be the second argument.

#  [PREFIX dir]

The *_DIR options specify directories for the project, with default directories computed as follows. If the PREFIX option is given to ExternalProject_Add() or the EP_PREFIX directory property is set, then an external project is built and installed under the specified prefix:

TMP_DIR = <prefix>/tmp
STAMP_DIR = <prefix>/src/<name>-stamp
DOWNLOAD_DIR = <prefix>/src
SOURCE_DIR = <prefix> /src/<name>
BINARY_DIR = <prefix>/src/<name>-build
INSTALL_DIR = <prefix>

If the EP_BASE directory property is set then components of an external project are stored under the specified base:

TMP_DIR = <base>/tmp/<name>
STAMP_DIR = <base>/Stamp/<name>
DOWNLOAD_DIR = <base>/Download/<name>
SOURCE_DIR = <base>/Source/<name>
BINARY_DIR = <base>/Build/<name>
INSTALL_DIR = <base>/Install/<name>

If no PREFIX, EP_PREFIX, or EP_BASE is specified then the default is to set PREFIX to “<name>-prefix”. Relative paths are interpreted with respect to the build directory corresponding to the source directory that invokes ExternalProject_Add.

# [LIST_SEPARATOR sep]
# Sep to be replaced by ; in cmd lines

Sometimes it’s necessary to pass a semi-colon delimited string through to CMake as a -D argument. It’s impossible to pass a semicolon in a string and then re-extract it later. So, if you need to pass a list, use some other string (that isn’t used elsewhere in your ExternalProject_Add arguments) to separate items. Then pass that other string as the LIST_SEPARATOR. For example, to pass:

CMAKE_ARGS -Dmylist:STRING=item1;item2;item3;item4

Use something like:

CMAKE_ARGS -Dmylist:
    STRING=item1^^item2^^item3^^item4
LIST_SEPARATOR ^^

# [TMP_DIR dir]

The temp directory is used as a place to extract *.tar.gz files, for other custom steps, or during configure and build of the calling project.

# [STAMP_DIR dir]

The stamp files that indicate the last time each external project build step ran are stored in this directory. For multiconfiguration build systems like Visual Studio and Xcode, the stamp files are stored in per-configuration subdirectories.

Download step For the download step specify a URL; CVS_REPOSITORY and CVS_MODULE; SVN_REPOSITORY; or DOWNLOAD_COMMAND. If all of these and SOURCE_DIR are omitted, it is an error. If there is no download step, then use an empty string for DOWNLOAD_COMMAND ””.

# [DOWNLOAD_DIR dir]

If URL points to a file to download from the internet then the downloaded file will be stored in this directory.

# [DOWNLOAD_COMMAND cmd...]

DOWNLOAD_COMMAND provides a custom download command. For example, to extract from a different archive format than the .tar.gz files that CMake can handle.

The download command should ensure that there is a source directory at <SOURCE_DIR> when it is done; it executes with the working directory set to <DOWNLOAD_DIR>.

# [CVS_REPOSITORY cvsroot]

“:pserver:” or “:ext:” (or other) protocol string to the root of the CVS repository.

# [CVS_MODULE mod]

Name of the module to checkout relative to the root of CVS_REPOSITORY.

# [CVS_TAG tag]

-r tagname or -r branchname or -D 2009-09-10. If omitted, the checkout gets no sticky tag or date, so you sync with the HEAD of the CVS repository.

# [SVN_REPOSITORY url]

URL to an svn repository. Usually includes “trunk”, a branch, or a tag and the source directory.

# [SVN_REVISION rev]

-r revisionNumber or -r {2009-09-10} date stamp. If omitted, you sync with the latest revision from the SVN repository.

# [URL /.../src.tgz]

URL to an internet-based file or name of a local file or preexisting already extracted source directory. Examples:

URL ${CMAKE_CURRENT_SOURCE_DIR}/Externals/Proj1
URL ${CMAKE_CURRENT_SOURCE_DIR}/tarballs/proj2.tgz
URL http://www.cmake.org/.../cmake-2.6.4.tar.gz

The download step will copy the local dir or the extracted contents of the archive file to <SOURCE_DIR>. To support other archive types, or avoid the copy, provide a custom DOWNLOAD_COMMAND instead of using URL.

Update/Patch Step
If the download step is based on a CVS_REPOSITORY or SVN_ REPOSITORY then the default value for the update command will be a call to cvs or svn to update the source tree. In this case, the update command will always re-run at build time. To avoid the update step use UPDATE_COMMAND ””.

# [UPDATE_COMMAND cmd...]

The default UPDATE_COMMAND for non-repository download steps is ””. The update command executes with the working directory set to <SOURCE_DIR>.

# [PATCH_COMMAND cmd...]

By default, there is no patch step. If you have a patch file to apply on top of a download or cvs checkout, use PATCH_COMMAND. For example:

PATCH_COMMAND patch -p2 -t -N < ${dir}/proj1.patch

The patch command executes with the working directory set to <SOURCE_DIR>.

Configure step
If SOURCE_DIR is explicitly set to an existing directory, the project will be built from it.

# [SOURCE_DIR dir]

Otherwise a download step must be specified using one of the DOWNLOAD_COMMAND, CVS_*, SVN_*, or URL options. The URL option may refer to a local directory or source tarball, or to a remote tarball (e.g., http://.../src.tgz).

# [CONFIGURE_COMMAND cmd...]

Command used to configure the project before building it. If a project uses a configure script to achieve this then use:

CONFIGURE_COMMAND <SOURCE_DIR>/configure
    --prefix=<INSTALL_DIR>

The configure command executes with the working directory set to <BINARY_DIR>.

# [CMAKE_COMMAND /.../cmake]

If project uses CMake to configure, and it uses a different CMake than the one being used to configure the calling project, use:

CMAKE_COMMAND ${full_path_to_other_cmake}

A project that configures with CMake uses the same CMake as the calling project. The CMake command executes with the working directory set to <BINARY_DIR>.

# [CMAKE_GENERATOR gen]

If project requires a different CMAKE_GENERATOR than the one used for the calling project, use:

CMAKE_GENERATOR “Visual Studio 8 2005”

# [CMAKE_ARGS args...]

To make builds consistent, specify all of the configuration settings used to build the project using CMAKE_ARGS.

CMAKE_ARGS -DCMAKE_BUILD_TYPE:STRING=Release
     -DPROJ_OPTION1:BOOL=ON
     -DPROJ_OPTION2:BOOL=OFF

You do not need to specify the “-G generator” part or the source directory part of the CMake command line in CMAKE_ ARGS. The source directory is always automatically appended as the last argument to the CMake configure command line, and the generator defaults to the same as the calling project unless you use the CMAKE_GENERATOR arg.

If you need to pass a list of items through a CMAKE_ARGS -D argument, see the LIST_SEPARATOR documentation.

Build Step
There is a reasonable location for the binary tree for the external project, underneath ${CMAKE_BINARY_DIR} for the calling project.

# [BINARY_DIR dir]

If you would like to specify a different binary directory, use:

BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR}/MyBinaryDir
# [BUILD_COMMAND cmd...]

CMake-based projects build with ‘cmake--build’. Others build using ‘make’. Use BUILD_COMMAND to customize the build step. The build command executes with the working directory set to <BINARY_DIR>.

# [BUILD_IN_SOURCE 1]

For projects that do not build properly in a separate binary dir, or that want to build in the source tree, BUILD_IN_SOURCE 1 means set BINARY_DIR to be the same as <SOURCE_DIR>.

Install Step
The INSTALL_DIR is underneath the calling project’s binary directory. Use INSTALL_DIR to specify a different location. Note that in addition to setting INSTALL_DIR, you also have to pass -DCMAKE_INSTALL_PREFIX or --prefix to the CMake or configure command. It is not used automatically in the configure step since not all projects follow this convention.

# [INSTALL_DIR dir]

You can refer to the install directory in your configure command, for example:

CONFIGURE_COMMAND SOURCE_DIR/configure
     --prefix=INSTALL_DIR

# [INSTALL_COMMAND cmd...]

CMake-based projects use ‘cmake--build’ to build the install target. Other projects use ‘make install’. Use INSTALL_COMMAND to customize the install step. Use INSTALL_COMMAND “” to omit the install step. The install command executes with the working directory set to <BINARY_DIR>.

Test Step
The test step is omitted to save time based on the assumption that external projects are stable and well-tested. The code you depend on will ultimately be tested by your own project’s tests, anyway, right? It’s easy to activate the test step, however. Simply add TEST_BEFORE_INSTALL 1 or TEST_ AFTER_INSTALL 1 to execute the test command either before or after the install step. Or add a custom TEST_COMMAND if the default one isn’t sufficient.

# [TEST_BEFORE_INSTALL  1]
# [TEST_AFTER_INSTALL  1]
# [TEST_COMMAND cmd...]

By default, cmake based projects use ‘ctest’ to test the build tree. Other projects use ‘make test’. Use TEST_COMMAND to customize the test step.

Adding Custom Steps
Perhaps you still have a build step that the standard sequence does not account for.

For even finer grained control than ExternalProject_Add already gives you, you can add your own custom build steps in lieu of, or in addition to, the steps already provided. To add your own custom build step, use the function ExternalProject_Add_Step after a call to ExternalProject_Add.

The ‘ExternalProject_Add_Step’ function adds a custom step to an external project. The signature looks like this:

ExternalProject_Add_Step(<name> <step>
 [COMMAND cmd...]
 [COMMENT “text...”]
 [DEPENDEES steps...]
 [DEPENDERS steps...]
 [DEPENDS files...]
 [ALWAYS 1]
 [WORKING_DIRECTORY dir]
)

“Silly example” calls that just echo some strings to the build output stream after the standard build and install steps for the external project “proj1” look like this:

ExternalProject_Add_Step(proj1 e1
   COMMAND ${CMAKE_COMMAND} -E echo e1
   DEPENDEES build
   DEPENDERS install
)
ExternalProject_Add_Step(proj1 e2
   COMMAND ${CMAKE_COMMAND} -E echo e2
   DEPENDEES install
)

These silly example calls would result in two additional build steps. Step e1 executes after the build step and before the install step, while step e2 executes after the install step.

The command line, comment, and working directory of every standard and custom step is processed to replace tokens <SOURCE_DIR>, <BINARY_DIR>, <INSTALL_DIR>, and <TMP_DIR> with corresponding property values.

The following names are reserved step names used for the standard external project build steps. Do not use these as step names in calls to ExternalProject_Add_Step. These are mine: mkdir, download, update, patch, configure, build, test, install, complete, done. Use your own.

Properties
After an ExternalProject_Add call, you can call ExternalProject_Get_Property to retrieve the properties of that project. For example, if you need to use its install location in another project’s configuration command…

ExternalProject_Get_Property(proj1 install_dir)

…puts the value of the INSTALL_DIR property for proj1 into the CMake variable “install_dir”. You may then refer to it in a subsequent configure command for proj2. For example:

CMAKE_ARGS -Dproj1_DIR:PATH=${install_dir}

The names of the variables used in an ExternalProject_Get_Property call must match the names of the corresponding properties that you retrieve.

Conclusion
The “external project” approach allows for consistent builds across many machines because it is dependent upon a call to ExternalProject_Add in your CMakeLists.txt file. Whereas, relying on several people to set different variables to the same set of values may not always yield consistent results.

In addition to consistency, automatically building external projects saves developers time because they don’t have to spend time manually configuring and building project dependent components. Developers can configure a project, crank up the build and voila: all those components are built for them by ExternalProject’s automagic goodness.

One drawback, however, is ExternalProject’s lack of full dependency analysis. Changes in header files of an external project may not cause an incremental rebuild of the affected sources that depend on those headers. This is because the build step might not re-run. A complete list of header files is not given as input to the build step, so there is nothing to indicate that the build step should re-run. This problem can be circumvented by touching or deleting a build step’s corresponding stamp file which will force it to run.

For more information on ExternalProject, please visit the CMake mailing list.

David Cole  David Cole is an R&D Engineer in Kitware’s Clifton Park, NY office. David has contributed code to the VTK, CMake, ITK, ParaView, KWWidgets and gccxml opensource projects. He has also contributed to Kitware’s proprietary products including ActiViz and VolView.