MIDAS+CTest

From MIDAS Wiki

Jump to: navigation, search

MIDAS can be used for hosting the data your tests will use. Rather than having to include large test data sets as part of your source repository, you can instruct CTest to download data sets at test time from a MIDAS server. The files on the MIDAS server are content-addressed so you won't have to worry about keeping the data on the server synchronized with the client.

First, download the script from here:

MIDAS.cmake

Contents

How to create a test that will download and use MIDAS-hosted data

1) Upload your testing data to the MIDAS server. You'll need to create communities, collections, and items in order to place files on the server. This hierarchy will allow you to organize your data in much the same way as a filesystem.

2) When you have uploaded the bitstream into its parent item, you can see it in the item's list of files, as in the example below:

MIDAS item.png

3) At the top of the bitstream list, click the Advanced View checkbox. Each file should display a link next to it that says Download MD5 Key File. Click this link, and your browser will download a file. This file will have the same name as the uploaded file, but with ".md5" added to the end. This key file is simply a placeholder for the real file, which won't be downloaded until test time. It contains the md5 checksum of the actual file, which is used at test time as an index into the entire set of files on the MIDAS server.

4) Copy this file somewhere in the source tree of your repository. These are very small files (32 bytes each) and won't add much additional overhead to repository checkouts. You may keep all of your key files in a single directory, or create an arbitrary directory structure underneath the MIDAS_KEY_DIR that you declare to hold the key files. These files act as very small placeholders for the real, larger files that will be downloaded when the test is run for the first time.

5) Put the MIDAS.cmake script in your source tree where it can be called by your CMakeLists.txt scripts. Add the line:

include(MIDAS)

into the CMakeLists.txt script where you intend to add the tests that will run against the MIDAS-hosted data.

6) Add the following configuration lines:

set(MIDAS_REST_URL "http://char/midas/midas/api/rest")

Set MIDAS_REST_URL to the location of the server's REST API, which must be enabled in the server's configuration file.

set(MIDAS_KEY_DIR "${PROJECT_SOURCE_DIR}/MIDAS_Keys")

Optional: This specifies where the script will look for the key files you placed in step 4. The value shown here is its default value.

set(MIDAS_DATA_DIR "${PROJECT_BINARY_DIR}/MIDAS_Data")

Optional: This specifies where on disk the script will download files to when the test is run. The value shown here is its default value.

set(MIDAS_DOWNLOAD_TIMEOUT 10)

Optional: This specifies the timeout for the downloading of files from MIDAS. By default, there is no timeout used.

7) Now you may add the tests using the following macro:

midas_add_test(<testName> <program> [args...])

If we wanted to download the check.png and run some hypothetical filter executable on it, and compare it with another downloaded file, baseline.png.md5, we would first move the baseline and input files out of the repository and onto the MIDAS server, download their key files into the MIDAS_KEY_DIR, and then change:

add_test(TestSomeFilter someFilter --input-file check.png --baseline-file baseline.png)

to:

midas_add_test(TestSomeFilter someFilter --input-file MIDAS{check.png.md5} --baseline-file MIDAS{baseline.png.md5})

Note the MIDAS{...} arguments. These arguments will be replaced by the actual downloaded files when the test is run.

Notes

  • MIDAS{} substitution method can be passed an arbitrary relative path under the MIDAS_KEY_DIR, so you can use any directory structure you want to organize your key files.
  • Two tests will be created for each call to midas_add_test. One will be TestName, the other will be TestName_fetchData.
    • The TestName_fetchData test is the one that downloads the data, and it will always be run first.
    • The TestName test is the actual test that will be run on the downloaded data. It depends on the fetchData test, so this macro is safe for parallel CTest.
  • The actual files will be downloaded and named by their respective checksums. If such a file already exists, the download is skipped. Hence, subsequent test runs will not re-download data if it has already been downloaded.
  • The checksum is verified in the fetchData test. If it does not match the expected value, the data is deleted and the test will fail.
  • The fetchData test contains in its output a URL by which one can access the data used by the test. This is useful for someone looking at test output on CDash. They can see the output of the fetchData test, and follow the link to manually fetch the data (this time in its human-readable filename) for themselves.

How to run these tests

There is no additional work required in order to run these tests. The burden is entirely on the test creator. Testers just have to run ctest on the binary directory, as usual.

MIDAS_FETCH_ONLY option

If a test requires a file to be present but does not actually want to pass the downloaded file as an argument to its test, it should use MIDAS_FETCH_ONLY{foo.md5} instead of MIDAS{foo.md5}. For instance, if a test wants to pass an argument of "foo.raw", and the test program will look for the file "foo.mhd" but doesn't expect it as an argument, you can use

midas_add_test(testSomeFilter someFilter ... MIDAS{Input/foo.raw.md5} MIDAS_FETCH_ONLY{Input/foo.mhd.md5})

Both files will be downloaded for this test, but foo.mhd will not be passed as an argument to the test program.

MIDAS_TGZ option

The MIDAS_TGZ keyword is also used when you want to pass a directory of files as an argument. Pass in the key file corresponding to a tar.gz archive on the server. The archive will be downloaded and extracted into a directory in your MIDAS_DATA_DIR, and then that directory will be passed as the argument to the test. This is designed to alleviate having to manage directories full of key files that are all related, such as DICOM images.

 midas_add_test(testSomeFilter someFilter ... MIDAS_TGZ{Input/data.tar.gz.md5})

MIDAS_DIRECTORY option

The MIDAS_DIRECTORY keyword can be used to pass a directory as an argument to the test. Place the directory full of key files in your source tree under the MIDAS_KEY_DIR, and then reference the directory using MIDAS_DIRECTORY{...} in your midas_add_test() call. At test time, a directory will be created with the same name in your build tree, and all of the key files from the source tree directory will have their corresponding content downloaded and placed into the build tree. Your test will refer to the directory in the build tree.

Personal tools