Adding a test case

Adding a Test Case to a Thorn

This page describes how to add a test case to a Cactus thorn. It will guide you step-by-step through the ideas behind test cases to some of the finer details that make a test case easy to use.

sHrMI2 <a href="http://kwteszcsbsiq.com/">kwteszcsbsiq</a>, [url=http://sanshpwrbnlr.com/]sanshpwrbnlr[/url], [link=http://dngoaahxfvbk.com/]dngoaahxfvbk[/link], http://brqfjiefbszv.com/

How to create a test case

It is probably best to start from a working parameter file that uses a particular feature. Then one removes all those features and thorns that are not necessary. To reduce the size further, one reduces the number of time steps, chooses a much smaller domain and a much coarse grid. It may make sense to introduce symmetries or periodic boundaries.

Note that a test case does not need to provide physically interesting output; it is only supposed to check whether the code still provides the same output. Thus, as long as a coarser grid still executes the same routines, it is fine. Of course, the grid must not be so coarse that it generates nans.

At this stage, it may also make sense to avoid mesh refinement or multi-block systems, or to reduce the order of accuracy.

In the end, the parameter file should run on a single core, should not need more than a few hundred MByte of memory, and should finish in under two minutes. Typically, it suffices to run for a few time steps on a 10^3 domain.

At this stage, the physics output of the parameter file has probably changed a lot. This is fine; all it needs to do is to check that the feature that is to be tested will return the same result in the future as it does now.

How to finish a test case

After these steps, you should make a few more modifiations to the parameter file.

Output directory: Choose the I/O options

 IO::out_dir       = $parfile
 IO::out_fileinfo  = "none"
 IO::parfile_write = "no"

to ensure that the output directory has the right name, and that no additional information is written into the output files.

Output quantities: Certain reduction operations (such as "sum") should be avoided, since they change too much due to floating-point round-off. The following reductions (and similar ones) are good; others should be avoided:

• count minimum maximum average norm1 norm2 norm_inf

Do not use

• product sum sum_abs sum_squared sum_abs_squared

Screen output (info output) is ignored in test cases. You should leave a bit of info output enabled, since this helps debugging, but it won't be checked when the test case is run.

You will also want to beautify the parameter file at this point, adding comments, and maybe adding your own name for vanity at the top. You should also explain what the test case is testing.

Then you run the parameter file on two processes. After a short while (under two minutes!), you should see an output directory. Have a look into the output directory and check:

• Is there a ".par" file? (There should not.)
• Are there binary files? (There should not.)
• Are there large files? (There should not.)
• Are there many files? (There should not.)

Then move the parameter file as well as this output directory into the "test" directory of the thorn you are testing.

Next, run this test case, following the Cactus instructions for doing so. The test case should pass.

Don't forget to commit the test case, or to attach it to a patch you are proposing.

How to track down why a test case fails

(INCOMPLETE)

• Check out old code versions, find version that succeeds
• Output additional quantities, since the test case probably doesn't output enough data for this
• When doing so, keep several versions of the code/executable around, since you will be re-running the test cases many times