Contents

1 Introduction

Casacore comes with an extensive set of test programs. The tests can be run in the cmake/ctest environment used by casacore. A few scripts are doing the actual execution of a test.

A test can use a script for optional preprocessing or postprocessing of a test or to run a test in multiple ways. For regression testing purposes the standard output of a test run can be compared with the expected output.

2 Creating a test

Casacore consists of several modules (e.g. scimath, tables), where a module can consists of several packages (e.g. casa/IO, casa/OS). Each package contains one or more classes, usually one class per file. Test programs should be creaed in the test directory of a package.

Usually a test for a class X (in source file X.cc) is written in C++. The casacore convention is to name such a test tX, thus the source code for a test is in <module>/<package>/test/tX.cc. In order to build the test program, it needs to be added to the tests list in the CMakeLists.txt file in that test directory.

The test executable is created in a build directory that is different from the source directory.

Besides the .cc file, a few more files can optionally be created for a test. They need to reside in the source test directory and will be copied automatically to the build test directory.

• tX.run is a script. It can contain commands to do some preprocessing, postprocessing, to run the test in different ways, etc. Note that such a script is usually a shell script, but could also be a python script or any other executable script. If the script is not executable, it is run through sh.
  Note that the invocation of the test program in a script should be preceeded by $casa_checktool. This environment variable defines an optional check tool (like valgrind) as described in the last section.
  Also note that a test program should be run as ./tX, because . does not need to be part of the PATH.
• tX.in is a text file. If existing, the standard input of the test program is redirected to this file.
• tX.out is a text file containing the expected standard output of the test.
• tX.in_* are other inputs that can be used by the test. They can be files, but also directories (e.g., casacore tables).

Note that to run test tX, the test system will only copy files/directories as named above to the build trest directory. If test input files are named differently, they can be used in one of the two following ways.

• The test system defines the environment variable testsrcdir. It can be used in test programs or scripts to copy a test file from the test source directory or to use it directly. If using it directly, it should (of course) be readonly.
• Another option to copy files from the source to working directory is by adding appropriate lines to test/CMakeLists.txt like:
  set (datafiles  
  amp_0.fits  
  ampErr_0.fits  
  )  
  foreach (file ${datafiles})  
      configure_file (${CMAKE_CURRENT_SOURCE_DIR}/${file} ${CMAKE_CURRENT_BINARY_DIR}/${file} COPYONLY)  
  endforeach (file)

  to get the file copied to the working directory. An entire directory cannot be done in a single line; each individual file has to be mentioned.

If the test program or script creates output files, it is best to call them tX_tmp<something>. In this way they will be removed automatically after the test is run.

3 Running tests

The standard way to run a test program is using ctest. It makes it possible to run all or some tests.

If such a command is run in the casacore top build directory, all casacore tests are considered. However, it is also possible to run it in a subdirectory to execute only tests of a specific module or package.

The system runs a test as follows:

1. ctest starts the script cmake_assay to run the test.
2. cmake_assay copies the tX files mentioned in the previous section to the build test directory where the test is run. Thereafter it starts casacore_assay to run the actual test.
3. casacore_assay executes the test as follows:
   • If tX.run exists, that script will be executed. otherwise tX itself.
   • If tX.in exists, it will be redirected as input to tX.
   • If tX.out exists, it will be compared with the standard output of the test. The comparison is not done if the test exits with an error status. Before the comparison is done the output is massaged as follows to cater for system specific output.
     • The working directory is removed from the test output.
     • Text enclosed in >>> and <<< on a single line is removed from the test output and the expected output.
     • Text demarked by lines starting with >>> and <<< is removed from the test output and the expected output.
     • If the results are equal, the test passes. If any non-numerical value mismatches, the test fails.
     • Otherwise casacore_floatcheck is run to compare all numerical values with a relative error margin of 10e-5. The test fails if a value is found unequal.
   • At the end of the test the tX_tmp* files/directories are removed by casacore_assay.

3.1 Return status

The return of a test program or script can be:
- 0 indicates success.
- 3 indicates that the test is skipped. This might be used if an expected file is not available or if the test is run on a platform not supporting it. cmake_assay will change this to 0, because ctest can only deal with success and failure.
- Any other value indicates a failure.

4 Valgrind support

If the environment variable CASACORE_CHECK has the value 1, the casacore_assay script runs the test of an executable through casacore_memcheck. This script runs the test through valgrind’s memcheck tool and test for errors, definite and possible leaks, and leaked file descriptors. If any such problem is found, the valgrind output is put into a file tX.checktool.valgrind in the working directory.

If a .run file is used to execute the test, each invocation of the test program in the .run file should be preceeded by $casa_checktool. This environment variable defines the valgrind command. If valgrind is not used, the variable is empty.

4.1 Using other checking tools

The same mechanism can be used to use a check tool different from valgrind’s memcheck. CASACORE_CHECK can be defined as the name of a script or a command to use. In fact, defining CASACORE_CHECK as casacore_memcheck would have the same effect as defining it as 1.

4.2 Omitting tests in check tool

Sometimes a test run though a check tool can take too much time. There is no direct way to tell that a specific test should be omitted. However, it can easily be achieved by having a .run file for such a test without the $casa_checktool prefix.