Creating test cases

Autotester reads test configurations from ATC files (Autotester Test Configuration). Each test has a name and consists of a description of the program under test, how to call the program for this test (including parameters and their values) as well as a number of validations which will be performed after the program was run successfully. Tests can also have a description and a list of variables which can be used to avoid duplicate values in the configuration. ATC files can define one or more tests. A schema description for the XML format can be found schema/atc.xsd.

By default, Autotester will recursively search files with the ending .atc and process them. Thus, it is possible to define a hierarchy of tests accompanied by input and reference data.

For example, a folder structure for a simple test setup could look like this:

.
├── group1
│   ├── compare
│   │   └── reference.dat
│   └── group1.atc
├── group2
│   ├── compare
│   │   └── reference.dat
│   ├── group2.atc
│   └── input
│       └── config.txt
└── main.atc

Important: Only one ATC file is allowed per directory!

If Autotester is called in the main directory of this folder structure, Autotester will find all ATC files and run the tests in the respective folder or subfolder. In case the option -e|--regex [pattern] is specified, tests will be skipped if their name does not match the pattern.

It is also possible to run a single ATC file using the option -s|--single [FILE]:

autotester -s examples/example.atc

In the test description it’s possible to reference the folders compare/, input/ and output/ which reside in the same directory as the ATC file by using the pre-defined variables $COMPARE_PATH$, $INPUTDATA_PATH$ and $OUTPUT_PATH$. The default values for those variables can be overwritten in the file $HOME/.autotester/config.xml (see schema/config.xsd for a description of the expected format of the configuration file).

Example ATC file

The following is an example ATC file that can be run with Autotester:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<atc>
  <test name="example system test">
    <description format="markdown">This is a simple test which copies a file</description>
    <program name="cp" />
    <call expected-return-code="0">
      <command>$program$ $program_options$ $file1$ $file2$</command><!-- just copy a file using cp -->
      <timeout unit="s" kill-signal="9">1</timeout>
    </call>

    <variables>
      <variable name="program_options">-f</variable>
      <variable name="file1">$COMPARE_PATH$/example_data.txt</variable><!-- the compare path is relative to the current
          directory -->
      <variable name="file2">$OUTPUT_PATH$/output.txt</variable><!-- the output folder will be created next to the
          .atc file in the same parent folder. The benefit of using is that it will be cleaned up by autotester when
          the test succeeds by default (this behaviour can be changed using the options "keeponsuccess" and
          "deleteonerror") -->
    </variables>

    <validations>
      <validation>
        <description>Output and compare file must match</description>
        <call expected-return-code="0">
          <command>cmp $options$ $file1$ $file2$</command>
          <timeout unit="s" kill-signal="9">12</timeout>
        </call>
        <variables>
          <variable name="options">-s -i 10</variable> <!-- for example: ignore the first 10 bytes of data -->
        </variables>
      </validation>
    </validations>
  </test>
  <!-- add more tests here if necessary -->
</atc>

Find this and other examples in the folder examples/.

Caveats

• For some characters it is necessary to use escaped sequences in the ATC file, for example use > instead of > for piping to a file. Alternatively, use the CDATA directive:

<command><![CDATA[pandoc --version > /dev/null]]></command>

• Using bash variables or expressions like $(...) currently leads to problems because $ will be interpreted as the start of a variable to be replaced by Autotester.