Creating test cases
~~~~~~~~~~~~~~~~~~~

Autotester reads test configurations from ATC files (**A**\ utotester
**T**\ est **C**\ onfiguration). Each test has a name and consists of a
description of the program under test, how to call the program for this
test (including parameters and their values) as well as a number of
validations which will be performed after the program was run
successfully. Tests can also have a description and a list of variables
which can be used to avoid duplicate values in the configuration. ATC
files can define one or more tests. A schema description for the XML
format can be found ``schema/atc.xsd``.

By default, Autotester will recursively search files with the ending
``.atc`` and process them. Thus, it is possible to define a hierarchy of
tests accompanied by input and reference data.

For example, a folder structure for a simple test setup could look like
this:

::

   .
   ├── group1
   │   ├── compare
   │   │   └── reference.dat
   │   └── group1.atc
   ├── group2
   │   ├── compare
   │   │   └── reference.dat
   │   ├── group2.atc
   │   └── input
   │       └── config.txt
   └── main.atc

**Important**: Only one ATC file is allowed per directory!

If Autotester is called in the main directory of this folder structure,
Autotester will find all ATC files and run the tests in the respective
folder or subfolder. In case the option ``-e|--regex [pattern]`` is
specified, tests will be skipped if their name does not match the
pattern.

It is also possible to run a single ATC file using the option
``-s|--single [FILE]``:

::

   autotester -s examples/example.atc

In the test description it’s possible to reference the folders
``compare/``, ``input/`` and ``output/`` which reside in the same
directory as the ATC file by using the pre-defined variables
``$COMPARE_PATH$``, ``$INPUTDATA_PATH$`` and ``$OUTPUT_PATH$``. The
default values for those variables can be overwritten in the file
``$HOME/.autotester/config.xml`` (see ``schema/config.xsd`` for a
description of the expected format of the configuration file).


Example ATC file
^^^^^^^^^^^^^^^^

The following is an example ATC file that can be run with Autotester:

.. literalinclude:: ../../../examples/structured_tests/main.atc
  :language: xml

Find this and other examples in the folder `examples/ <https://git.opencarp.org/iam-cms/autotester/-/tree/master/examples>`__.

Caveats
^^^^^^^

-  For some characters it is necessary to use escaped sequences in the
   ATC file, for example use ``&gt;`` instead of ``>`` for piping to a
   file. Alternatively, use the CDATA directive:

.. code-block:: xml

  <command><![CDATA[pandoc --version > /dev/null]]></command>


-  Using bash variables or expressions like ``$(...)`` currently leads
   to problems because ``$`` will be interpreted as the start of a
   variable to be replaced by Autotester.