How tests will be processed by Autotester
Tests will be processed by calling the specified command in the
<command> section of the test description. In case the command
contains variables, they will be replaced first.
If the test configuration is invalid (invalid syntax, unknown variables etc.), the result will be ATC_FAIL.
If the command fails (expected return code does not match the actual return code), the result will be EXECUTION_FAIL. The expected return code defaults to 0 but can be specified using an attribute of the
<call>tag in the test description:<call expected_return_code="7">.If a timeout for the program call is specified (using the optional
<timeout>tag) and the specified duration is exceeded by the running process, the process will be killed and the result will be EXECUTION_TIMEOUT.
In case none of the above error events occurred, the validations of the
test will be processed. Similar to the processing of tests, the each
validation will be started as described in its <call> section. The
result of the test will be VALIDATION_FAIL in case any of the
validation fails, VALIDATION_TIMEOUT in case one or more validations
exceeded their limit, or PASS in case all validations were
successful. When there are different validation errors for one test, the
most severe failure determines the result status of the test case.
VALIDATION_FAIL will be considered more severe than
VALIDATION_TIMEOUT.
Autotesters return code
The return code of Autotester will be the number of testcases that have failed, independently for the reason they failed, or how many of their validations have failed.