Goals and Motivation

When doing quality assurance for scientific software, it is often necessary to look at the overall results of a scientific simulation or calculation, and compare the outcome carefully with those of previous runs. The validation or comparison of results is often highly specific to the software under test and the actual test case. Sometimes a discrepancy between the expected and the actual result can be a consequence of improvement rather than failure, for example when there was an increase in accuracy. In some cases it might be necessary to allow a discrepancy with certain conditions or boundaries, or there might be the need for more complex analysis of the results to decide if a certain validation fails or succeeds.

This is a perfect use-case for black-box testing as performed with Autotester: Program internals will not be taken into account, instead the focus lies on validating and comparing results with reference data after the program under test as a whole was executed as defined in the test case. Unit tests and other more fine-grained tests instead only enforce certain conditions for small parts of the code like functions or interfaces, and are often tied to specific data formats and programming languages. This is why frameworks for unit testing can not easily be adapted for black-box testing in all cases. Manually performing such tests is of course cumbersome, which is why an automated solution like Autotester with convenient features like summarizing test results, machine-readable results and statistics can help with the task of black-box testing of software in general.

Autotester allows describing and running automated test suits for black-box testing. Each test consists of a command to be executed, and any number of validations to be performed after the command was run successfully. This way, there are no restrictions in terms of programming language or specialized data file formats of the tested application, as long as suitable tooling is available to be integrated with Autotester. In most cases those tools can reuse code or library functions from the scientific framework they work with, and thus understand the specifics of involved data formats. This way, Autotester can provide testing functionality for any software or framework without growing in complexity.