Core features required to perform Golden file testing.
We provide the core functionality to run a *single* golden file test, or
a whole test tree.
This allows the developer freedom to use as is or design the rest of the
test harness to their liking.
This was originally used as part of Idris2's own test suite and
the core functionality is useful for the many and not the few.
Please see Idris2 test harness for example usage.
# Test Structure
This harness works from the assumption that each individual golden test
comprises of a directory with the following structure:
+ `run` a *shell* script that runs the test. We expect it to:
* Use `$1` as the variable standing for the idris executable to be tested
* May use `${IDRIS2_TESTS_CG}` to pick a codegen that ought to work
* Clean up after itself (e.g. by running `rm -rf build/`)
+ `expected` a file containing the expected output of `run`
During testing, the test harness will generate an artefact named `output`
and display both outputs if there is a failure.
During an interactive session the following command is used to compare them
as they are:
```sh
git diff --no-index --exit-code --word-diff-regex=. --color expected output
```
If `git` fails then the runner will simply present the expected and 'given'
files side-by-side.
Of note, it is helpful to add `output` to a local `.gitignore` instance
to ensure that it is not mistakenly versioned.
# Options
The test harness has several options that may be set:
+ `idris2` The path of the executable we are testing.
+ `codegen` The backend to use for code generation.
+ `onlyNames` The tests to run relative to the generated executable.
+ `onlyFile` The file listing the tests to run relative to the generated executable.
+ `interactive` Whether to offer to update the expected file or not.
+ `timing` Whether to display time taken for each test.
+ `threads` The maximum numbers to use (default: number of cores).
+ `failureFile` The file in which to write the list of failing tests.
We provide an options parser (`options`) that takes the list of command line
arguments and constructs this for you.
# Usage
When compiled to an executable the expected usage is:
```sh
runtests
[--timing]
[--interactive]
[--only-file PATH]
[--failure-file PATH]
[--threads N]
[--cg CODEGEN]
[[--only|--except] [NAMES]]
```
assuming that the test runner is compiled to an executable named `runtests`.