If the special variable
TESTS is defined, its value is taken to be
a list of programs or scripts to run in order to do the testing. Under
the appropriate circumstances, it’s possible for
TESTS to list
also data files to be passed to one or more test scripts defined by
different means (the so-called “log compilers”, see Parallel Test Harness).
Test scripts can be executed serially or concurrently. Automake supports
both these kinds of test execution, with the parallel test harness being
the default. The concurrent test harness relies on the concurrence
capabilities (if any) offered by the underlying
implementation, and can thus only be as good as those are.
By default, only the exit statuses of the test scripts are considered when determining the testsuite outcome. But Automake allows also the use of more complex test protocols, either standard (see Using the TAP test protocol) or custom (see Custom Test Drivers). Note that you can’t enable such protocols when the serial harness is used, though. In the rest of this section we are going to concentrate mostly on protocol-less tests, since we cover test protocols in a later section (again, see Custom Test Drivers).
When no test protocol is in use, an exit status of 0 from a test script will denote a success, an exit status of 77 a skipped test, an exit status of 99 an hard error, and any other exit status will denote a failure.
You may define the variable
XFAIL_TESTS to a list of tests
(usually a subset of
TESTS) that are expected to fail; this will
effectively reverse the result of those tests (with the provision that
skips and hard errors remain untouched). You may also instruct the
testsuite harness to treat hard errors like simple failures, by defining
DISABLE_HARD_ERRORS make variable to a nonempty value.
Note however that, for tests based on more complex test protocols,
the exact effects of
might change, or they might even have no effect at all (for example,
in tests using TAP, there is not way to disable hard errors, and the
DISABLE_HARD_ERRORS variable has no effect on them).
The result of each test case run by the scripts in
TESTS will be
printed on standard output, along with the test name. For test protocols
that allow more test cases per test script (such as TAP), a number,
identifier and/or brief description specific for the single test case is
expected to be printed in addition to the name of the test script. The
possible results (whose meanings should be clear from the previous
Generalities about Testing) are
ERROR. Here is an
example of output from an hypothetical testsuite that uses both plain
and TAP tests:
PASS: foo.sh PASS: zardoz.tap 1 - Daemon started PASS: zardoz.tap 2 - Daemon responding SKIP: zardoz.tap 3 - Daemon uses /proc # SKIP /proc is not mounted PASS: zardoz.tap 4 - Daemon stopped SKIP: bar.sh PASS: mu.tap 1 XFAIL: mu.tap 2 # TODO frobnication not yet implemented
A testsuite summary (expected to report at least the number of run, skipped and failed tests) will be printed at the end of the testsuite run.
If the standard output is connected to a capable terminal, then the test
results and the summary are colored appropriately. The developer and the
user can disable colored output by setting the
‘AM_COLOR_TESTS=no’; the user can in addition force colored output
even without a connecting terminal with ‘AM_COLOR_TESTS=always’.
It’s also worth noting that some
when used in parallel mode, have slightly different semantics
(see Parallel Make), which can
break the automatic detection of a connection to a capable terminal.
If this is the case, the user will have to resort to the use of
‘AM_COLOR_TESTS=always’ in order to have the testsuite output
Test programs that need data files should look for them in
(which is both a make variable and an environment variable made available
to the tests), so that they work when building in a separate directory
(see Build Directories), and in particular for the
(see Checking the Distribution).
TESTS_ENVIRONMENT variables can
be used to run initialization code and set environment variables for the
test scripts. The former variable is developer-reserved, and can be
defined in the Makefile.am, while the latter is reserved for the
user, which can employ it to extend or override the settings in the
former; for this to work portably, however, the contents of a non-empty
AM_TESTS_ENVIRONMENT must be terminated by a semicolon.
AM_TESTS_FD_REDIRECT variable can be used to define file
descriptor redirections for the test scripts. One might think that
AM_TESTS_ENVIRONMENT could be used for this purpose, but experience
has shown that doing so portably is practically impossible. The main
hurdle is constituted by Korn shells, which usually set the close-on-exec
flag on file descriptors opened with the
exec builtin, thus
rendering an idiom like
AM_TESTS_ENVIRONMENT = exec 9>&2;
ineffectual. This issue also affects some Bourne shells, such as the
AM_TESTS_ENVIRONMENT = \ ## Some environment initializations are kept in a separate shell ## file 'tests-env.sh', which can make it easier to also run tests ## from the command line. . $(srcdir)/tests-env.sh; \ ## On Solaris, prefer more POSIX-compliant versions of the standard ## tools by default. if test -d /usr/xpg4/bin; then \ PATH=/usr/xpg4/bin:$$PATH; export PATH; \ fi; ## With this, the test scripts will be able to print diagnostic ## messages to the original standard error stream, even if the test ## driver redirects the stderr of the test scripts to a log file ## before executing them. AM_TESTS_FD_REDIRECT = 9>&2
Note however that
AM_TESTS_ENVIRONMENT is, for historical and
implementation reasons, not supported by the serial harness
(see Serial Test Harness).
Automake ensures that each file listed in
TESTS is built before
it is run; you can list both source and derived programs (or scripts)
TESTS; the generated rule will look both in
.. For instance, you might want to run a C program as a test.
To do this you would list its name in
TESTS and also in
check_PROGRAMS, and then specify it as you would any other
Programs listed in
check_LTLIBRARIES...) are only built during
make all. You should list there any program needed
by your tests that does not need to be built by
make all. Note
check_PROGRAMS are not automatically added to
check_PROGRAMS usually lists programs used
by the tests, not the tests themselves. Of course you can set
TESTS = $(check_PROGRAMS) if all your programs are test cases.