12.1.4.2 Use TAP with the Automake test harness

Currently, the TAP driver that comes with Automake requires some by-hand steps on the developer’s part (this situation should hopefully be improved in future Automake versions). You’ll have to grab the tap-driver.sh script from the Automake distribution by hand, copy it in your source tree, and use the Automake support for third-party test drivers to instruct the harness to use the tap-driver.sh script and the awk program found by AM_INIT_AUTOMAKE to run your TAP-producing tests. See the example below for clarification.

Apart from the options common to all the Automake test drivers (see Command-line arguments for test drivers), the tap-driver.sh supports the following options, whose names are chosen for enhanced compatibility with the prove utility.

--ignore-exit

Causes the test driver to ignore the exit status of the test scripts; by default, the driver will report an error if the script exits with a non-zero status. This option has effect also on non-zero exit statuses due to termination by a signal.

--comments

Instruct the test driver to display TAP diagnostic (i.e., lines beginning with the ‘#’ character) in the testsuite progress output too; by default, TAP diagnostic is only copied to the .log file.

--no-comments

Revert the effects of --comments.

--merge

Instruct the test driver to merge the test scripts’ standard error into their standard output. This is necessary if you want to ensure that diagnostics from the test scripts are displayed in the correct order relative to test results; this can be of great help in debugging (especially if your test scripts are shell scripts run with shell tracing active). As a downside, this option might cause the test harness to get confused if anything that appears on standard error looks like a test result.

--no-merge

Revert the effects of --merge.

--diagnostic-string=STRING

Change the string that introduces TAP diagnostic from the default value of “#” to STRING. This can be useful if your TAP-based test scripts produce verbose output on which they have limited control (because, say, the output comes from other tools invoked in the scripts), and it might contain text that gets spuriously interpreted as TAP diagnostic: such an issue can be solved by redefining the string that activates TAP diagnostic to a value you know won’t appear by chance in the tests’ output. Note however that this feature is non-standard, as the “official” TAP protocol does not allow for such a customization; so don’t use it if you can avoid it.

Here is an example of how the TAP driver can be set up and used.

% cat configure.ac
AC_INIT([GNU Try Tap], [1.0], [bug-automake@gnu.org])
AC_CONFIG_AUX_DIR([build-aux])
AM_INIT_AUTOMAKE([foreign -Wall -Werror])
AC_CONFIG_FILES([Makefile])
AC_REQUIRE_AUX_FILE([tap-driver.sh])
AC_OUTPUT

% cat Makefile.am
TEST_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) \
                  $(top_srcdir)/build-aux/tap-driver.sh
TESTS = foo.test bar.test baz.test
EXTRA_DIST = $(TESTS)

% cat foo.test
#!/bin/sh
echo 1..4 # Number of tests to be executed.
echo 'ok 1 - Swallows fly'
echo 'not ok 2 - Caterpillars fly # TODO metamorphosis in progress'
echo 'ok 3 - Pigs fly # SKIP not enough acid'
echo '# I just love word plays ...'
echo 'ok 4 - Flies fly too :-)'

% cat bar.test
#!/bin/sh
echo 1..3
echo 'not ok 1 - Bummer, this test has failed.'
echo 'ok 2 - This passed though.'
echo 'Bail out! Ennui kicking in, sorry...'
echo 'ok 3 - This will not be seen.'

% cat baz.test
#!/bin/sh
echo 1..1
echo ok 1
# Exit with error, even if all the tests have been successful.
exit 7

% cp PREFIX/share/automake-APIVERSION/tap-driver.sh .
% autoreconf -vi && ./configure && make check
...
PASS: foo.test 1 - Swallows fly
XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
PASS: foo.test 4 - Flies fly too :-)
FAIL: bar.test 1 - Bummer, this test has failed.
PASS: bar.test 2 - This passed though.
ERROR: bar.test - Bail out! Ennui kicking in, sorry...
PASS: baz.test 1
ERROR: baz.test - exited with status 7
...
Please report to bug-automake@gnu.org
...
% echo exit status: $?
exit status: 1

% env TEST_LOG_DRIVER_FLAGS='--comments --ignore-exit' \
      TESTS='foo.test baz.test' make -e check
...
PASS: foo.test 1 - Swallows fly
XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
# foo.test: I just love word plays...
PASS: foo.test 4 - Flies fly too :-)
PASS: baz.test 1
...
% echo exit status: $?
exit status: 0