12.1.3.5 Log files generation and test results recording

The test driver must correctly generate the files specified by the --log-file and --trs-file option (even when the tested program fails or crashes).

The .log file should ideally contain all the output produced by the tested program, plus optionally other information that might facilitate debugging or analysis of bug reports. Apart from that, its format is basically free.

The .trs file is used to register some metadata through the use of custom reStructuredText fields. This metadata is expected to be employed in various ways by the parallel test harness; for example, to count the test results when printing the testsuite summary, or to decide which tests to re-run upon make recheck. Unrecognized metadata in a .trs file is currently ignored by the harness, but this might change in the future. The list of currently recognized metadata follows.

:test-result:

The test driver must use this field to register the results of each test case run by a test script file. Several :test-result: fields can be present in the same .trs file; this is done in order to support test protocols that allow a single test script to run more test cases.

The only recognized test results are currently PASS, XFAIL, SKIP, FAIL, XPASS and ERROR. These results, when declared with :test-result:, can be optionally followed by text holding the name and/or a brief description of the corresponding test; the harness will ignore such extra text when generating test-suite.log and preparing the testsuite summary.

:recheck:

If this field is present and defined to no, then the corresponding test script will not be run upon a make recheck. What happens when two or more :recheck: fields are present in the same .trs file is undefined behaviour.

:copy-in-global-log:

If this field is present and defined to no, then the content of the .log file will not be copied into the global test-suite.log. We allow to forsake such copying because, while it can be useful in debugging and analysis of bug report, it can also be just a waste of space in normal situations, e.g., when a test script is successful. What happens when two or more :copy-in-global-log: fields are present in the same .trs file is undefined behaviour.

:test-global-result:

This is used to declare the "global result" of the script. Currently, the value of this field is needed only to be reported (more or less verbatim) in the generated global log file $(TEST_SUITE_LOG), so it’s quite free-form. For example, a test script which run 10 test cases, 6 of which pass and 4 of which are skipped, could reasonably have a PASS/SKIP value for this field, while a test script which run 19 successful tests and one failed test could have an ALMOST PASSED value. What happens when two or more :test-global-result: fields are present in the same .trs file is undefined behaviour.

Let’s see a small example. Assume a .trs file contains the following lines:

:test-result: PASS server starts
:global-log-copy: no
:test-result: PASS HTTP/1.1 request
:test-result: FAIL HTTP/1.0 request
:recheck: yes
:test-result: SKIP HTTPS request (TLS library wasn't available)
:test-result: PASS server stops

Then the corresponding test script will be re-run by make check, will contribute with five test results to the testsuite summary (three of these tests being successful, one failed, and one skipped), and the content of the corresponding .log file will not be copied in the global log file test-suite.log.