[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

16. Generating Test Suites with Autotest

Note: This section describes an experimental feature which will
be part of Autoconf in a forthcoming release.  Although we believe
Autotest is stabilizing, this documentation describes an interface which
might change in the future: do not depend upon Autotest without
subscribing to the Autoconf mailing lists.

It is paradoxical that portable projects depend on nonportable tools to run their test suite. Autoconf by itself is the paragon of this problem: although it aims at perfectly portability, up to 2.13, its test suite was using DejaGNU, a rich and complex testing framework, but which is far from being standard on Unix systems. Worse yet, it was likely to be missing on the most fragile platforms, the very platforms that are most likely to torture Autoconf and exhibit deficiencies.

To circumvent this problem many package maintainers have developed their own testing framework, based on simple shell scripts whose sole output are their exit status: the test succeeded, or failed. In addition, most of these tests share some common patterns, what results in lots of duplicated code, tedious maintenance etc.

Following exactly the same reasoning that yielded to the inception of Autoconf, Autotest provides a test suite generation frame work, based on M4 macros, building a portable shell script. The suite itself is equipped with automatic logging and tracing facilities which greatly diminish the interaction with bug reporters, and simple timing reports.

Autoconf itself has been using Autotest for years, and we do attest that it has considerably improved the strength of the test suite, and the quality of bug reports. Other projects are known to use some generation of Autotest, such as Bison, Free Recode, Free Wdiff, GNU Tar, each of them having different needs, what slowly polishes Autotest as a general testing framework.

Nonetheless, compared to DejaGNU, Autotest is inadequate for interactive tool testing, which is probably its main limitation.

16.1 Using an Autotest Test Suite  Autotest and the user
16.2 Writing `testsuite.at'  Autotest macros
16.3 Running testsuite Scripts  Running testsuite scripts
16.4 Making testsuite Scripts  Using autom4te to create testsuite

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

16.1 Using an Autotest Test Suite

16.1.1 testsuite Scripts  The concepts of Autotest
16.1.2 Autotest Logs  Their contents

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

16.1.1 testsuite Scripts

Generating testing or validation suites using Autotest is rather easy. The whole validation suite is held in a file to be processed through autom4te, itself using GNU M4 under the scene, to produce a stand-alone Bourne shell script which then gets distributed. Neither autom4te nor GNU M4 are not needed anymore at the installer end.

Each test of the validation suite should be part of some test group. A test group is a sequence of interwoven tests that ought to be executed together, usually because one test in the group creates data files than a later test in the same group needs to read. Complex test groups make later debugging more tedious. It is much better keeping keep only a few tests per test group, and if you can put only one test per test group, this is just ideal.

For all but the simplest packages, some file such as `testsuite.at' does not fully hold all test sources, as these are often easier to maintain in separate files. Each of these separate files holds a single test group, or a sequence of test groups all addressing some common functionality in the package. In such cases, file `testsuite.at' only initializes the whole validation suite, and sometimes do elementary health checking, before listing include statements for all other test files. The special file `package.m4', containing the identification of the package, is automatically included if found.

The validation scripts that Autotest produces are by convention called testsuite. When run, testsuite executes each test group in turn, producing only one summary line per test to say if that particular test succeeded or failed. At end of all tests, summarizing counters get printed. If any test failed, one debugging script gets automatically generated for each test group which failed. These debugging scripts are named `testsuite.nn', where nn is the sequence number of the test group. In the ideal situation, none of the tests fail, and consequently, no debugging script is generated out of validation.

The automatic generation of debugging scripts for failed test has the purpose of easing the chase for bugs.

It often happens in practice that individual tests in the validation suite need to get information coming out of the configuration process. Some of this information, common for all validation suites, is provided through the file `atconfig', automatically created by AC_CONFIG_TESTDIR. For configuration informations which your testing environment specifically needs, you might prepare an optional file named `atlocal.in', instantiated by AC_CONFIG_FILES. The configuration process produces `atconfig' and `atlocal' out of these two input files, and these two produced files are automatically read by the `testsuite' script.

Here is a diagram showing the relationship between files.

Files used in preparing a software package for distribution:

subfile-1.at ->.
    ...         \
subfile-i.at ---->-- testsuite.at -->.
    ...         /                     \
subfile-n.at ->'                       >-- autom4te* -->testsuite
                      [package.m4] ->'

Files used in configuring a software package:

                                     .--> atconfig
[atlocal.in] -->  config.status* --<
                                     `--> [atlocal]

Files created during the test suite execution:

atconfig -->.                    .--> testsuite.log
             \                  /
              >-- testsuite* --<
             /                  \
[atlocal] ->'                    `--> [testsuite.nn*]

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

16.1.2 Autotest Logs

When run, the test suite creates a log file named after itself, e.g., a test suite named testsuite creates `testsuite.log'. It contains a lot of information, usually more than maintainers actually need, but therefore most of the time it contains all that is needed:

command line arguments
A very bad Unix habit which is unfortunately wide spread consists of setting environment variables before the command, such as in `CC=my-home-grown-cc ./testsuite'. This results in the test suite not knowing this change, hence (i) it can't report it to you, and (ii) it cannot preserve the value of CC for subsequent runs(5). Autoconf faced exactly the same problem, and solved it by asking users to pass the variable definitions as command line arguments. Autotest requires this rule too, but has no means to enforce it; the log then contains a trace of the variables the user changed.

`ChangeLog' excerpts
The topmost lines of all the `ChangeLog's found in the source hierarchy. This is especially useful when bugs are reported against development versions of the package, since the version string does not provide sufficient information to know the exact state of the sources the user compiled. Of course this relies on the use of a `ChangeLog'.

build machine
Running a test suite in a cross-compile environment is not an easy task, since it would mean having the test suite run on a machine build, while running programs on a machine host. It is much simpler to run both the test suite and the programs on host, but then, from the point of view of the test suite, there remains a single environment, host = build. The log contains relevant information on the state of the build machine, including some important environment variables.

tested programs
The absolute path and answers to `--version' of the tested programs (see 16.2 Writing `testsuite.at', AT_TESTED).

configuration log
The contents of `config.log', as created by configure, are appended. It contains the configuration flags and a detailed report on the configuration itself.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

16.2 Writing `testsuite.at'

The `testsuite.at' is a Bourne shell script making use of special Autotest M4 macros. It often contains a call to AT_INIT nears its beginning followed by one call to m4_include per source file for tests. Each such included file, or the remainder of `testsuite.at' if include files are not used, contain a sequence of test groups. Each test group begins with one call to AT_SETUP, it contains an arbitrary number of shell commands or calls to AT_CHECK, and it completes with one call to AT_CLEANUP.

Macro: AT_INIT ([name])
Initialize Autotest. Giving a name to the test suite is encouraged if your package includes several test suites. In any case, the test suite always displays the package name and version. It also inherits the package bug report address.

Macro: AT_TESTED (executables)
Log the path and answer to `--version' of each program in space-separated list executables. Several invocations register new executables, in other words, don't fear registering one program several times.

Autotest test suites rely on the PATH to find the tested program. This saves from generating the absolute paths to the various tools, and makes it possible to test installed programs. Therefore, knowing what programs are being exercised is crucial to understand some problems in the test suite itself, or its occasional misuses. It is a good idea to also subscribe foreign programs you depend upon, to ease incompatibility diagnostics.

Macro: AT_SETUP (test-group-name)
This macro starts a group of related tests, all to be executed in the same subshell. It accepts a single argument, which holds a few words (no more than about 30 or 40 characters) quickly describing the purpose of the test group being started.

Macro: AT_KEYWORDS (keywords)
Associate the space-separated list of keywords to the enclosing test group. This makes it possible to run "slices" of the test suite. For instance if some of your test groups exercise some `foo' feature, then using `AT_KEYWORDS(foo)' lets you run `./testsuite -k foo' to run exclusively these test groups. The title of the test group is automatically recorded to AT_KEYWORDS.

Several invocations within a test group accumulate new keywords. In other words, don't fear registering several times the same keyword in a test group.

End the current test group.

Macro: AT_DATA (file, contents)
Initialize an input data file with given contents. Of course, the contents have to be properly quoted between square brackets to protect against included commas or spurious M4 expansion. The contents ought to end with an end of line.

Macro: AT_CHECK (commands, [status = ``0''], [stdout], [stderr])
Execute a test by performing given shell commands. These commands should normally exit with status, while producing expected stdout and stderr contents. If commands exit with status 77, then the whole test group is skipped.

The commands must not redirect the standard output, nor the standard error.

If status, or stdout, or stderr is `ignore', then the corresponding value is not checked.

The special value `expout' for stdout means the expected output of the commands is the content of the file `expout'. If stdout is `stdout', then the standard output of the commands is available for further tests in the file `stdout'. Similarly for stderr with `expout' and `stderr'.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

16.3 Running testsuite Scripts

Autotest test suites support the following arguments:

Display the list of options and exit successfully.

Display the version of the test suite and exit successfully.

Remove all the files the test suite might have created and exit. Meant for clean Makefile targets.

List all the tests (or only the selection), including their possible keywords.

By default all the tests are performed (or described with `--list') in the default environment first silently, then verbosely, but the environment, set of tests, and verbosity level can be tuned:

Set the environment variable to value. Do not run `FOO=foo ./testsuite' as debugging scripts would then run in a different environment.

The variable AUTOTEST_PATH specifies the testing path to prepend to PATH. It handles specially relative paths (not starting with `/'): they are considered to be relative to the top level of the package being built. All the directories are made absolute, first starting from the top level build tree, then from the source tree. For instance `./testsuite AUTOTEST_PATH=tests:bin' for a `/src/foo-1.0' source package built in `/tmp/foo' results in `/tmp/foo/tests:/tmp/foo/bin' and then `/src/foo-1.0/tests:/src/foo-1.0/bin' being prepended to PATH.

Add the corresponding test groups, with obvious semantics, to the selection.

`-k keywords'
Add to the selection the test groups which title or keywords (arguments to AT_SETUP or AT_KEYWORDS) match all the keywords of the comma separated list keywords.

Running `./testsuite -k autoupdate,FUNC' will select all the tests tagged with `autoupdate' and `FUNC' (as in `AC_CHECK_FUNC', `AC_FUNC_FNMATCH' etc.) while `./testsuite -k autoupdate -k FUNC' runs all the tests tagged with `autoupdate' or `FUNC'.

If any test fails, immediately abort testing. It implies `--debug': post test group clean up, debugging script generation, and logging are inhibited. This option is meant for the full test suite, it is not really useful for generated debugging scripts.

Force more verbosity in the detailed output of what is being done. This is the default for debugging scripts.

Do not remove the files after a test group was performed --but they are still removed before, therefore using this option is sane when running several test groups. Do not create debugging scripts. Do not log (in order to preserve supposedly existing full log file). This is the default for debugging scripts.

Trigger shell tracing of the test groups.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

16.4 Making testsuite Scripts

For putting Autotest into movement, you need some configuration and Makefile machinery. We recommend, at least if your package uses deep or shallow hierarchies, that you use `tests/' as the name of the directory holding all your tests and their `Makefile'. Here is a check list of things to do.

With Automake, here is a minimal example about how to link `make check' with a validation suite.

EXTRA_DIST = testsuite.at testsuite
TESTSUITE = $(srcdir)/testsuite
check-local: atconfig atlocal $(TESTSUITE)
        $(SHELL) $(TESTSUITE)

AUTOTEST = $(AUTOM4TE) --language=autotest
$(TESTSUITE): $(srcdir)/testsuite.at
        $(AUTOTEST) -I $(srcdir) $@.at -o $@.tmp
        mv $@.tmp $@

You might want to list explicitly the dependencies, i.e., the list of the files `testsuite.at' includes.

With strict Autoconf, you might need to add lines inspired from the following:

subdir = tests

atconfig: $(top_builddir)/config.status
        cd $(top_builddir) && \
           $(SHELL) ./config.status $(subdir)/$@

atlocal: $(srcdir)/atlocal.in $(top_builddir)/config.status
        cd $(top_builddir) && \
           $(SHELL) ./config.status $(subdir)/$@

and manage to have `atconfig.in' and $(EXTRA_DIST) distributed.

[ << ] [ >> ]           [Top] [Contents] [Index] [ ? ]

This document was generated by Jeff Bailey on December, 24 2002 using texi2html