[lttng-dev] babeltrace tests (master/2.0)

[Nathan Lynch]

How is the babeltrace test suite on the master and stable-2.0 branches
intended to be used?

Effectively there are four test suites, one for each definition of TESTS
in the tests/ directory.  If one of them has a failure, 'make check'
will not proceed to the next unless the -k/--keep-going flag is used.
Is this intended?  I unwittingly was running only the tests under
tests/cli (some of which always fail) until I discovered this.

Defining multiple test suites this way seems to be unusual; other
projects I've checked just have one top-level definition of TESTS.

I'm aware of the rationale for doing it this way:

commit 6ca1931cb32ca2eb33252896d2a42a4c48af436a
Author: Philippe Proulx <eeppeliteloop at gmail.com>
Date:   Fri May 5 16:45:11 2017 -0400

    tests: put TESTS list in each Makefile.am
    
    So that you can do `make check` in any subdirectory to run only those
    tests.

But it still feels non-idiomatic to me, and I'm wondering if there's a
better way.  There are other ways to limit the set of tests to run,
detailed here:

https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html

Here are my current results from running all test suites; let me know if
you want more detail and I'll follow up; there are likely several
different issues to investigate.

$ find tests/ -name test-suite.log -exec head {} +          
==> tests/lib/test-suite.log <==
=====================================================
   babeltrace 2.0.0-pre1: tests/lib/test-suite.log
=====================================================

# TOTAL: 1200
# PASS:  1195
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0

==> tests/bindings/python/bt2/test-suite.log <==
=====================================================================
   babeltrace 2.0.0-pre1: tests/bindings/python/bt2/test-suite.log
=====================================================================

# TOTAL: 11152
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  11151
# XPASS: 0

==> tests/cli/test-suite.log <==
=====================================================
   babeltrace 2.0.0-pre1: tests/cli/test-suite.log
=====================================================

# TOTAL: 153
# PASS:  143
# SKIP:  0
# XFAIL: 0
# FAIL:  7
# XPASS: 0

==> tests/plugins/test-suite.log <==
=========================================================
   babeltrace 2.0.0-pre1: tests/plugins/test-suite.log
=========================================================

# TOTAL: 2
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0

(BTW, since the tests don't pass/skip/xfail 100%, 'make distcheck' is failing.)