[lttng-dev] babeltrace tests (master/2.0)

[Nathan Lynch]

Ping?


Nathan Lynch <nathan_lynch at mentor.com> writes:
> How is the babeltrace test suite on the master and stable-2.0 branches
> intended to be used?
>
> Effectively there are four test suites, one for each definition of TESTS
> in the tests/ directory.  If one of them has a failure, 'make check'
> will not proceed to the next unless the -k/--keep-going flag is used.
> Is this intended?  I unwittingly was running only the tests under
> tests/cli (some of which always fail) until I discovered this.
>
> Defining multiple test suites this way seems to be unusual; other
> projects I've checked just have one top-level definition of TESTS.
>
> I'm aware of the rationale for doing it this way:
>
> commit 6ca1931cb32ca2eb33252896d2a42a4c48af436a
> Author: Philippe Proulx <eeppeliteloop at gmail.com>
> Date:   Fri May 5 16:45:11 2017 -0400
>
>     tests: put TESTS list in each Makefile.am
>     
>     So that you can do `make check` in any subdirectory to run only those
>     tests.
>
> But it still feels non-idiomatic to me, and I'm wondering if there's a
> better way.  There are other ways to limit the set of tests to run,
> detailed here:
>
> https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html
>
> Here are my current results from running all test suites; let me know if
> you want more detail and I'll follow up; there are likely several
> different issues to investigate.
>
> $ find tests/ -name test-suite.log -exec head {} +          
> ==> tests/lib/test-suite.log <==
> =====================================================
>    babeltrace 2.0.0-pre1: tests/lib/test-suite.log
> =====================================================
>
> # TOTAL: 1200
> # PASS:  1195
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  1
> # XPASS: 0
>
> ==> tests/bindings/python/bt2/test-suite.log <==
> =====================================================================
>    babeltrace 2.0.0-pre1: tests/bindings/python/bt2/test-suite.log
> =====================================================================
>
> # TOTAL: 11152
> # PASS:  0
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  11151
> # XPASS: 0
>
> ==> tests/cli/test-suite.log <==
> =====================================================
>    babeltrace 2.0.0-pre1: tests/cli/test-suite.log
> =====================================================
>
> # TOTAL: 153
> # PASS:  143
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  7
> # XPASS: 0
>
> ==> tests/plugins/test-suite.log <==
> =========================================================
>    babeltrace 2.0.0-pre1: tests/plugins/test-suite.log
> =========================================================
>
> # TOTAL: 2
> # PASS:  0
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  0
> # XPASS: 0
>
> (BTW, since the tests don't pass/skip/xfail 100%, 'make distcheck' is failing.)