[lttng-dev] [RELEASE] Babeltrace 2.0.0-pre5
Jérémie Galarneau
jeremie.galarneau at efficios.com
Fri May 3 19:37:27 EDT 2019
Hi everyone!
We are happy to announce the release of Babeltrace 2.0.0-pre5.
What's new since pre4?
----------------------
- New features
-- Trace merging
It is now possible to merge CTF traces that share the same UUID to a
single CTF trace. This is especially useful when combined with the
tracing session rotation feature introduced by LTTng 2.11.
For instance, it is possible to combine an arbitrary number of trace
archives and merge them back together.
Example:
A tracing session configured to automatically rotate every two seconds
will result in a large number of independent traces, as follows:
/path/to/my_session-20190501-181356
└── archives
├── 20190501T181356-0400-20190501T181358-0400-1
│ └── ust
│ └── uid
│ └── 1000
│ └── 64-bit
│ ├── channel0_0
│ ├── channel0_1
│ ├── channel0_2
│ ├── channel0_3
│ ├── index
│ └── metadata
├── 20190501T181358-0400-20190501T181400-0400-2
│ └── ust
│ └── uid
│ └── 1000
│ └── 64-bit
│ ├── channel0_0
│ ├── channel0_1
│ ├── channel0_2
│ ├── channel0_3
│ ├── index
│ └── metadata
├── 20190501T181400-0400-20190501T181402-0400-3
│ └── ust
│ └── uid
│ └── 1000
│ └── 64-bit
│ ├── channel0_0
│ ├── channel0_1
│ ├── channel0_2
│ ├── channel0_3
│ ├── index
│ └── metadata
...
With this feature, it is now possible to combine all, or a subset, of
the archived trace chunks. The following example combines two of the
archived trace chunks into a single trace.
babeltrace [...]/archives/20190501T181358-0400-20190501T181400-0400-2 \
[...]archives/20190501T181402-0400-20190501T181404-0400-4 \
--output-format=ctf --output /tmp/merged-trace
This command produces the following trace:
/tmp/merged-trace
└── myhostname
└── my_session-20190501T181356-0400
└── ust
└── uid
└── 1000
└── 64-bit
├── channel0_0
├── channel0_1
├── channel0_2
├── channel0_3
└── metadata
Note that this command uses new trace environment fields which will be
produced by the final release of LTTng 2.11. In the meantime, using
the existing LTTng 2.11-rc1 release will result in a different output
trace hierarchy.
-- LTTng live "subscribe" mode
It is now possible to configure the `src.ctf.lttng-live` component in
"subscribe" mode. In this mode, the source will attempt to consume an
LTTng relay daemon's session and retry periodically if the session
does not exist.
This is in contrast with the default behaviour which will return (with
success) immediately if the session to consume does not exist on the
LTTng relay daemon.
The following command demonstrates the use of this new mode:
babeltrace --component=src.ctf.lttng-live \
--url=net://relayhost/host/tgthost/my-session \
--params='session-not-found-action="continue"'
Note that we are planning the addition of a dedicated option to the CLI by the
time of the final release.
-- CLI support for array component parameters (`--params`)
Users of the CLI can now use array parameters when configuring a
component. See the following example:
babeltrace --component=src.my.class \
--params='tokens=[23, 17, [42, "meow"]], path="/path/to/trace"'
-- CTF source can now recursively search paths (and path arrays)
Using the feature above, it is now possible to provide a list of paths
to the `src.ctf.fs` component and rely on the component to recursively
open CTF traces it finds.
See the following example:
babeltrace --component=src.ctf.fs \
--params='paths=["trace1", "trace2", "trace3"]'
- Performance improvements
One of the primary focus areas of the pre5 release was optimization
work to bring Babeltrace 2's CTF reader back to performance parity
with Babeltrace 1.x and reduce the overhead of the graph processing
infrastructure. As of this release, our benchmarks indicate that
Babeltrace 2 is now slightly faster (~4%) than the latest Babeltrace 1
release with a dummy output.
This work (and performance comparison) was the subject of a talk by
Philippe Proulx at the latest Tracing Summit. The video [1] and slides
[2] of this presentation are available and provide the rationale
behind the biggest changes to the API.
Beyond API changes, this work resulted in the addition of a new
"developer mode". When Babeltrace 2 is built in this mode, a large
number of checks are added to enforce most preconditions required by
the API.
This makes it easier to develop and test new plugins while ensuring
that they honour the contract of the various functions they use. Since
components are now assumed to have been tested in this mode by their
authors, the release configuration of Babeltrace 2 can eschew most
precondition checks at run time, resulting in improved performance.
To build Babeltrace 2 in this mode and test your own plugins, set the
`BABELTRACE_DEV_MODE` environment variable `1` at configure time, for
example:
BABELTRACE_DEV_MODE=1 ./configure
Moreover, you can use the `BABELTRACE_MINIMAL_LOG_LEVEL` variable to
compile out all logging below a given level (`VERBOSE`, `DEBUG`,
`INFO`, `WARN`, `ERROR`, and `FATAL`), resulting in further
performance improvements. The performance impact of enabling a logging
level of INFO (and up) is minimal as such statements are seldom used
in hot paths.
The current default logging level is left at `VERBOSE` and will remain
so until the final release. This setting incurs a performance penalty,
but will make it easier for users of this pre-release to report bugs
with detailed logs.
- API changes
The performance improvement work has required significant API changes.
These changes were intrusive and have made it difficult to make
intermediary releases available, which is the main reason for the long
time between releases pre4 and pre5.
The APIs introduced by Babeltrace 2 made use of systematic reference
counting which was identified as a key culprit behind the performance
gap between Babeltrace 1.x and 2.x. A number of APIs routinely used in
the "hot paths" of trace processing now allow references to be
borrowed at no performance cost.
Among the biggest changes is the split between the CTF IR API
(previously introduced by the CTF writer library in Babeltrace 1.x)
and trace IR.
Trace IR is a higher-level trace access API that decouples the trace
processing graph from trace format implementation details. Since it is
decoupled from the CTF specification, we took the opportunity to adopt
a more generic (and sometimes clearer) terminology.
Observations made during the development of the new features of this
release and comments by users who have started using the pre4 API have
led us to adding "self" versions of objects which make use of
compile-time type checking to ensure that using functions that should
only be used in a given context results in warnings when they are
misused.
Moreover, const-related changes to the API will make it easier for
plugin authors to write const-correct code and provide an indication
when certain operations should not be performed.
Finally, an optional message iterator seeking API has been added to
allow seeking to a trace's beginning or to an arbitrary point
expressed in nanoseconds since the trace's origin.
We expect the API to undergo some minor changes before the first
release candidate. Among them figure:
- Addition of ranges associated to variant field classes.
- Introduction of a dependency link between upstream and
downstream message iterators.
- Introduction of canceler objects, allowing long graph operations to be
canceled (mainly meant for interactive use cases).
- Introduction of new types and user attributes in preparation for
CTF 2.
- Versioning API to allow plugins to query graph capabilities.
- Introduction of unsigned/signed value objects.
A number of users had reported having started implementing plugins
using the pre4 API. Hence, we welcome any feedback regarding the API
changes.
- Co-installation with Babeltrace 1.x
In order to facilitate the transition from Babeltrace 1.x to 2.x, we
will be making it easier to co-install both release series of
Babeltrace. To do so, the following changes are upcoming planned:
- Rename the CLI tool to `babeltrace2`. It is planned that
packagers will symlink it as `babeltrace` when preferred, as
is done for CPython packages.
- Rename the include directory to `babeltrace2`.
- Rename the library to `libbabeltrace2`.
Since co-installing both series will be possible, we plan to:
- Drop support for the `babeltrace` Python package (superseded by `bt2`).
- Drop support for the `babeltrace-log` util (you can do the
equivalent with the Babeltrace 2 CLI).
- Bug fixes
A large number of bug fixes are introduced as part of this release,
notably:
- Network efficiency improvements of the `src.ctf.lttng-live`
component class.
- Fixes to the argument parsing of the CLI.
- CTF writer correctness fixes.
- Miscellaneous bug fixes in the various plugins.
- Road to the final release
Here is a non-exhaustive list of changes planned before we make the
final 2.0 release available. Note that intermediate pre-releases and
release candidate releases will be made available as significant
milestones are met.
- Minor API changes (detailed above).
- Windows support.
- Update the `bt2` Python package to match the current library API.
- Restore failing `bt2` Python package tests.
- Update the C and Python API documentation to reflect pre5 changes.
- Update man pages and add relevant/useful/common examples.
- Create a deterministic sink component class for testing and debugging
purposes.
- Create `flt.lttng-utils.debug-info` test cases.
- Ensure all public header files have `__cplusplus` guards and
that it is possible to build a C++ applications using
the Babeltrace 2 library.
- Fully support `lttng-crash` traces.
- Allow the co-installation of Babeltrace 1 and 2.
- Warn or fail on unknown component parameters.
Project website: http://diamon.org/babeltrace/
Download link: https://www.efficios.com/files/babeltrace/babeltrace-2.0.0-pre5.tar.bz2
[1] https://www.youtube.com/watch?v=7BcXbvVewQY
[2] https://tracingsummit.org/w/images/3/38/Tracingsummit2018-babeltrace2-proulx.pdf
More information about the lttng-dev
mailing list