[lttng-dev] [RELEASE] Babeltrace 2.0.0

Jérémie Galarneau jeremie.galarneau at efficios.com
Wed Jan 22 12:35:51 EST 2020


Hello people!

This is it: the day you've been waiting for has finally arrived.

Today we're releasing Babeltrace 2.0 “Amqui”! 🥳

Amqui [1] (/ɒmkwiː/) is a town in eastern Québec [2], Canada, at the
base of the Gaspé peninsula [3] in Bas-Saint-Laurent [4].

Located at the confluence of the Humqui and Matapédia [5] Rivers, its
proximity to woodlands makes it a great destination for outdoor
activities such as camping, hiking, and mountain biking.

By the way, Babeltrace has a new dedicated domain name:
`babeltrace.org`.

NOTE: An HTML version of this announcement is available at
<https://diamon.org/babeltrace/docs/release-notes/babeltrace-2.0.0-release-notes.html>.

Background
==========
Just a few words about why Babeltrace 2 took about five years to
complete.

First, let us clarify that from day one, one of the most important
imperatives of the project was **long-term stability**: we wanted
Babeltrace 2 to last for a long time and for its library's API to become
a well-known reference in the trace processing territory, a bit like
Libxml2 [6] has had major version 2 for about 20 years. All the other
Babeltrace 2 parts depend on the library's C API [7]: the command-line
interface [8] (CLI), the plugins, and the Python bindings.

So, as we needed to modify the library, CLI, and plugins to adapt to new
requirements during the development phase, we postponed the release so
as not to have to release Babeltrace 3 a year after and Babeltrace 4
another year after.

Second, the scope and requirements of the project evolved. Because there
were at most two developers working on Babeltrace 2 at the same time
during its first years, the project took enough time to develop for new
ideas to emerge and translate into changes, sometimes significant, to
the project's modules.

For example, the very first requirement was for the libbabeltrace2 API
to use the same objects as Babeltrace 1's CTF writer API. We put a _lot_
of effort backfitting new concepts into the CTF writer API and making
sure not to break existing user code. But in the end, we realized that
this condition was too limiting and that we had to split the APIs
completely to have the freedom to introduce all the features we wanted.

When the API finally met our needs feature-wise in 2018, we measured and
found that a simple CTF-to-text conversion task with Babeltrace 2 was
approximately 23 times slower than the equivalent with Babeltrace 1!
Hence we entered an optimization phase [9] in which we had to redefine
core concepts of the library (for example, introduce _unique_ objects to
create object pools [10]) so that, today, Babeltrace 2's performance is
comparable to Babeltrace 1's [11].

Revamping the C API to optimize the library means we had to readapt the
CLI, plugins, and Python bindings. We spent the beginning of 2019
synchronizing the project's parts, but also improving and uplifting the
test suite and fixing a lot of lingering bugs.

The last few months were invested in writing a complete C API
documentation [7] and in more bug fixes, tests, and build system
improvements.

Today ends this journey to Babeltrace 2.0. It's been a long one, but we
made it. All the next releases of Babeltrace 2 will be minor ones for
the foreseeable future, never breaking the library's API.

Babeltrace 2 is:

* 120,000 lines of C.
* 7000 lines of Python.
* 21,000 tests.
* 30,000 lines of C API documentation.
* 24 manual pages

We hope you enjoy using our software as much as we enjoyed making and
refining it over the last years.

What's new since Babeltrace 1?
==============================
The Babeltrace 2 project is completely independent from Babeltrace 1. In
fact, you can install both on the same system as they share no common
files. We only made sure to keep the `babeltrace2` CLI [8] tool backward
compatible with the original `babeltrace` tool [12].

See `babeltrace2-intro(7)` [13] to learn more about Babeltrace 2.

General:
    * Full plugin support: any user can distribute a Babeltrace 2 plugin
      and, as long as libbabeltrace2 finds it, any application linked to
      libbabeltrace2 can load it and use it.

      Plugins are not just trace format encoders and decoders: they
      package source, filter, and sink component classes so that you can
      connect specialized, reusable components together in a trace
      processing graph to create a customized trace conversion or
      analysis device.

      This modular strategy is much like how the FFmpeg [14],
      GStreamer [15], and DirectShow [16] projects approach media stream
      processing.

    * All the parts of the Babeltrace 2 project run on the major
      operating systems, including Linux, macOS, and Windows.

    * Some component classes, such as `sink.text.pretty` [17] (similar
      to the `text` output format of the `babeltrace` CLI [12]) and
      `sink.text.details` [18], can write color codes to the standard
      output when it's connected to a color-enabled terminal.

      The Babeltrace 2 log, printed to the standard output, can also be
      colorized.

Command-line interface:
    * Whereas you can convert traces from one format to another with
      Babeltrace 1's CLI tool, `babeltrace` [12], you can also execute a
      custom trace manipulation task with `babeltrace2` [8] thanks to
      its `run` [19] command.

    * The `convert` [20] command features an automatic source component
      discovery algorithm to find the best suited components to create
      for a given non-option argument (file or directory path, or custom
      string like an LTTng live [21] URL).

      For example:

          $ babeltrace2 /path/to/ctf/trace

          $ babeltrace2 net://localhost/host/myhost/my-session

CTF input/output [22]:
    * The `source.ctf.fs` [23] component class, which is more or less
      the equivalent of Babeltrace 1's `ctf` input format, has features
      not found in Babeltrace 1:

      * The component handles many trace quirks [24] which are the
        results of known tracer bugs (which are fixed today) and corner
        cases in LTTng-UST, LTTng-modules, and barectf [25], making it
        possible to decode malformed packets.

      * The component merges CTF traces sharing the same UUID into a
        single, logical trace [26].

        This feature supports LTTng 2.11's tracing session rotation [27]
        trace chunks.

    * With a `sink.ctf.fs` [28] component, you can create CTF traces on
      the file system.

      With the `babeltrace2` tool, you can use the `--output-format=ctf`
      and `--output` options to create an implicit `sink.ctf.fs`
      component.

      For example:

          $ babeltrace2 /path/to/input/trace --output-format=ctf \
                                             --output=trace-dir

LTTng live [21] input:
    Babeltrace 1's `babeltrace` command [12] exits successfully when it
    cannot find an LTTng live (`--input-format=lttng-live` option)
    tracing session.

    The `session-not-found-action` [29] initialization parameter
    controls what a `source.ctf.lttng-live` [30] message iterator does
    when it cannot find the remote tracing session:

    * If the action is `end`, the message iterator does like
      `babeltrace` and simply ends successfully.

    * If the action is `continue` (the default), the message iterator
      never ends: it keeps on trying until the tracing session exists,
      indeed “subscribing” to the session.

Library:
    * libbabeltrace2 shares nothing with libbabeltrace.

      The Babeltrace 2 library C API [7] has features such as:

      * A single header file.
      * Function precondition and postcondition checking.
      * Object-oriented model with shared and unique objects.
      * Strict C typing and `const` correctness.
      * User-extensible classes.
      * Rich, thread-safe error reporting.
      * Per-component and per-subsystem logging levels.
      * Trace intermediate representation (IR) objects to make the API
        trace-format-agnostic.
      * A versioned protocol for the exchange of messages between
        components to enable forward and backward compatibility.

    * You can build the library in _developer mode_ to enable an
      extensive set of function precondition and postcondition checks.

      The developer mode can help detect programming errors early when
      you develop a Babeltrace 2 plugin or an application using
      libbabeltrace2.

      See the project's README [31] for build-time requirements and
      detailed build instructions.

Python 3 bindings:
    * The new `bt2` package offers all the functionality of
      libbabeltrace2 and even more through a brand new, Pythonic
      programming interface.

    * With the `bt2` package, you can load existing plugins and create a
      trace processing graph, but also create Babeltrace 2 plugins.

      libbabeltrace2 can load both shared object and Python plugins.

    * What has the `const` qualifier in libbabeltrace2 has a dedicated
      constant type in the `bt2` package so that methods to modify it
      simply do not exist. This can help catch programming errors.

      For example, when you create an event message, you get a
      `bt2._EventMessage` object. When you consume an event message, you
      get an immutable `bt2._EventMessageConst` object. Both objects
      wrap the same C `bt_event *` (and `const bt_event *`) object.

What's new since Babeltrace 2.0.0-rc4?
======================================

Improvements
~~~~~~~~~~~~
General:
    * Hide private symbols.

    * Add a mechanism to add extra information (name, description, and
      names of applied patches) to a custom build's version.

      See `1f316595`.

Command-line interface:
    Print full version, including the release's name and name
    description, the Git revision's description, and the extra
    information, with the `--version` option.

    See `72fa419d`.

Documentation:
    Write a comprehensive Babeltrace 2 library C API documentation [7].

    See `7704a0af`.

Library:
    * Remove the `bt_packet_context_field` API (not needed anymore).

      See `9b98c3bf`.

    * Standardize variant field option function names.

      See `7e0b3e01`.

    * Graph API: remove “ports connected” listeners (not needed
      anymore).

      See `d9d764cc`.

    * Graph API: remove the “listener removed” callbacks (not needed
      anymore).

      See `fecbdb3e`.

    * Remove `bt_graph_interrupt()`; use the new
      `bt_graph_borrow_default_interrupter()` function instead.

      See `f09b3ae9`.

    * Remove `bt_query_executor_interrupt()`; use the new
      `bt_query_executor_borrow_default_interrupter()` function instead.

      See `d9a20b20`.

    * Add the _message iterator class_ concept.

      Instead of having an implicit message iterator class within source
      and filter component classes, make its creation explicit and pass
      it to `bt_component_class_source_create()` and
      `bt_component_class_filter_create()`.

      See `68e2deed`, `8e1fe2a4`, and `f615b250`.

    * Rename `BT_PLUGIN_INITIALIZE()` to `BT_PLUGIN_INITIALIZE_FUNC()`
      and `BT_PLUGIN_FINALIZE()` to `BT_PLUGIN_FINALIZE_FUNC()`.

      See `1906e012`.

    * For many functions of the graph and component APIs, return a
      _borrowed_ reference of an added object instead of a new
      reference.

      See `246d5223`.

    * Rename
      `bt_message_message_iterator_inactivity_borrow_default_clock_snapshot_const()`
      to
      `bt_message_message_iterator_inactivity_borrow_clock_snapshot_const()`.

      See `62988c56`.

    * Rename `bt_version_get_extra()` to
      `bt_version_get_development_stage()`.

      See `19e20ba9`.

    * Add the following functions to get more information about the
      library's version:

      * `bt_version_get_vcs_revision_description()`
      * `bt_version_get_name()`
      * `bt_version_get_name_description()`
      * `bt_version_get_extra_name()`
      * `bt_version_get_extra_description()`
      * `bt_version_get_extra_patch_names()`

      See `2491e1b5`, `58df4c80`, and `a0446fa8`.

Python bindings:
    * Adapt to the library changes above.

    * Make `bt2.Graph.add_*_listener()` return `None`.

      See `d9d764cc`.

    * Reverse the order of the printed causes in `bt2._Error.__str__()`
      to match what the CLI does.

      See `701698c8`.

`source.ctf.fs` component class [23]:
    Sort the paths of the data stream files to handle so that the output
    port order is predictable (for debugging purposes).

    See `18118efb`.

Notable bug fixes
~~~~~~~~~~~~~~~~~
Command-line interface:
    Print error causes in all error paths.

    See `f2ff3e07`.

Python bindings:
    * Fix shared reference issues in listener handles.

      See `d4024c86`.

    * Don't print previous error causes in causes created from
      `bt2._Error`.

      See `b53cd768`.

What's next?
============
The current road map includes:

* Make it easier to create an efficient filter message iterator which
  needs to modify or augment event messages.

  Currently, if you need to modify an event message, considering that
  you cannot change an event class once you use it, you need to create a
  new event class, which means you also need to create a new stream
  class and a new trace class. You also need to copy all the metadata
  and data objects you don't alter. This is what
  `filter.lttng-utils.debug-info` [32] does.

  We do realize that trace-transforming filter message iterators
  represent an important Babeltrace 2 use case. While it's possible to
  write one for Babeltrace 2.0, it is cumbersome to do and the result is
  not as performant as we'd like. We decided to release Babeltrace 2 now
  and reserve improvements to this part of the library for a future
  minor release.

* Add a filter component class to drop messages (like events) based on a
  filter expression, similar to the `--filter` option [33] of the
  `lttng` command.

* Make the `ctf` [34] plugin's component classes support CTF 2, the
  upcoming major revision of the Common Trace Format [22].

* Optimize specific project component classes, especially
  `source.ctf.fs` [23], `source.ctf.lttng-live` [30], and
  `filter.utils.muxer` [35].

  For `source.ctf.fs`, implement the “seek ns from origin” method using
  the packet indexes to make a message iterator seek a specific time
  faster.

== Important links

Important links
===============
Babeltrace 2.0.0 tarball:
    https://www.efficios.com/files/babeltrace/babeltrace-2.0.0.tar.bz2

Babeltrace website:
    https://diamon.org/babeltrace

Mailing list (for support and development):
    `lttng-dev at lists.lttng.org`

IRC channel:
    `#lttng` on `irc.oftc.net`

Git repository:
    https://bugs.lttng.org/projects/babeltrace

GitHub project:
    https://github.com/efficios/babeltrace

Continuous integration:
    https://ci.lttng.org/view/Babeltrace/

Code review:
    https://review.lttng.org/q/project:babeltrace

References
==========
[1]: https://en.wikipedia.org/wiki/Amqui
[2]: https://en.wikipedia.org/wiki/Quebec
[3]: https://en.wikipedia.org/wiki/Gasp%C3%A9_Peninsula
[4]: https://en.wikipedia.org/wiki/Bas-Saint-Laurent
[5]: https://en.wikipedia.org/wiki/Matapedia_River
[6]: http://www.xmlsoft.org/
[7]: https://babeltrace.org/docs/v2.0/libbabeltrace2/
[8]: https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/
[9]: https://tracingsummit.org/ts/2018/files/Tracingsummit2018-babeltrace2-proulx.pdf
[10]: https://en.wikipedia.org/wiki/Object_pool_pattern
[11]: https://babeltrace.org/docs/release-notes/babeltrace-benchmark-2020-01-20.pdf
[12]: http://man7.org/linux/man-pages/man1/babeltrace.1.html
[13]: https://babeltrace.org/docs/v2.0/man7/babeltrace2-intro.7
[14]: https://www.ffmpeg.org/
[15]: https://gstreamer.freedesktop.org/
[16]: https://en.wikipedia.org/wiki/DirectShow
[17]: https://babeltrace.org/docs/v2.0/man7/babeltrace2-sink.text.pretty.7
[18]: https://babeltrace.org/docs/v2.0/man7/babeltrace2-sink.text.details.7
[19]: https://babeltrace.org/docs/v2.0/man1/babeltrace2-run.1
[20]: https://babeltrace.org/docs/v2.0/man1/babeltrace2-convert.1
[21]: https://lttng.org/docs/#doc-lttng-live
[22]: https://diamon.org/ctf/
[23]: https://babeltrace.org/docs/v2.0/man7/babeltrace2-source.ctf.fs.7
[24]: https://babeltrace.org/docs/v2.0/man7/babeltrace2-source.ctf.fs.7/#doc-_trace_quirks
[25]: https://github.com/efficios/barectf
[26]: https://babeltrace.org/docs/v2.0/man7/babeltrace2-source.ctf.fs.7/#doc-input
[27]: https://lttng.org/docs/#doc-session-rotation
[28]: https://babeltrace.org/docs/v2.0/man7/babeltrace2-sink.ctf.fs.7
[29]: https://babeltrace.org/docs/v2.0/man7/babeltrace2-source.ctf.lttng-live.7/#doc-param-session-not-found-action
[30]: https://babeltrace.org/docs/v2.0/man7/babeltrace2-source.ctf.lttng-live.7
[31]: https://github.com/efficios/babeltrace/blob/stable-2.0/README.adoc
[32]: https://babeltrace.org/docs/v2.0/man7/babeltrace2-filter.lttng-utils.debug-info.7
[33]: https://lttng.org/man/1/lttng-enable-event/v2.11/#doc-opt--filter
[34]: https://babeltrace.org/docs/v2.0/man7/babeltrace2-plugin-ctf.7
[35]: https://babeltrace.org/docs/v2.0/man7/babeltrace2-filter.utils.muxer.7


More information about the lttng-dev mailing list