[lttng-dev] Improving the enable-event and disable-event commands

Philippe Proulx eeppeliteloop at gmail.com
Tue Oct 27 18:59:56 EDT 2015

Hello LTTngers!

This is a request for comments regarding the suggested solutions to
two fundamental problems with the `enable-event`/`disable-event` CLI
command (and their associated API functions).

Both problems and their proposed solution are independent, and it is
recommended that problem 1 be solved before problem 2, as its proposed
solution is easier to implement.

The solutions suggested below are designed to maintain full backward
compatibility with LTTng 2.7 while still improving the `enable-event`
and `disable-event` commands.

Feel free to contribute your own ideas!

Philippe Proulx


LTTng came to this world with the concept of switching the tracing
activity of possible system events on and off from the command-line,
hence the _enable_ and _disable_ names.

Eventually, wildcards in event names and other event activation
criteria were added (dynamic filters, log levels). All those features
were added as new options to the `enable-event` command. In this case,
however, we're not simply switching existing events on, but rather
_creating_ new **tuples** of conditions to be met for an event, existing
or not, to be emitted by the LTTng tracers. Those tuples, as of
LTTng 2.7, have the following components:

  * **Domain**: `--kernel`, `--userspace`, `--jul`, `--log4j`, and
    `--python` options
  * **Event name** (may contain a wildcard at the end, in which case a list
    of specific event names to exclude (`--exclude` option) may also exist):
    1st positional argument
  * **Event log level(s)**: `--loglevel` and `--loglevel-only` options
    (default: any log level)
  * **Filter string**: `--filter` option (default: `1`, which is always true)

For example, the command

    lttng enable-event -u hello:world --loglevel TRACE_INFO

creates and enables this tuple:

  * Domain:             user space
  * Event name:         match `hello:world`
  * Event log level(s): <= `TRACE_INFO`
  * Filter string:      `1`

As long as this tuple is enabled, any user space event matching those
conditions will be emitted by the tracer, even if there was no such
matching event when the `enable-event` command was executed.

Problem 1: `enable-event` cannot enable more than one event tuple at a time

An issue is encountered when using the `--all` option of the
`enable-event` command. The `--all` option, as it is implemented
today, is the equivalent of providing the event name `*`.

For example:

    lttng enable-event -u --all
    lttng enable-event -u something --filter 'id >= 10'
    lttng enable-event -u 'els*' --exclude else,elsy

creates and enables those tuples:

  * Tuple 1:
    * Domain:             user space
    * Event name:         match anything
    * Event log level(s): any
    * Filter string:      `1`
  * Tuple 2:
    * Domain:             user space
    * Event name:         match `something`
    * Event log level(s): any
    * Filter string:      `id >= 10`
  * Tuple 3:
    * Domain:             user space
    * Event name:         match `els` followed by anything,
                          except `else` and `elsy`
    * Event log level(s): any
    * Filter string:      `1`

Now, let's disable all those tuples:

    lttng disable-event -u -a

All three tuples are disabled. What if we need to enable all three
tuples? Using `--all` seems intuitive:

    lttng enable-event -u --all

But, as mentioned above, this is the equivalent of:

    lttng enable-event -u '*'

and so, only tuple 1 is enabled, leaving the other two
disabled. In other words, it is not possible, today, to enable more than
one event tuple matching specific conditions.

More generally, this problem can be exhibited when event names with
wildcards are used.

Proposed solution

Ideally, the `enable-event` command should be split in two individual
commands: `create-event` for creating new tuples and `enable-event` for
enabling existing _tuples_ matching specific conditions.

Unfortunately, the `enable-event` command already fullfills both
roles, and its current behaviour cannot be changed with a minor

It was suggested that a `--match` option be added to the `enable-event`
command. The presence of the `--match` option affects the command's
behaviour as such:

  * Absent: **legacy mode**. The `enable-event` command behaves as it
    does today. Enabling a nonexistent tuple creates it. There is no way
    to enable more than one tuple at a time in this mode.
  * Present: **match mode**. Enabling a nonexistent tuple has no effect
    (this version of the command does not create tuples). Any unspecified
    tuple criterion is considered a _match all_ (don't care). This mode
    can enable more than one tuple at a time.

For example, it is now possible to enable all the _existing tuples_
having a log level equal to `TRACE_WARNING`:

    lttng enable-event --match -u --loglevel-only TRACE_WARNING

Or all the _existing_ tuples having an event name starting with `hello`:

    lttng enable-event --match -u hello'*'

The same command without `--match`:

    lttng enable-event -u hello'*'

would create a new tuple with the event name `hello*`.

When the desired action is to enable all tuples with a specific event name
containing a wildcard, the latter must be escaped with `\`:

    lttng enable-event -u test
    lttng enable-event -u test23
    lttng enable-event -u test'*'
    lttng enable-event -u test'*' --loglevel TRACE_INFO
    lttng enable-event -u test'*' --filter 'yeah != 23'


    lttng enable-event --match -u test'*'

enables all the events above, whereas

    lttng enable-event --match -u test'\*'

enables the last three because their name contains the actual
wildcard character.

Even though the event type is not part of the tuple conditions,
since two events with different types cannot share the same name,
the `enable-event` command in match mode still offers the event
type options to match existing events. For example,

    lttng enable-event --match -k --function

matches all existing kernel event tuples having the dynamic function
probe type.

The domain option is not mandatory in match mode: it's a criterion
to match. So this:

    lttng enable-event --match eye

enables all the event tuples which belong to all the possible tracing
domains and have the name `eye`.

A `--match` option is also added to the `disable-event` command.
This option behaves like its equivalent on the `enable-event` command:
it allows to disable multiple event tuples at the same time.

It has to be noted that the current behaviour of `disable-event -a`
implies a match mode: it targets all the enabled event tuples, not
the `*` event like `enable-event -a` does. In other words, `--match`
has no effect on `disable-event -a`, but it does on `enable-event -a`.
This asymmetry must remain to ensure full backward compatibility.

Problem 2: enabling an existing event tuple requires all its conditions

Let us create two event tuples:

    lttng enable-event -u hello:world --loglevel TRACE_INFO
    lttng enable-event -u hello:world

which creates and enables the following tuples:

  * Tuple 1:
    * Domain:             user space
    * Event name:         match `hello:world`
    * Event log level(s): <= `TRACE_INFO`
    * Filter string:      `1`
  * Tuple 2:
    * Domain:             user space
    * Event name:         match `hello:world`
    * Event log level(s): any
    * Filter string:      `1`

Now, we have two enabled tuples sharing the same event name: their
only difference is their log level condition.

The `disable-event` command matches existing event names; this command
cannot, as for LTTng 2.7, match other conditions. In this case, our
only way to disable the first tuple, but leave the second one enabled,
is disabling both first:

    lttng disable-event -u hello:world

and then enabling the first one again:

    lttng enable-event -u hello:world --loglevel TRACE_INFO

A mechanism to uniquely identify both tuples would be interesting, since
right now, all the conditions must be repeated on the command-line to
target a specific tuple.

Proposed solution

The problem of uniquely identifying event tuples can be solved by
assigning a unique _rule name_ (or _alias name_) to each tuple.

When creating a tuple, a rule name is automatically assigned to it
using the event name (which is a mandatory criterion). Since many
event tuples may coexist with the same event names, a numeric suffix
is appended to the rule name. For example:

    lttng enable-event -u hello
    lttng enable-event -u hello --loglevel TRACE_INFO
    lttng enable-event -u hello --filter 'a < 23'
    lttng enable-event -u hello'*'
    lttng enable-event -u hello.all

creates five rules named:

  * `hello`
  * `hello.2`
  * `hello.3`
  * `hello.all`
  * `hello.all.2`

A rule name can be assigned manually to a tuple when creating it
using the new `--rule` option of the `enable-event` command:

    lttng enable-event -u --rule hi           hello
    lttng enable-event -u --rule meow         hello --loglevel TRACE_INFO
    lttng enable-event -u --rule hi-with-filt hello --filter 'a < 23'
    lttng enable-event -k --rule fn:get-opt   get_opt --function get_option

Rule names are unique within the tracing session.

The `list` command shows the rule names assigned to each event tuple,
be it automatically or manually. The `disable-event` and `enable-event`
commands can target existing rule names using the `--rule` option:

    lttng disable-event --rule hi-with-filt
    lttng disable-event --rule meow
    lttng enable-event  --rule hi-with-filt

Another example; all system calls can be grouped under the same
rule name:

    lttng enable-event -k --syscall --all --rule syscalls

Wildcards can be used when enabling/disabling rules. With carefully
crafted rule names, a hierarchy can be created and parent rules can
be enabled/disabled at once. For example, consider the following rule

    User space:


All driver event tuples can be disabled at once:

    lttng disable-event --rule sys:driver:'*'

MariaDB tuples can be turned on and off in one command:

    lttng disable-event --rule maria:'*'
    lttng enable-event  --rule maria:'*'

It is possible to use the `enable-event` command in match mode (as
described in the previous solution). The command

    lttng enable-event --match --rule apache:'*' --loglevel-only TRACE_INFO

only enables the existing Apache HTTP Server event tuples having a log
level set to `TRACE_INFO`.

More information about the lttng-dev mailing list