[lttng-dev] Proposal: Make lttng-analyses part of lttng-tools

[Jonathan Rajotte-Julien]

Hi Geneviève,

On Tue, Jul 17, 2018 at 02:34:54PM -0400, Geneviève Bastien wrote:
> Hi all,
> 
> 
> Here's an idea to improve first impression of lttng and hopefully help
> the UX of the toolchain:
> 
> 
> tl;dr; I propose to make lttng-analyses part of lttng-tools to provide
> some basic recording/analysis facilities and make them the blessed way
> of getting started with lttng, so that end users can quickly obtain
> meaningful results after installing lttng.

As previously said in person and on IRC, I'm 100% in favour of putting an
emphasis on lttng-analyses. Still, I'm not sure that making the lttng-analyses
project part of the lttng-tools source tree is the way to go.

> 
> Here follows my argumentation. I'll focus on kernel tracing.
> 
> 
> Facts:
> 
> * People don't read documentation, at least not from the beginning.

Yup...

> 
> * First impressions stick

Pretty sure some people still thinks that you need to recompile your kernel for
kernel tracing to work. For those skimming, lttng-modules DOES NOT
requires a kernel compilation.

> 
> * Potential users trying to evaluate a tool will typically spend a few
> minutes/hours getting that first impression by reading an as
> minimalistic as possible readme file.

Yup

> 
> 
> Right now, with lttng, the first thing people are invited to do for
> kernel tracing, is to enable all the events (-a -k), which results in a
> number of lost events and a lot of noise when trying to analyze the

If I remember correctly, at some time, the default subbuffer size was bumped at the package
(Ubuntu) level. There is a commit in tools that bump the default size and
should fix the lost events stuff.

    commit c9eb1ddcd4af613f4e10938ab957f2a4a945db61
    Author: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
    Date:   Fri Mar 31 10:04:20 2017 -0400

        Bump default kernel, and UST per-uid/per-pid buffer size

        LTTng with current default buffer size often lead to discarded events,
        which is not something we want as a first user impression.

        The choice of default buffer size were made conservatively around 2010.
        Since then, the memory available on typical systems has increased, and
        so has the amount of instrumentation available. As an example, the
        mid-2010 Macbook Pro had 2GB ram. The current 2017 Macbook specification
        states 8GB ram, for a 4-fold installed memory size increase.

        Increase the kernel tracer buffer size from:
        4 x 256kB per core
        to:
        4 x 1MB per core

        Increase the UST tracer per-uid buffer size from:
        4 x 128kB per core
        to
        4 x 512kB per core

        Increase the UST tracer per-pid buffer size from:
        4 x 4kB per core
        to
        4 x 16kB per core


> results, and then 'lttng view' to display the babeltrace output! (see
> the Usage section in the lttng-modules readme page here:
> https://github.com/lttng/lttng-modules)

The Usage section of modules does not seems to instruct user to do "enable -k
-a" but I get your point.

> 
> That is not imho a good first impressions. Other tracers like ftrace or
> perf have a more per-event approach to tracing, their beginner's

The doc here seems to have a good basic primer indicating that the user could
choose to enable all events if necessary and the different projects for
analysis (Following section).

https://lttng.org/docs/v2.10/#doc-tracing-the-linux-kernel

> documentation showing how to enable 1 or 2 tracepoints. But to get
> interesting results, we usually need more than that and listing them
> individually can be cumbersome.
> Looking at my outbox, I sent my lttng
> script to enable events to get the critical path to at least 15 people

I'm not sure it is the responsibility of the tracer to define "pertinent" events.
The critical analysis is mostly a Trace Compass thing. The Trace Compass project
might want to have clearer event requirements (matching to lttng events name or
other tracer if possible) for their analysis.

> in the last years... It would be easy to have a predefined script for
> that purpose. Or many scripts (or profiles?) to enable events for
> various interesting analyses types.

Scripts could be done (lttng-analyses), not sure profiles at the tracer level make sense.

I think lttng-analyses already provides the essential part of a possible
solution. It would be a matter wrapping the lttng-analyses-record command and
the analyses scripts.

> 
> By comparison, take another kernel tracing tool: ebpf / bcc
> (https://github.com/iovisor/bcc). Installation is easy, then the readme
> points to a number of tools and example scripts of all kind. Running
> them after installation is easy and gives instant results. That makes
> one happy and feel good about bcc! Because having to write an ebpf
> script would have many users run for their sanity! ;-) But since many
> use cases are already covered by easily available scripts, people adopt
> the tool, then a few months later, when they need more, some may get to
> actually read the script, modify it, make them own.
> 
> lttng should aim for that kind of ease of use for beginners, have them
> reach that good feeling in a few minutes trial. And lttng-analyses
> already provides such scripts to record and analyze traces, and it
> should really be put forward and improved to give the user a more
> seamless first experience. I'd suggest to include it with lttng-tools,
> in a tools/ directory, as it would be one less repo to fork or one less

I would be more inclined to have it as a "recommended" package of
lttng-modules (since only kernel analyses are provided)
packages for package manager supporting this kind of feature (e.g apt).

Installing from source should not be the "go to" way of installing lttng for
beginners or people with limited involvement in the development of the project.

> package to install and it's a clear sign that it goes together with
> lttng. Keeping it as a separate project makes it appear as a side-dish,
> while it should be the center-plate!

I agree on center-plate stuff, not so much on how :).

I'm eager to get some feedback from Mathieu and Jérémie on all this.

Cheers.

-- 
Jonathan Rajotte-Julien
EfficiOS