[lttng-dev] Proposal: Make lttng-analyses part of lttng-tools

[Genevieve Bastien]

On 2018-07-18 03:35 PM, Jérémie Galarneau wrote:
> On 18 July 2018 at 12:48, Geneviève Bastien <gbastien at versatic.net> wrote:
>> On 2018-07-18 12:11 PM, Philippe Proulx wrote:
>>> On Wed, Jul 18, 2018 at 11:37 AM Geneviève Bastien
>>> <gbastien at versatic.net> wrote:
>>>>> I'm eager to get some feedback from Mathieu and Jérémie on all this.
>>>> So am I!
>>> Adding mine even if you're not eager to get it ;-).
>> I'm eager to hear anyone's opinion! And I think from the 2 opinions I
>> got so far, analyses will not be part of tools. I'll take that as
>> granted and consider all future comments of mine for lttng-analyses repo
>> alone...
>>> As someone who knows a few parts of the LTTng analyses project, it is
>>> still considered experimental, prototypal, in other words not ready at
>>> all to be integrated within any stable project. It is with good reason
>>> that its version is still 0.x.
>>>
>>> LTTng analyses helps us prove that such a tool can be useful (we
>>> received good feedback in general), and it's a great project to quickly
>>> develop a tracing analysis, but it's lacking thorough documentation
>>> (outside its README file), tests, and by the nature of Python is very
>>> slow and will probably remain like this forever. LTTng analyses also
>>> does not handle exceptional scenarios very well.
>>>
>>> Also, do not forget that this project has hard dependencies on Python
>>> and Babeltrace. What would this mean, on the packaging level, when
>>> making it part of LTTng-tools, which has a very limited set of
>>> dependencies on purpose?
>> Interesting remark: You highlight that lttng-"analyses" package has 2
>> purposes: trace collection and trace analysis.
>>
>> Trace collection can only be bash, does not require any other
>> dependencies than lttng and that needs to be available on all machines
>> to trace (with this mechanism)
>>
>> Trace analyses use babeltrace python bindings and needs to be run on the
>> "analysis" machine, which may not even be one that is traced.
>>
>> 2 very different use cases... Should they be split in 2 packages? Could
>> the trace collection part of analyses be moved to tools, leaving only
>> the analyses in analyses? (oh I lied up there, I did mention merging
>> with lttng-tools again ;-)
> There is always the possibility of shipping session configurations along
> with lttng-tools and/or scripts if the configuration needs to be "smart".
>
> That's something that can be improved right now, we just need to
> identify presets that make sense. Is that a good first step?
I thought from previous conversations that session configurations (what
you save and load in lttng) is much more complex than just the traced
events and is really machine-specific, so the workflow is typically for
a user to configure his session, then save it so he can load it after.
Are you saying that right now, it would be possible to just define
presets of events, leaving the rest of the sessions configuration as
package defaults?

But if it's easily done, then yes, it's definitely a good first step!

>
>>> I believe our mid/long-term goal is to:
>>>
>>> 1. Release Babeltrace 2 ASAP.
>>> 2. Optimize Babeltrace 2 for the typical CTF input with stream muxing
>>>    scenario.
>>> 3. Create/transfer LTTng analyses as Babeltrace 2 filter or sink
>>>    components.
>>> 4. Encourage people to use Babeltrace 2 to analyse LTTng traces.
>> Indeed our goals at Poly are different. But lttng being an open source
>> project, I hope our goals can be reconciled for the good of the project :)
>>
>> Here are my short/mid-term goals:
>>
>> 1- Make trace collection with lttng for newcomers a painless process,
>> easy to document and explain, to get fast results (whether with
>> lttng-analyses or Trace Compass)
>> 2- Develop and document methods to collect traces in an OpenStack/ceph
>> cloud, in conjunction with monitoring tools.
>>
> I share Philippe's goals, but I also understand you want to improve the
> situation in the short term.
>
> Without integrating lttng-analyses in lttng-tools, I think Jonathan has
> a good point in terms of marking lttng-analyses as a recommended
> package or making it part of the 'lttng' metapackages (if applicable).
>
> Packagers (Michael?) may want to comment here as I guess this
> is a distro-specific issue.
>
>> In the short term, I'm more interested in the trace collection part of
>> lttng-analyses (or tools?), and that, in the [very!] short term, I'm
>> ready to bring to 1.0.
> I'm a lot more comfortable shipping this than the analysis part.
#define "shipping this"

Do you mean to say your would be open to have trace collection scripts
ship with lttng-tools? So basically, to move and modify the
lttng-analysis-record script from analyses to tools, leaving
lttng-analyses as a suggested package to tools and modules.

#define "comfortable"

For the next release? :p And you're using present tense, not
conditional, that's good!

Another option would be to have trace collection as another
package/project, so that collection and analysis are 2 different things.
But if so, I'd like for this repo to be official and somewhat blessed by
the project, to use in tutorial and getting started documentation, and
not my own shady repo. It can eventually also contain scripts to manage
tracing sessions (configure, start, stop, snapshot) on multiple
machines, then gather those traces together for later analysis.

Anyway, it seems that there's a script out there that needs a home
downtown lttng and it's making buying offers here and there. Will it go
in tools, evolve in lttng-analyses or get a brand new place? We can
discuss that once the said script has reached maturity. But we all seem
to agree on the need for it and that it has a place in the lttng
project. Which I'm happy with right now.

Also, in case we decide to split trace collection and analyses somehow,
Francis Giraldeau had also done a trace recording script as part of
workload-kit[1]. That one is in python and may be more along the lines
of tracing profiles (for events) and including ust traces. That may also
be a good starting point. Everybody has python these days, right?

[1] https://github.com/giraldeau/workload-kit/blob/master/utils/lttng-simple

Thanks for all the feedback and comments,

Geneviève