[lttng-dev] Userspace LTTng tracing in a non-privileged container

[Jonathan Rajotte-Julien]

Hi Sasha
On Tue, Jan 30, 2018 at 09:50:06AM +0000, Sasha Goldshtein wrote:
> Hi,
> 
> I'd like to use LTTng from within a non-privileged Docker container to
> perform
> user-space tracing (enable-event --userspace --tracepoint etc.).
> Specifically,
> I would like to trace a .NET Core application, which uses LTTng events to
> trace
> GC, JIT, assembly loading, and other events. My testing environment is a
> FC25
> host with an Ubuntu 14.04 container.
> 
> I can install LTTng within the target container, but the session daemon
> produces
> the following message while loading:
> 
> modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could
> not open moddep file '/lib/modules /4.12.9-200.fc25.x86_64/modules.dep.bin'
> Error: Unable to load module lttng-tracer
> DEBUG1 [1830/1830]: Failed to open /proc/lttng (in init_kernel_tracer() at
> main.c:2382)
> Error: Unable to remove module lttng-tracer
> Warning: No kernel tracer available

This should not be a problem, you can even start lttng-sessiond with the
--no-kernel options to bypass this check. Userspace tracing should be available
in both cases.

> 
> Subsequent commands seem to work, but there are no traces produced.
> What I ran was:
> 
> lttng create mysession --output /tmp
> lttng enable-event --userspace --tracepoint DotNETRuntime:GC*
> lttng start
> lttng stop
> lttng destroy

Did you try with all events enabled? For testing purpose only.

If it does not work, could you open a bug here [1] and put as much information
in it?

 - Logs from lttng-sessiond in verbose mode (-vvv --verbose-consumer),
 - Application log launched with the LTTNG_UST_DEBUG env.variable set,
 - How lttng was installed.

[1] https://bugs.lttng.org/

> 
> Is there a way to use the LTTng session daemon in non-privileged mode,
> if I'm only interested in userspace tracing? I would like to avoid
> installing
> LTTng on the host system, or running privileged. In fact, the ideal workflow
> for me is to have a separate container image with LTTng and "attach" that
> container image (using docker --pid --net) to the target container that I
> want
> to analyze.

Never tried that but it look quite interesting.

Michael Jeanson might have more information on all this, he's been working on
container tracing lately. He is also presenting at FOSDEM [2] on this subject (host
side tracing) in a couple of days.

[2] https://fosdem.org/2018/schedule/event/containers_lttng/

Cheers,

>But I'd settle for having LTTng in the target container if
> that's the
> only option.
> 
> By the way, running LTTng from the host when the target application is in a
> container doesn't work either (i.e. no traces are produced). Running LTTng
> from the host when the target application is also on the host produces
> results successfully.
> 
> Any suggestions would be very much appreciated.
> 
> Thanks,
> Sasha

> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


-- 
Jonathan Rajotte-Julien
EfficiOS