[lttng-dev] Userspace LTTng tracing in a non-privileged container

[Michael Jeanson]

On 2018-01-30 09:58, Jonathan Rajotte-Julien wrote:
> Hi Sasha
> On Tue, Jan 30, 2018 at 09:50:06AM +0000, Sasha Goldshtein wrote:
>> Hi,
>>
>> I'd like to use LTTng from within a non-privileged Docker container to
>> perform
>> user-space tracing (enable-event --userspace --tracepoint etc.).
>> Specifically,
>> I would like to trace a .NET Core application, which uses LTTng events to
>> trace
>> GC, JIT, assembly loading, and other events. My testing environment is a
>> FC25
>> host with an Ubuntu 14.04 container.
>>
>> I can install LTTng within the target container, but the session daemon
>> produces
>> the following message while loading:
>>
>> modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could
>> not open moddep file '/lib/modules /4.12.9-200.fc25.x86_64/modules.dep.bin'
>> Error: Unable to load module lttng-tracer
>> DEBUG1 [1830/1830]: Failed to open /proc/lttng (in init_kernel_tracer() at
>> main.c:2382)
>> Error: Unable to remove module lttng-tracer
>> Warning: No kernel tracer available
> 
> This should not be a problem, you can even start lttng-sessiond with the
> --no-kernel options to bypass this check. Userspace tracing should be available
> in both cases.
> 
>>
>> Subsequent commands seem to work, but there are no traces produced.
>> What I ran was:
>>
>> lttng create mysession --output /tmp
>> lttng enable-event --userspace --tracepoint DotNETRuntime:GC*

You would need to single quote 'DotNETRuntime:GC*' to make sure your
shell doesn't expand the '*'.

>> lttng start
>> lttng stop
>> lttng destroy
> 
> Did you try with all events enabled? For testing purpose only.
> 
> If it does not work, could you open a bug here [1] and put as much information
> in it?
> 
>  - Logs from lttng-sessiond in verbose mode (-vvv --verbose-consumer),
>  - Application log launched with the LTTNG_UST_DEBUG env.variable set,
>  - How lttng was installed.
> 
> [1] https://bugs.lttng.org/

That would be useful.

> 
>>
>> Is there a way to use the LTTng session daemon in non-privileged mode,
>> if I'm only interested in userspace tracing? I would like to avoid
>> installing
>> LTTng on the host system, or running privileged. In fact, the ideal workflow
>> for me is to have a separate container image with LTTng and "attach" that
>> container image (using docker --pid --net) to the target container that I
>> want
>> to analyze.

Not sure about the specifics of docker "attach" but for it to work the
lttng session daemon and the traced application would have to be in the
same pid namespace and share a common "/proc". The communication between
app and daemon is trough a socket in "/run" if the sessiond is running
as root or in $LTTNG_HOME otherwise, so that directory would also need
to be shared.

> 
> Never tried that but it look quite interesting.
> 
> Michael Jeanson might have more information on all this, he's been working on
> container tracing lately. He is also presenting at FOSDEM [2] on this subject (host
> side tracing) in a couple of days.
> 
> [2] https://fosdem.org/2018/schedule/event/containers_lttng/
> 
> Cheers,
> 
>> But I'd settle for having LTTng in the target container if
>> that's the
>> only option.

That would be the easiest way.

>>
>> By the way, running LTTng from the host when the target application is in a
>> container doesn't work either (i.e. no traces are produced). Running LTTng
>> from the host when the target application is also on the host produces
>> results successfully.

That's correct, apps in a container have no way to reach the sessiond
sockets on the host. Not that it couldn't be done but depending on the
container runtime and the level of isolation fro  the host it can get a
bit complicated.

>>
>> Any suggestions would be very much appreciated.
>>
>> Thanks,
>> Sasha
> 
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev at lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 
>