[lttng-dev] Userspace tracing in docker containers
eqbalzee at gmail.com
Mon May 3 19:45:07 EDT 2021
Thanks for the responses. The reasoning makes sense. We have decided to run
lttng-sessiond on the host. Our trace generating application will run in a
container and so will our libbabeltrace based trace consumer app (using
On Tue, Apr 6, 2021 at 7:07 AM Jonathan Rajotte-Julien <
jonathan.rajotte-julien at efficios.com> wrote:
> On Mon, Apr 05, 2021 at 11:09:39AM -0700, Eqbal via lttng-dev wrote:
> > Hi,
> > I am trying to get user space tracing working for an application running
> > a docker container. I am running lttng session daemon in another
> > I mounted the unix socket locations (either /var/run/lttng for root or
> > $HOME/.lttng for another user). By doing that I can run commands like
> > create or lttng list <session-name>, but the tracepoint events from the
> > application don't get registered and there is no trace output.
> > I enabled LTTNG_UST_DEBUG an ran lttng-sessiond in verbose mode (-vvv and
> > --verbose-consumer) and got the following error message:
> > "*Unix socket credential pid=0. Refusing application in distinct,
> > non-nested pid namespace.*"
> > It appears that for some calls to the session daemon there is a
> > syscall made with *SO_PEERCRED* which returns 0 for pid and the call is
> > failed with *LTTNG_UST_ERR_PEERCRED_PID* error (see get_cred call in
> > ustctl.c).
> > If I comment out the getsockopt call, my application tracing starts to
> > From what I found, docker cannot support getsockopt/SO_PEERCRED call to
> > peer pid on the unix socket which would make sense as it's in a separate
> > namespace.
> > I have a few questions on this:
> > 1. What is the reason for the get_cred/getsockopt call with SO_PEERCRED?
> > would like to understand why it's required for some and not other calls.
> More information is found in the introducing commit:
> commit a834901f2890deadb815d7f9e3ab79c3ba673994
> Author: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
> Date: Mon Oct 12 16:52:03 2020 -0400
> Fix: Use unix socket peercred for pid, uid, gid credentials
> Currently, the session daemon trust the pid, ppid, uid, and gid values
> passed by the application, but should really validate the uid using
> socket peercred. This fix uses the peercred values rather than the
> values provided by the application on registration for:
> - pid, uid and gid on Linux,
> - uid and gid on FreeBSD.
> This should improve how the session daemon deals with containerized
> applications on Linux as well. Applications are required to be either
> the same pid namespace, or in a pid namespace nested within the pid
> namespace of the lttng-sessiond, so the session daemon can map the
> application pid to something meaningful within its own pid namespace.
> Applications in a unrelated (disjoint) pid namespace will be refused by
> the session daemon.
> About the uid and gid with user namespaces on Linux, those will provide
> meaningful IDs if the application user namespace is either the same as
> the user namespace of the session daemon, or a nested user namespace.
> Otherwise, the IDs will be that of /proc/sys/kernel/overflowuid and
> /proc/sys/kernel/overflowgid, which typically maps to nobody.nogroup on
> current distributions.
> Given that fetching the parent pid (ppid) of the application would
> require to use /proc/<pid>/status (which is racy wrt pid reuse), expose
> the ppid provided by the application on registration instead, but only
> in situations where the application sits in the same pid namespace as
> the session daemon (on Linux), which is detected by checking if the pid
> provided by the application matches the pid obtained using unix socket
> credentials. The ppid is only used for logging/debugging purposes in
> session daemon anyway, so it is OK to use the value provided by the
> application for that purpose.
> Fixes: #1286
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
> Change-Id: I94742e57dad642106908d09e2c7e395993c2c48f
> As for "why it's required for some and not other calls.", there is a
> between communicating with a lttng-sessiond daemon (using the lttng CLI)
> userspace application registering. They are essentially two distinct
> communication interface. Now, to be honest, I'm not certain of the complete
> "security" policy for the lttng-sessiond <-> CLI interface and if we
> should be
> more strict or not.
> > 2. Is there any workaround for this problem, so that I can get this to
> > with the container topology I am working with (app in one container and
> > lttng daemons in another).
> Based on the commit message, lttng-ust explicitly cannot be used across
> non-nested pid namespace.
> Could you give us more information on the goal for the topology you plan
> to use?
> This could lead to further discussion and/or alternative solution based on
> goal and constraints of your deployment.
> > 3. Related to 2, are there any gotchas to bypassing the getsockopt call
> > get_cred?
> Based on the content of the mentioned bug (1286) , the principal
> concern is:
> This means a non-root application could theoretically impersonate a root
> application from a tracing perspective, and thus access root tracing
> buffers in
> a per-uid configuration, which is unwanted.
>  https://bugs.lttng.org/issues/1286
> Jonathan Rajotte-Julien
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the lttng-dev