<div dir="ltr">Thanks for the responses. The reasoning makes sense. We have decided to run lttng-sessiond on the host. Our trace generating application will run in a container and so will our libbabeltrace based trace consumer app (using live sessions).<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 6, 2021 at 7:07 AM Jonathan Rajotte-Julien <<a href="mailto:jonathan.rajotte-julien@efficios.com">jonathan.rajotte-julien@efficios.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
On Mon, Apr 05, 2021 at 11:09:39AM -0700, Eqbal via lttng-dev wrote:<br>
> Hi,<br>
> <br>
> I am trying to get user space tracing working for an application running in<br>
> a docker container. I am running lttng session daemon in another container.<br>
> I mounted the unix socket locations (either /var/run/lttng for root or<br>
> $HOME/.lttng for another user). By doing that I can run commands like lttng<br>
> create or lttng list <session-name>, but the tracepoint events from the<br>
> application don't get registered and there is no trace output.<br>
> <br>
> I enabled LTTNG_UST_DEBUG an ran lttng-sessiond in verbose mode (-vvv and<br>
> --verbose-consumer) and got the following error message:<br>
> <br>
> "*Unix socket credential pid=0. Refusing application in distinct,<br>
> non-nested pid namespace.*"<br>
> <br>
> It appears that for some calls to the session daemon there is a getsockopt<br>
> syscall made with *SO_PEERCRED* which returns 0 for pid and the call is<br>
> failed with *LTTNG_UST_ERR_PEERCRED_PID* error (see get_cred call in<br>
> ustctl.c).<br>
> <br>
> If I comment out the getsockopt call, my application tracing starts to work.<br>
> <br>
> From what I found, docker cannot support getsockopt/SO_PEERCRED call to get<br>
> peer pid on the unix socket which would make sense as it's in a separate<br>
> namespace.<br>
> <br>
> I have a few questions on this:<br>
> 1. What is the reason for the get_cred/getsockopt call with SO_PEERCRED? I<br>
> would like to understand why it's required for some and not other calls.<br>
<br>
<br>
More information is found in the introducing commit:<br>
<br>
commit a834901f2890deadb815d7f9e3ab79c3ba673994<br>
Author: Mathieu Desnoyers <<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>><br>
Date: Mon Oct 12 16:52:03 2020 -0400<br>
<br>
Fix: Use unix socket peercred for pid, uid, gid credentials<br>
<br>
Currently, the session daemon trust the pid, ppid, uid, and gid values<br>
passed by the application, but should really validate the uid using unix<br>
socket peercred. This fix uses the peercred values rather than the<br>
values provided by the application on registration for:<br>
<br>
- pid, uid and gid on Linux,<br>
- uid and gid on FreeBSD.<br>
<br>
This should improve how the session daemon deals with containerized<br>
applications on Linux as well. Applications are required to be either in<br>
the same pid namespace, or in a pid namespace nested within the pid<br>
namespace of the lttng-sessiond, so the session daemon can map the<br>
application pid to something meaningful within its own pid namespace.<br>
Applications in a unrelated (disjoint) pid namespace will be refused by<br>
the session daemon.<br>
<br>
About the uid and gid with user namespaces on Linux, those will provide<br>
meaningful IDs if the application user namespace is either the same as<br>
the user namespace of the session daemon, or a nested user namespace.<br>
Otherwise, the IDs will be that of /proc/sys/kernel/overflowuid and<br>
/proc/sys/kernel/overflowgid, which typically maps to nobody.nogroup on<br>
current distributions.<br>
<br>
Given that fetching the parent pid (ppid) of the application would<br>
require to use /proc/<pid>/status (which is racy wrt pid reuse), expose<br>
the ppid provided by the application on registration instead, but only<br>
in situations where the application sits in the same pid namespace as<br>
the session daemon (on Linux), which is detected by checking if the pid<br>
provided by the application matches the pid obtained using unix socket<br>
credentials. The ppid is only used for logging/debugging purposes in the<br>
session daemon anyway, so it is OK to use the value provided by the<br>
application for that purpose.<br>
<br>
Fixes: #1286<br>
Signed-off-by: Mathieu Desnoyers <<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>><br>
Change-Id: I94742e57dad642106908d09e2c7e395993c2c48f<br>
<br>
As for "why it's required for some and not other calls.", there is a difference<br>
between communicating with a lttng-sessiond daemon (using the lttng CLI) and<br>
userspace application registering. They are essentially two distinct<br>
communication interface. Now, to be honest, I'm not certain of the complete<br>
"security" policy for the lttng-sessiond <-> CLI interface and if we should be<br>
more strict or not.<br>
<br>
> 2. Is there any workaround for this problem, so that I can get this to work<br>
> with the container topology I am working with (app in one container and<br>
> lttng daemons in another).<br>
<br>
Based on the commit message, lttng-ust explicitly cannot be used across<br>
non-nested pid namespace.<br>
<br>
Could you give us more information on the goal for the topology you plan to use?<br>
This could lead to further discussion and/or alternative solution based on the<br>
goal and constraints of your deployment.<br>
<br>
> 3. Related to 2, are there any gotchas to bypassing the getsockopt call in<br>
> get_cred?<br>
<br>
Based on the content of the mentioned bug (1286) [1], the principal concern is:<br>
<br>
"<br>
This means a non-root application could theoretically impersonate a root<br>
application from a tracing perspective, and thus access root tracing buffers in<br>
a per-uid configuration, which is unwanted.<br>
"<br>
<br>
[1] <a href="https://bugs.lttng.org/issues/1286" rel="noreferrer" target="_blank">https://bugs.lttng.org/issues/1286</a><br>
<br>
Cheers<br>
<br>
-- <br>
Jonathan Rajotte-Julien<br>
EfficiOS<br>
</blockquote></div>