[ltt-dev] [RFC UST and LTTNG] Daemon model proposal v0.3

[Mathieu Desnoyers]

* David Goulet (david.goulet at polymtl.ca) wrote:
> (missing the RFC :P)
>
> Hi everyone,
>
> Introduce a complete new proposal for LTTng and UST tracing  
> infrastructure and control.
>
> This is the third revision (v0.3) of that proposal. Comments from Nils  
> Carlson and Michel Dagenais were added. Nothing final, we are still in  
> the development process.
>
> Again, questions and comments are more than welcome!
>
> Cheers
> David
>
> -- 
> David Goulet
> LTTng project, DORSAL Lab.
>
> PGP/GPG : 1024D/16BD8563
> BE3C 672B 9331 9796 291A  14C6 4AF7 C14B 16BD 8563

> RFC - New processes model for UST and LTTng
> 
> Author: David Goulet <david.goulet at polymtl.ca>
> 
> Contributors:
>     * Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
>     * Yannick Brosseau <yannick.brosseau at polymtl.ca>
>     * Nils Carlson <nils.carlson at ericsson.com>
>     * Michel Dagenais <michel.dagenais at polymtl.ca>
>     * Stefan Hajnoczi <stefanha at gmail.com>
> 
> Version:
>     - v0.1: 17/01/2011
>         * Initial proposal
> 
>     - v0.2: 19/01/2011 
>         After multiple reply from all the contributors above, here is the list
>         of what has changed:
>         * Change/Add Terminology elements from the initial model
>         * New figures for four new scenarios
>         * Add inprocess library section
>         * LTTng kernel tracer support proposition
>         * More details for the Model and Components
>         * Improve the basic model. Quite different from the last one
>     
>     - v0.3: 28/01/2011
>         In response from Michel Dagenais and Nils Carlson comments:
>         * Add scaling to reasons of this re-engineering
>         * Purpose of the session ID
>         * Explain why ltt-sessiond creates the tracing buffers
>         * ust-consumerd interaction schema
>         * Clarify inprocess library behavior
> 
> Terminology
> -----------------
> 
> ltt-sessiond - Main daemon for trace session registry for UST and LTTng
> 
> ust-consumerd - Daemon that consume UST buffers for a specific application
> 
> ltt-consumerd - Daemon that consume LTTng buffers
> 
> tracing session - A trace linked to a set of specific tracepoints

"A trace linked to a set of specific tracepoints and to a set of tracing
buffers"

> 
> tracing buffers - Buffers containing tracing data
> 
> tracing data - Data created by tracing an application
> 
> inprocess library - UST library linked with the application
> 
> shared memory - system V shared memory
> 
> Model
> -----------------
> 
> This RFC propose brand new UST and LTTng daemon model. This re-engineering was
> mostly driven by the need of better security in terms of access rights, having
> tracing session, scaling for the tracing infrastructure,

scaling in terms of what ? Number of threads/processes needed to perform
tracing ? (it could be number of CPUs too...)

> LTTng and UST
> integration and networking such as streaming and remote control over different
> traces.

You could also add that we want to make the use of both tool easier by combining
the traces gathered in a combined LTTng-UST session by saving them beneath the
same top-level directory.

> 
> The new model follows the basic principles of having a session registry
> (ltt-sessiond) and consumers for each tracing buffers (ust-consumerd and
> ltt-consumerd).
> 
> With this proposal, LTTng and UST will share the same tracing session, be
> manage by the same tool and bring a complete integration between these two

be managed

> powerful tools. Note that this proposal does NOT makes UST dependent on LTTng
> and vice versa.

We'll have to address this before we get to the final version of this document.
Possibly adding one extra section describing how LTTng and UST work when in
standalone mode. And also adding a section to describe how UST would work in a
system where no root access is available (not even for UST installation).

Normally, these extra concerns should fit well into the overall design: they
should just be a "specialization" of the overall design to a more specific
configuration.

> Each component's roles are explained in the following sections.
> 
> LTT-SESSIOND:
> 
> The ltt-sessiond daemon acts as a session registry i.e. by keeping reference to
> all active session and, by active, it means a session in any state other than
> destroyed. Each entity we are keeping track of, here session, will have a
> unique identifier (ID) assign to it. The purpose of this ID is to be able for

assigned

> drtrace to apply any kind of actions (Ex: Attach, Destroy). It MUST be human
> readable in order to facilitate the session identification when listed by
> drtrace.
> 
> This unique identifier SHOULD be a unique string consisting of the username, a
> session name and the number of already created session plus one. The last
> number makes sure that the ID is unique in the ltt-sessiond context. In the
> implementation, a simple number could be use for performance purpose instead of
> comparing string at each iteration.
> 
> Ex: Username - "dave", Session name - "mysession", Num. sessions - fifth one 
>     --> ID: dave.mysession-5

We will run into problems across reboots: session IDs might be reused, thus
trying to overwrite older traces. Other ideas, possibly including UUID, would be
welcome.

My recommandation would be to allow two schemes:

1 - allow the user to specify the trace session name.
    (fails if the trace session name is already taken, or if the target trace is
    already there)
2 - automatically assign trace session names with:
    username.UUID

Thoughts ?

> This daemon is also responsible for tracing buffer creation. Two main reasons
> motivate this design:
>     
>     * The shared memory segment needs to be in the tracing group. We can only
>     assume that ltt-sessiond, being root (UID = 0), is always able to do that.

As we discussed, I think this argument does not hold: any user can set the file
GID to "tracing", right ?

> 
>     * The ltt-sessiond needs to keep track of all the shared memory segments in
>     order to be able to give reference to any other consumer since it is this
>     daemon that controls the access to all tracing buffers.

I'm not convinced that ltt-sessiond needs to _create_ these buffers with this
argument, only that it needs some way to know what they are and to be able to
pass a reference to them to the consumers.

However, the following point might point us towards the requirement for
allocating the shared-mem buffers (shm) from the registry: if we ever want to
share the tracing buffers between all userspace applications (on a system where
security is less of a concern, but where scalability and performance is utterly
important), allocating them from the session registry will allow that very
easily, but if we allocate them from within the inprocess lib, then we will need
to redesign the whole thing.

> 
> A tracing session is associated to a user of the system (UID). Unless a user is
> in the tracing group, no other users can access the tracing data for that
> session.

This is unclear, and does not seem to match our discussion. Access in terms of
"write" and "read" accesses to the buffers and control pipes are unclealy
specified. I would rather say:

1) if a user is in the tracing group, its tracing sessions can consume
   information from all tracing buffers, from all applications and the kernel.
2) if a user is not in the tracing group, its tracing sessions can only consume
   data from its own applications' buffers.

Big note: I am really tempted to go for the approach of letting users which are
not part of the tracing group spawn their own sessiond. Actually, this would
also apply for users part of the tracing group when the system-wide sessiond is
not reachable. As soon as a user who needs to talk to sessiond discovers that
the sessiond is not reachable, it starts its own sessiond daemon.

Applications which need to talk to sessiond (to indicate their availability for
trace production) would have to register to both the system-wide sessiond and
the per-user sessiond. The idea is that a sessiond spawned by a user is only
allowed to see trace data coming from applications which effective UID is the
same user.

> 
> The trace roles of ltt-sessiond:
> 
>     Trace interaction - Create, Destroy, Pause, Stop, Start, Set options
> 
>     Registry - keep track of tracing session information:
>         * shared memory location (only the keyid)
>         * application PID
>         * session name

For this next one:

>         * UID/GID of the session user

Hrm, I would simplify it though: if a user is not part of the tracing group, it
does not have access to the system-wide sessiond, and needs to spawn its own. It
will simplify things in terms of credential management, so a bug is less likely
to creep in. We need a per-user sessiond for non-tracing-group users anyway.

> 
>     Session access control - allow or deny session interaction based on the user
>                              credentials and/or session ID.
> 
>     Buffers creation - creates shared memory for the tracing buffers.
> 
> UST-CONSUMERD:
> 
> The purpose of this daemon is to consume the UST trace buffers for only a
> specific session. The session MAY have several traces for example two different
> applications. The client tool, drtrace, is in charge of creating a
> ust-consumerd if NONE is available for that session. It is very important to
> understand that for a tracing session, there is only one ust-consumerd for all
> the traced applications.
> 
> This daemon basically empty the tracing buffers when asked for and write that

writes

> data to disk for future analysis using LTTv or/and TMF (Tracing Monitoring
> Frameworks). Upon creation, that daemon UID/GID is set to the user credentials
> and so are the data files on disk.
> 
> Here is a flow of action to illustrate the ust-consumerd life span:
> 
> 1) 
> +-----------+    ops     +--------------+
> | drtrace A |<---------->| ltt-sessiond |
> +-----------+            +--------------+
> 
> drtrace ask for tracing an application using the PID and the session ID. If it
> is allowed, meaning that drtrace needs to be either in the tracing group or the
> same UID as the application, the shared memory reference is given to drtrace
> and the ust-consumerd communication socket if ust-consumerd already exist.
> 
> 2a) If ust-consumerd EXIST
> 
> +-----------+        
> | drtrace A |        
> +-----------+        
>     | mem ref.               
>     |   +---------------+   read   +------------+    
>     +-->| ust-consumerd |--------->| shared mem |
>         +---------------+          +------------+
> 
> In that case, drtrace only ask ust-consumerd to consume the buffers using
> the reference it previously got from ltt-sessiond.

Please take into account a flight recorder session too, which is a slight
variation: drtrace can ask to create the trace session and start tracing in one
execution, and a different execution and hook on the trace session and request
to gather a buffer snapshot (which would spawn a consumerd that does exactly
that and possibly disconnects afterward).

> 
> 2b) If ust-consumerd DOES NOT EXIST
> 
> +-----------+              +--------------+
> | drtrace A |        +---->| ltt-sessiond |
> +-----------+        |     +--------------+
>     | ID             |
>     | mem ref.       | register
>     |   +---------------+
>     +-->| ust-consumerd |
>         +---------------+
> 
> drtrace spawns the ust-consumerd for the session using the session ID in order
> for the daemon to register as a consumer to ltt-sessiond for that session.
> 
> On the security side, the ust-consumerd gets UID/GID from the drtrace
> credentials since it was spawned by drtrace and so the files containing the
> tracing data will also be set to UID/GID of the drtrace client.
> 
> The roles of ust-consumerd:
>     
>     Register to ltt-sessiond - Using a session ID and credentials (UID/GID)
> 
>     Consume buffers - Write data to a file descriptor (on disk, network, ...)
> 
>     Communicate with the application for buffers options
> 
> LTT-CONSUMERD:
> 
> The purpose of this daemon is to consume the LTTng trace buffers for only a
> specific session. 
> 
> For that kernel consumer, ltt-sessiond will pass different anonymous file
> descriptors to the ltt-consumerd using a Unix socket. From these file
> desriptors, it will be able to get the data from a special function export by
> the LTTng kernel.
> 
> ltt-consumerd will be manage by the exact same way as ust-consumerd. However,

managed

> in order to trace the kernel, you are either root (UID=0) or in the tracing
> group.

Tracing applications should also be allowed from root user (not just tracing
group).

> 
> The roles of ltt-consumerd:
> 
>     Register to ltt-sessiond - Using a session ID and credentials (UID/GID)
> 
>     Consume buffers - Write data to a file descriptor (on disk, network, ...)
> 
>     Communicate with the kernel for buffers information and with ltt-sessiond
>     for command.
> 
> UST INPROCESS LIBRARY:
> 
> When the application starts, this library will check for the global name pipe
> of ltt-sessiond.  If it is present, it MUST validate that root:root is the
> owner. This check is very important to prevent ltt-sessiond spoofing. If the
> pipe is root, we have the assurance that it's the privilege user that operates

we have the assurance -> we are certain

priviledged

> tracing.

It should also check for its own user's ltt-sessiond named pipe (possibly in
/tmp/ltt-sessiond-username or /HOME/.ltt-sessiond ?).

Cleanup of the named pipe when the sessiond dies is a very important aspect to
test, because we have to keep in mind that if the pipe is left there when there
is no sessiond active, the applications might be stucked. I would personally try
not to rely on the presence of the named pipe alone to indicate that sessiond is
there: if the named pipe exists and has the correct user, we should try to talk
to the sessiond (and fail if it does not respond), rather that waiting
endlessly. I'm afraid this is the only solid way to deal with improper cleanup
of ltt-sessiond (which is something might always happen, and which should not
hang all the traced applications).

Same thing when we start a ltt-sessiond: if the named pipe already exists, we
should check if it is associated with an actual sessiond that responds to
commands.

> From there, it will register as a valid traceable application using the apps
> credentials and will open a pipe in order to receive an eventual shared memory
> reference.
> 
> If not present, the application opens a unix socket using the PID has unique

If ltt-sessiond is not present

as unique

> identifier and wait on it for request from ltt-sessiond. If one bytes is
> written to that pipe, the application will try again to register to
> ltt-sessiond. Again, the application try to connect to ltt-sessiond and not the
> contrary since we don't want any unauthorized user being able to talk to the
> library and control tracing.
> 
> (Future works, private ltt-sessiond if no global ltt-sessiond is found.)

I was getting to this :)  We should describe how this works when:

1) no sessiond is there at all
2) global sessiond is there, but not local
2) local is there, but not global

We should probably use the same app unix socket to wait for both global and
local daemons to talk to us, when either of the daemon is not running.

> 
> SHARED MEMORY
> 
> For UST, this is the memory area where the tracing buffers will be held and
> given access in write mode for the inprocess library of the application and in

given access in read-write mode for the inprocess lib.

> readonly mode for the ust-consumerd.
> 
> On the LTTng side (for ltt-consumerd), these buffers are in the kernel space
> and given access by opening a file in the debugfs file system. With an
> anonymous file desriptor, this consumer will be able to extract the data.
> 
> This memory is ONLY used for the tracing data. No communication between
> components is done using that memory.
> 
> A shared memory segment for tracing MUST be set with the tracing group GID for
> the UST buffers. This is the job of ltt-sessiond.
> 
> PREREQUISITES:
> 
> The ltt-sessiond daemon MUST always be running as "root" or an equivalent user
> having the same privilege as root (UID = 0).

We should update this to reflect the "local per-user sessiond" execution.

> 
> The ltt-sessiond daemon MUST be up and running at all time in order to trace a
> tracable application. (Future works will propose multiple daemon coexistence).
> 
> The new drtrace command line tool MUST be use to interact with the

be used

> ltt-sessiond registry daemon for every trace action needed by the user. 

Actually, we might want to allow various applications to do tracing, drtrace
being the main one. If we modularize the pieces of drtrace into various
libraries, it will be easier to re-use for others.

> 
> A tracing group MUST be created. Whoever is in that group is able to access the
> tracing data of any buffers and is able to trace any application or the kernel.
> 
> WARNING: This group name MAY interfere with other linux apps using the same
> group name. Care should be put at install time for that (from source and
> packages)
> 
> Access Control
> -----------------
> 
> The next section illustrates different use cases using that new model.
> 
> Use Cases
> -----------------
> 
> Each case considers these :
> 
> * user A - UID: A; GID: A, tracing
> * user B - UID: B; GID: B, tracing
> 
> Scenario 1 - Single user tracing app_1
> 
> This first scenario shows how user A will start a trace for application app_1
> that is not running.
> 
> 1) drtrace ask ltt-sessiond for a new session through a Unix socket. If
> allowed, ltt-sessiond returns a session ID to the client. 
> (Ex: ops --> new session)
> 
> +-----------+    ops     +--------------+
> | drtrace A |<---------->| ltt-sessiond |
> +-----------+            +--------------+
> 
> 2) The app_1 is spawned by drtrace having the user A credentials. Then, app_1
> automatically register to ltt-sessiond has a "tracable apps" through the global
> name pipe of ltt-sessiond using the UID/GID and session ID.
> 
> The shared memory is created with the app_1 UID (rw-) and tracing group GID
> (r--) and a reference is given back to app_1
> 
> +-----------+            +--------------+
> | drtrace A |            | ltt-sessiond |
> +-----------+            +--------------+
>     |                     ^    |
>     |   +-------+         |    |   +-------------+
>     +-->| app_1 |<--------+    +-->| shared mem. |
>         +-------+                  +-------------+
> 
> 3) app_1 connect to the shared memory and ust-consumerd is spawned with the
> session ID and drtrace credentials (user A).  It then register to ltt-sessiond
> for a valid session to consume using the previous session ID and credentials.
> 
> +-----------+            +--------------+
> | drtrace A |        +-->| ltt-sessiond |
> +-----------+        |   +--------------+
>     |                |                  
>     |   +---------------+    read       
>     +-->| ust-consumerd |---------+           
>         +---------------+         v 
>           ^  |                  +-------------+
>           |  v          +------>| shared mem. |
>         +-------+       |       +-------------+
>         | app_1 |--------            
>         +-------+     write        
> 
> 
> Scenario 2 - Single user tracing already running app_1
> 
> 1) drtrace ask ltt-sessiond for a new session through a Unix socket. If
> allowed, ltt-sessiond returns a session ID to the client.
> 
> +-----------+    ops     +--------------+
> | drtrace A |<---------->| ltt-sessiond |
> +-----------+            +--------------+
>                            ^ 
>         +-------+   read   |
>         | app_1 |----------+
>         +-------+
> 
> NOTE: At this stage, since app_1 is already running, the registration of app_1
> to ltt-sessiond has already been done. However, the shared memory segment is
> not allocated yet until a trace session is initiated. Having no shared memory,
> the inprocess library of app_1 will wait on a name pipe connected to
> ltt-sessiond for the reference.

I would argue that even if the app has a set of buffers being used by an active
tracing session, it should still wait on its named pipe for ltt-sessiond
connexions for two reasons:

- Connexion may come from the per-user sessiond (local ltt-sessiond)
- We might create another concurrent tracing session, which can ask for yet
  another set of tracing buffers to be created. It seems more convenient to use
  the same communication scheme independently of if there is 0 or 1 active
  tracing session.

> 
> +-----------+            +--------------+
> | drtrace A |            | ltt-sessiond |
> +-----------+            +--------------+
>                           ^    |
>         +-------+         |    |   +-------------+
>         | app_1 |<--------+    +-->| shared mem. |
>         +-------+                  +-------------+
>             |                           ^
>             +---------- write ----------+
> 
> 2) drtrace spawns a ust-consumerd for the session. We get the same figure as
> step 3 in the first scenario.
> 
> There is a small difference though. The application MAY NOT be using the same
> credentials as user A (drtrace).  However, the shared memory is always GID of
> the tracing group. So, in order for user A to trace app_1, is MUST be in the
> tracing group otherwise, if the application is not set with the user
> credentials, user A will not be able to trace app_1
> 
> Scenario 3 - Multiple users tracing the same running application
> 
> 1) Session are created for the two users. Using the same exact mechanism as
> before, the shared memory and consumers are created. Two users, two sessions,
> two consumers and two shared memories for the same application.
> 
> +-----------+                     +--------------+          
> | drtrace A |-------- ops ------->| ltt-sessiond |          
> +-----------+      ^              +--------------+          
>                    |                    
> +-----------+      |               +-------+               
> | drtrace B |------+          +--->| app_1 |------- write -----+
> +-----------+                 |    +-------+                   | 
>                               |                                | 
>          +-----------------+  |               +-------------+  |
>          | ust-consumerd A |--O--- read ----->| shared mem. |<-+
>          +-----------------+  |               +-------------+  |  
>                               |                                |    
>          +-----------------+  v               +-------------+  |
>          | ust-consumerd B |--+--- read ----->| shared mem. |<-+
>          +-----------------+                  +-------------+    
> 
> NOTE: No link was made to ltt-sessiond, but all consumers are connected to the
> session daemon
> 
> ust-consumerd A - UID: user A (rw-), GID: tracing (r--)
> ust-consumerd B - UID: user B (rw-), GID: tracing (r--)
> 
> Scenario 4 - User not in the tracing group
> 
> For this particular case, it's all goes back to the first scenario. The user
> MUST start the application using his credentials. The session will be created
> by ltt-sessiond but he will not be able to trace anything that the user does
> not owned.

I would move this last use-case into the per-user ltt-sessiond.

Thanks for putting this together!

Mathieu



-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com