[ltt-dev] [RFC UST and LTTng] Daemon model proposal

[David Goulet]

Sorry for late reply...

On 11-01-20 05:08 AM, Nils Carlson wrote:
> David Goulet wrote:
>> Hi everyone,
>>
>> After a lot of feedbacks and talks! Here is the v0.2 with an almost
>> complete new design.
>>
>> This time, the proposal is for both LTTng and UST which get integrated
>> together.
>>
>> Again, questions and comments are more than welcome.
>>
>> Cheers!
>> David
>>
>
> Hi David,
>
> Some simple comments to begin with:
>
> 1. We can not spawn the consumer daemon. We have a libustconsumer for a
> reason,
> that reason being that we will have multiple different consumer daemons
> doing different things. We will have one writing to disk, one streaming
> and possibly
> some doing post-processing. Spawning won't work, the consumer
> functionality has to
> be kept in a library.
>

I'm confused here. What exactly do you mean by having consumer 
functionality in a library and not being able to spawn... ?

The next version of the document will explain the ust-consumerd clearly. 
I can give you a heads-up.

ust-consumerd is on a per tracing session basis. It will be a thread 
pool and using a thread to consume an application buffer for a specific 
session. So, I don't see how we cannot spawn the consumer daemon from 
the client (drtrace.. lttngtrace... whatever name)? If the session 
already exist, the client will simply attach to the currently running 
consumer daemon.

>
> 3. ust-sessiond and ltt-sessiond integrated, I'm not sure I see the
> point in this either.
> It's probably wiser to keep them apart lest we try to do to much in one
> program,
> also it places a burden on the maintainers to keep things synchronized
> and complicates
> testing. If you want to build a tool that uses both its better to use
> the external api
> from both lttng and ust.

Exactly. They will use the libustsession and liblttsession for the 
functionality they need. Again, why merging them? The goal is to have 
lttng tracing and ust tracing in the same session. Merging traces from 
kernel and userspace, easy control over all tracers, etc...

>
> 4. Using a common group for consumer, session, controller and app is
> spontaneously a good
> idea. Try to expand on it. Also, a lot of UST architecture is already in
> place... re-writing
> things is expensive.

I will explain more on security and why that group is so important.

I agree on that, UST has already a lot of good codes. Evidently, code 
will be reused. It's not a complete rewriting.. yet lol

Thanks for the comments!
David

>
> /Nils
>
>> RFC - New processes model for UST and LTTng
>>
>> Author: David Goulet <david.goulet at polymtl.ca>
>>
>> Contributors:
>> * Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
>> * Yannick Brosseau <yannick.brosseau at polymtl.ca>
>> * Nils Carlson <nils.carlson at ericsson.com>
>> * Michel Dagenais <michel.dagenais at polymtl.ca>
>> * Stefan Hajnoczi <stefanha at gmail.com>
>>
>> Version:
>> - v0.1: 17/01/2011
>> * Initial proposal
>>
>> - v0.2: 19/01/2011 After multiple reply from all the contributors
>> above, here is the list
>> of what has changed:
>> * Change/Add Terminology elements from the initial model
>> * New figures for four new scenarios
>> * Add inprocess library section
>> * LTTng kernel tracer support proposition
>> * More details for the Model and Components
>> * Improve the basic model. Quite different from the last one
>>
>> Terminology
>> -----------------
>>
>> ltt-sessiond - Main daemon for trace session registry for UST and LTTng
>>
>> ust-consumerd - Daemon that consume UST buffers for a specific
>> application
>>
>> ltt-consumerd - Daemon that consume LTTng buffers
>>
>> tracing session - A trace linked to a set of specific tracepoints
>>
>> tracing buffers - Buffers containing tracing data
>>
>> tracing data - Data created by tracing an application
>>
>> inprocess library - UST library linked with the application
>>
>> shared memory - system V shared memory
>>
>> Model
>> -----------------
>>
>> This RFC propose brand new UST and LTTng daemon model. This
>> re-engineering was
>> mostly driven by the need of better security in terms of access
>> rights, tracing
>> session, LTTng and UST integration and networking such as streaming
>> and remote
>> control over different traces and
>> The new model follows the basic principles of having a session registry
>> (ltt-sessiond), consumers for each tracing buffers (ust-consumerd and
>> ltt-consumerd).
>>
>> With this proposal, LTTng and UST will share the same tracing session, be
>> manage by the same tool and bring a complete integration between these
>> two
>> powerful tools.
>>
>> Each component's roles are explained in the following sections.
>>
>> LTT-SESSIOND:
>>
>> The ltt-sessiond daemon acts as a trace registry i.e. by keeping
>> reference to
>> all active session and, by active, it means a session in any state
>> other than
>> destroyed. Each entity we are keeping track of, here session, will have a
>> unique identifier or pidunique (ID) assign to it.
>>
>> This unique identifier SHOULD be a unique hash consisting of the PID,
>> session
>> name and the number of already created session plus one.
>>
>> Ex: PID - 4242, Session name - "joe_session", Num. sessions - fifth
>> one --> ID: 4242.joe_session-5
>>
>> The trace roles of ltt-sessiond:
>>
>> Trace interaction - Create, Destroy, Pause, Stop, Start, Set options
>>
>> Registry - keep track of tracing session information:
>> * shared memory location (only the keyid)
>> * application PID
>> * session name
>> * UID/GID of the session user
>>
>> Session access control - allow or deny session interaction based on
>> the user
>> credentials and/or session ID.
>>
>> Buffers creation - creates shared memory for the tracing buffers.
>>
>> UST-CONSUMERD:
>>
>> The purpose of this daemon is to consume the UST trace buffers for only a
>> specific session. The session MAY have several traces for example two
>> different
>> applications.
>>
>> This daemon basically empty the tracing buffers when asked for and
>> write that
>> data to disk for future analysis using LTTv or/and TMF (Tracing
>> Monitoring
>> Frameworks). Upon creation, that daemon UID/GID is set to the user
>> credentials
>> and so are the data files on disk.
>>
>> The roles of ust-consumerd:
>> Register to ltt-sessiond - Using a session ID and credentials (UID/GID)
>>
>> Consume buffers - Write data to a file descriptor (on disk, network, ...)
>>
>> Communicate with the application for buffers information
>>
>> LTT-CONSUMERD:
>>
>> The purpose of this daemon is to consume the LTTng trace buffers for
>> only a
>> specific session.
>> For that kernel consumer, ltt-sessiond will pass different anonymous file
>> descriptors to the ltt-consumerd using a Unix socket. From these file
>> desriptors, it will be able to get the data from a special function
>> export by
>> the LTTng kernel.
>>
>> The roles of ltt-consumerd:
>>
>> Register to ltt-sessiond - Using a session ID and credentials (UID/GID)
>>
>> Consume buffers - Write data to a file descriptor (on disk, network, ...)
>>
>> Communicate with the kernel for buffers information and with ltt-sessiond
>> for command.
>>
>> UST INPROCESS LIBRARY:
>>
>> When the application starts, this library will check for the global
>> name pipe
>> of ltt-sessiond. If it is present, it must validate that root:root is the
>> owner. From there, it will register as a valid traceable application
>> using the
>> apps credentials and will open a pipe in order to receive an eventual
>> shared
>> memory reference.
>>
>> If not present, the application opens a name pipe using the PID has
>> unique
>> identifier and wait on it for request from ltt-sessiond. If one bytes is
>> written to that pipe, the application will try again to register to
>> ltt-sessiond.
>>
>> (Future works, private ltt-sessiond if no global ltt-sessiond is found)
>>
>> SHARED MEMORY - UST part
>>
>> This is the memory area where the tracing buffers will be held and
>> given access
>> in write mode for the inprocess library of the application and in
>> readonly mode
>> for the ust-consumerd.
>>
>> On the LTTng side (for ltt-consumerd), these buffers are in the kernel
>> space
>> and given access by opening a file in the debugfs file system. With an
>> anonymous file desriptor, this consumer will be able to extract the data.
>>
>> This memory is ONLY used for the tracing data. No communication between
>> components is done using that memory.
>>
>> A shared memory segment for tracing MUST be set with the tracing group
>> GID for
>> the UST buffers.
>>
>> PREREQUISITES:
>>
>> The ltt-sessiond daemon MUST always be running as "root" or an
>> equivalent user
>> having the same privilege as root (UID = 0).
>>
>> The ltt-sessiond daemon MUST be up and running at all time in order to
>> trace a
>> tracable application. (Future works will propose multiple daemon
>> coexistence).
>>
>> The new drtrace command line tool MUST be use to interact with the
>> ltt-sessiond registry daemon for every trace action needed by the user.
>> A tracing group MUST be created. Whoever is in that group is able to
>> access the
>> tracing data of any buffers and is able to trace any application or
>> the kernel.
>>
>> WARNING: This group name MAY interfere with other linux apps using the
>> same
>> group name. Care should be put at install time for that (from source and
>> packages)
>>
>> Access Control
>> -----------------
>>
>> The next section illustrates different use cases using that new model.
>>
>> Use Cases
>> -----------------
>>
>> Each case considers these :
>>
>> * user A - UID: A; GID: A, tracing
>> * user B - UID: B; GID: B, tracing
>>
>> Scenario 1 - Single user tracing app_1
>>
>> This first scenario shows how user A will start a trace for
>> application app_1
>> that is not running.
>>
>> 1) drtrace ask ltt-sessiond for a new session through a Unix socket. If
>> allowed, ltt-sessiond returns a session ID to the client. (Ex: ops -->
>> new session)
>>
>> +-----------+ ops +--------------+
>> | drtrace A |<---------->| ltt-sessiond |
>> +-----------+ +--------------+
>>
>> 2) The app_1 is spawned by drtrace having the user A credentials.
>> Then, app_1
>> automatically register to ltt-sessiond has a "tracable apps" through
>> the global
>> name pipe of ltt-sessiond using the UID/GID and session ID.
>>
>> The shared memory is created with the app_1 UID (rw-) and tracing
>> group GID
>> (r--) and a reference is given back to app_1
>>
>> +-----------+ +--------------+
>> | drtrace A | | ltt-sessiond |
>> +-----------+ +--------------+
>> | ^ |
>> | +-------+ | | +-------------+
>> +-->| app_1 |<--------+ +-->| shared mem. |
>> +-------+ +-------------+
>>
>> 3) app_1 connect to the shared memory and ust-consumerd is spawned
>> with the
>> session ID and drtrace credentials (user A). It then register to
>> ltt-sessiond
>> for a valid session to consume using the previous session ID and
>> credentials.
>>
>> +-----------+ +--------------+
>> | drtrace A | +-->| ltt-sessiond |
>> +-----------+ | +--------------+
>> | | | +---------------+ read +-->| ust-consumerd |---------+
>> +---------------+ v ^ | +-------------+
>> | v +------>| shared mem. |
>> +-------+ | +-------------+
>> | app_1 |-------- +-------+ write
>>
>> Scenario 2 - Single user tracing already running app_1
>>
>> 1) drtrace ask ltt-sessiond for a new session through a Unix socket. If
>> allowed, ltt-sessiond returns a session ID to the client.
>>
>> +-----------+ ops +--------------+
>> | drtrace A |<---------->| ltt-sessiond |
>> +-----------+ +--------------+
>> ^ +-------+ read |
>> | app_1 |----------+
>> +-------+
>>
>> NOTE: At this stage, since app_1 is already running, the registration
>> of app_1
>> to ltt-sessiond has already been done. However, the shared memory
>> segment is
>> not allocated yet until a trace session is initiated. Having no shared
>> memory,
>> the inprocess library of app_1 will wait on a name pipe connected to
>> ltt-sessiond for the reference.
>>
>> +-----------+ +--------------+
>> | drtrace A | | ltt-sessiond |
>> +-----------+ +--------------+
>> ^ |
>> +-------+ | | +-------------+
>> | app_1 |<--------+ +-->| shared mem. |
>> +-------+ +-------------+
>> | ^
>> +---------- write ----------+
>>
>> 2) drtrace spawns a ust-consumerd for the session. We get the same
>> figure as
>> step 3 in the first scenario.
>>
>> There is a small difference though. The application MAY NOT be using
>> the same
>> credentials as user A (drtrace). However, the shared memory is always
>> GID of
>> the tracing group. So, in order for user A to trace app_1, is MUST be
>> in the
>> tracing group otherwise, if the application is not set with the user
>> credentials, user A will not be able to trace app_1
>>
>> Scenario 3 - Multiple users tracing the same running application
>>
>> 1) Session are created for the two users. Using the same exact
>> mechanism as
>> before, the shared memory and consumers are created. Two users, two
>> sessions,
>> two consumers and two shared memories for the same application.
>>
>> +-----------+ +--------------+ | drtrace A |-------- ops ------->|
>> ltt-sessiond | +-----------+ ^ +--------------+ | +-----------+ |
>> +-------+ | drtrace B |------+ +--->| app_1 |------- write -----+
>> +-----------+ | +-------+ | | | +-----------------+ | +-------------+ |
>> | ust-consumerd A |--O--- read ----->| shared mem. |<-+
>> +-----------------+ | +-------------+ | | | +-----------------+ v
>> +-------------+ |
>> | ust-consumerd B |--+--- read ----->| shared mem. |<-+
>> +-----------------+ +-------------+
>> NOTE: No link was made to ltt-sessiond, but all consumers are
>> connected to the
>> session daemon
>>
>> ust-consumerd A - UID: user A (rw-), GID: tracing (r--)
>> ust-consumerd B - UID: user B (rw-), GID: tracing (r--)
>>
>> Scenario 4 - User not in the tracing group
>>
>> For this particular case, it's all goes back to the first scenario.
>> The user
>> MUST start the application using his credentials. The session will be
>> created
>> by ltt-sessiond but he will not be able to trace anything that the
>> user does
>> not owned.
>

-- 
David Goulet
LTTng project, DORSAL Lab.

PGP/GPG : 1024D/16BD8563
BE3C 672B 9331 9796 291A  14C6 4AF7 C14B 16BD 8563