[ltt-dev] [RFC UST] Processes model
Nils Carlson
nils.carlson at ericsson.com
Tue Jan 18 05:24:35 EST 2011
My comments as well. :-)
David Goulet wrote:
> Hi everyone,
>
> I'm submitting a RFC about a "new daemon model" for UST.
>
> This is the *first* draft... so a lot of feedback will be appreciated
> for improvements and ideas!
>
> English might be bad sometimes, please feel free to submit correction
> about that!
>
> Thanks to all
>
> RFC - New processes model for UST
>
> Author: David Goulet <david.goulet at polymtl.ca>
>
> Acknowledgement for helping creating that model:
> - Mathieu Desnoyers
> - Yannick Brosseau
>
> Version:
> - v0.1: 17/01/2011
> Initial proposal
>
> Terminology
> -----------------
>
> ustd - Main daemon that act as a trace session registry
I would name this ust-sessiond, mostly because ustd has already been used
and it might be good to not reuse the name.
>
> ust-consumerd - Daemon that reads UST buffers for a speficic application
>
> tracing session - A trace linked to set of specific tracepoints
>
> tracing buffers - Buffers containing tracing data
>
> tracing data - Data created by tracing an application
>
> inprocess library - UST library linked with the application
>
> Model
> -----------------
>
> This RFC propose a complete new UST daemon model. This re-engineering was
> mostly driven by the needs of better security in terms of access rights,
> tracing session and networking such as streaming and remote control over
> different traces.
>
> The new model follows the basic principles of having a trace registry
> (ustd),
> consumers for each tracing buffers (ust-consumerd) and having these
> buffers
> shared between the application and one consumer (shared memory).
>
> From now on, each component mention above is presented with their
> associated
> roles and existence meaning.
>
> USTD:
>
> The ustd daemon act as a trace registry i.e by keeping reference to
> all active
> session and, by active, it means a session in any state other then
> destroyed.
> Each entity we are keeping track of, here traces, will have a unique
> identifier
> (ID) assign to it.
>
> An ID SHOULD be a unique hash of the session name, trace path name,
> date/time, PID and/or UID
>
What is the ID used for?
> The trace roles of ustd:
>
> Trace interaction - Create, Destroy, Pause, Stop, Start, Set options
>
> Registry - keep track of all tracing session information (basics):
> * shared memory location
> * UID and GID
> * application PID
> * trace ID (unique identifier)
> * session name
>
This is a bit of a tricky one... I've been thinking a bit about this too.
If we look at a traceable application we could say that we have a hierachy
like so:
+--------+
| app_1 |
+---+----+
|
+-------------+-------------+
+---+------+ +---+------+
| trace_1 | | trace_2 |
+---+------+ +---+------+
| |
+-------------+ +....
| | |
+---+------+ +---+------+ +---+------+
| buffer_1 | | buffer_2 | | buffer_2 |
+---+------+ +---+------+ +---+------+....
(Hope this picture is legible...)
Now, the the consumer obviosuly has to be aware of the lowest levels of
the hierarchy
but does the session daemon need to be aware? I'm not sure...
> Basic access control - allow or deny trace interaction based on
> the user
> UID/GID.
>
> Consumer creation - spawns ust-consumerd.
>
Don't agree with this... At boot ust-consumerd should start.
> Buffers creation - creates shared memory for the tracing buffers.
>
> In order for ustd to be stateless, it should save all the registry
> information
> to disk. If ustd is killed or crashes, it will be able to restart and
> get the
> tracing state back without loosing track of all sessions.
>
> UST-CONSUMERD:
>
> This daemon basically consume the tracing buffers and write that data
> to disk
> for future analysis using LTTv or/and TMF (Tracing Monitoring
> Frameworks). Upon
> creation, that daemon UID is set to the application user.
hmhm.... I will start bottom up now as I think people will be reading
linearly.
As I seet it we will always start two daemons at boot, ust-sessiond and
ust-consumerd
ust-consumerd is the default disk consumer daemon, ust-sessiond
coordinates traces.
Sequence of events
Figure 1 ust-sessiond started:
+--------------+
| ust-sessiond |
+---+----------+
Figure 2 ust-consumerd started, ust-consumerd connect to ust-sessiond
and registers
itself as a consumer of type "disk":
+--------------+
| ust-sessiond |
+---+----------+
^
|
register consumer
|
+---+-----------+
| ust-consumerd |
+---+-----------+
Figure 3 application starts, connects to ust-sessiond and registers itself.
+--------------+ +--------+
| ust-sessiond |<--register--| app_1 |
+---+----------+ +---+----+
|
|
|
|
+---+-----------+
| ust-consumerd |
+---+-----------+
Figure 4 ustctl starts a trace, telling the sessiond what
the name of the trace should be, which application pid and what consumer it
wants.
+--------+
| ustctl |
+---+----+
|
Start trace
|
v
+--------------+ +--------+
| ust-sessiond |-------------| app_1 |
+---+----------+ +---+----+
|
|
|
|
+---+-----------+
| ust-consumerd |
+---+-----------+
Figure 5 trace still starting, ust-sessiond creates two socketpairs and
sends one fd
to the consumer and one to the app, connecting them bidirectionally (the
consumer can
send commands to the app across one fd and receive commands on the other
fd.)
+--------+
| ustctl |
+---+----+
|
Start trace
|
v
+--------------+ +--------+
| ust-sessiond |--socket fd->| app_1 |
+---+----------+ +---+----+
|
socket fd
|
v
+---+-----------+
| ust-consumerd |
+---+-----------+
Figure 6 trace still starting, app and consumerd are now bidirectionally
connected.
+--------+
| ustctl |
+---+----+
|
Start trace
|
v
+--------------+ +--------+
| ust-sessiond |-------------| app_1 |
+---+----------+ +---+----+
| ^
| |
| |
| |
+---+-----------+ |
| ust-consumerd +<---------------+
+---+-----------+
Figure 7 trace still starting, app allocates buffers in shared memory
and passes
each to the consumerd. The consumerd now maps each of them in turn and
checks which
cpu the belong to and passes them to the corresponding cpu thread where
they are added
to the epoll set.
+---------------+ +--------+
| ust-consumerd |<-map buffers-| app_1 |
+---------------+ +---+----+
I would draw internal diagrams for ust-consumerd but I'm getting tired
of drawing. :-/
What is different is the following:
1. We have only one instance of a libustconsumer normally. This instance
can handle multiple
traces. We thus scale well.
c = number of cpus
n = number of libustconsumer instances
total thread count = n * c + 1
For a typical 12 cpu system tracing 20 applications to disk we then have
c = 12
n = 1
total thread count = 13
In the current model we have a scalability of
c = number of cpus
ch = number of channels
i = number of libustconsumer instances (one per trace)
total thread count = c * ch * i + i
c = 12
ch = 2
i = 20 (one trace per app assumed)
total thread count = 500
2. There is no need for the sessiond to keep track of indivdual buffers.
3. If we want to implement access control we can do it by passing
credentials across the socket (see man unix). These can then be used
by both the consumerd to open files in the right places with the right
credentials. DBUS does this. We don't want to re-implement DBUS though.
4. ust-sessiond doesn't have to work as a router... Routing is painful
and I don't want to have to implement it. Connecting the consumerd with
the app is much simpler. I thought about this, hard. I even had a prototype
routing messages, and I promise, you don't want to do this.
This is the general idea, a lot of stuff is still loitering in my head.
> SHARED MEMORY:
>
> This is the memory area where the tracing buffers of the application
> will be
> held and given access in write mode for the inprocess library of the
> application and in readonly mode for the ust-consumerd.
>
> This memory is ONLY used for the tracing buffers. No communication
> between
> components is done by that memory.
>
> PREREQUISITES:
>
> The ustd daemon MUST always be running as "root" or an equivalent user
> having
> the same privilege as root.
>
> This daemon MUST be up and running at all time in order to trace a
> tracable
> application. (Future works will propose multiple registry coexistence).
>
> A "tracing" group SHOULD be created. Whoever is in that group is ABLE
> to access
> the tracing data of ANY buffers and is ABLE to operate a tracing
> session for
> ANY application.
>
> WARNING: This group name might interfer with other linux apps using
> the same
> group name. Carefull care should be put at install time for that.
>
> The new "lttngtrace" command line tool MUST be use to interact with
> the ustd
> registry daemon for every trace action needed by the user.
>
We currently have a library (libustcmd) for this? ustctl wraps
libustcmd. I have
a small plan of renaming the library libustctl because it sounds better.
The library has to stay as it will in not too distant a future be used
to implement
RPC calls to ust using tcf or some such.
Have to get to work now... If someone feels like reading up on
credential passing that
would be great.
/Nils
> The next section illustrates different use cases using that new model.
>
> Use Cases
> -----------------
>
> Every case considers these :
>
> * user A - UID: A; GID: A, tracing
> * user B - UID: B; GID: B, tracing
> * user C - UID: C; GID: C
>
> This first scenario shows how a certain user A will start a trace for the
> specific already running application app_1:
>
> Single user:
>
> +--------+ +------+ +-------+
> | user A |---- new ---->| ustd |<------>| app_1 |
> +--------+ +------+ +-------+
> | | | write
> fork create v
> +-----------------+ | | +--------+
> | ust-consumerd A |<----+ +--------->| shared |
> +-----------------+ | memory |
> ^ +--------+
> | |
> +-------------- read -------------+
>
>
> The user A ask for a tracing session (new) to ustd using the PID of
> app_1. At
> that point, the buffers (shared memory) that will contains the traced
> data is
> created by the ustd registry daemon and gives to app_1 the location of
> that
> memory with write access (write). Important to understand here that
> this memory
> segment will be set with UID of the application (app_1) and with the
> tracing
> group GID (tracing).
>
> Then, the ust-consumerd is spawned (fork) having these security steps
> taken:
>
> * setuid(<user A>) and seteuid()
> * setgid(<tracing>) and setegid()
> * attach to shared memory in readonly (shmat)
>
> ust-consumerd A - UID: user A (rw-), GID: tracing (r--)
>
> Note here that the GID is set to "tracing" because user A is in that
> group. So,
> for example, the user B, who is also in the group tracing, will be
> ABLE to read
> the tracing data of that application since ust-consumerd has the read
> access.
>
> Two users tracing the same application:
>
> +--------+ +------+ +-------+
> | user A |-------- new ------->| ustd |<------------>| app_1 |--+
> +--------+ ^ +------+ +-------+ |
> | | | |
> +--------+ | | | |
> | user B |------+ fork create | write
> +--------+ | | |
> | | |
> +-----------------+ | | +-------------+ |
> | ust-consumerd A |<---+ +-------->| shared mem. |<-+
> +-----------------+ | | +-------------+ |
> ^ | | | |
> +-------------O----O---- read ------+ |
> | | |
> +-----------------+ | | +-------------+ |
> | ust-consumerd B |<---+ +-------->| shared mem. |<-+
> +-----------------+ +-------------+
> ^ |
> +------------- read ----------------+
>
> This is essentially the same scheme as before but the previous figure
> shows the
> memory and daemon separation for two different users tracing the same
> application (app_1). This applies for two or more users. Again, as
> mention
> earlier, access rights for ust-consumerd:
>
> ust-consumerd A - UID: user A (rw-), GID: tracing (r--)
> ust-consumerd B - UID: user B (rw-), GID: tracing (r--)
>
> The next case shows a single user NOT in the tracing group and shows the
> process scheme associate to that particular case which is different then
> before since ustd will NOT register that session.
>
> Single user NOT in tracing group:
>
> +--------+ +------+
> | user C |---- new ---->| ustd |
> +--------+ +------+
> | +-------+
> spawn +------------------->| app_1 |
> | v +-------+
> | +-----------------+ |
> +-->| ust-consumerd C |-create+ | write
> +-----------------+ | |
> ^ v v
> | +--------+
> | | shared |
> +----- read -----| memory |
> +--------+
>
> Access rights will look like this:
>
> ust-consumerd C - UID: user C (rw-), GID: C (r--)
>
> For someone NOT in the tracing group, he CAN NOT use the global
> registry for
> creating a new session in order to prevent any user of being able to
> trace any
> application.
>
> So ustd will simply denied lttngtrace the ability to create a session
> and thus
> lttngtrace will have to spawn and inform ust-consumerd to create the
> shared
> memory and give it to the application.
More information about the lttng-dev
mailing list