[lttng-dev] lttng, lttng-sessiond and the tracing group

[Thibault, Daniel]

   The lttng man pages are a little vague as to the relationship between lttng, the lttng-sessiond, and the tracing group.  My experiments have shown that a user belonging to the tracing group should more or less be able to issue lttng commands as if they were prefixed with sudo, but that's not quite right, as we'll see.  Is there a document outlining the operational concept?

   First observation

   A user in the tracing group who issues an lttng command will launch a user lttng-sessiond only if the root lttng-sessiond isn't already running.  That user can easily set up both lttng-sessiond daemons by first stopping the root lttng-sessiond, and then issuing a couple of 'lttng list' commands, the first at user level, the second at root level (using sudo).  If he then creates a user-level trace, it'll be listed by both 'lttng trace' and 'sudo lttng trace'.  A subsequent root-level trace ('sudo lttng create ...') will be listed only by the root daemon.  A problem that arises at this point is that both daemons use the same ~/.lttngrc file to track the current session, so since in this scenario the root trace was created last any defaulting at the user level will be erroneous.  Specifically:

(user belongs to tracing and sudo groups)
(~/.lttngrc does not exist)

$ sudo stop lttng-sessiond
lttng-sessiond stop/waiting
$ lttng -vvv list
Spawning a session daemon                                                                                  <-- user-level lttng-sessiond
DEBUG1: SIGUSR1 caught [in sighandler() at lttng.c:198]
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 0 [in list_sessions() at commands/list.c:588]
Currently no available tracing session
$ sudo lttng -vvv list
Spawning a session daemon                                                                                  <-- root-level lttng-sessiond
DEBUG1: SIGUSR1 caught [in sighandler() at lttng.c:198]
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 0 [in list_sessions() at commands/list.c:588]
Currently no available tracing session
$ lttng -vvv create userlevel
DEBUG3: URI string: file:///home/daniel/lttng-traces/userlevel-20130206-091913 [in uri_parse() at uri.c:253]
DEBUG3: URI file destination: /home/daniel/lttng-traces/userlevel-20130206-091913 [in uri_parse() at uri.c:290]
DEBUG3: URI dtype: 3, proto: 0, host: , subdir: , ctrl: 0, data: 0 [in uri_parse() at uri.c:467]
DEBUG1: LSM cmd type : 8 [in send_session_msg() at lttng-ctl.c:261]
Session userlevel created.
Traces will be written in /home/daniel/lttng-traces/userlevel-20130206-091913
DEBUG1: Init config session in /home/daniel [in config_init() at conf.c:294]
$ sudo lttng -vvv list
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 1 [in list_sessions() at commands/list.c:588]
Available tracing sessions:
  1) userlevel (/home/daniel/lttng-traces/userlevel-20130206-091913) [inactive]

Use lttng list <session_name> for more details
$ lttng -vvv list
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 1 [in list_sessions() at commands/list.c:588]
Available tracing sessions:
  1) userlevel (/home/daniel/lttng-traces/userlevel-20130206-091913) [inactive]

Use lttng list <session_name> for more details
$ sudo lttng -vvv create rootlevel
DEBUG3: URI string: file:///home/daniel/lttng-traces/rootlevel-20130206-091954 [in uri_parse() at uri.c:253]
DEBUG3: URI file destination: /home/daniel/lttng-traces/rootlevel-20130206-091954 [in uri_parse() at uri.c:290]
DEBUG3: URI dtype: 3, proto: 0, host: , subdir: , ctrl: 0, data: 0 [in uri_parse() at uri.c:467]
DEBUG1: LSM cmd type : 8 [in send_session_msg() at lttng-ctl.c:261]
Session rootlevel created.
Traces will be written in /home/daniel/lttng-traces/rootlevel-20130206-091954
DEBUG1: Init config session in /home/daniel [in config_init() at conf.c:294]
$ sudo lttng -vvv list                                                                                                              <-- at this point .lttngrc contains 'session=rootlevel'
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 2 [in list_sessions() at commands/list.c:588]
Available tracing sessions:
  1) rootlevel (/home/daniel/lttng-traces/rootlevel-20130206-091954) [inactive]
  2) userlevel (/home/daniel/lttng-traces/userlevel-20130206-091913) [inactive]

Use lttng list <session_name> for more details
$ lttng -vvv list
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 1 [in list_sessions() at commands/list.c:588]
Available tracing sessions:
  1) userlevel (/home/daniel/lttng-traces/userlevel-20130206-091913) [inactive]

Use lttng list <session_name> for more details
$ lttng -vvv list userlevel
DEBUG2: Session name: userlevel [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 1 [in list_sessions() at commands/list.c:588]
Tracing session userlevel: [inactive]
    Trace path: /home/daniel/lttng-traces/userlevel-20130206-091913

DEBUG1: LSM cmd type : 11 [in send_session_msg() at lttng-ctl.c:261]
$ lttng -vvv list rootlevel
DEBUG2: Session name: rootlevel [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 1 [in list_sessions() at commands/list.c:588]
Error: Session 'rootlevel' not found
Error: Command error
DEBUG1: Clean exit [in clean_exit() at lttng.c:169]
$ lttng -vvv start                                                                                                                 <-- you'd expect this to start the userlevel trace
DEBUG2: Config file path found: /home/daniel [in get_session_name() at utils.c:50]
DEBUG1: Session name found: rootlevel [in get_session_name() at utils.c:51]
DEBUG1: Starting tracing for session rootlevel [in start_tracing() at commands/start.c:81]
DEBUG1: LSM cmd type : 16 [in send_session_msg() at lttng-ctl.c:261]
Error: Permission denied
DEBUG1: Clean exit [in clean_exit() at lttng.c:169]

   At this point the user cannot use 'service lttng-sessiond stop' or 'stop lttng-sessiond' to stop the user-level daemon: his only recourse is the kill command.  This is normal, since service and start/stop both deal with System V init and/or Upstart.

   Of course the above is a little contrived because it's all from a single console, but you could easily have another user running the root-level lttng commands in parallel to an unknowing user.

   Second observation

   Contention for ~/.lttngrc can lead to odd behaviour.  This can only occur with users in the tracing group, I think (the root lttng-sessiond has no business writing to a non-tracing user's home directory).  Witness this:

(user belongs to tracing and sudo groups)
(~/.lttngrc does not exist)

$ sudo lttng -vvv create rootlevel
DEBUG3: URI string: file:///home/daniel/lttng-traces/rootlevel-20130205-162212 [in uri_parse() at uri.c:253]
DEBUG3: URI file destination: /home/daniel/lttng-traces/rootlevel-20130205-162212 [in uri_parse() at uri.c:290]
DEBUG3: URI dtype: 3, proto: 0, host: , subdir: , ctrl: 0, data: 0 [in uri_parse() at uri.c:467]
DEBUG1: LSM cmd type : 8 [in send_session_msg() at lttng-ctl.c:261]
Session rootlevel created.
Traces will be written in /home/daniel/lttng-traces/rootlevel-20130205-164435
DEBUG1: Init config session in /home/daniel [in config_init() at conf.c:294]
$ sudo lttng -vvv list                                                                                                              <-- At this point ~./.lttngrc is root-owned, contents 'session=rootlevel'
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 1 [in list_sessions() at commands/list.c:588]
Available tracing sessions:
  1) rootlevel (/home/daniel/lttng-traces/rootlevel-20130205-164435) [inactive]

Use lttng list <session_name> for more details
$ lttng -vvv list
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 0 [in list_sessions() at commands/list.c:588]
Currently no available tracing session
$ lttng -vvv create rootlevel
DEBUG3: URI string: file:///home/daniel/lttng-traces/rootlevel-20130205-164617 [in uri_parse() at uri.c:253]
DEBUG3: URI file destination: /home/daniel/lttng-traces/rootlevel-20130205-164617 [in uri_parse() at uri.c:290]
DEBUG3: URI dtype: 3, proto: 0, host: , subdir: , ctrl: 0, data: 0 [in uri_parse() at uri.c:467]
DEBUG1: LSM cmd type : 8 [in send_session_msg() at lttng-ctl.c:261]
Warning: Session rootlevel already exists
Error: Session name already exist
DEBUG1: Clean exit [in clean_exit() at lttng.c:169]                                                    <-- user will be puzzled that a session name can already exist if the list of sessions is empty
$ sudo lttng -vvv list
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 1 [in list_sessions() at commands/list.c:588]
Available tracing sessions:
  1) rootlevel (/home/daniel/lttng-traces/rootlevel-20130205-164435) [inactive]

Use lttng list <session_name> for more details
$ lttng -vvv list
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 0 [in list_sessions() at commands/list.c:588]
Currently no available tracing session
$ lttng -vvv create userlevel
DEBUG3: URI string: file:///home/daniel/lttng-traces/userlevel-20130205-164835 [in uri_parse() at uri.c:253]
DEBUG3: URI file destination: /home/daniel/lttng-traces/userlevel-20130205-164835 [in uri_parse() at uri.c:290]
DEBUG3: URI dtype: 3, proto: 0, host: , subdir: , ctrl: 0, data: 0 [in uri_parse() at uri.c:467]
DEBUG1: LSM cmd type : 8 [in send_session_msg() at lttng-ctl.c:261]
Session userlevel created.
Traces will be written in /home/daniel/lttng-traces/userlevel-20130205-164835
Error: Unable to create config file                                                                                               <-- user can't access a root-owned .lttngrc
Error: Command error
DEBUG1: Clean exit [in clean_exit() at lttng.c:169]
$ sudo lttng -vvv list
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 2 [in list_sessions() at commands/list.c:588]
Available tracing sessions:
  1) userlevel (/home/daniel/lttng-traces/userlevel-20130205-164835) [inactive]    <-- despite the error, the userlevel session is created
  2) rootlevel (/home/daniel/lttng-traces/rootlevel-20130205-164435) [inactive]

Use lttng list <session_name> for more details
$ lttng -vvv list
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 1 [in list_sessions() at commands/list.c:588]
Available tracing sessions:
  1) userlevel (/home/daniel/lttng-traces/userlevel-20130205-164835) [inactive]

Use lttng list <session_name> for more details                                                                        <-- .lttngrc is still root-owned, still contains 'session=rootlevel'
$ lttng -vvv destroy                                                                                                                               <-- user expects to destroy the session he just created
DEBUG2: Config file path found: /home/daniel [in get_session_name() at utils.c:50]
DEBUG1: Session name found: rootlevel [in get_session_name() at utils.c:51]
DEBUG1: LSM cmd type : 9 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Clean exit [in clean_exit() at lttng.c:169]                                                                  <-- note the lack of feedback, even though the destroy failed completely
$ sudo lttng -vvv list
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 2 [in list_sessions() at commands/list.c:588]
Available tracing sessions:
  1) userlevel (/home/daniel/lttng-traces/userlevel-20130205-164835) [inactive]
  2) rootlevel (/home/daniel/lttng-traces/rootlevel-20130205-164435) [inactive]

Use lttng list <session_name> for more details
$ lttng -vvv list
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 1 [in list_sessions() at commands/list.c:588]
Available tracing sessions:
  1) userlevel (/home/daniel/lttng-traces/userlevel-20130205-164835) [inactive]

Use lttng list <session_name> for more details
$ lttng -vvv destroy userlevel
DEBUG1: LSM cmd type : 9 [in send_session_msg() at lttng-ctl.c:261]
Session userlevel destroyed
DEBUG1: Removing /home/daniel/.lttngrc                                                                               <-- note how user destroyed a root-owned ~./.lttngrc
 [in config_destroy() at conf.c:151]
$ sudo lttng -vvv list
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 1 [in list_sessions() at commands/list.c:588]
Available tracing sessions:
  1) rootlevel (/home/daniel/lttng-traces/rootlevel-20130205-164435) [inactive]

Use lttng list <session_name> for more details
$ lttng -vvv list
DEBUG2: Session name: (null) [in cmd_list() at commands/list.c:720]
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: Session count 0 [in list_sessions() at commands/list.c:588]
Currently no available tracing session
$ lttng -vvv destroy -a
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
No session found, nothing to do.
$ sudo lttng -vvv destroy -a
DEBUG1: LSM cmd type : 13 [in send_session_msg() at lttng-ctl.c:261]
DEBUG1: LSM cmd type : 9 [in send_session_msg() at lttng-ctl.c:261]
Session rootlevel destroyed

   Third observation

   'lttng set-session' issues an error only if there is no name specified or if the name is too long (or if .lttngrc is root-owned, of course): it'll happily change the contents to whatever the argument is, including non-existent sessions or even illegal session names (containing embedded '/', for example).  A too-long session name (246 characters or more) also has the unfortunate side effect of wiping the contents of .lttngrc.  You can also end up with sessions which are "sort of created":

$ lttng -vvv create 123456781012345678201234567830123456784012345678501234567860123456787012345678801234567890123456710012345671101234567120123456713012345671401234567150123456716012345671701234567180123456719012345672001234567210123456722012345672301234567240123456
DEBUG3: URI string: file:///home/daniel/lttng-traces/123456781012345678201234567830123456784012345678501234567860123456787012345678801234567890123456710012345671101234567120123456713012345671401234567150123456716012345671701234567180123456719012345672001234567210123456722012345672301234567240123456-2013020 [in uri_parse() at uri.c:253]
DEBUG3: URI file destination: /home/daniel/lttng-traces/123456781012345678201234567830123456784012345678501234567860123456787012345678801234567890123456710012345671101234567120123456713012345671401234567150123456716012345671701234567180123456719012345672001234567210123456722012345672301234567240123456-2013020 [in uri_parse() at uri.c:290]
DEBUG3: URI dtype: 3, proto: 0, host: , subdir: , ctrl: 0, data: 0 [in uri_parse() at uri.c:467]
DEBUG1: LSM cmd type : 8 [in send_session_msg() at lttng-ctl.c:261]
Session 123456781012345678201234567830123456784012345678501234567860123456787012345678801234567890123456710012345671101234567120123456713012345671401234567150123456716012345671701234567180123456719012345672001234567210123456722012345672301234567240123456 created.
Traces will be written in /home/daniel/lttng-traces/123456781012345678201234567830123456784012345678501234567860123456787012345678801234567890123456710012345671101234567120123456713012345671401234567150123456716012345671701234567180123456719012345672001234567210123456722012345672301234567240123456-2013020
Error: Command error
DEBUG1: Clean exit [in clean_exit() at lttng.c:169]

   In this case I created a session with a 246-character name but got an error, and the output directory has an unexpectedly truncated name.  A session with a 245-character name will be created without an error but will still have an unexpectedly truncated output path.  Odds are this will cause trouble further down the line.  NAME_MAX is 255 on my system.

Daniel U. Thibault
R & D pour la défense Canada - Valcartier (RDDC Valcartier) / Defence R&D Canada - Valcartier (DRDC Valcartier)
Cyber sécurité pour les missions essentielles (CME) / Mission Critical Cyber Security (MCCS)
Protection des systèmes et contremesures (PSC) / Systems Protection & Countermeasures (SPC)
2459 route de la Bravoure
Québec, QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada / Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>