[lttng-dev] [RFC] Dynamic instrumentation support in UST

[Thibault, Daniel]

----------------------------------------------------------------------
> Date: Fri, 26 Jul 2013 23:40:33 +0800
> From: Zifei Tong <soariez at gmail.com>
>
> Command Line Interface
> ----------------------
>
> To enable a dynamic probe in a running process at given address, you can use:
>     lttng enable-event NAME -u --pid PID
>         --probe (addr | symbol | symbol+offset)
>                                Dynamic UST probe.
>                                Addr and offset can be octal (0NNN...),
>                                decimal (NNN...) or hexadecimal (0xNNN...)
>
> This will place a bare probe at certain address.

   Unless I'm mistaken, currently lttng's 'enable-event -u ...' works only on already-running user-space processes.  It should be fairly easy to change this so the event gets enabled for any new user-space process that registers itself with the session daemon while the session is in existence.  This would be possible only for instrumented applications, of course.  It should also be possible to extend this to uninstrumented applications, as long as the session daemon watches for new processes being launched.

   I would allow the above command line to specify a process name instead of a process ID, in order to capture multi-threaded and multi-instance processes.  When using process IDs, one also has to be careful of PID namespaces: for a member of the 'tracing' group, a PID issued from a local lttng command (e.g. '$ lttng enable-event --pid ...') may need to be translated by the time it reaches the root lttng-sessiond.  Similar concerns exist with user namespaces.  The implications of namespaces are not peculiar to this extension of lttng, however: I would expect lttng to provide namespace-handling utility functions or to otherwise handle namespaces consistently throughout its services.

> Being only able to attach and instrument existing processes is sometimes restricted. If we can have #15 [2] implemented, then we can extend the command to support dynamic instrumentation.
>
>    lttng trace -c COMMAND
>        --probe (addr | symbol | symbol+offset)
>        --function (addr | symbol | symbol+offset)
>
> This will execute given program and place probes at certain places.

   The -c command option by itself may be insufficient in some cases.  For instance, if the lttng command is issued from an su shell (e.g. '# lttng trace -c ...'), from a 'tracing' group member (e.g. '$ lttng trace -c ...'), or from some ordinary account (e.g. '$ sudo -H lttng trace -c ...'), the intent may be to specify which user account to run the COMMAND as.  An 'lttng trace' command would be problematic anyway since it may need to specify several events to enable, each assigned to a different channel and possibly with a filter as well.  The overall command line could become very unwieldy.  If enable-event is able to set up latent events (as described in my previous comment block), then 'lttng trace' is unnecessary: the user need only set up his events, channels, etc., using --processname and then issue a proper 'sudo' command to launch the process (sudo has all the requisite options to assign the process to the proper user account with the proper environment settings, etc.).
------------------------------
Date: Fri, 26 Jul 2013 13:40:03 -0400
From: Simon Marchi <simon.marchi at polymtl.ca>

> What is the difference if I do
>  --function foo
>vs
>  --probe foo
> ?

   If it follows the lttng kernel implementation, --function sets up a kretprobe and --probe sets up a kprobe, so in user-space this should work similarly: --probe sets up a function entry event while --function sets up a function entry AND a function return event.

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>