[lttng-dev] I'm still getting empty ust traces using tracef

Brian Hutchinson b.hutchman at gmail.com
Thu May 11 14:57:34 EDT 2023 

On Thu, May 11, 2023 at 12:36 PM Brian Hutchinson <b.hutchman at gmail.com> wrote:
>
> ... more background.  I've always used ltt in the kernel so I don't
> have much experience with the user side of it and especially
> multi-threaded, multi-core so I'm probably missing some fundamental
> concepts that I need to understand.
>
> Regards,
>
> Brian

I'm currently reading up on the new v2.13 "Recording event rule and
event record" features.  I came to 2.13 because I want to be able to
trigger snapshots on an event so I don't fill up storage on an
embedded device.

After I start lttng-systemd --daemonize and then start my app, I see
this when I do lttng list --userspace:

# lttng-sessiond --daemonize
[   85.180717] LTTng: Loaded modules v2.13.9 (Nordicit�é)
[   85.186103] LTTng: Experimental bitwise enum enabled.

# lttng list --userspace
UST events:
-------------

PID: 617 - Name: sh
     lttng_ust_libc:posix_memalign (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_libc:memalign (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_libc:realloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:calloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:free (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:malloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_unlock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_trylock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_acq (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_req (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlclose (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_dl:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlmopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG (loglevel:
TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE
(loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION
(loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT
(loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE
(loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS
(loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM
(loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM
(loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO (loglevel:
TRACE_INFO (6)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE
(loglevel: TRACE_NOTICE (5)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING
(loglevel: TRACE_WARNING (4)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR (loglevel:
TRACE_ERR (3)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT (loglevel:
TRACE_CRIT (2)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT (loglevel:
TRACE_ALERT (1)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG (loglevel:
TRACE_EMERG (0)) (type: tracepoint)
     lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_statedump:procname (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)

PID: 601 - Name: /usr/loal/bin/my_app
     lttng_ust_libc:posix_memalign (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_libc:memalign (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_libc:realloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:calloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:free (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:malloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_unlock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_trylock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_acq (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_req (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlclose (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_dl:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlmopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG (loglevel:
TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE
(loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION
(loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT
(loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE
(loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS
(loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM
(loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM
(loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO (loglevel:
TRACE_INFO (6)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE
(loglevel: TRACE_NOTICE (5)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING
(loglevel: TRACE_WARNING (4)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR (loglevel:
TRACE_ERR (3)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT (loglevel:
TRACE_CRIT (2)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT (loglevel:
TRACE_ALERT (1)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG (loglevel:
TRACE_EMERG (0)) (type: tracepoint)
     lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_statedump:procname (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)

PID: 626 - Name: xz
     lttng_ust_libc:posix_memalign (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_libc:memalign (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_libc:realloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:calloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:free (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:malloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_unlock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_trylock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_acq (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_req (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlclose (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_dl:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlmopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG (loglevel:
TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE
(loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION
(loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT
(loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE
(loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS
(loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM
(loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM
(loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO (loglevel:
TRACE_INFO (6)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE
(loglevel: TRACE_NOTICE (5)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING
(loglevel: TRACE_WARNING (4)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR (loglevel:
TRACE_ERR (3)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT (loglevel:
TRACE_CRIT (2)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT (loglevel:
TRACE_ALERT (1)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG (loglevel:
TRACE_EMERG (0)) (type: tracepoint)
     lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_statedump:procname (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)

I'm not sure why all this showed up 3 times.  Could be because I have
systemd unit file ExecStart calling a script so that my LD_PRELOAD=
line can work.  ExecStart= didn't like LD_PRELOAD.

So now I'm guessing ... I haven't read the fine documentation far
enough yet, that I have to somehow connect my record event rules to
PID 601 somehow and I'm getting empty trace records cause I'm not
smart enough (yet) to figure out how to do that.

Hopefully I'm getting "warmer".

Regards,

Brian

More information about the lttng-dev mailing list