[lttng-dev] I'm still getting empty ust traces using tracef

Fri May 12 10:52:27 EDT 2023

Hi Mathieu,

On Fri, May 12, 2023 at 9:33 AM Mathieu Desnoyers
<mathieu.desnoyers at efficios.com> wrote:
>
> On 2023-05-12 00:10, Brian Hutchinson wrote:
> > Hmm, I missed this earlier somehow.
> >
> > So, I'm not the greatest at updating OE and Yocto recipes.  I'm
> > currently using this recipe:
> > http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-kernel/lttng/lttng-ust_2.13.5.bb?h=master
> >
> > ... and it looks like the commit you are talking about is newer.
> >
> > I always think, oh, I'll just update the source URI in the recipe but
> > it's never that simple ... and there are patches in the recipe etc.
> >
> > I've got a sdk (external toolchain) built for my embedded platform.
> > Would it be too hard to just download stable-2.13 of everything and
> > cross compile it outside of Yocto?
> >
> > What do you suggest?
> >
> > And do I need to do anything besides just get 2.13 stable working?  I
> > was kind of confused if I need to put a #define
> > LTTNG_TRACEPOINT_DEFINE somewhere in my code.  I'm not using a
> > tracepoint provider packages at this point
>
> Hi Brian,
>
> You might want to provide a trimmed-down reproducer of your issue:
> example .c compile unit instrumented with tracepoints, example .c
> compile unit containing the tracepoint probes, and the log of the
> console when this application is run with LTTNG_UST_DEBUG=1.

The code has two different areas where I'm trying to use tracef.  The
way the app is put together, each of these areas end up becoming
static libs that all get lumped together to make the final executable
(which is then linked with -llttng-ust and -ldl).

If I'm reading between the lines correctly with respect to the commit
you pointed out (that I'm missing), if I reduce the inclusion of I
#include <lttng/tracef.h> to one instance (like with the hello world
that worked), I'm thinking the version I have might work.

I don't know how I could trim down the large multi threaded app I'm
trying to debug to share.

Another dynamic I should mention in full disclosure, the app in
question has been ported from a different OS and was on a single core
cpu.  The new host ( imx8) is a quad core A53 and since the app wasn't
written for multicore, the cpu's are isolated and systemd is starting
the app on cpu 0 but once it's up it switches it's affinity to cpu 1
so I don't know if that's a factor here or not so just mentioning it.

I did try with LTTNG_UST_DEBUG=1 last night and it didn't put out much:

export LTTNG_UST_DEBUG=1
# systemctl start my_app

#lttng create my_tc_trace --output=/tmp/my_tc_trace
Spawning a session daemon
libringbuffer-clients[711/711]
: LTT : ltt ring buffer client
"relay-metadata-mmap" init
(in lttng_ring_buffer_metadata_client_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:364)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-overwrite-mmap" init
(in lttng_ring_buffer_client_overwrite_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-overwrite-rt-mmap" init
(in lttng_ring_buffer_client_overwrite_rt_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-discard-mmap" init
(in lttng_ring_buffer_client_discard_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-discard-rt-mmap" init
(in lttng_ring_buffer_client_discard_rt_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
[  179.384456] LTTng: Loaded modules v2.13.9 (Nordicit�é)
[  179.390366] LTTng: Experimental bitwise enum enabled.
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-discard-rt-mmap" exit
(in lttng_ring_buffer_client_discard_rt_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-discard-mmap" exit
(in lttng_ring_buffer_client_discard_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-overwrite-rt-mmap" exit
(in lttng_ring_buffer_client_overwrite_rt_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-overwrite-mmap" exit
(in lttng_ring_buffer_client_overwrite_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-metadata-mmap" exit
(in lttng_ring_buffer_metadata_client_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:371)
Session my_tc_trace created.
Traces will be output to /tmp/my_tc_trace

# lttng enable-event --userspace 'lttng_ust_tracef:*'
ust event lttng_ust_tracef:* created in channel channel0

# lttng start
Tracing started for session my_tc_trace

# lttng destroy
Destroying session my_tc_trace..
Session my_tc_trace destroyed

root at localhost:/tmp/my_tc_trace/ust/uid/0# cd 64-bit/
root at localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# ls
channel0_0  channel0_1  channel0_2  channel0_3  index  metadata
root at localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# ls -al
total 20
drwxrwx--- 3 root root  160 Sep 20 10:48 .
drwxrwx--- 3 root root   60 Sep 20 10:48 ..
-rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_0
-rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_1
-rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_2
-rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_3
drwxrwx--- 2 root root  120 Sep 20 10:48 index
-rw-rw---- 1 root root 4096 Sep 20 10:48 metadata
root at localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# cd index/
root at localhost:/tmp/my_tc_trace/ust/uid/0/64-bit/index# ls
channel0_0.idx  channel0_1.idx  channel0_2.idx  channel0_3.idx
root at localhost:/tmp/my_tc_trace/ust/uid/0/64-bit/index# ls -al
total 16
drwxrwx--- 2 root root 120 Sep 20 10:48 .
drwxrwx--- 3 root root 160 Sep 20 10:48 ..
-rw-rw---- 1 root root  88 Sep 20 10:48 channel0_0.idx
-rw-rw---- 1 root root  88 Sep 20 10:48 channel0_1.idx
-rw-rw---- 1 root root  88 Sep 20 10:48 channel0_2.idx
-rw-rw---- 1 root root  88 Sep 20 10:48 channel0_3.idx

No events in the logs, if it were working there would be tons of stuff in there.

But it sounds like my main problem is I need stable-2.13 and I need to
figure out the best way to do that.  Compile from lttng source outside
of yocto (probably not a good idea with all the dependencies that
would also need to be cross compiled) with my generated sdk for the
imx8 or try to update the OE recipe that I'm currently using.

Regards,

Brian

>
> Thanks,
>
> Mathieu
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com
>