[lttng-dev] I'm still getting empty ust traces using tracef

Fri May 12 10:57:05 EDT 2023

On 2023-05-12 10:52, Brian Hutchinson wrote:
> Hi Mathieu,
> 
> On Fri, May 12, 2023 at 9:33 AM Mathieu Desnoyers
> <mathieu.desnoyers at efficios.com> wrote:
>>
>> On 2023-05-12 00:10, Brian Hutchinson wrote:
>>> Hmm, I missed this earlier somehow.
>>>
>>> So, I'm not the greatest at updating OE and Yocto recipes.  I'm
>>> currently using this recipe:
>>> http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-kernel/lttng/lttng-ust_2.13.5.bb?h=master
>>>
>>> ... and it looks like the commit you are talking about is newer.
>>>
>>> I always think, oh, I'll just update the source URI in the recipe but
>>> it's never that simple ... and there are patches in the recipe etc.
>>>
>>> I've got a sdk (external toolchain) built for my embedded platform.
>>> Would it be too hard to just download stable-2.13 of everything and
>>> cross compile it outside of Yocto?
>>>
>>> What do you suggest?
>>>
>>> And do I need to do anything besides just get 2.13 stable working?  I
>>> was kind of confused if I need to put a #define
>>> LTTNG_TRACEPOINT_DEFINE somewhere in my code.  I'm not using a
>>> tracepoint provider packages at this point
>>
>> Hi Brian,
>>
>> You might want to provide a trimmed-down reproducer of your issue:
>> example .c compile unit instrumented with tracepoints, example .c
>> compile unit containing the tracepoint probes, and the log of the
>> console when this application is run with LTTNG_UST_DEBUG=1.
> 
> The code has two different areas where I'm trying to use tracef.  The
> way the app is put together, each of these areas end up becoming
> static libs that all get lumped together to make the final executable
> (which is then linked with -llttng-ust and -ldl).
> 
> If I'm reading between the lines correctly with respect to the commit
> you pointed out (that I'm missing), if I reduce the inclusion of I
> #include <lttng/tracef.h> to one instance (like with the hello world
> that worked), I'm thinking the version I have might work.
> 
> I don't know how I could trim down the large multi threaded app I'm
> trying to debug to share.
> 
> Another dynamic I should mention in full disclosure, the app in
> question has been ported from a different OS and was on a single core
> cpu.  The new host ( imx8) is a quad core A53 and since the app wasn't
> written for multicore, the cpu's are isolated and systemd is starting
> the app on cpu 0 but once it's up it switches it's affinity to cpu 1
> so I don't know if that's a factor here or not so just mentioning it.
> 
> I did try with LTTNG_UST_DEBUG=1 last night and it didn't put out much:
> 
> export LTTNG_UST_DEBUG=1
> # systemctl start my_app

I suspect that because you run your application under systemctl, we are 
not seeing the console output from the application.

The console output below appears to come from liblttng-ust-ctl.so linked 
within lttng-sessiond/consumerd, not the application.

Can you find a way to run your application and capture the console output ?

Thanks,

Mathieu

> 
> #lttng create my_tc_trace --output=/tmp/my_tc_trace
> Spawning a session daemon
> libringbuffer-clients[711/711]
> : LTT : ltt ring buffer client
> "relay-metadata-mmap" init
> (in lttng_ring_buffer_metadata_client_init() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:364)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-overwrite-mmap" init
> (in lttng_ring_buffer_client_overwrite_init() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-overwrite-rt-mmap" init
> (in lttng_ring_buffer_client_overwrite_rt_init() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-discard-mmap" init
> (in lttng_ring_buffer_client_discard_init() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-discard-rt-mmap" init
> (in lttng_ring_buffer_client_discard_rt_init() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
> [  179.384456] LTTng: Loaded modules v2.13.9 (Nordicit�é)
> [  179.390366] LTTng: Experimental bitwise enum enabled.
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-discard-rt-mmap" exit
> (in lttng_ring_buffer_client_discard_rt_exit() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-discard-mmap" exit
> (in lttng_ring_buffer_client_discard_exit() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-overwrite-rt-mmap" exit
> (in lttng_ring_buffer_client_overwrite_rt_exit() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-overwrite-mmap" exit
> (in lttng_ring_buffer_client_overwrite_exit() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-metadata-mmap" exit
> (in lttng_ring_buffer_metadata_client_exit() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:371)
> Session my_tc_trace created.
> Traces will be output to /tmp/my_tc_trace
> 
> # lttng enable-event --userspace 'lttng_ust_tracef:*'
> ust event lttng_ust_tracef:* created in channel channel0
> 
> # lttng start
> Tracing started for session my_tc_trace
> 
> # lttng destroy
> Destroying session my_tc_trace..
> Session my_tc_trace destroyed
> 
> root at localhost:/tmp/my_tc_trace/ust/uid/0# cd 64-bit/
> root at localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# ls
> channel0_0  channel0_1  channel0_2  channel0_3  index  metadata
> root at localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# ls -al
> total 20
> drwxrwx--- 3 root root  160 Sep 20 10:48 .
> drwxrwx--- 3 root root   60 Sep 20 10:48 ..
> -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_0
> -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_1
> -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_2
> -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_3
> drwxrwx--- 2 root root  120 Sep 20 10:48 index
> -rw-rw---- 1 root root 4096 Sep 20 10:48 metadata
> root at localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# cd index/
> root at localhost:/tmp/my_tc_trace/ust/uid/0/64-bit/index# ls
> channel0_0.idx  channel0_1.idx  channel0_2.idx  channel0_3.idx
> root at localhost:/tmp/my_tc_trace/ust/uid/0/64-bit/index# ls -al
> total 16
> drwxrwx--- 2 root root 120 Sep 20 10:48 .
> drwxrwx--- 3 root root 160 Sep 20 10:48 ..
> -rw-rw---- 1 root root  88 Sep 20 10:48 channel0_0.idx
> -rw-rw---- 1 root root  88 Sep 20 10:48 channel0_1.idx
> -rw-rw---- 1 root root  88 Sep 20 10:48 channel0_2.idx
> -rw-rw---- 1 root root  88 Sep 20 10:48 channel0_3.idx
> 
> No events in the logs, if it were working there would be tons of stuff in there.
> 
> But it sounds like my main problem is I need stable-2.13 and I need to
> figure out the best way to do that.  Compile from lttng source outside
> of yocto (probably not a good idea with all the dependencies that
> would also need to be cross compiled) with my generated sdk for the
> imx8 or try to update the OE recipe that I'm currently using.
> 
> Regards,
> 
> Brian
> 
> 
> 
> 
> 
>>
>> Thanks,
>>
>> Mathieu
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> https://www.efficios.com
>>

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com