[lttng-dev] Using lttng 2.11 and UST doesn't appear to work - getting empty trace files
Brian Hutchinson
b.hutchman at gmail.com
Tue Apr 25 08:30:14 EDT 2023
So looking at my lttng-sessiond log I found some concerning issues of
things being "not found":
DEBUG1 - 10:53:52.130237000 [9674/9735]: Enable event command for event
'lttng_ust_tracef:*' (in _cmd_enable_event() at
../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/cmd.c:2036)
DEBUG2 - 10:53:52.130399500 [9674/9735]: Trace UST channel channel0 not
found by name (in trace_ust_find_channel_by_name() at
../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:197)
DEBUG1 - 10:53:52.130450125 [9674/9735]: Enabling channel for session
auto-20210324-105310 (in cmd_enable_channel() at
../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/cmd.c:1510)
DEBUG2 - 10:53:52.130496375 [9674/9735]: Trace UST channel channel0 not
found by name (in trace_ust_find_channel_by_name() at
../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:197)
DEBUG3 - 10:53:52.130572000 [9674/9735]: Created hashtable size 4 at
0xffff90017170 of type 0 (in lttng_ht_new() at
../../../../lttng-tools-2.11.5/src/common/hashtable/hashtable.c:145)
DEBUG3 - 10:53:52.130619625 [9674/9735]: Created hashtable size 4 at
0xffff900175b0 of type 1 (in lttng_ht_new() at
../../../../lttng-tools-2.11.5/src/common/hashtable/hashtable.c:145)
DEBUG2 - 10:53:52.130666000 [9674/9735]: Trace UST channel channel0 created
(in trace_ust_create_channel() at
../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:396)
DEBUG2 - 10:53:52.130713625 [9674/9735]: Channel channel0 is being created
for UST with buffer 1 and id 0 (in channel_ust_create() at
../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/channel.c:461)
DEBUG2 - 10:53:52.130762250 [9674/9735]: Channel channel0 created
successfully (in channel_ust_create() at
../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/channel.c:500)
DEBUG2 - 10:53:52.130807500 [9674/9735]: Trace UST channel channel0 found
by name (in trace_ust_find_channel_by_name() at
../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:192)
DEBUG2 - 10:53:52.130851750 [9674/9735]: Trace UST event lttng_ust_tracef:*
NOT found (in trace_ust_find_event() at
../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:235)
DEBUG2 - 10:53:52.130896500 [9674/9735]: Trace UST event
lttng_ust_tracef:*, loglevel (0,-1) created (in trace_ust_create_event() at
../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:517)
DEBUG1 - 10:53:52.130968375 [9674/9735]: Sending response (size: 20,
retcode: Success (10)) (in thread_manage_clients() at
../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/client.c:2236)
DEBUG1 - 10:53:52.134962250 [9674/9735]: Clean command context structure
(in clean_command_ctx() at
../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/client.c:659)
Not quite sure what to do about that.
Regards,
Brian
On Tue, Apr 25, 2023 at 12:56 AM Brian Hutchinson <b.hutchman at gmail.com>
wrote:
> So instructions for tracef just say to link in lttng-ust, not dl, I didn't
> use liblttng-ust.a so maybe I misspoke.
>
> https://lttng.org/docs/v2.11/#doc-tracef
>
> ... so that's what I did.
>
> When I compile in a tracepoint with trace provider, I link in lttng-ust
> and dl. I'm assuming you can mix both tracef and tracepoint calls?
>
> I included a lttng-sessiond -vvv log in original post. Are attachments
> allowed or should I have posted the context as a plain text post? It's
> quite big.
>
> When I do lttng start, my app is already running via systemd at startup.
> Do I need to disable that and start the executable manually after enabling
> the lttng ust events?
>
> Like I said, kernel trace is working fine, it's UST that appears to not be
> working for some reason. I have less experience with UST but I've been
> researching and searching archives and I see others complaining about what
> looks to be my problem but either don't see a solution or maybe I just
> don't understand it.
>
> Thanks,
>
> Brian
>
> On Tue, Apr 25, 2023 at 12:13 AM Yitschak, Yehuda <yehuday at amazon.com>
> wrote:
>
>> It might be related to the static linking of lttng-ust
>> https://lists.lttng.org/pipermail/lttng-dev/2018-January/027705.html
>>
>> I also went down that path just to figure it’s not supported.
>>
>> Perhaps lttng documentation should be updated to make a clear note static
>> linking is not supported (or maybe I missed it)
>>
>> Yehuda
>>
>> Hi,
>>
>>
>>
>> I'm on an embedded platform, i.MX8 using Lttng 2.11 with 5.6.69 kernel.
>>
>>
>>
>> Tracing kernel events with lttng works fine but userspace (tracef and
>> trace provider type tracing) appears to have an issue I can't figure out.
>>
>>
>>
>> systemd is starting up my application ( a bunch of processes and threads
>> on multiple cpu's) on boot.
>>
>>
>>
>> I'm statically compiling and linking in lttng-ust. If I use just tracef
>> type tracing, when I do lttng list --userspace I get nothing. If I add in
>> a trace provider and a tracepoint, then I see my trace provider tracepoint
>> and:
>>
>>
>>
>> lttng_ust_tracelog:TRACE_DEBUG (loglevel: TRACE_DEBUG (14)) (type:
>> tracepoint)
>> lttng_ust_tracelog:TRACE_DEBUG_LINE (loglevel: TRACE_DEBUG_LINE
>> (13)) (type: tracepoint)
>> lttng_ust_tracelog:TRACE_DEBUG_FUNCTION (loglevel:
>> TRACE_DEBUG_FUNCTION (12)) (type: tracepoint)
>> lttng_ust_tracelog:TRACE_DEBUG_UNIT (loglevel: TRACE_DEBUG_UNIT
>> (11)) (type: tracepoint)
>> lttng_ust_tracelog:TRACE_DEBUG_MODULE (loglevel: TRACE_DEBUG_MODULE
>> (10)) (type: tracepoint)
>> lttng_ust_tracelog:TRACE_DEBUG_PROCESS (loglevel:
>> TRACE_DEBUG_PROCESS (9)) (type: tracepoint)
>> lttng_ust_tracelog:TRACE_DEBUG_PROGRAM (loglevel:
>> TRACE_DEBUG_PROGRAM (8)) (type: tracepoint)
>> lttng_ust_tracelog:TRACE_DEBUG_SYSTEM (loglevel: TRACE_DEBUG_SYSTEM
>> (7)) (type: tracepoint)
>> lttng_ust_tracelog:TRACE_INFO (loglevel: TRACE_INFO (6)) (type:
>> tracepoint)
>> lttng_ust_tracelog:TRACE_NOTICE (loglevel: TRACE_NOTICE (5)) (type:
>> tracepoint)
>> lttng_ust_tracelog:TRACE_WARNING (loglevel: TRACE_WARNING (4))
>> (type: tracepoint)
>> lttng_ust_tracelog:TRACE_ERR (loglevel: TRACE_ERR (3)) (type:
>> tracepoint)
>> lttng_ust_tracelog:TRACE_CRIT (loglevel: TRACE_CRIT (2)) (type:
>> tracepoint)
>> lttng_ust_tracelog:TRACE_ALERT (loglevel: TRACE_ALERT (1)) (type:
>> tracepoint)
>> lttng_ust_tracelog:TRACE_EMERG (loglevel: TRACE_EMERG (0)) (type:
>> tracepoint)
>> lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type:
>> tracepoint)
>> lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type:
>> tracepoint)
>> lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
>> tracepoint)
>> lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type:
>> tracepoint)
>> lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type:
>> tracepoint)
>> lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type:
>> tracepoint)
>> lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13))
>> (type: tracepoint)
>> lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13))
>> (type: tracepoint)
>> lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13))
>> (type: tracepoint)
>> lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13)) (type:
>> tracepoint)
>>
>>
>>
>> When I do lttng start, I get a directory structure in the trace directory
>> but the file sizes do not ever increase. It's just a directory structure
>> with files created and then that's it, no events being written to the
>> files. At first I thought things just weren't linked in right and not
>> working but then discovered how to run lttng-sessiond in -vvv mode and then
>> could see lots of things are going on, tracing appears to be trying to work
>> as there is tons of activity being output by lttng-sessiond, but like I
>> said the trace files aren't being added to.
>>
>>
>>
>> I looked at the lttng-sessiond log I captured and can't quite make out
>> the problem so attaching it.
>>
>>
>>
>> I'm probably not doing something right but can't figure out what that is
>> so any pointers appreciated.
>>
>>
>>
>> Regards,
>>
>>
>>
>> Brian
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20230425/828ef1fa/attachment.htm>
More information about the lttng-dev
mailing list