<div dir="ltr"><div>So looking at my lttng-sessiond log I found some concerning issues of things being "not found":</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">DEBUG1 - 10:53:52.130237000 [9674/9735]: Enable event command for event 'lttng_ust_tracef:*' (in _cmd_enable_event() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/cmd.c:2036) </span><br>DEBUG2 - 10:53:52.130399500 [9674/9735]: Trace UST channel channel0 not found by name (in trace_ust_find_channel_by_name() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:197) <br>DEBUG1 - 10:53:52.130450125 [9674/9735]: Enabling channel for session auto-20210324-105310 (in cmd_enable_channel() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/cmd.c:1510) <br>DEBUG2 - 10:53:52.130496375 [9674/9735]: Trace UST channel channel0 not found by name (in trace_ust_find_channel_by_name() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:197) <br>DEBUG3 - 10:53:52.130572000 [9674/9735]: Created hashtable size 4 at 0xffff90017170 of type 0 (in lttng_ht_new() at ../../../../lttng-tools-2.11.5/src/common/hashtable/hashtable.c:145) <br>DEBUG3 - 10:53:52.130619625 [9674/9735]: Created hashtable size 4 at 0xffff900175b0 of type 1 (in lttng_ht_new() at ../../../../lttng-tools-2.11.5/src/common/hashtable/hashtable.c:145) <br>DEBUG2 - 10:53:52.130666000 [9674/9735]: Trace UST channel channel0 created (in trace_ust_create_channel() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:396) <br>DEBUG2 - 10:53:52.130713625 [9674/9735]: Channel channel0 is being created for UST with buffer 1 and id 0 (in channel_ust_create() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/channel.c:461) <br>DEBUG2 - 10:53:52.130762250 [9674/9735]: Channel channel0 created successfully (in channel_ust_create() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/channel.c:500) <br>DEBUG2 - 10:53:52.130807500 [9674/9735]: Trace UST channel channel0 found by name (in trace_ust_find_channel_by_name() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:192) <br>DEBUG2 - 10:53:52.130851750 [9674/9735]: Trace UST event lttng_ust_tracef:* NOT found (in trace_ust_find_event() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:235) <br>DEBUG2 - 10:53:52.130896500 [9674/9735]: Trace UST event lttng_ust_tracef:*, loglevel (0,-1) created (in trace_ust_create_event() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:517) <br>DEBUG1 - 10:53:52.130968375 [9674/9735]: Sending response (size: 20, retcode: Success (10)) (in thread_manage_clients() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/client.c:2236) <br>DEBUG1 - 10:53:52.134962250 [9674/9735]: Clean command context structure (in clean_command_ctx() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/client.c:659)<br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">Not quite sure what to do about that.</span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">Regards,</span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">Brian<br></span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 25, 2023 at 12:56 AM Brian Hutchinson <<a href="mailto:b.hutchman@gmail.com">b.hutchman@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>So instructions for tracef just say to link in lttng-ust, not dl, I didn't use liblttng-ust.a so maybe I misspoke.</div><div><br></div><div><a href="https://lttng.org/docs/v2.11/#doc-tracef" target="_blank">https://lttng.org/docs/v2.11/#doc-tracef</a></div><div><br></div><div>... so that's what I did.</div><div><br></div><div>When I compile in a tracepoint with trace provider, I link in lttng-ust and dl. I'm assuming you can mix both tracef and tracepoint calls?</div><div><br></div><div>I included a lttng-sessiond -vvv log in original post. Are attachments allowed or should I have posted the context as a plain text post? It's quite big.</div><div><br></div><div>When I do lttng start, my app is already running via systemd at startup. Do I need to disable that and start the executable manually after enabling the lttng ust events?</div><div><br></div><div>Like I said, kernel trace is working fine, it's UST that appears to not be working for some reason. I have less experience with UST but I've been researching and searching archives and I see others complaining about what looks to be my problem but either don't see a solution or maybe I just don't understand it.<br></div><div><br></div><div>Thanks,</div><div><br></div><div>Brian<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 25, 2023 at 12:13 AM Yitschak, Yehuda <<a href="mailto:yehuday@amazon.com" target="_blank">yehuday@amazon.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div lang="EN-US">
<div>
<p class="MsoNormal">It might be related to the static linking of lttng-ust<br>
<a href="https://lists.lttng.org/pipermail/lttng-dev/2018-January/027705.html" target="_blank">https://lists.lttng.org/pipermail/lttng-dev/2018-January/027705.html</a><br>
<br>
<u></u><u></u></p>
<p class="MsoNormal">I also went down that path just to figure it’s not supported.<u></u><u></u></p>
<p class="MsoNormal">Perhaps lttng documentation should be updated to make a clear note static linking is not supported (or maybe I missed it)<br>
<br>
<u></u><u></u></p>
<p class="MsoNormal">Yehuda <br>
<br>
<u></u><u></u></p>
<div style="border-color:currentcolor currentcolor currentcolor blue;border-style:none none none solid;border-width:medium medium medium 1.5pt;padding:0cm 0cm 0cm 4pt">
<div>
<div>
<div>
<p class="MsoNormal">Hi,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'm on an embedded platform, i.MX8 using Lttng 2.11 with 5.6.69 kernel.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Tracing kernel events with lttng works fine but userspace (tracef and trace provider type tracing) appears to have an issue I can't figure out.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">systemd is starting up my application ( a bunch of processes and threads on multiple cpu's) on boot.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'm statically compiling and linking in lttng-ust. If I use just tracef type tracing, when I do lttng list --userspace I get nothing. If I add in a trace provider and a tracepoint, then I see my trace provider tracepoint and:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black;background:white"> lttng_ust_tracelog:TRACE_DEBUG (loglevel: TRACE_DEBUG (14)) (type: tracepoint)
</span><span style="font-family:"Courier New""><br>
lttng_ust_tracelog:TRACE_DEBUG_LINE (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
<br>
lttng_ust_tracelog:TRACE_DEBUG_FUNCTION (loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint)
<br>
lttng_ust_tracelog:TRACE_DEBUG_UNIT (loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint)
<br>
lttng_ust_tracelog:TRACE_DEBUG_MODULE (loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint)
<br>
lttng_ust_tracelog:TRACE_DEBUG_PROCESS (loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint)
<br>
lttng_ust_tracelog:TRACE_DEBUG_PROGRAM (loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint)
<br>
lttng_ust_tracelog:TRACE_DEBUG_SYSTEM (loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint)
<br>
lttng_ust_tracelog:TRACE_INFO (loglevel: TRACE_INFO (6)) (type: tracepoint) <br>
lttng_ust_tracelog:TRACE_NOTICE (loglevel: TRACE_NOTICE (5)) (type: tracepoint)
<br>
lttng_ust_tracelog:TRACE_WARNING (loglevel: TRACE_WARNING (4)) (type: tracepoint)
<br>
lttng_ust_tracelog:TRACE_ERR (loglevel: TRACE_ERR (3)) (type: tracepoint) <br>
lttng_ust_tracelog:TRACE_CRIT (loglevel: TRACE_CRIT (2)) (type: tracepoint) <br>
lttng_ust_tracelog:TRACE_ALERT (loglevel: TRACE_ALERT (1)) (type: tracepoint)
<br>
lttng_ust_tracelog:TRACE_EMERG (loglevel: TRACE_EMERG (0)) (type: tracepoint)
<br>
lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint) <br>
lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) <br>
lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
<br>
lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) <br>
lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) <br>
lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
<br>
lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
<br>
lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
<br>
lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
<br>
lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">When I do lttng start, I get a directory structure in the trace directory but the file sizes do not ever increase. It's just a directory structure with files created and then that's it, no events
being written to the files. At first I thought things just weren't linked in right and not working but then discovered how to run lttng-sessiond in -vvv mode and then could see lots of things are going on, tracing appears to be trying to work as there is
tons of activity being output by lttng-sessiond, but like I said the trace files aren't being added to.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">I looked at the lttng-sessiond log I captured and can't quite make out the problem so attaching it.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">I'm probably not doing something right but can't figure out what that is so any pointers appreciated.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">Regards,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">Brian</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div></div>
</blockquote></div>