<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 17, 2024 at 10:13 AM Kienan Stewart <<a href="mailto:kstewart@efficios.com" target="_blank">kstewart@efficios.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Brian,<br>
<br>
thanks! I'll take a look at these and see if I can get any ideas from them.<br>
<br>
To go back to your original e-mail, a detail that I hadn't noticed in <br>
your original backtrace was pointed out to me and I'd like to verify <br>
with you:<br>
<br>
```<br>
Core was generated by `./my_app'.<br>
Program terminated with signal SIGSEGV, Segmentation fault.<br>
#0 __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96<br>
96 ../sysdeps/aarch64/multiarch/strlen_asimd.S: No such file or <br>
directory.<br>
[Current thread is 1 (Thread 0xffffb8e2d040 (LWP 757))]<br>
(gdb) bt<br>
#0 __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96<br>
#1 0x0000ffffb86c5330 in lttng_ust_tracepoint_module_register () from <br>
/usr/lib/liblttng-ust-tracepoint.so.1<br>
#2 0x0000aaaab4c8da18 in lttng_ust__tracepoints__ptrs_init () at <br>
/opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h:629<br>
#3 0x0000ffffb872b30c in call_init (env=<optimized out>, <br>
argv=0xffffe2d66098, argc=1) at ../csu/libc-start.c:145<br>
#4 __libc_start_main_impl (main=0xaaaab4bd94e0 <main>, argc=1, <br>
argv=0xffffe2d66098, init=<optimized out>, fini=<optimized out>, <br>
rtld_fini=<optimized out>, stack_end=<optimized out>) at <br>
../csu/libc-start.c:376<br>
#5 0x0000aaaab4bd9230 in _start () at ../sysdeps/aarch64/start.S:81<br>
```<br>
<br>
The library loaded at frame 1 is <br>
`/usr/lib/liblttng-ust-tracepoint.so.1`; however, in frame 2 the <br>
reference is from <br>
`/opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h`.</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Are the LTTng libraries in `/usr/lib` the exact same version as the <br>
headers in <br>
`/opt/poky/4.0.18/sysroots/cortexa53-cryptop-poky-linux/usr/include/lttng`?<br></blockquote><div><br></div><div>Yes. The sdk was created by yocto using populate_sdk which includes the
target image sysroot in the sdk. Hopefully I explained that right. The target image drives the contents of the sdk and the generated sdk is what is used to cross compile the application specifically for the target we are running on (NXP imx8mm).<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
If some of your app is compiled using one version of headers and a <br>
different version of the library is loaded at runtime, there could be an <br>
ABI mismatch.<br></blockquote><div></div><div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
You could check what the include paths are during compilation and <br>
`LD_LIBRARY_PATH` at runtime. Running with the environment variable <br>
`LD_DEBUG=bindings,libs` (see `man ld.so` for more info).<br></blockquote><div> </div><div>I've been down that road a few times before with the same line of thinking (ABI issue) and believe I've proven I don't have those issues. That's one of the reasons I hand built lttng-ust in native environment (on the actual target) to verify it wasn't an issue with a yocto recipe, cross compiling etc..<br></div><div><br></div><div>I've been thru this process with different OS versions (yocto Dunfell before), kernel version, toolchain version etc., and now the same with Kirkstone based components.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
In any case, I'll take a look over the most recent backtraces and see if <br>
anything else jumps out.<br></blockquote><div><br></div><div>Thanks! I'm running out of things to try.</div><div></div><div><br></div><div>I don't fully understand the implications of some of this documentation, but I have learned enough to know LD_PRELOAD means nothing if everything is static built. Our application is heavily multi threaded, uses fork, clone and who knows if it's doing double closes or not ... so I've tried these LD_PRELOAD "helpers" or whatever they are, before and didn't get a different result, but now know it's only for shared library object. So now I will experiment with making our tpp a shared object and try LD_PRELOAD of fork, fd and pthread helpers and starting my app that way.</div><div><br></div><div>This is what I'm referring to above from the lttng docs:<br></div><div><h5><a href="https://lttng.org/docs/v2.13/#doc-using-lttng-ust-with-daemons" target="_blank">Use <span>LTTng-UST</span> with daemons</a></h5><p>If your instrumented application calls <a href="https://man7.org/linux/man-pages/man2/fork.2.html" target="_blank">fork<span>(2)</span></a>, <a href="https://man7.org/linux/man-pages/man2/clone.2.html" target="_blank">clone<span>(2)</span></a>,
or BSD’s <a href="http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE" target="_blank">rfork<span>(2)</span></a>, without a following <a href="https://man7.org/linux/man-pages/man3/exec.3.html" target="_blank">exec<span>(3)</span></a>-family
system call, you must preload the <code>liblttng-ust-fork.so</code> shared
object when you start the application.</p><div></div><div><pre>LD_PRELOAD=liblttng-ust-fork.so ./my-app</pre></div><p>If your tracepoint provider package is
a shared library which you also preload, you must put both
shared objects in <code>LD_PRELOAD</code>:</p><div></div><div><pre>LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app</pre></div><h5><a href="https://lttng.org/docs/v2.13/#doc-liblttng-ust-fd" target="_blank">Use <span>LTTng-UST</span> with applications which close file descriptors that don’t belong to them</a></h5><div>Since 2.9</div><p>If your instrumented application closes one or more file descriptors
which it did not open itself, you must preload the
<code>liblttng-ust-fd.so</code> shared object when you start the application:</p><div></div><div><pre>LD_PRELOAD=liblttng-ust-fd.so ./my-app</pre></div><p>Typical use cases include closing all the file descriptors after
<a href="https://man7.org/linux/man-pages/man2/fork.2.html" target="_blank">fork<span>(2)</span></a> or <a href="http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE" target="_blank">rfork<span>(2)</span></a> and buggy applications doing
“double closes”.</p><p><br></p><p></p><p>Thanks,</p><p></p><p>Brian<br></p></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
On 7/16/24 9:27 PM, Brian Hutchinson wrote:<br>
> Hi Kienan,<br>
> <br>
> On Thu, Jul 11, 2024 at 8:38 AM Kienan Stewart <<a href="mailto:kstewart@efficios.com" target="_blank">kstewart@efficios.com</a> <br>
> <mailto:<a href="mailto:kstewart@efficios.com" target="_blank">kstewart@efficios.com</a>>> wrote:<br>
> <br>
> Hi Brian,<br>
> <br>
> On 7/11/24 8:33 AM, Brian Hutchinson via lttng-dev wrote:<br>
> > ...<br>
> ><br>
> > I'm not sure about this, still trying to comprehend everything,<br>
> but I<br>
> > think my issue could be related to all this gcc TLS, heap,<br>
> trampoline<br>
> > talk here:<br>
> ><br>
> ><br>
> <a href="https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html" rel="noreferrer" target="_blank">https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html</a><br>
> <<a href="https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html" rel="noreferrer" target="_blank">https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html</a>><br>
> ><br>
> <<a href="https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html" rel="noreferrer" target="_blank">https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html</a> <<a href="https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html" rel="noreferrer" target="_blank">https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html</a>>><br>
> ><br>
> > Regards,<br>
> ><br>
> > Brian<br>
> ><br>
> <br>
> You could try using lttng-ust 2.13.8, which includes a fix for the<br>
> discussion you referenced.<br>
> <br>
> <a href="https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836" rel="noreferrer" target="_blank">https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836</a> <<a href="https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836" rel="noreferrer" target="_blank">https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836</a>><br>
> <br>
> Please let us know if it doesn't solve it.<br>
> <br>
> thanks,<br>
> kienan<br>
> <br>
> <br>
> I updated to lttng-ust 2.13.8 as you suggested. At first, I pulled in <br>
> the recipe from master branch of Yocto, but had the same problem as <br>
> before after rebuilding.<br>
> <br>
> Next, I downloaded the 2.13.8 lttng-ust tarball from the lttng site and <br>
> setup a native build environment on my target and built lttng-ust 2.13.8 <br>
> and my app as a second independent build method for sanity check. I can <br>
> report that I get the exact same result as before this way too.<br>
> <br>
> In both cases, I verified that I had the commit you referenced above.<br>
> <br>
> I did a backtrace of the three threads captured in the coredump and show <br>
> them below in hopes it will shed light on what I need to try next.<br>
> <br>
> (gdb) thread apply all backtrace<br>
> <br>
> Thread 3 (Thread 0xffffbe7de7e0 (LWP 559)):<br>
> #0 syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38<br>
> #1 0x0000ffffbeef1bb8 in lttng_ust_futex (val3=0, uaddr2=0x0, <br>
> timeout=0x0, val=0, op=0, uaddr=0xffffbefd8000) at <br>
> ../../../src/lib/lttng-ust/futex.h:56<br>
> #2 lttng_ust_futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0, <br>
> uaddr=0xffffbefd8000) at ../../../src/lib/lttng-ust/futex.h:88<br>
> #3 wait_for_sessiond (sock_info=0xffffbef6b638 <global_apps>) at <br>
> lttng-ust-comm.c:1761<br>
> #4 ust_listener_thread (arg=0xffffbef6b638 <global_apps>) at <br>
> lttng-ust-comm.c:1832<br>
> #5 0x0000ffffbe9206a8 in start_thread (arg=0xffffc1bd8687) at <br>
> pthread_create.c:442<br>
> #6 0x0000ffffbe98801c in thread_start () at <br>
> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79<br>
> <br>
> Thread 2 (Thread 0xffffbdfce7e0 (LWP 560)):<br>
> #0 syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38<br>
> #1 0x0000ffffbeef1bb8 in lttng_ust_futex (val3=0, uaddr2=0x0, <br>
> timeout=0x0, val=0, op=0, uaddr=0xffffbefd7000) at <br>
> ../../../src/lib/lttng-ust/futex.h:56<br>
> #2 lttng_ust_futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0, <br>
> uaddr=0xffffbefd7000) at ../../../src/lib/lttng-ust/futex.h:88<br>
> #3 wait_for_sessiond (sock_info=0xffffbef695e0 <local_apps>) at <br>
> lttng-ust-comm.c:1761<br>
> #4 ust_listener_thread (arg=0xffffbef695e0 <local_apps>) at <br>
> lttng-ust-comm.c:1832<br>
> #5 0x0000ffffbe9206a8 in start_thread (arg=0xffffc1bd8687) at <br>
> pthread_create.c:442<br>
> #6 0x0000ffffbe98801c in thread_start () at <br>
> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79<br>
> <br>
> Thread 1 (Thread 0xffffbefcf460 (LWP 545)):<br>
> #0 __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96<br>
> #1 0x0000ffffbe834bd0 in lttng_ust_tp_validate_event_name <br>
> (tp=0xaaaab8f14de0 <br>
> <lttng_ust_tracepoint_hello_world___my_first_tracepoint>, <br>
> tp=0xaaaab8f14de0 <br>
> <lttng_ust_tracepoint_hello_world___my_first_tracepoint>) at <br>
> tracepoint.c:143<br>
> #2 add_callsite (tp=0xaaaab8f14de0 <br>
> <lttng_ust_tracepoint_hello_world___my_first_tracepoint>, <br>
> lib=0xaaaae74fad30) at tracepoint.c:412<br>
> #3 lib_register_callsites (lib=0xaaaae74fad30) at tracepoint.c:552<br>
> #4 lttng_ust_tracepoint_module_register (tracepoints_start=<optimized <br>
> out>, tracepoints_count=<optimized out>) at tracepoint.c:956<br>
> #5 0x0000aaaab8758204 in lttng_ust.tracepoints.ptrs_init ()<br>
> #6 0x0000ffffbe8cb30c in call_init (env=<optimized out>, <br>
> argv=0xffffc1bd8c88, argc=1) at ../csu/libc-start.c:145<br>
> #7 __libc_start_main_impl (main=0xaaaab867aabc <main>, argc=1, <br>
> argv=0xffffc1bd8c88, init=<optimized out>, fini=<optimized out>, <br>
> rtld_fini=<optimized out>, stack_end=<optimized out>) at <br>
> ../csu/libc-start.c:376<br>
> #8 0x0000aaaab867a7f0 in _start () at ../sysdeps/aarch64/start.S:81<br>
> <br>
> Regards,<br>
> <br>
> Brian<br>
</blockquote></div></div>