[lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.
Brian Hutchinson
b.hutchman at gmail.com
Wed Jul 17 12:27:35 EDT 2024
On Wed, Jul 17, 2024 at 10:13 AM Kienan Stewart <kstewart at efficios.com>
wrote:
> Hi Brian,
>
> thanks! I'll take a look at these and see if I can get any ideas from them.
>
> To go back to your original e-mail, a detail that I hadn't noticed in
> your original backtrace was pointed out to me and I'd like to verify
> with you:
>
> ```
> Core was generated by `./my_app'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> 96 ../sysdeps/aarch64/multiarch/strlen_asimd.S: No such file or
> directory.
> [Current thread is 1 (Thread 0xffffb8e2d040 (LWP 757))]
> (gdb) bt
> #0 __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> #1 0x0000ffffb86c5330 in lttng_ust_tracepoint_module_register () from
> /usr/lib/liblttng-ust-tracepoint.so.1
> #2 0x0000aaaab4c8da18 in lttng_ust__tracepoints__ptrs_init () at
>
> /opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h:629
> #3 0x0000ffffb872b30c in call_init (env=<optimized out>,
> argv=0xffffe2d66098, argc=1) at ../csu/libc-start.c:145
> #4 __libc_start_main_impl (main=0xaaaab4bd94e0 <main>, argc=1,
> argv=0xffffe2d66098, init=<optimized out>, fini=<optimized out>,
> rtld_fini=<optimized out>, stack_end=<optimized out>) at
> ../csu/libc-start.c:376
> #5 0x0000aaaab4bd9230 in _start () at ../sysdeps/aarch64/start.S:81
> ```
>
> The library loaded at frame 1 is
> `/usr/lib/liblttng-ust-tracepoint.so.1`; however, in frame 2 the
> reference is from
>
> `/opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h`.
> Are the LTTng libraries in `/usr/lib` the exact same version as the
> headers in
> `/opt/poky/4.0.18/sysroots/cortexa53-cryptop-poky-linux/usr/include/lttng`?
>
Yes. The sdk was created by yocto using populate_sdk which includes the
target image sysroot in the sdk. Hopefully I explained that right. The
target image drives the contents of the sdk and the generated sdk is what
is used to cross compile the application specifically for the target we are
running on (NXP imx8mm).
> If some of your app is compiled using one version of headers and a
> different version of the library is loaded at runtime, there could be an
> ABI mismatch.
>
> You could check what the include paths are during compilation and
> `LD_LIBRARY_PATH` at runtime. Running with the environment variable
> `LD_DEBUG=bindings,libs` (see `man ld.so` for more info).
>
I've been down that road a few times before with the same line of thinking
(ABI issue) and believe I've proven I don't have those issues. That's one
of the reasons I hand built lttng-ust in native environment (on the actual
target) to verify it wasn't an issue with a yocto recipe, cross compiling
etc..
I've been thru this process with different OS versions (yocto Dunfell
before), kernel version, toolchain version etc., and now the same with
Kirkstone based components.
> In any case, I'll take a look over the most recent backtraces and see if
> anything else jumps out.
>
Thanks! I'm running out of things to try.
I don't fully understand the implications of some of this documentation,
but I have learned enough to know LD_PRELOAD means nothing if everything is
static built. Our application is heavily multi threaded, uses fork, clone
and who knows if it's doing double closes or not ... so I've tried these
LD_PRELOAD "helpers" or whatever they are, before and didn't get a
different result, but now know it's only for shared library object. So now
I will experiment with making our tpp a shared object and try LD_PRELOAD of
fork, fd and pthread helpers and starting my app that way.
This is what I'm referring to above from the lttng docs:
Use LTTng-UST with daemons
<https://lttng.org/docs/v2.13/#doc-using-lttng-ust-with-daemons>
If your instrumented application calls fork(2)
<https://man7.org/linux/man-pages/man2/fork.2.html>, clone(2)
<https://man7.org/linux/man-pages/man2/clone.2.html>, or BSD’s rfork(2)
<http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE>,
without a following exec(3)
<https://man7.org/linux/man-pages/man3/exec.3.html>-family system call, you
must preload the liblttng-ust-fork.so shared object when you start the
application.
LD_PRELOAD=liblttng-ust-fork.so ./my-app
If your tracepoint provider package is a shared library which you also
preload, you must put both shared objects in LD_PRELOAD:
LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
Use LTTng-UST with applications which close file descriptors that don’t
belong to them <https://lttng.org/docs/v2.13/#doc-liblttng-ust-fd>
Since 2.9
If your instrumented application closes one or more file descriptors which
it did not open itself, you must preload the liblttng-ust-fd.so shared
object when you start the application:
LD_PRELOAD=liblttng-ust-fd.so ./my-app
Typical use cases include closing all the file descriptors after fork(2)
<https://man7.org/linux/man-pages/man2/fork.2.html> or rfork(2)
<http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE>
and buggy applications doing “double closes”.
Thanks,
Brian
>
>
> On 7/16/24 9:27 PM, Brian Hutchinson wrote:
> > Hi Kienan,
> >
> > On Thu, Jul 11, 2024 at 8:38 AM Kienan Stewart <kstewart at efficios.com
> > <mailto:kstewart at efficios.com>> wrote:
> >
> > Hi Brian,
> >
> > On 7/11/24 8:33 AM, Brian Hutchinson via lttng-dev wrote:
> > > ...
> > >
> > > I'm not sure about this, still trying to comprehend everything,
> > but I
> > > think my issue could be related to all this gcc TLS, heap,
> > trampoline
> > > talk here:
> > >
> > >
> > https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html
> > <
> https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>
> > >
> > <
> https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html <
> https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>>
> > >
> > > Regards,
> > >
> > > Brian
> > >
> >
> > You could try using lttng-ust 2.13.8, which includes a fix for the
> > discussion you referenced.
> >
> >
> https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836
> <
> https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836
> >
> >
> > Please let us know if it doesn't solve it.
> >
> > thanks,
> > kienan
> >
> >
> > I updated to lttng-ust 2.13.8 as you suggested. At first, I pulled in
> > the recipe from master branch of Yocto, but had the same problem as
> > before after rebuilding.
> >
> > Next, I downloaded the 2.13.8 lttng-ust tarball from the lttng site and
> > setup a native build environment on my target and built lttng-ust 2.13.8
> > and my app as a second independent build method for sanity check. I can
> > report that I get the exact same result as before this way too.
> >
> > In both cases, I verified that I had the commit you referenced above.
> >
> > I did a backtrace of the three threads captured in the coredump and show
> > them below in hopes it will shed light on what I need to try next.
> >
> > (gdb) thread apply all backtrace
> >
> > Thread 3 (Thread 0xffffbe7de7e0 (LWP 559)):
> > #0 syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
> > #1 0x0000ffffbeef1bb8 in lttng_ust_futex (val3=0, uaddr2=0x0,
> > timeout=0x0, val=0, op=0, uaddr=0xffffbefd8000) at
> > ../../../src/lib/lttng-ust/futex.h:56
> > #2 lttng_ust_futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0,
> > uaddr=0xffffbefd8000) at ../../../src/lib/lttng-ust/futex.h:88
> > #3 wait_for_sessiond (sock_info=0xffffbef6b638 <global_apps>) at
> > lttng-ust-comm.c:1761
> > #4 ust_listener_thread (arg=0xffffbef6b638 <global_apps>) at
> > lttng-ust-comm.c:1832
> > #5 0x0000ffffbe9206a8 in start_thread (arg=0xffffc1bd8687) at
> > pthread_create.c:442
> > #6 0x0000ffffbe98801c in thread_start () at
> > ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
> >
> > Thread 2 (Thread 0xffffbdfce7e0 (LWP 560)):
> > #0 syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
> > #1 0x0000ffffbeef1bb8 in lttng_ust_futex (val3=0, uaddr2=0x0,
> > timeout=0x0, val=0, op=0, uaddr=0xffffbefd7000) at
> > ../../../src/lib/lttng-ust/futex.h:56
> > #2 lttng_ust_futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0,
> > uaddr=0xffffbefd7000) at ../../../src/lib/lttng-ust/futex.h:88
> > #3 wait_for_sessiond (sock_info=0xffffbef695e0 <local_apps>) at
> > lttng-ust-comm.c:1761
> > #4 ust_listener_thread (arg=0xffffbef695e0 <local_apps>) at
> > lttng-ust-comm.c:1832
> > #5 0x0000ffffbe9206a8 in start_thread (arg=0xffffc1bd8687) at
> > pthread_create.c:442
> > #6 0x0000ffffbe98801c in thread_start () at
> > ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
> >
> > Thread 1 (Thread 0xffffbefcf460 (LWP 545)):
> > #0 __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> > #1 0x0000ffffbe834bd0 in lttng_ust_tp_validate_event_name
> > (tp=0xaaaab8f14de0
> > <lttng_ust_tracepoint_hello_world___my_first_tracepoint>,
> > tp=0xaaaab8f14de0
> > <lttng_ust_tracepoint_hello_world___my_first_tracepoint>) at
> > tracepoint.c:143
> > #2 add_callsite (tp=0xaaaab8f14de0
> > <lttng_ust_tracepoint_hello_world___my_first_tracepoint>,
> > lib=0xaaaae74fad30) at tracepoint.c:412
> > #3 lib_register_callsites (lib=0xaaaae74fad30) at tracepoint.c:552
> > #4 lttng_ust_tracepoint_module_register (tracepoints_start=<optimized
> > out>, tracepoints_count=<optimized out>) at tracepoint.c:956
> > #5 0x0000aaaab8758204 in lttng_ust.tracepoints.ptrs_init ()
> > #6 0x0000ffffbe8cb30c in call_init (env=<optimized out>,
> > argv=0xffffc1bd8c88, argc=1) at ../csu/libc-start.c:145
> > #7 __libc_start_main_impl (main=0xaaaab867aabc <main>, argc=1,
> > argv=0xffffc1bd8c88, init=<optimized out>, fini=<optimized out>,
> > rtld_fini=<optimized out>, stack_end=<optimized out>) at
> > ../csu/libc-start.c:376
> > #8 0x0000aaaab867a7f0 in _start () at ../sysdeps/aarch64/start.S:81
> >
> > Regards,
> >
> > Brian
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20240717/3cd44420/attachment-0001.htm>
More information about the lttng-dev
mailing list