[lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Kienan Stewart]

Hi Brian,

thanks! I'll take a look at these and see if I can get any ideas from them.

To go back to your original e-mail, a detail that I hadn't noticed in 
your original backtrace was pointed out to me and I'd like to verify 
with you:

```
Core was generated by `./my_app'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
96      ../sysdeps/aarch64/multiarch/strlen_asimd.S: No such file or 
directory.
[Current thread is 1 (Thread 0xffffb8e2d040 (LWP 757))]
(gdb) bt
#0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
#1  0x0000ffffb86c5330 in lttng_ust_tracepoint_module_register () from 
/usr/lib/liblttng-ust-tracepoint.so.1
#2  0x0000aaaab4c8da18 in lttng_ust__tracepoints__ptrs_init () at 
/opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h:629
#3  0x0000ffffb872b30c in call_init (env=<optimized out>, 
argv=0xffffe2d66098, argc=1) at ../csu/libc-start.c:145
#4  __libc_start_main_impl (main=0xaaaab4bd94e0 <main>, argc=1, 
argv=0xffffe2d66098, init=<optimized out>, fini=<optimized out>, 
rtld_fini=<optimized out>, stack_end=<optimized out>) at 
../csu/libc-start.c:376
#5  0x0000aaaab4bd9230 in _start () at ../sysdeps/aarch64/start.S:81
```

The library loaded at frame 1 is 
`/usr/lib/liblttng-ust-tracepoint.so.1`; however, in frame 2 the 
reference is from 
`/opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h`.

Are the LTTng libraries in `/usr/lib` the exact same version as the 
headers in 
`/opt/poky/4.0.18/sysroots/cortexa53-cryptop-poky-linux/usr/include/lttng`?

If some of your app is compiled using one version of headers and a 
different version of the library is loaded at runtime, there could be an 
ABI mismatch.

You could check what the include paths are during compilation and 
`LD_LIBRARY_PATH` at runtime. Running with the environment variable 
`LD_DEBUG=bindings,libs` (see `man ld.so` for more info).

In any case, I'll take a look over the most recent backtraces and see if 
anything else jumps out.

thanks,
kienan


On 7/16/24 9:27 PM, Brian Hutchinson wrote:
> Hi Kienan,
> 
> On Thu, Jul 11, 2024 at 8:38 AM Kienan Stewart <kstewart at efficios.com 
> <mailto:kstewart at efficios.com>> wrote:
> 
>     Hi Brian,
> 
>     On 7/11/24 8:33 AM, Brian Hutchinson via lttng-dev wrote:
>       > ...
>       >
>       > I'm not sure about this, still trying to comprehend everything,
>     but I
>       > think my issue could be related to all this gcc TLS, heap,
>     trampoline
>       > talk here:
>       >
>       >
>     https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html
>     <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>
>       >
>     <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>>
>       >
>       > Regards,
>       >
>       > Brian
>       >
> 
>     You could try using lttng-ust 2.13.8, which includes a fix for the
>     discussion you referenced.
> 
>     https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836 <https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836>
> 
>     Please let us know if it doesn't solve it.
> 
>     thanks,
>     kienan
> 
> 
> I updated to lttng-ust 2.13.8 as you suggested.  At first, I pulled in 
> the recipe from master branch of Yocto, but had the same problem as 
> before after rebuilding.
> 
> Next, I downloaded the 2.13.8 lttng-ust tarball from the lttng site and 
> setup a native build environment on my target and built lttng-ust 2.13.8 
> and my app as a second independent build method for sanity check.  I can 
> report that I get the exact same result as before this way too.
> 
> In both cases, I verified that I had the commit you referenced above.
> 
> I did a backtrace of the three threads captured in the coredump and show 
> them below in hopes it will shed light on what I need to try next.
> 
> (gdb) thread apply all backtrace
> 
> Thread 3 (Thread 0xffffbe7de7e0 (LWP 559)):
> #0  syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
> #1  0x0000ffffbeef1bb8 in lttng_ust_futex (val3=0, uaddr2=0x0, 
> timeout=0x0, val=0, op=0, uaddr=0xffffbefd8000) at 
> ../../../src/lib/lttng-ust/futex.h:56
> #2  lttng_ust_futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0, 
> uaddr=0xffffbefd8000) at ../../../src/lib/lttng-ust/futex.h:88
> #3  wait_for_sessiond (sock_info=0xffffbef6b638 <global_apps>) at 
> lttng-ust-comm.c:1761
> #4  ust_listener_thread (arg=0xffffbef6b638 <global_apps>) at 
> lttng-ust-comm.c:1832
> #5  0x0000ffffbe9206a8 in start_thread (arg=0xffffc1bd8687) at 
> pthread_create.c:442
> #6  0x0000ffffbe98801c in thread_start () at 
> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
> 
> Thread 2 (Thread 0xffffbdfce7e0 (LWP 560)):
> #0  syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
> #1  0x0000ffffbeef1bb8 in lttng_ust_futex (val3=0, uaddr2=0x0, 
> timeout=0x0, val=0, op=0, uaddr=0xffffbefd7000) at 
> ../../../src/lib/lttng-ust/futex.h:56
> #2  lttng_ust_futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0, 
> uaddr=0xffffbefd7000) at ../../../src/lib/lttng-ust/futex.h:88
> #3  wait_for_sessiond (sock_info=0xffffbef695e0 <local_apps>) at 
> lttng-ust-comm.c:1761
> #4  ust_listener_thread (arg=0xffffbef695e0 <local_apps>) at 
> lttng-ust-comm.c:1832
> #5  0x0000ffffbe9206a8 in start_thread (arg=0xffffc1bd8687) at 
> pthread_create.c:442
> #6  0x0000ffffbe98801c in thread_start () at 
> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
> 
> Thread 1 (Thread 0xffffbefcf460 (LWP 545)):
> #0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> #1  0x0000ffffbe834bd0 in lttng_ust_tp_validate_event_name 
> (tp=0xaaaab8f14de0 
> <lttng_ust_tracepoint_hello_world___my_first_tracepoint>, 
> tp=0xaaaab8f14de0 
> <lttng_ust_tracepoint_hello_world___my_first_tracepoint>) at 
> tracepoint.c:143
> #2  add_callsite (tp=0xaaaab8f14de0 
> <lttng_ust_tracepoint_hello_world___my_first_tracepoint>, 
> lib=0xaaaae74fad30) at tracepoint.c:412
> #3  lib_register_callsites (lib=0xaaaae74fad30) at tracepoint.c:552
> #4  lttng_ust_tracepoint_module_register (tracepoints_start=<optimized 
> out>, tracepoints_count=<optimized out>) at tracepoint.c:956
> #5  0x0000aaaab8758204 in lttng_ust.tracepoints.ptrs_init ()
> #6  0x0000ffffbe8cb30c in call_init (env=<optimized out>, 
> argv=0xffffc1bd8c88, argc=1) at ../csu/libc-start.c:145
> #7  __libc_start_main_impl (main=0xaaaab867aabc <main>, argc=1, 
> argv=0xffffc1bd8c88, init=<optimized out>, fini=<optimized out>, 
> rtld_fini=<optimized out>, stack_end=<optimized out>) at 
> ../csu/libc-start.c:376
> #8  0x0000aaaab867a7f0 in _start () at ../sysdeps/aarch64/start.S:81
> 
> Regards,
> 
> Brian