[lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.
Brian Hutchinson
b.hutchman at gmail.com
Thu Jul 11 08:33:59 EDT 2024
On Wed, Jul 10, 2024 at 9:53 PM Brian Hutchinson <b.hutchman at gmail.com>
wrote:
> Hi,
>
> I'm getting a weird segfault when I compile in a lttng-ust tracepoint into
> my application.
>
> I'm able to run the lttng-ust hello_world example just fine on my platform
> which one would think should prove everything out, but when I use the exact
> same hello world trace provider and link it into my multi-threadded app and
> simply call:
>
> lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
>
> I get a segfault.
>
> Background. I'm on aarch64 platform (imx8mm-lpddr4-evk) with sdk (gcc
> 11.4 glibc 2.35) and rootfs generated from Yocto Kirkstone 4.0.18. Built
> lttng-tools (2.13.9) and lttng-ust (2.13.6) from yocto recipe. Built
> out-of-tree kernel modules for lttng-modules (2.13.9) and I'm on kernel
> 6.1.38.
>
> If I do a backtrace with gdb on the core this is what I see:
>
> GNU gdb (GDB) 11.2
> Copyright (C) 2022 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "aarch64-poky-linux".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <https://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
>
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from ./my_app...
>
> warning: core file may not match specified executable file.
> [New LWP 757]
> [New LWP 759]
> [New LWP 758]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/libthread_db.so.1".
> Core was generated by `./my_app'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> 96 ../sysdeps/aarch64/multiarch/strlen_asimd.S: No such file or
> directory.
> [Current thread is 1 (Thread 0xffffb8e2d040 (LWP 757))]
> (gdb) bt
> #0 __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> #1 0x0000ffffb86c5330 in lttng_ust_tracepoint_module_register () from
> /usr/lib/liblttng-ust-tracepoint.so.1
> #2 0x0000aaaab4c8da18 in lttng_ust__tracepoints__ptrs_init () at
> /opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h:629
>
> #3 0x0000ffffb872b30c in call_init (env=<optimized out>,
> argv=0xffffe2d66098, argc=1) at ../csu/libc-start.c:145
> #4 __libc_start_main_impl (main=0xaaaab4bd94e0 <main>, argc=1,
> argv=0xffffe2d66098, init=<optimized out>, fini=<optimized out>,
> rtld_fini=<optimized out>, stack_end=<optimized out>) at
> ../csu/libc-start.c:376
> #5 0x0000aaaab4bd9230 in _start () at ../sysdeps/aarch64/start.S:81
> (gdb)
>
> If I run valgrind, this is what I see:
>
> ==627== Memcheck, a memory error detector
> ==627== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> ==627== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
> ==627== Command: ./my_app
> ==627==
> LTTng-UST: Error while registering tracepoint probe.
> ==627==
> ==627== Process terminating with default action of signal 6 (SIGABRT)
> ==627== at 0x6772278: __pthread_kill_implementation (pthread_kill.c:44)
> ==627== by 0x672DD4F: raise (raise.c:26)
> ==627== by 0x671AEE3: abort (abort.c:79)
> ==627== by 0x2C572B: lttng_ust__events_init__hello_world
> (ust-tracepoint-event.h:1221)
> ==627== by 0x2C5B97: lttng_ust_constructor_hello_world
> (ust-tracepoint-event.h:1239)
> ==627== by 0x671B30B: call_init (libc-start.c:145)
> ==627== by 0x671B30B: __libc_start_main@@GLIBC_2.34 (libc-start.c:376)
> ==627== by 0x21122F: (below main) (start.S:81)
> ==627==
> ==627== HEAP SUMMARY:
> ==627== in use at exit: 151,712 bytes in 90 blocks
> ==627== total heap usage: 1,184 allocs, 1,094 frees, 202,570 bytes
> allocated
> ==627==
> ==627== 48 bytes in 1 blocks are still reachable in loss record 1 of 17
> ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627== by 0x68D50CB: lttng_ust_tracepoint_module_register (in
> /usr/lib/liblttng-ust-tracepoint.so.1.0.0)
> ==627== by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627== by 0x5923767: call_init (dl-init.c:29)
> ==627== by 0x5923767: _dl_init (dl-init.c:121)
> ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 48 bytes in 1 blocks are still reachable in loss record 2 of 17
> ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627== by 0x61E8143: lttng_ust_probe_register (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x61FD49B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627== by 0x5923767: call_init (dl-init.c:29)
> ==627== by 0x5923767: _dl_init (dl-init.c:121)
> ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 48 bytes in 1 blocks are still reachable in loss record 3 of 17
> ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627== by 0x61E8143: lttng_ust_probe_register (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x61DFF4F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627== by 0x5923767: call_init (dl-init.c:29)
> ==627== by 0x5923767: _dl_init (dl-init.c:121)
> ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 48 bytes in 1 blocks are still reachable in loss record 4 of 17
> ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627== by 0x61E8143: lttng_ust_probe_register (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x61E026F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627== by 0x5923767: call_init (dl-init.c:29)
> ==627== by 0x5923767: _dl_init (dl-init.c:121)
> ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 48 bytes in 1 blocks are still reachable in loss record 5 of 17
> ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627== by 0x61E8143: lttng_ust_probe_register (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x61E058F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627== by 0x5923767: call_init (dl-init.c:29)
> ==627== by 0x5923767: _dl_init (dl-init.c:121)
> ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 48 bytes in 1 blocks are still reachable in loss record 6 of 17
> ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627== by 0x68D50CB: lttng_ust_tracepoint_module_register (in
> /usr/lib/liblttng-ust-tracepoint.so.1.0.0)
> ==627== by 0x2C5A17: lttng_ust__tracepoints__ptrs_init
> (tracepoint.h:629)
> ==627== by 0x671B30B: call_init (libc-start.c:145)
> ==627== by 0x671B30B: __libc_start_main@@GLIBC_2.34 (libc-start.c:376)
> ==627== by 0x21122F: (below main) (start.S:81)
> ==627==
> ==627== 128 bytes in 1 blocks are still reachable in loss record 7 of 17
> ==627== at 0x618551C: malloc (vg_replace_malloc.c:381)
> ==627== by 0x68A388B: ??? (in /usr/lib/liblttng-ust-common.so.1.0.0)
> ==627== by 0x68A2FBF: lttng_ust_common_ctor (in
> /usr/lib/liblttng-ust-common.so.1.0.0)
> ==627== by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627== by 0x5923767: call_init (dl-init.c:29)
> ==627== by 0x5923767: _dl_init (dl-init.c:121)
> ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 200 bytes in 1 blocks are still reachable in loss record 8 of 17
> ==627== at 0x618551C: malloc (vg_replace_malloc.c:381)
> ==627== by 0x5929E37: malloc (rtld-malloc.h:56)
> ==627== by 0x5929E37: add_to_global_resize (dl-open.c:152)
> ==627== by 0x592AA8F: dl_open_worker_begin (dl-open.c:716)
> ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627== by 0x5929C77: dl_open_worker (dl-open.c:782)
> ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627== by 0x592A07B: _dl_open (dl-open.c:884)
> ==627== by 0x676C467: dlopen_doit (dlopen.c:56)
> ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627== by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227)
> ==627== by 0x676BF4F: _dlerror_run (dlerror.c:138)
> ==627== by 0x676C52B: dlopen_implementation (dlopen.c:71)
> ==627== by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81)
> ==627==
> ==627== 240 bytes in 2 blocks are still reachable in loss record 9 of 17
> ==627== at 0x618551C: malloc (vg_replace_malloc.c:381)
> ==627== by 0x59214C7: malloc (rtld-malloc.h:56)
> ==627== by 0x59214C7: _dl_map_object_deps (dl-deps.c:479)
> ==627== by 0x592A50B: dl_open_worker_begin (dl-open.c:592)
> ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627== by 0x5929C77: dl_open_worker (dl-open.c:782)
> ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627== by 0x592A07B: _dl_open (dl-open.c:884)
> ==627== by 0x676C467: dlopen_doit (dlopen.c:56)
> ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627== by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227)
> ==627== by 0x676BF4F: _dlerror_run (dlerror.c:138)
> ==627== by 0x676C52B: dlopen_implementation (dlopen.c:71)
> ==627== by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81)
> ==627==
> ==627== 292 bytes in 15 blocks are possibly lost in loss record 10 of 17
> ==627== at 0x618551C: malloc (vg_replace_malloc.c:381)
> ==627== by 0x6783993: strdup (strdup.c:42)
> ==627== by 0x61FD92F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74)
> ==627== by 0x61FE653: lttng_ust_dl_update (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627== by 0x5923767: call_init (dl-init.c:29)
> ==627== by 0x5923767: _dl_init (dl-init.c:121)
> ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 336 bytes in 1 blocks are possibly lost in loss record 11 of 17
> ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627== by 0x592ECB7: calloc (rtld-malloc.h:44)
> ==627== by 0x592ECB7: allocate_dtv (dl-tls.c:376)
> ==627== by 0x592F723: _dl_allocate_tls (dl-tls.c:635)
> ==627== by 0x677114F: allocate_stack (allocatestack.c:428)
> ==627== by 0x677114F: pthread_create@@GLIBC_2.34
> (pthread_create.c:647)
> ==627== by 0x61DEF03: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627== by 0x5923767: call_init (dl-init.c:29)
> ==627== by 0x5923767: _dl_init (dl-init.c:121)
> ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 336 bytes in 1 blocks are possibly lost in loss record 12 of 17
> ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627== by 0x592ECB7: calloc (rtld-malloc.h:44)
> ==627== by 0x592ECB7: allocate_dtv (dl-tls.c:376)
> ==627== by 0x592F723: _dl_allocate_tls (dl-tls.c:635)
> ==627== by 0x677114F: allocate_stack (allocatestack.c:428)
> ==627== by 0x677114F: pthread_create@@GLIBC_2.34
> (pthread_create.c:647)
> ==627== by 0x61DEEB7: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627== by 0x5923767: call_init (dl-init.c:29)
> ==627== by 0x5923767: _dl_init (dl-init.c:121)
> ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 340 bytes in 17 blocks are possibly lost in loss record 13 of 17
> ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627== by 0x61FD94F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74)
> ==627== by 0x61FE653: lttng_ust_dl_update (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627== by 0x5923767: call_init (dl-init.c:29)
> ==627== by 0x5923767: _dl_init (dl-init.c:121)
> ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 1,248 bytes in 26 blocks are still reachable in loss record 14 of
> 17
> ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627== by 0x68D529F: lttng_ust_tracepoint_module_register (in
> /usr/lib/liblttng-ust-tracepoint.so.1.0.0)
> ==627== by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627== by 0x5923767: call_init (dl-init.c:29)
> ==627== by 0x5923767: _dl_init (dl-init.c:121)
> ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 4,608 bytes in 2 blocks are possibly lost in loss record 15 of 17
> ==627== at 0x618551C: malloc (vg_replace_malloc.c:381)
> ==627== by 0x5922FFB: malloc (rtld-malloc.h:56)
> ==627== by 0x5922FFB: _dlfo_mappings_segment_allocate
> (dl-find_object.c:217)
> ==627== by 0x5922FFB: _dl_find_object_update_1 (dl-find_object.c:671)
> ==627== by 0x5922FFB: _dl_find_object_update (dl-find_object.c:805)
> ==627== by 0x592A803: dl_open_worker_begin (dl-open.c:735)
> ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627== by 0x5929C77: dl_open_worker (dl-open.c:782)
> ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627== by 0x592A07B: _dl_open (dl-open.c:884)
> ==627== by 0x676C467: dlopen_doit (dlopen.c:56)
> ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627== by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227)
> ==627== by 0x676BF4F: _dlerror_run (dlerror.c:138)
> ==627== by 0x676C52B: dlopen_implementation (dlopen.c:71)
> ==627== by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81)
> ==627==
> ==627== 70,992 bytes in 17 blocks are possibly lost in loss record 16 of
> 17
> ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627== by 0x61FD91B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74)
> ==627== by 0x61FE653: lttng_ust_dl_update (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627== by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627== by 0x5923767: call_init (dl-init.c:29)
> ==627== by 0x5923767: _dl_init (dl-init.c:121)
> ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 72,704 bytes in 1 blocks are still reachable in loss record 17 of
> 17
> ==627== at 0x618551C: malloc (vg_replace_malloc.c:381)
> ==627== by 0x653E137: ??? (in /usr/lib/libstdc++.so.6.0.29)
> ==627== by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627== by 0x5923767: call_init (dl-init.c:29)
> ==627== by 0x5923767: _dl_init (dl-init.c:121)
> ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== LEAK SUMMARY:
> ==627== definitely lost: 0 bytes in 0 blocks
> ==627== indirectly lost: 0 bytes in 0 blocks
> ==627== possibly lost: 76,904 bytes in 53 blocks
> ==627== still reachable: 74,808 bytes in 37 blocks
> ==627== suppressed: 0 bytes in 0 blocks
> ==627==
> ==627== For lists of detected and suppressed errors, rerun with: -s
> ==627== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 0 from 0)
> Aborted
>
> I'm sure I'm probably doing something wrong but I've tried static linking,
> dynamic linking using LD_PRELOAD helpers like LD_PRELOAD=liblttng-ust-pthread-wrapper.so.1:liblttng-ust-fd.so.1:liblttng-ust-fork.so.1
> and I'm out of options that I know to try.
>
> Looking for pointers on what else I can try and what I could be doing
> wrong.
>
> I had this same segfault issue using Dunfell version of Yocto and earlier
> versions of lttng (2.11) but never dug into it because was planning on
> upgrading everything soon, so recently stepped up to newer versions of
> everything to see if that solved the issue but apparently not.
>
>
I'm not sure about this, still trying to comprehend everything, but I think
my issue could be related to all this gcc TLS, heap, trampoline talk here:
https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html
Regards,
Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20240711/f6119173/attachment-0001.htm>
More information about the lttng-dev
mailing list