[lttng-dev] lttng sessiond hash table crashes.

Fri Oct 7 12:59:12 UTC 2016

----- On Oct 7, 2016, at 6:46 AM, Aravind HT <aravind.ht at gmail.com> wrote: 

> Hi,

> I am encountering a few crashes which are all happening around the hash table
> functions, the below are from my 2.6 version for which I have reproduced them
> with debug symbols enabled.

Hi Aravind, 

We do not support LTTng 2.6 anymore. Let us know if you can reproduce with 
LTTng 2.8 or master. 

Thanks, 

Mathieu 

> 2.6 SCENARIO 1

> (gdb) bt

> Python Exception <type 'exceptions.ImportError'> No module named gdb.frames:

> #0 lttng_ht_lookup (ht=0x6888f0, key=key at entry=0x4b,
> iter=iter at entry=0x7f318fffeae0) at hashtable.c:236

> #1 0x0000000000429900 in ust_app_unregister (sock=sock at entry=75) at
> ust-app.c:3105

> #2 0x000000000043918a in thread_manage_apps (data=<optimized out>) at
> main.c:1558

> #3 0x00007f3197c74294 in start_thread (arg=0x7f318ffff700) at
> pthread_create.c:336

> #4 0x00007f31979b213d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

> 2.6 SCENARIO 2 : Not sure why cmd_init() gets called here

> #0 0x00007f024d68fd2c in lttng_cds_lfht_lookup (ht=0x68ac90,
> hash=8123239074643410387, match=0x418be2 <ht_match_reg_uid>,
> key=0x7f023affb8a0, iter=0x7f023affb900) at rculfhash.c:1535

> #1 0x0000000000418be2 in cmd_init () at cmd.c:3332

> #2 0x00000000004191e3 in buffer_reg_uid_find (session_id=<optimized out>,
> bits_per_long=64, uid=0) at buffer-registry.c:198

> #3 0x000000000042df89 in ust_consumer_metadata_request
> (socket=socket at entry=0x67f1f0 <ustconsumer64_data+176>) at ust-consumer.c:464

> #4 0x000000000043b0e6 in thread_manage_consumer (data=0x67f140
> <ustconsumer64_data>) at main.c:1353

> #5 0x00007f024c4ae294 in start_thread (arg=0x7f023affd700) at
> pthread_create.c:336

> #6 0x00007f024c1ec13d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

> (gdb)

> 2.6 SCENARIO 3

> (gdb) bt

> #0 0x00007f444c017740 in ?? ()

> #1 0x00007f44676a9b6b in bucket_at (index=<optimized out>, ht=0x7f444c00d4e0) at
> rculfhash.c:853

> #2 lookup_bucket (hash=4405530133935945645, size=<optimized out>,
> ht=0x7f444c00d4e0) at rculfhash.c:861

> #3 lttng_cds_lfht_lookup (ht=0x7f444c00d4e0, hash=4405530133935945645,
> match=0x7f44676a7980 <match_ulong>, key=0x29, iter=iter at entry=0x7f44613d59d0)
> at rculfhash.c:1542

> #4 0x00007f44676a7e20 in lttng_ht_lookup (ht=<optimized out>, key=<optimized
> out>, iter=iter at entry=0x7f44613d59d0) at hashtable.c:240

> #5 0x0000000000412217 in consumer_find_socket (key=<optimized out>,
> consumer=<optimized out>) at consumer.c:359

> #6 0x0000000000000000 in ?? ()

> I am also trying to upgrade to 2.8.1 and see similar crashes there,
> unfortunately do not have debug symbols enabled, but crashes always in one of
> the hash table related functions.
> I'm in the process of enabling debug symbols on 2.8.1 so that the info that i
> provide applies to the current version, also trying out Valgrind.

> Let me know if you can figure something out with this.

> 2.8.1 SCENARIO 1

> GNU gdb (GDB) 7.8.1

> Copyright (C) 2014 Free Software Foundation, Inc.

> License GPLv3+: GNU GPL version 3 or later < [ http://gnu.org/licenses/gpl.html
> | http://gnu.org/licenses/gpl.html ] >

> This is free software: you are free to change and redistribute it.

> There is NO WARRANTY, to the extent permitted by law. Type "show copying"

> and "show warranty" for details.

> This GDB was configured as "x86_64-poky-linux".

> Type "show configuration" for configuration details.

> For bug reporting instructions, please see:

> < [ http://www.gnu.org/software/gdb/bugs/ |
> http://www.gnu.org/software/gdb/bugs/ ] >.

> Find the GDB manual and other documentation resources online at:

> < [ http://www.gnu.org/software/gdb/documentation/ |
> http://www.gnu.org/software/gdb/documentation/ ] >.

> For help, type "help".

> Type "apropos word" to search for commands related to "word"...

> Reading symbols from /usr/bin/lttng-sessiond...(no debugging symbols
> found)...done.

> [New LWP 1134]

> [New LWP 1090]

> [New LWP 1091]

> [New LWP 1092]

> [New LWP 1093]

> [New LWP 1095]

> [New LWP 1098]

> [New LWP 1102]

> [New LWP 1110]

> [New LWP 1096]

> [New LWP 1094]

> [New LWP 1097]

> warning: Could not load shared library symbols for linux-vdso.so.1.

> Do you need "set solib-search-path" or "set sysroot"?

> [Thread debugging using libthread_db enabled]

> Using host libthread_db library "/lib64/libthread_db.so.1".

> Core was generated by `/usr/bin/lttng-sessiond --consumerd32-path
> /usr/lib/lttng/libexec/lttng-consume'.

> Program terminated with signal SIGSEGV, Segmentation fault.

> #0 0x0000000000000000 in ?? ()

> (gdb) bt

> Python Exception <type 'exceptions.ImportError'> No module named gdb.frames:

> #0 0x0000000000000000 in ?? ()

> #1 0x00007f1b39b1db6b in lttng_cds_lfht_lookup () from
> /usr/lib64/liblttng-ctl.so.0

> #2 0x0000000000412217 in ?? ()

> #3 0x0000000000412263 in ?? ()

> #4 0x0000000000427de8 in ?? ()

> #5 0x0000000000427ed1 in ?? ()

> #6 0x00000000004280d3 in ?? ()

> #7 0x00007f1b398facad in ?? () from /usr/lib64/liburcu.so.2

> #8 0x00007f1b3893d294 in start_thread (arg=0x7f1b1e7fc700) at
> pthread_create.c:336

> #9 0x00007f1b3867b13d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

> (gdb) q

> 2.8.1 SCENARIO 2

> GNU gdb (GDB) 7.8.1

> Copyright (C) 2014 Free Software Foundation, Inc.

> License GPLv3+: GNU GPL version 3 or later < [ http://gnu.org/licenses/gpl.html
> | http://gnu.org/licenses/gpl.html ] >

> This is free software: you are free to change and redistribute it.

> There is NO WARRANTY, to the extent permitted by law. Type "show copying"

> and "show warranty" for details.

> This GDB was configured as "x86_64-poky-linux".

> Type "show configuration" for configuration details.

> For bug reporting instructions, please see:

> < [ http://www.gnu.org/software/gdb/bugs/ |
> http://www.gnu.org/software/gdb/bugs/ ] >.

> Find the GDB manual and other documentation resources online at:

> < [ http://www.gnu.org/software/gdb/documentation/ |
> http://www.gnu.org/software/gdb/documentation/ ] >.

> For help, type "help".

> Type "apropos word" to search for commands related to "word"...

> Reading symbols from /usr/bin/lttng-sessiond...(no debugging symbols
> found)...done.

> [New LWP 637]

> [New LWP 610]

> [New LWP 613]

> [New LWP 624]

> [New LWP 605]

> [New LWP 608]

> [New LWP 607]

> [New LWP 609]

> [New LWP 612]

> [New LWP 606]

> [New LWP 611]

> [New LWP 616]

> warning: Could not load shared library symbols for linux-vdso.so.1.

> Do you need "set solib-search-path" or "set sysroot"?

> [Thread debugging using libthread_db enabled]

> Using host libthread_db library "/lib64/libthread_db.so.1".

> Core was generated by `/usr/bin/lttng-sessiond --consumerd32-path
> /usr/lib/lttng/libexec/lttng-consume'.

> Program terminated with signal SIGSEGV, Segmentation fault.

> #0 0x0000000000000020 in ?? ()

> (gdb) bt

> Python Exception <type 'exceptions.ImportError'> No module named gdb.frames:

> #0 0x0000000000000020 in ?? ()

> #1 0x00007ff010197e02 in lttng_ht_lookup () from /usr/lib64/liblttng-ctl.so.0

> #2 0x0000000000412217 in ?? ()

> #3 0x0000000000412263 in ?? ()

> #4 0x0000000000427de8 in ?? ()

> #5 0x0000000000427ed1 in ?? ()

> #6 0x00000000004280d3 in ?? ()

> #7 0x00007ff00ff76cad in ?? () from /usr/lib64/liburcu.so.2

> #8 0x00007ff00efb9294 in start_thread (arg=0x7ff0096c5700) at
> pthread_create.c:336

> #9 0x00007ff00ecf713d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

> (gdb) q

> Regards,
> Aravind.

> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20161007/1d5b2d49/attachment-0001.html>