[lttng-dev] lttng sessiond hash table crashes.

Fri Oct 7 14:35:15 UTC 2016

----- On Oct 7, 2016, at 8:59 AM, Mathieu Desnoyers <mathieu.desnoyers at efficios.com> wrote: 

> ----- On Oct 7, 2016, at 6:46 AM, Aravind HT <aravind.ht at gmail.com> wrote:

>> Hi,

>> I am encountering a few crashes which are all happening around the hash table
>> functions, the below are from my 2.6 version for which I have reproduced them
>> with debug symbols enabled.

> Hi Aravind,

> We do not support LTTng 2.6 anymore. Let us know if you can reproduce with
> LTTng 2.8 or master.

Hi, 

I noticed that further down below, you reproduced with 2.8. Please 
provide backtrace with debug information, of a build without optimisation. 
If you can provide a reproducer it would also help. 

Ref. https://lttng.org/community/#bug-reporting-guidelines 

Thanks, 

Mathieu 

> Thanks,

> Mathieu

>> 2.6 SCENARIO 1

>> (gdb) bt

>> Python Exception <type 'exceptions.ImportError'> No module named gdb.frames:

>> #0 lttng_ht_lookup (ht=0x6888f0, key=key at entry=0x4b,
>> iter=iter at entry=0x7f318fffeae0) at hashtable.c:236

>> #1 0x0000000000429900 in ust_app_unregister (sock=sock at entry=75) at
>> ust-app.c:3105

>> #2 0x000000000043918a in thread_manage_apps (data=<optimized out>) at
>> main.c:1558

>> #3 0x00007f3197c74294 in start_thread (arg=0x7f318ffff700) at
>> pthread_create.c:336

>> #4 0x00007f31979b213d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

>> 2.6 SCENARIO 2 : Not sure why cmd_init() gets called here

>> #0 0x00007f024d68fd2c in lttng_cds_lfht_lookup (ht=0x68ac90,
>> hash=8123239074643410387, match=0x418be2 <ht_match_reg_uid>,
>> key=0x7f023affb8a0, iter=0x7f023affb900) at rculfhash.c:1535

>> #1 0x0000000000418be2 in cmd_init () at cmd.c:3332

>> #2 0x00000000004191e3 in buffer_reg_uid_find (session_id=<optimized out>,
>> bits_per_long=64, uid=0) at buffer-registry.c:198

>> #3 0x000000000042df89 in ust_consumer_metadata_request
>> (socket=socket at entry=0x67f1f0 <ustconsumer64_data+176>) at ust-consumer.c:464

>> #4 0x000000000043b0e6 in thread_manage_consumer (data=0x67f140
>> <ustconsumer64_data>) at main.c:1353

>> #5 0x00007f024c4ae294 in start_thread (arg=0x7f023affd700) at
>> pthread_create.c:336

>> #6 0x00007f024c1ec13d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

>> (gdb)

>> 2.6 SCENARIO 3

>> (gdb) bt

>> #0 0x00007f444c017740 in ?? ()

>> #1 0x00007f44676a9b6b in bucket_at (index=<optimized out>, ht=0x7f444c00d4e0) at
>> rculfhash.c:853

>> #2 lookup_bucket (hash=4405530133935945645, size=<optimized out>,
>> ht=0x7f444c00d4e0) at rculfhash.c:861

>> #3 lttng_cds_lfht_lookup (ht=0x7f444c00d4e0, hash=4405530133935945645,
>> match=0x7f44676a7980 <match_ulong>, key=0x29, iter=iter at entry=0x7f44613d59d0)
>> at rculfhash.c:1542

>> #4 0x00007f44676a7e20 in lttng_ht_lookup (ht=<optimized out>, key=<optimized
>> out>, iter=iter at entry=0x7f44613d59d0) at hashtable.c:240

>> #5 0x0000000000412217 in consumer_find_socket (key=<optimized out>,
>> consumer=<optimized out>) at consumer.c:359

>> #6 0x0000000000000000 in ?? ()

>> I am also trying to upgrade to 2.8.1 and see similar crashes there,
>> unfortunately do not have debug symbols enabled, but crashes always in one of
>> the hash table related functions.
>> I'm in the process of enabling debug symbols on 2.8.1 so that the info that i
>> provide applies to the current version, also trying out Valgrind.

>> Let me know if you can figure something out with this.

>> 2.8.1 SCENARIO 1

>> GNU gdb (GDB) 7.8.1

>> Copyright (C) 2014 Free Software Foundation, Inc.

>> License GPLv3+: GNU GPL version 3 or later < [ http://gnu.org/licenses/gpl.html
>> | http://gnu.org/licenses/gpl.html ] >

>> This is free software: you are free to change and redistribute it.

>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"

>> and "show warranty" for details.

>> This GDB was configured as "x86_64-poky-linux".

>> Type "show configuration" for configuration details.

>> For bug reporting instructions, please see:

>> < [ http://www.gnu.org/software/gdb/bugs/ |
>> http://www.gnu.org/software/gdb/bugs/ ] >.

>> Find the GDB manual and other documentation resources online at:

>> < [ http://www.gnu.org/software/gdb/documentation/ |
>> http://www.gnu.org/software/gdb/documentation/ ] >.

>> For help, type "help".

>> Type "apropos word" to search for commands related to "word"...

>> Reading symbols from /usr/bin/lttng-sessiond...(no debugging symbols
>> found)...done.

>> [New LWP 1134]

>> [New LWP 1090]

>> [New LWP 1091]

>> [New LWP 1092]

>> [New LWP 1093]

>> [New LWP 1095]

>> [New LWP 1098]

>> [New LWP 1102]

>> [New LWP 1110]

>> [New LWP 1096]

>> [New LWP 1094]

>> [New LWP 1097]

>> warning: Could not load shared library symbols for linux-vdso.so.1.

>> Do you need "set solib-search-path" or "set sysroot"?

>> [Thread debugging using libthread_db enabled]

>> Using host libthread_db library "/lib64/libthread_db.so.1".

>> Core was generated by `/usr/bin/lttng-sessiond --consumerd32-path
>> /usr/lib/lttng/libexec/lttng-consume'.

>> Program terminated with signal SIGSEGV, Segmentation fault.

>> #0 0x0000000000000000 in ?? ()

>> (gdb) bt

>> Python Exception <type 'exceptions.ImportError'> No module named gdb.frames:

>> #0 0x0000000000000000 in ?? ()

>> #1 0x00007f1b39b1db6b in lttng_cds_lfht_lookup () from
>> /usr/lib64/liblttng-ctl.so.0

>> #2 0x0000000000412217 in ?? ()

>> #3 0x0000000000412263 in ?? ()

>> #4 0x0000000000427de8 in ?? ()

>> #5 0x0000000000427ed1 in ?? ()

>> #6 0x00000000004280d3 in ?? ()

>> #7 0x00007f1b398facad in ?? () from /usr/lib64/liburcu.so.2

>> #8 0x00007f1b3893d294 in start_thread (arg=0x7f1b1e7fc700) at
>> pthread_create.c:336

>> #9 0x00007f1b3867b13d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

>> (gdb) q

>> 2.8.1 SCENARIO 2

>> GNU gdb (GDB) 7.8.1

>> Copyright (C) 2014 Free Software Foundation, Inc.

>> License GPLv3+: GNU GPL version 3 or later < [ http://gnu.org/licenses/gpl.html
>> | http://gnu.org/licenses/gpl.html ] >

>> This is free software: you are free to change and redistribute it.

>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"

>> and "show warranty" for details.

>> This GDB was configured as "x86_64-poky-linux".

>> Type "show configuration" for configuration details.

>> For bug reporting instructions, please see:

>> < [ http://www.gnu.org/software/gdb/bugs/ |
>> http://www.gnu.org/software/gdb/bugs/ ] >.

>> Find the GDB manual and other documentation resources online at:

>> < [ http://www.gnu.org/software/gdb/documentation/ |
>> http://www.gnu.org/software/gdb/documentation/ ] >.

>> For help, type "help".

>> Type "apropos word" to search for commands related to "word"...

>> Reading symbols from /usr/bin/lttng-sessiond...(no debugging symbols
>> found)...done.

>> [New LWP 637]

>> [New LWP 610]

>> [New LWP 613]

>> [New LWP 624]

>> [New LWP 605]

>> [New LWP 608]

>> [New LWP 607]

>> [New LWP 609]

>> [New LWP 612]

>> [New LWP 606]

>> [New LWP 611]

>> [New LWP 616]

>> warning: Could not load shared library symbols for linux-vdso.so.1.

>> Do you need "set solib-search-path" or "set sysroot"?

>> [Thread debugging using libthread_db enabled]

>> Using host libthread_db library "/lib64/libthread_db.so.1".

>> Core was generated by `/usr/bin/lttng-sessiond --consumerd32-path
>> /usr/lib/lttng/libexec/lttng-consume'.

>> Program terminated with signal SIGSEGV, Segmentation fault.

>> #0 0x0000000000000020 in ?? ()

>> (gdb) bt

>> Python Exception <type 'exceptions.ImportError'> No module named gdb.frames:

>> #0 0x0000000000000020 in ?? ()

>> #1 0x00007ff010197e02 in lttng_ht_lookup () from /usr/lib64/liblttng-ctl.so.0

>> #2 0x0000000000412217 in ?? ()

>> #3 0x0000000000412263 in ?? ()

>> #4 0x0000000000427de8 in ?? ()

>> #5 0x0000000000427ed1 in ?? ()

>> #6 0x00000000004280d3 in ?? ()

>> #7 0x00007ff00ff76cad in ?? () from /usr/lib64/liburcu.so.2

>> #8 0x00007ff00efb9294 in start_thread (arg=0x7ff0096c5700) at
>> pthread_create.c:336

>> #9 0x00007ff00ecf713d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

>> (gdb) q

>> Regards,
>> Aravind.

>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev at lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20161007/b4234a58/attachment-0001.html>