[lttng-dev] lttng sessiond hash table crashes.

Fri Oct 7 10:46:00 UTC 2016

Hi,

I am encountering a few crashes which are all happening around the hash
table functions, the below are from my 2.6 version for which I have
reproduced them with debug symbols enabled.

*2.6 SCENARIO 1*

(gdb) bt

Python Exception <type 'exceptions.ImportError'> No module named gdb.frames:

#0  lttng_ht_lookup (ht=0x6888f0, key=key at entry=0x4b,
iter=iter at entry=0x7f318fffeae0)
at hashtable.c:236

#1  0x0000000000429900 in ust_app_unregister (sock=sock at entry=75) at
ust-app.c:3105

#2  0x000000000043918a in thread_manage_apps (data=<optimized out>) at
main.c:1558

#3  0x00007f3197c74294 in start_thread (arg=0x7f318ffff700) at
pthread_create.c:336

#4  0x00007f31979b213d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109

*2.6 **SCENARIO 2 : Not sure why cmd_init() gets called here *

#0  0x00007f024d68fd2c in lttng_cds_lfht_lookup (ht=0x68ac90,
hash=8123239074643410387, match=0x418be2 <ht_match_reg_uid>,
key=0x7f023affb8a0, iter=0x7f023affb900) at rculfhash.c:1535

#1  0x0000000000418be2 in cmd_init () at cmd.c:3332

#2  0x00000000004191e3 in buffer_reg_uid_find (session_id=<optimized out>,
bits_per_long=64, uid=0) at buffer-registry.c:198

#3  0x000000000042df89 in ust_consumer_metadata_request
(socket=socket at entry=0x67f1f0
<ustconsumer64_data+176>) at ust-consumer.c:464

#4  0x000000000043b0e6 in thread_manage_consumer (data=0x67f140
<ustconsumer64_data>) at main.c:1353

#5  0x00007f024c4ae294 in start_thread (arg=0x7f023affd700) at
pthread_create.c:336

#6  0x00007f024c1ec13d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109

(gdb)

*2.6 **SCENARIO 3*

(gdb) bt

#0  0x00007f444c017740 in ?? ()

#1  0x00007f44676a9b6b in bucket_at (index=<optimized out>,
ht=0x7f444c00d4e0) at rculfhash.c:853

#2  lookup_bucket (hash=4405530133935945645, size=<optimized out>,
ht=0x7f444c00d4e0) at rculfhash.c:861

#3  lttng_cds_lfht_lookup (ht=0x7f444c00d4e0, hash=4405530133935945645,
match=0x7f44676a7980 <match_ulong>, key=0x29, iter=iter at entry=0x7f44613d59d0)
at rculfhash.c:1542

#4  0x00007f44676a7e20 in lttng_ht_lookup (ht=<optimized out>,
key=<optimized out>, iter=iter at entry=0x7f44613d59d0) at hashtable.c:240

#5  0x0000000000412217 in consumer_find_socket (key=<optimized out>,
consumer=<optimized out>) at consumer.c:359

#6  0x0000000000000000 in ?? ()

I am also trying to upgrade to 2.8.1 and see similar crashes there,
unfortunately do not have debug symbols enabled, but crashes always in one
of the hash table related functions.
I'm in the process of enabling debug symbols on 2.8.1 so that the info that
i provide applies to the current version, also trying out Valgrind.

Let me know if you can figure something out with this.

*2.8.1 SCENARIO 1*

GNU gdb (GDB) 7.8.1

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-poky-linux".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from /usr/bin/lttng-sessiond...(no debugging symbols
found)...done.

[New LWP 1134]

[New LWP 1090]

[New LWP 1091]

[New LWP 1092]

[New LWP 1093]

[New LWP 1095]

[New LWP 1098]

[New LWP 1102]

[New LWP 1110]

[New LWP 1096]

[New LWP 1094]

[New LWP 1097]

warning: Could not load shared library symbols for linux-vdso.so.1.

Do you need "set solib-search-path" or "set sysroot"?

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

Core was generated by `/usr/bin/lttng-sessiond --consumerd32-path
/usr/lib/lttng/libexec/lttng-consume'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0  0x0000000000000000 in ?? ()

(gdb) bt

Python Exception <type 'exceptions.ImportError'> No module named
gdb.frames:

#0  0x0000000000000000 in ?? ()

#1  0x00007f1b39b1db6b in lttng_cds_lfht_lookup () from
/usr/lib64/liblttng-ctl.so.0

#2  0x0000000000412217 in ?? ()

#3  0x0000000000412263 in ?? ()

#4  0x0000000000427de8 in ?? ()

#5  0x0000000000427ed1 in ?? ()

#6  0x00000000004280d3 in ?? ()

#7  0x00007f1b398facad in ?? () from /usr/lib64/liburcu.so.2

#8  0x00007f1b3893d294 in start_thread (arg=0x7f1b1e7fc700) at
pthread_create.c:336

#9  0x00007f1b3867b13d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109

(gdb) q

*2.8.1 SCENARIO 2*

GNU gdb (GDB) 7.8.1

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>