[lttng-dev] core seen in relayd_cleanup()

[Aravind HT]

Hi,


I was in the processes of upgrading to 2.8.1 and see the below relayd core.
I am trying to get the logs for this but is proving hard to reproduce this
scenario with full logs enabled and also as this is happening in a complex
environment, not sure why relayd is exiting.

For the meantime, from https://lwn.net/Articles/573432/  , I see that
cds_lfht_destroy() may fail in case the ht is not empty. Should the
assert() be there in relayd_cleanup() or do you know what can
cause/simulate this crash ?
Ive tried killing relayd with SIGINT, SIGTERM when sesions are being active
and that doesnt reproduce this.

55      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7fd7e7c0e980 (LWP 29366))]
(gdb) info threads
  Id   Target Id         Frame
  2    Thread 0x7fd7e1b1c700 (LWP 29375) 0x00007fd7e6701996 in _int_free
(av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:4059
* 1    Thread 0x7fd7e7c0e980 (LWP 29366) 0x00007fd7e66bd367 in __GI_raise
(sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
(gdb) bt full
#0  0x00007fd7e66bd367 in __GI_raise (sig=sig at entry=6) at
../sysdeps/unix/sysv/linux/raise.c:55
        resultvar = 0
        pid = 29366
        selftid = 29366
#1  0x00007fd7e66c033a in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x4, sa_sigaction =
0x4}, sa_mask = {__val = {5, 48, 140732823353744, 3924933264, 0, 0, 0,
21474836480, 140732823353896, 140565261773123,
              140732823353696, 140565261807824, 140565261789000,
140732823354144, 140565282869248, 140565261789000}}, sa_flags = 4462201,
sa_restorer = 0x4416c0 <__PRETTY_FUNCTION__.5555>}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007fd7e66b644d in __assert_fail_base (fmt=0x7fd7e67f2748
"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion at entry=0x441679
"!ret", file=file at entry=0x4415da "hashtable.c",
    line=line at entry=162, function=function at entry=0x4416c0
<__PRETTY_FUNCTION__.5555> "lttng_ht_destroy") at assert.c:92
        str = 0x65b310 ""
        total = 4096
#3  0x00007fd7e66b6502 in __GI___assert_fail (assertion=0x441679 "!ret",
file=0x4415da "hashtable.c", line=162, function=0x4416c0
<__PRETTY_FUNCTION__.5555> "lttng_ht_destroy") at assert.c:101
No locals.
#4  0x0000000000422f17 in lttng_ht_destroy (ht=0x65a0e0) at hashtable.c:162
        ret = -1
        __PRETTY_FUNCTION__ = "lttng_ht_destroy"
#5  0x000000000040803b in relayd_cleanup () at main.c:497
        __func__ = "relayd_cleanup"
#6  0x000000000040ec4c in main (argc=5, argv=0x7ffee9f1bd68) at main.c:2939
        ret = 0
        retval = 0
        status = 0x0
        __func__ = "main"

(gdb) f 4
#4  0x0000000000422f17 in lttng_ht_destroy (ht=0x65a0e0) at hashtable.c:162
162     hashtable.c: No such file or directory.
(gdb) p *ht->ht
$1 = {max_nr_buckets = 9223372036854775808, mm = 0x7fd7e6e50920
<cds_lfht_mm_order>,
flavor = 0x7fd7e77b6340 <rcu_flavor_memb>, count = -1024, resize_mutex =
{__data = {
__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0,
      __elision = 0, __list = {__prev = 0x0, __next = 0x0}},
    __size = '\000' <repeats 39 times>, __align = 0}, resize_attr = 0x0,
  in_progress_resize = 0, in_progress_destroy = 1, resize_target = 2048,
  resize_initiated = 0, flags = 3, min_alloc_buckets_order = 0,
min_nr_alloc_buckets = 1,
  split_count = 0x65b170, .., bucket_at = 0x7fd7e6c4e080 <bucket_at>, {
    tbl_order = {0x65b280, 0x65b2a0, 0x65b2c0, 0x7fd7c40008c0,
0x7fd7c4000910,
      0x7fd7c40009a0, 0x7fd7c4000ab0, 0x7fd7c4001070, 0x7fd7c4001480,
0x7fd7c4001c90,
      0x7fd7c4003350, 0x7fd7c4005360, 0x0 <repeats 52 times>}, tbl_chunk =
0x65af60,
    tbl_mmap = 0x65b280}}


Regards,
Aravind.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20161130/77678311/attachment.html>