[lttng-dev] 答复: lttng list -u coredump

Mathieu Desnoyers mathieu.desnoyers at efficios.com
Mon Oct 21 05:20:46 EDT 2013 

Hi Zhenyu, 

Added lttng-dev mailing list to CC. 

----- Original Message -----

> From: "zhenyu.ren" <zhenyu.ren at aliyun.com>
> To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
> Sent: Monday, October 21, 2013 9:41:39 AM
> Subject: 答复: lttng list -u coredump

> Hi,Mathieu
> I have known why lttng-sessiond coredumps. Suppose lfhash table to be empty :

> int ust_app_list_events(struct lttng_event **events)
> ...
> cds_lfht_for_each_entry(ust_app_ht->ht, &iter.iter, app, pid_n.node) {
> ...

> #define cds_lfht_for_each_entry(ht, iter, pos, member)\
> for (cds_lfht_first(ht, iter),\
> pos = caa_container_of(cds_lfht_iter_get_node(iter), \
> __typeof__(*(pos)), member);\
> &(pos)->member != NULL;\

> ...

> If cds_lfht_iter_get_node(iter) returns NULL,then pos points to an unvalid
> value ,this is where core happens.

The NULL check will ensure that this "invalid" value is never actually dereferenced: 

whenever cds_lfht_first() populates the iterator "iter" with a NULL node, the "pos" returned by caa_container_of() will be a very large pointer value which corresponds to a structure where the lfht node fields is the NULL pointer. Therefore, the &(pos)->member != NULL check will break out of the loop immediately, 

For the backtrace I see below, it appears that there might be mixup with respect to struct ust_app hash table node usage somewhere. I don't see where for the moment. David, any idea ? 

could be a mixup between: 
struct lttng_ht_node_ulong pid_n; 
struct lttng_ht_node_ulong sock_n; 
struct lttng_ht_node_ulong notify_sock_n; 
causing the start of loop not to succeed its NULL pointer check. 

Please provide details about the lttng-tools, lttng-ust, lttng-modules, userspace RCU versions you are using. 

Thanks, 

Mathieu 

> thanks
> zhenyu.ren

> ------------------------------------------------------------------
> 发件人：Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
> 发送时间：2013年10月17日(星期四) 23:27
> 收件人：zhenyu.ren <zhenyu.ren at aliyun.com>
> 主 题：Re: lttng list -u coredump

> Please ask this on lttng-dev at lists.lttng.org, or open a bug report report on
> bugs.lttng.org, detailing versions used and detailing the error you
> encounter.

> Thanks,

> Mathieu

> ----- Original Message -----

> > From: "zhenyu.ren" <zhenyu.ren at aliyun.com>
> 
> > To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
> 
> > Sent: Thursday, October 17, 2013 4:56:52 AM
> 
> > Subject: lttng list -u coredump
> 

> > Hi,Mathieu
> 
> > regardless or not running as root,lttng list -u coredumps
> 
> > #lttng list -u
> 
> > Spawning a session daemon
> 
> > FATAL: Module lttng_tracer not found.
> 
> > Error: Unable to load module lttng-tracer
> 
> > Error: Unable to remove module lttng-tracer
> 
> > Warning: No kernel tracer available
> 
> > Warning: No tracing group detected
> 
> > Error: Session daemon died (exit status 0)
> 
> > UST events:
> 
> > -------------
> 
> > None
> 

> > *** glibc detected *** lttng: free(): invalid pointer: 0x000000000061ad00
> > ***
> 
> > ======= Backtrace: =========
> 
> > /lib64/libc.so.6[0x34b64722ef]
> 
> > /lib64/libc.so.6(cfree+0x4b)[0x34b647273b]
> 
> > lttng[0x404ef9]
> 
> > lttng[0x40cf6a]
> 
> > /lib64/libc.so.6(__libc_start_main+0xf4)[0x34b641d994]
> 
> > lttng[0x403109]
> 
> > ======= Memory map: ========
> 
> > 00400000-0041a000 r-xp 00000000 08:02 2494453 /usr/local/bin/lttng
> 
> > 00619000-0061b000 rw-p 00019000 08:02 2494453 /usr/local/bin/lttng
> 
> > 0061b000-0063d000 rw-p 00000000 00:00 0 [heap]
> 
> > 34b6000000-34b601c000 r-xp 00000000 08:02 1884196 /lib64/ld-2.5.so
> 
> > 34b621b000-34b621c000 r--p 0001b000 08:02 1884196 /lib64/ld-2.5.so
> 
> > 34b621c000-34b621d000 rw-p 0001c000 08:02 1884196 /lib64/ld-2.5.so
> 
> > 34b6400000-34b654d000 r-xp 00000000 08:02 1884473 /lib64/libc-2.5.so
> 
> > 34b654d000-34b674d000 ---p 0014d000 08:02 1884473 /lib64/libc-2.5.so
> 
> > 34b674d000-34b6751000 r--p 0014d000 08:02 1884473 /lib64/libc-2.5.so
> 
> > 34b6751000-34b6752000 rw-p 00151000 08:02 1884473 /lib64/libc-2.5.so
> 
> > 34b6752000-34b6757000 rw-p 00000000 00:00 0
> 
> > 34b6800000-34b6802000 r-xp 00000000 08:02 1884474 /lib64/libdl-2.5.so
> 
> > 34b6802000-34b6a02000 ---p 00002000 08:02 1884474 /lib64/libdl-2.5.so
> 
> > 34b6a02000-34b6a03000 r--p 00002000 08:02 1884474 /lib64/libdl-2.5.so
> 
> > 34b6a03000-34b6a04000 rw-p 00003000 08:02 1884474 /lib64/libdl-2.5.so
> 
> > 34b6c00000-34b6c16000 r-xp 00000000 08:02 1884478 /lib64/libpthread-2.5.so
> 
> > 34b6c16000-34b6e15000 ---p 00016000 08:02 1884478 /lib64/libpthread-2.5.so
> 
> > 34b6e15000-34b6e16000 r--p 00015000 08:02 1884478 /lib64/libpthread-2.5.so
> 
> > 34b6e16000-34b6e17000 rw-p 00016000 08:02 1884478 /lib64/libpthread-2.5.so
> 
> > 34b6e17000-34b6e1b000 rw-p 00000000 00:00 0
> 
> > 34b7400000-34b7407000 r-xp 00000000 08:02 1884480 /lib64/librt-2.5.so
> 
> > 34b7407000-34b7607000 ---p 00007000 08:02 1884480 /lib64/librt-2.5.so
> 
> > 34b7607000-34b7608000 r--p 00007000 08:02 1884480 /lib64/librt-2.5.so
> 
> > 34b7608000-34b7609000 rw-p 00008000 08:02 1884480 /lib64/librt-2.5.so
> 
> > 34b8400000-34b840d000 r-xp 00000000 08:02 1884371
> > /lib64/libgcc_s-4.1.2-20080825.so.1
> 
> > 34b840d000-34b860d000 ---p 0000d000 08:02 1884371
> > /lib64/libgcc_s-4.1.2-20080825.so.1
> 
> > 34b860d000-34b860e000 rw-p 0000d000 08:02 1884371
> > /lib64/libgcc_s-4.1.2-20080825.so.1
> 
> > 34b9400000-34b9407000 r-xp 00000000 08:02 2493847
> > /usr/lib64/libpopt.so.0.0.0
> 
> > 34b9407000-34b9607000 ---p 00007000 08:02 2493847
> > /usr/lib64/libpopt.so.0.0.0
> 
> > 34b9607000-34b9608000 rw-p 00007000 08:02 2493847
> > /usr/lib64/libpopt.so.0.0.0
> 
> > 37a5e00000-37a5e04000 r-xp 00000000 08:02 1884499 /lib64/libuuid.so.1.2
> 
> > 37a5e04000-37a6003000 ---p 00004000 08:02 1884499 /lib64/libuuid.so.1.2
> 
> > 37a6003000-37a6004000 rw-p 00003000 08:02 1884499 /lib64/libuuid.so.1.2
> 
> > 7f596db29000-7f596db2d000 rw-p 00000000 00:00 0
> 
> > 7f596db2d000-7f596db32000 r-xp 00000000 08:02 2296098
> > /usr/local/lib/liburcu.so.2.0.0
> 
> > 7f596db32000-7f596dd31000 ---p 00005000 08:02 2296098
> > /usr/local/lib/liburcu.so.2.0.0
> 
> > 7f596dd31000-7f596dd32000 rw-p 00004000 08:02 2296098
> > /usr/local/lib/liburcu.so.2.0.0
> 
> > 7f596dd32000-7f596dd33000 rw-p 00000000 00:00 0
> 
> > 7f596dd33000-7f596dd36000 r-xp 00000000 08:02 2296094
> > /usr/local/lib/liburcu-common.so.2.0.0
> 
> > 7f596dd36000-7f596df35000 ---p 00003000 08:02 2296094
> > /usr/local/lib/liburcu-common.so.2.0.0
> 
> > 7f596df35000-7f596df36000 rw-p 00002000 08:02 2296094
> > /usr/local/lib/liburcu-common.so.2.0.0
> 
> > 7f596df49000-7f596df4a000 rw-p 00000000 00:00 0
> 
> > 7f596df4a000-7f596df6d000 r-xp 00000000 08:02 2294055
> > /usr/local/lib/liblttng-ctl.so.0.0.0
> 
> > 7f596df6d000-7f596e16c000 ---p 00023000 08:02 2294055
> > /usr/local/lib/liblttng-ctl.so.0.0.0
> 
> > 7f596e16c000-7f596e16e000 rw-p 00022000 08:02 2294055
> > /usr/local/lib/liblttng-ctl.so.0.0.0
> 
> > 7f596e16e000-7f596e170000 rw-p 00000000 00:00 0
> 
> > 7f596e170000-7f596e171000 r-xp 00000000 08:02 1427686
> > /home/tops/lib/snoopy.so
> 
> > 7f596e171000-7f596e371000 ---p 00001000 08:02 1427686
> > /home/tops/lib/snoopy.so
> 
> > 7f596e371000-7f596e372000 rw-p 00001000 08:02 1427686
> > /home/tops/lib/snoopy.so
> 
> > 7f596e372000-7f596e375000 rw-p 00000000 00:00 0
> 
> > 7fffe92bf000-7fffe92d4000 rw-p 00000000 00:00 0 [stack]
> 
> > 7fffe93ff000-7fffe9400000 r-xp 00000000 00:00 0 [vdso]
> 
> > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
> 
> > Aborted (core dumped)
> 
> > $ls -l core.12535*
> 
> > -rw------- 1 root root 471040 Oct 17 16:46 core.125350
> 
> > -rw------- 1 root root 84475904 Oct 17 16:46 core.125351
> 
> > #file core.12535*
> 
> > core.125350: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV),
> > SVR4-style, from 'lttng'
> 
> > core.125351: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV),
> > SVR4-style, from 'lttng-sessiond'
> 
> > #gdb --core=./core.125351 /usr/local/bin/lttng-sessiond
> 
> > warning: Can't read pathname for load map: Input/output error.
> 
> > Reading symbols from /home/tops/lib/snoopy.so...done.
> 
> > ...
> 
> > Core was generated by `lttng-sessiond --sig-parent --quiet'.
> 
> > Program terminated with signal 11, Segmentation fault.
> 
> > [New process 125356]
> 
> > [New process 125358]
> 
> > [New process 125359]
> 
> > [New process 125361]
> 
> > [New process 125351]
> 
> > [New process 125354]
> 
> > [New process 125357]
> 
> > [New process 125360]
> 
> > [New process 125355]
> 
> > #0 ust_app_list_events (events=0x40a62e10) at ust-app.c:3026
> 
> > 3026 if (!app->compatible) {
> 
> > (gdb) p ust_app_ht
> 
> > $1 = (struct lttng_ht *) 0x65d030
> 
> > (gdb) p ust_app_ht->ht
> 
> > $2 = (struct cds_lfht *) 0x65d0a0
> 
> > (gdb) p *(ust_app_ht->ht)
> 
> > $7 = {max_nr_buckets = 9223372036854775808, mm = 0x7f297bd6ac20, flavor =
> > 0x7f297b9445e0, count = 0, resize_mutex = {__data = {__lock = 0, __count =
> > 0, __owner = 0,
> 
> > __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next =
> > 0x0}}, __size = '\0' <repeats 39 times>, __align = 0}, resize_attr = 0x0,
> 
> > in_progress_resize = 0, in_progress_destroy = 0, resize_target = 4,
> > resize_initiated = 0, flags = 3, min_alloc_buckets_order = 0,
> > min_nr_alloc_buckets = 1,
> 
> > split_count = 0x65d340, size = 4, bucket_at = 0x7f297bb5aa50 <bucket_at>,
> > {tbl_order = {0x65d050, 0x65e350, 0x65e370, 0x0 <repeats 61 times>},
> > tbl_chunk = 0x65d130,
> 
> > tbl_mmap = 0x65d050}}
> 
> > (gdb) p app
> 
> > $8 = (struct ust_app *) 0xffffffffffffff88
> 
> > Since rcu hash table is unkown to me now,Could you please give me any
> > advices
> > on how to look at this issue?
> 
> > thanks in advance
> 
> > zhenyu.ren
> 

> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20131021/811f9809/attachment-0001.html>

More information about the lttng-dev mailing list