<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>Hi Zhenyu,<br></div><div><br></div><div>Added lttng-dev mailing list to CC.<br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"zhenyu.ren" <zhenyu.ren@aliyun.com><br><b>To: </b>"Mathieu Desnoyers" <mathieu.desnoyers@efficios.com><br><b>Sent: </b>Monday, October 21, 2013 9:41:39 AM<br><b>Subject: </b>答复: lttng list -u coredump<br><div><br></div><div class="__aliyun_email_body_block"><div style="" data-mce-style="">Hi,Mathieu</div><div style="" data-mce-style=""> I have known why lttng-sessiond coredumps. Suppose lfhash table to be empty :</div><div style="" data-mce-style=""><br></div><div style="" data-mce-style="">int ust_app_list_events(struct lttng_event **events)</div><div style="" data-mce-style="">...</div><div style="" data-mce-style="">cds_lfht_for_each_entry(ust_app_ht->ht, &iter.iter, app, pid_n.node) {</div><div style="" data-mce-style="">...</div><div style="" data-mce-style=""><br></div><div style="" data-mce-style=""><br></div><div class="__aliyun_signature_wrap"><div><div>#define cds_lfht_for_each_entry(ht, iter, pos, member)\</div><div>for (cds_lfht_first(ht, iter),\</div><div>pos = caa_container_of(cds_lfht_iter_get_node(iter), \</div><div>__typeof__(*(pos)), member);\</div><div>&(pos)->member != NULL;\</div><div><br></div></div><div>...</div><div><br></div><div>If cds_lfht_iter_get_node(iter) returns NULL,then pos points to an unvalid value ,this is where core happens.</div></div></div></blockquote><div><br></div><div>The NULL check will ensure that this "invalid" value is never actually dereferenced:<br></div><div><br></div><div>whenever cds_lfht_first() populates the iterator "iter" with a NULL node, the "pos" returned by caa_container_of() will be a very large pointer value which corresponds to a structure where the lfht node fields is the NULL pointer. Therefore, the &(pos)->member != NULL check will break out of the loop immediately,<br></div><div><br></div><div>For the backtrace I see below, it appears that there might be mixup with respect to struct ust_app hash table node usage somewhere. I don't see where for the moment. David, any idea ?<br></div><div><br></div><div>could be a mixup between:<br></div><div> struct lttng_ht_node_ulong pid_n;<br> struct lttng_ht_node_ulong sock_n;<br> struct lttng_ht_node_ulong notify_sock_n;<br></div><div>causing the start of loop not to succeed its NULL pointer check.<br></div><div><br></div><div>Please provide details about the lttng-tools, lttng-ust, lttng-modules, userspace RCU versions you are using.<br></div><div><br></div><div>Thanks,<br></div><div><br></div><div>Mathieu<br></div><div><br></div><div><br></div><div><br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><div class="__aliyun_email_body_block"><div class="__aliyun_signature_wrap"><div><br></div><div>thanks</div><div>zhenyu.ren</div><br><div><br></div></div><div class="__aliyun_previous_quote">------------------------------------------------------------------<br>发件人：Mathieu Desnoyers <mathieu.desnoyers@efficios.com><br>发送时间：2013年10月17日(星期四) 23:27<br>收件人：zhenyu.ren <zhenyu.ren@aliyun.com><br>主 题：Re: lttng list -u coredump<br><div><br></div><div style="font-family: times new roman , new york , times , serif;font-size: 12.0pt;color: #000000;" data-mce-style="font-family: times new roman , new york , times , serif; font-size: 12.0pt; color: #000000;"><div>Please ask this on lttng-dev@lists.lttng.org, or open a bug report report on bugs.lttng.org, detailing versions used and detailing the error you encounter.<br></div><div><br></div><div>Thanks,<br></div><div><br></div><div>Mathieu<br></div><div><br></div><hr id="zwchr"><blockquote style="border-left: 2.0px solid #1010ff;margin-left: 5.0px;padding-left: 5.0px;color: #000000;font-weight: normal;font-style: normal;text-decoration: none;font-family: helvetica , arial , sans-serif;font-size: 12.0pt;" data-mce-style="border-left: 2.0px solid #1010ff; margin-left: 5.0px; padding-left: 5.0px; color: #000000; font-weight: normal; font-style: normal; text-decoration: none; font-family: helvetica , arial , sans-serif; font-size: 12.0pt;"><b>From: </b>"zhenyu.ren" <zhenyu.ren@aliyun.com><br><b>To: </b>"Mathieu Desnoyers" <mathieu.desnoyers@efficios.com><br><b>Sent: </b>Thursday, October 17, 2013 4:56:52 AM<br><b>Subject: </b>lttng list -u coredump<br><div><br></div><div class="__aliyun_email_body_block"><div>Hi,Mathieu</div><div> regardless or not running as root,lttng list -u coredumps</div><div>#lttng list -u<br>Spawning a session daemon<br>FATAL: Module lttng_tracer not found.<br>Error: Unable to load module lttng-tracer<br>Error: Unable to remove module lttng-tracer<br>Warning: No kernel tracer available<br>Warning: No tracing group detected<br>Error: Session daemon died (exit status 0)<br>UST events:<br>-------------<br>None<br><div><br></div>*** glibc detected *** lttng: free(): invalid pointer: 0x000000000061ad00 ***<br>======= Backtrace: =========<br>/lib64/libc.so.6[0x34b64722ef]<br>/lib64/libc.so.6(cfree+0x4b)[0x34b647273b]<br>lttng[0x404ef9]<br>lttng[0x40cf6a]<br>/lib64/libc.so.6(__libc_start_main+0xf4)[0x34b641d994]<br>lttng[0x403109]<br>======= Memory map: ========<br>00400000-0041a000 r-xp 00000000 08:02 2494453 /usr/local/bin/lttng<br>00619000-0061b000 rw-p 00019000 08:02 2494453 /usr/local/bin/lttng<br>0061b000-0063d000 rw-p 00000000 00:00 0 [heap]<br>34b6000000-34b601c000 r-xp 00000000 08:02 1884196 /lib64/ld-2.5.so<br>34b621b000-34b621c000 r--p 0001b000 08:02 1884196 /lib64/ld-2.5.so<br>34b621c000-34b621d000 rw-p 0001c000 08:02 1884196 /lib64/ld-2.5.so<br>34b6400000-34b654d000 r-xp 00000000 08:02 1884473 /lib64/libc-2.5.so<br>34b654d000-34b674d000 ---p 0014d000 08:02 1884473 /lib64/libc-2.5.so<br>34b674d000-34b6751000 r--p 0014d000 08:02 1884473 /lib64/libc-2.5.so<br>34b6751000-34b6752000 rw-p 00151000 08:02 1884473 /lib64/libc-2.5.so<br>34b6752000-34b6757000 rw-p 00000000 00:00 0<br>34b6800000-34b6802000 r-xp 00000000 08:02 1884474 /lib64/libdl-2.5.so<br>34b6802000-34b6a02000 ---p 00002000 08:02 1884474 /lib64/libdl-2.5.so<br>34b6a02000-34b6a03000 r--p 00002000 08:02 1884474 /lib64/libdl-2.5.so<br>34b6a03000-34b6a04000 rw-p 00003000 08:02 1884474 /lib64/libdl-2.5.so<br>34b6c00000-34b6c16000 r-xp 00000000 08:02 1884478 /lib64/libpthread-2.5.so<br>34b6c16000-34b6e15000 ---p 00016000 08:02 1884478 /lib64/libpthread-2.5.so<br>34b6e15000-34b6e16000 r--p 00015000 08:02 1884478 /lib64/libpthread-2.5.so<br>34b6e16000-34b6e17000 rw-p 00016000 08:02 1884478 /lib64/libpthread-2.5.so<br>34b6e17000-34b6e1b000 rw-p 00000000 00:00 0<br>34b7400000-34b7407000 r-xp 00000000 08:02 1884480 /lib64/librt-2.5.so<br>34b7407000-34b7607000 ---p 00007000 08:02 1884480 /lib64/librt-2.5.so<br>34b7607000-34b7608000 r--p 00007000 08:02 1884480 /lib64/librt-2.5.so<br>34b7608000-34b7609000 rw-p 00008000 08:02 1884480 /lib64/librt-2.5.so<br>34b8400000-34b840d000 r-xp 00000000 08:02 1884371 /lib64/libgcc_s-4.1.2-20080825.so.1<br>34b840d000-34b860d000 ---p 0000d000 08:02 1884371 /lib64/libgcc_s-4.1.2-20080825.so.1<br>34b860d000-34b860e000 rw-p 0000d000 08:02 1884371 /lib64/libgcc_s-4.1.2-20080825.so.1<br>34b9400000-34b9407000 r-xp 00000000 08:02 2493847 /usr/lib64/libpopt.so.0.0.0<br>34b9407000-34b9607000 ---p 00007000 08:02 2493847 /usr/lib64/libpopt.so.0.0.0<br>34b9607000-34b9608000 rw-p 00007000 08:02 2493847 /usr/lib64/libpopt.so.0.0.0<br>37a5e00000-37a5e04000 r-xp 00000000 08:02 1884499 /lib64/libuuid.so.1.2<br>37a5e04000-37a6003000 ---p 00004000 08:02 1884499 /lib64/libuuid.so.1.2<br>37a6003000-37a6004000 rw-p 00003000 08:02 1884499 /lib64/libuuid.so.1.2<br>7f596db29000-7f596db2d000 rw-p 00000000 00:00 0<br>7f596db2d000-7f596db32000 r-xp 00000000 08:02 2296098 /usr/local/lib/liburcu.so.2.0.0<br>7f596db32000-7f596dd31000 ---p 00005000 08:02 2296098 /usr/local/lib/liburcu.so.2.0.0<br>7f596dd31000-7f596dd32000 rw-p 00004000 08:02 2296098 /usr/local/lib/liburcu.so.2.0.0<br>7f596dd32000-7f596dd33000 rw-p 00000000 00:00 0<br>7f596dd33000-7f596dd36000 r-xp 00000000 08:02 2296094 /usr/local/lib/liburcu-common.so.2.0.0<br>7f596dd36000-7f596df35000 ---p 00003000 08:02 2296094 /usr/local/lib/liburcu-common.so.2.0.0<br>7f596df35000-7f596df36000 rw-p 00002000 08:02 2296094 /usr/local/lib/liburcu-common.so.2.0.0<br>7f596df49000-7f596df4a000 rw-p 00000000 00:00 0<br>7f596df4a000-7f596df6d000 r-xp 00000000 08:02 2294055 /usr/local/lib/liblttng-ctl.so.0.0.0<br>7f596df6d000-7f596e16c000 ---p 00023000 08:02 2294055 /usr/local/lib/liblttng-ctl.so.0.0.0<br>7f596e16c000-7f596e16e000 rw-p 00022000 08:02 2294055 /usr/local/lib/liblttng-ctl.so.0.0.0<br>7f596e16e000-7f596e170000 rw-p 00000000 00:00 0<br>7f596e170000-7f596e171000 r-xp 00000000 08:02 1427686 /home/tops/lib/snoopy.so<br>7f596e171000-7f596e371000 ---p 00001000 08:02 1427686 /home/tops/lib/snoopy.so<br>7f596e371000-7f596e372000 rw-p 00001000 08:02 1427686 /home/tops/lib/snoopy.so<br>7f596e372000-7f596e375000 rw-p 00000000 00:00 0<br>7fffe92bf000-7fffe92d4000 rw-p 00000000 00:00 0 [stack]<br>7fffe93ff000-7fffe9400000 r-xp 00000000 00:00 0 [vdso]<br>ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]<br>Aborted (core dumped)</div><div>$ls -l core.12535*<br>-rw------- 1 root root 471040 Oct 17 16:46 core.125350<br>-rw------- 1 root root 84475904 Oct 17 16:46 core.125351</div><div>#file core.12535*<br>core.125350: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'lttng'<br>core.125351: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'lttng-sessiond'</div><div>#gdb --core=./core.125351 /usr/local/bin/lttng-sessiond</div><div>warning: Can't read pathname for load map: Input/output error.<br>Reading symbols from /home/tops/lib/snoopy.so...done.</div><div>...</div><div>Core was generated by `lttng-sessiond --sig-parent --quiet'.<br>Program terminated with signal 11, Segmentation fault.<br>[New process 125356]<br>[New process 125358]<br>[New process 125359]<br>[New process 125361]<br>[New process 125351]<br>[New process 125354]<br>[New process 125357]<br>[New process 125360]<br>[New process 125355]<br>#0 ust_app_list_events (events=0x40a62e10) at ust-app.c:3026<br>3026 if (!app->compatible) {</div><div>(gdb) p ust_app_ht<br>$1 = (struct lttng_ht *) 0x65d030<br>(gdb) p ust_app_ht->ht<br>$2 = (struct cds_lfht *) 0x65d0a0</div><div>(gdb) p *(ust_app_ht->ht)<br>$7 = {max_nr_buckets = 9223372036854775808, mm = 0x7f297bd6ac20, flavor = 0x7f297b9445e0, count = 0, resize_mutex = {__data = {__lock = 0, __count = 0, __owner = 0,<br> __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\0' <repeats 39 times>, __align = 0}, resize_attr = 0x0,<br> in_progress_resize = 0, in_progress_destroy = 0, resize_target = 4, resize_initiated = 0, flags = 3, min_alloc_buckets_order = 0, min_nr_alloc_buckets = 1,<br> split_count = 0x65d340, size = 4, bucket_at = 0x7f297bb5aa50 <bucket_at>, {tbl_order = {0x65d050, 0x65e350, 0x65e370, 0x0 <repeats 61 times>}, tbl_chunk = 0x65d130,<br> tbl_mmap = 0x65d050}}</div><div>(gdb) p app<br>$8 = (struct ust_app *) 0xffffffffffffff88</div><div> Since rcu hash table is unkown to me now,Could you please give me any advices on how to look at this issue?</div><div>thanks in advance</div><div>zhenyu.ren</div></div></blockquote><div><br><div><br></div></div><div><br></div><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com<br></div></div></div></div></blockquote><div><br><div><br></div></div><div><br></div><div>-- <br></div><div><span name="x"></span>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com<span name="x"></span><br></div></div></body></html>