<div dir="ltr"><div>Hi,</div><div><br></div><div>We have found a a crash in lttng-consumerd when a live session is running and the lttng-relayd is killed. The crash is from lttng-tools version 2.8.6, but the same problem/code is in newer versions.</div><div>This crash is from slightly modified version of 2.8.6, so the lines may not be correct, but the issue has been reproduced on 2.8.6 w/o patches on an arm target. The problem was introduced with</div><div>8dbd7d838dc2276e5a25057c76c2e219e1d2661b</div><div><br></div><div>(gdb) bt</div><div>#0 lttng_index_file_write (index_file=0x0, element=element@entry=0xb3dfea40) at index.c:132</div><div>#1 0x00024798 in consumer_stream_write_index (stream=stream@entry=0xb3f090b0, element=element@entry=0xb3dfea40) at consumer-stream.c:375</div><div>#2 0x00021fb0 in send_empty_index (stream_id=<optimized out>, ts=<optimized out>, stream=0xb3f090b0) at consumer-timer.c:125</div><div>#3 consumer_flush_ust_index (stream=0xb3f090b0) at consumer-timer.c:246</div><div>#4 0x0002323c in check_ust_stream (stream=0xb3f090b0) at consumer-timer.c:297</div><div>#5 live_timer (ctx=<optimized out>, sig=<optimized out>, si=0xb3dfebd0, uc=0x0) at consumer-timer.c:333</div><div>#6 consumer_timer_thread (data=0x0) at consumer-timer.c:591</div><div>#7 0xb6f000dc in start_thread (arg=0xb3dff340) at pthread_create.c:339</div><div>#8 0xb6e89130 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:89 from /proj/cpptemp/plf_tools/licop-rcs/CXP9031275_4-R9C22/sysroot/lib/libc.so.6</div><div>#9 0xb6e89130 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:89 from /proj/cpptemp/plf_tools/licop-rcs/CXP9031275_4-R9C22/sysroot/lib/libc.so.6</div><div>Backtrace stopped: previous frame identical to this frame (corrupt stack?)</div><div><br></div><div><br></div><div>The problem happens when;</div><div><br></div><div>./src/common/index/index.c</div><div>-------------------------------------</div><div>int consumer_stream_write_index(struct lttng_consumer_stream *stream,</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>struct ctf_packet_index *element)</div><div>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>int ret;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>struct consumer_relayd_sock_pair *relayd;</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>assert(stream);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>assert(element);</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>rcu_read_lock();</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>relayd = consumer_find_relayd(stream->net_seq_idx);</div><div>>> relayd is shutdown</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if (relayd) {</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>pthread_mutex_lock(&relayd->ctrl_sock_mutex);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ret = relayd_send_index(&relayd->control_sock, element,</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>stream->relayd_stream_id, stream->next_net_seq_num - 1);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>pthread_mutex_unlock(&relayd->ctrl_sock_mutex);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>} else {</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if (lttng_index_file_write(stream->index_file, element)) {</div><div>>> We get in here, but stream->index_file is set to NULL in consumer_stream_close()</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ret = -1;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>} else {</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ret = 0;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if (ret < 0) {</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>goto error;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}</div><div><br></div><div>src/common/consumer/consumer-stream.c</div><div>---------------------------------------------------------</div><div>void consumer_stream_close(struct lttng_consumer_stream *stream)</div><div>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>int ret;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>struct consumer_relayd_sock_pair *relayd;</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>assert(stream);</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>switch (consumer_data.type) {</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>case LTTNG_CONSUMER_KERNEL:</div><div>.....</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>case LTTNG_CONSUMER32_UST:</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>case LTTNG_CONSUMER64_UST:</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>{</div><div>...</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if (stream->index_file) {</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>lttng_index_file_put(stream->index_file);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>stream->index_file = NULL;</div><div>>> Here is stream->index_file set to NULL</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}</div><div>.....</div><div><br></div><div>The following patch fixes the crash, but it's a "band aid" patch since the flow is </div><div>broken and it only fixes the crash. I was not able to find the correct flow to fix the issue, but maybe the band aid patch should be included anyway!</div><div><br></div><div>From fc896fe08e30435b9d3c78fa4551b2dc5042fb03 Mon Sep 17 00:00:00 2001</div><div>From: Anders Wallin <<a href="mailto:wallinux@gmail.com">wallinux@gmail.com</a>></div><div>Date: Thu, 16 Mar 2017 11:15:23 +0100</div><div>Subject: [PATCH lttng-tools] Fix: crash in lttng-consumerd when lttng-relayd</div><div> is killed</div><div><br></div><div>Fixes this crash:</div><div>0 lttng_index_file_write (index_file=0x0, element=element@entry=0xb3dfea40) at index.c:132</div><div>1 0x00024798 in consumer_stream_write_index (stream=stream@entry=0xb3f090b0, element=element@entry=0xb3dfea40) at consumer-stream.c:375</div><div>2 0x00021fb0 in send_empty_index (stream_id=<optimized out>, ts=<optimized out>, stream=0xb3f090b0) at consumer-timer.c:125</div><div>3 consumer_flush_ust_index (stream=0xb3f090b0) at consumer-timer.c:246</div><div>4 0x0002323c in check_ust_stream (stream=0xb3f090b0) at consumer-timer.c:297</div><div>5 live_timer (ctx=<optimized out>, sig=<optimized out>, si=0xb3dfebd0, uc=0x0) at consumer-timer.c:333</div><div>6 consumer_timer_thread (data=0x0) at consumer-timer.c:591</div><div><br></div><div>Signed-off-by: Anders Wallin <<a href="mailto:wallinux@gmail.com">wallinux@gmail.com</a>></div><div>---</div><div> src/common/index/index.c | 12 ++++++++++--</div><div> 1 file changed, 10 insertions(+), 2 deletions(-)</div><div><br></div><div>diff --git a/src/common/index/index.c b/src/common/index/index.c</div><div>index b481badb..008d877b 100644</div><div>--- a/src/common/index/index.c</div><div>+++ b/src/common/index/index.c</div><div>@@ -129,11 +129,19 @@ int lttng_index_file_write(const struct lttng_index_file *index_file,</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span>const struct ctf_packet_index *element)</div><div> {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ssize_t ret;</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>int fd = index_file->fd;</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>size_t len = index_file->element_len;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>int fd;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>size_t len;</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span>assert(element);</div><div> </div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if (index_file == NULL) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>PERROR("index file is NULL");</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>goto error;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>fd = index_file->fd;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>len = index_file->element_len;</div><div>+</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if (fd < 0) {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span>goto error;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}</div><div><br></div><div>-- </div><div>2.11.0</div><div><br></div><div><div class="gmail_signature">Anders Wallin</div></div>
</div>