<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello all,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
My name is Bogdan Codres from Wind River.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Recently, we received a crash from one of our customer. <b>This happened only once</b></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<b>and we do not have a clear path on how to reproduce this</b>.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The crash happened on <span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important">
ARMv7</span> and the version of <span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important">
lttng-tools was 2.12.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important">This
is the backtrace of the crash:</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important">
<pre style="margin:0px;padding:9px 12px;max-height:30em;overflow:auto;font-size:12px;orphans:2;widows:2;background-color:rgb(244, 245, 247)">(gdb) bt
#0 __libc_do_syscall () at libc-do-syscall.S:49
#1 0xb6e13ad4 in __libc_signal_restore_set (set=0xb39f94e0) at ../sysdeps/unix/sysv/linux/internal-signals.h:84
#2 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:48
#3 0xb6e061a6 in __GI_abort () at abort.c:79
#4 0xb6e0ed90 in __assert_fail_base (fmt=0xb6ebfed0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x517e10 "!stream->trace_chunk", assertion@entry=0xb39fe300 "\001", file=0x51d844 "../../../../git/src/common/ust-consumer/ust-consumer.c", file@entry=0x0,
line=1124, line@entry=5363780, function=function@entry=0x51d234 <__PRETTY_FUNCTION__.15949> "snapshot_channel") at assert.c:92
#5 0xb6e0ee0e in __GI___assert_fail (assertion=0xb39fe300 "\001", file=0x0, line=5363780, line@entry=1124, function=0x51d234 <__PRETTY_FUNCTION__.15949> "snapshot_channel") at assert.c:101
#6 0x004f5840 in snapshot_channel (channel=0xb42008d0, key=1, path=path@entry=0xb39f9964 "ust/uid/0/32-bit", relayd_id=relayd_id@entry=18446744073709551615, nb_packets_per_stream=0, ctx=ctx@entry=0x544048) at ../../../../git/src/common/ust-consumer/ust-consumer.c:1124
#7 0x004f9a08 in lttng_ustconsumer_recv_cmd (ctx=0x544048, sock=30, consumer_sockpoll=<optimized out>) at ../../../../git/src/common/ust-consumer/ust-consumer.c:1790
#8 0x004dfac0 in consumer_thread_sessiond_poll (data=0x544048) at ../../../../git/src/common/consumer/consumer.c:3361
#9 0xb6ee7b00 in start_thread (arg=0x98396ec3) at pthread_create.c:486
#10 0xb6e853bc in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /sysroots/armv7at2-neon-wrs-linux-gnueabi/lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)</pre>
<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important">
<p style="margin:10px 0px 0px;orphans:2;widows:2;background-color:rgb(255, 255, 255)">
There's an assert( ! stream->trace_chunk ) which fails i.e. the stream trace_chunk exists.</p>
<p style="margin:10px 0px 0px;orphans:2;widows:2;background-color:rgb(255, 255, 255)">
There is a comment for the function, saying "the caller must take RCU read side lock and channel lock".<br>
The RCU read side lock is taken by the snapshot_channel, but from what I could see, nothing seems to take the channel lock in the functions calling the snapshot_channel.</p>
<p style="margin:10px 0px 0px;orphans:2;widows:2;background-color:rgb(255, 255, 255)">
As this crash looked like a race condition, and if the comments in the function are correct and the channel lock is missing,
<br>
</p>
<p style="margin:10px 0px 0px;orphans:2;widows:2;background-color:rgb(255, 255, 255)">
it could indeed be a race condition, and therefore I wondered if anyone else has seen it.</p>
<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important">I
did some source code investigation and I saw that in <br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important"><i><b><span style="background-color: rgb(244, 245, 247); display: inline !important; color: rgb(237, 92, 87);">lttng_kconsumer_recv_cmd
which have a similar structure</span></b></i></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important"><span style="orphans:2;widows:2;background-color:rgb(244, 245, 247);display:inline !important"><br>
</span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important"><span style="orphans:2;widows:2;background-color:rgb(244, 245, 247);display:inline !important">like
the lttng_ustconsumer_recv_cmd ... --><span> </span></span><b style="orphans:2;widows:2;background-color:rgb(244, 245, 247)">we see pthread_mutex_lock(&channel>lock)</b><span style="orphans:2;widows:2;background-color:rgb(244, 245, 247);display:inline !important">;
---> in LTTNG_CONSUMER_SNAPSHOT_CHANNE</span>L<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important">
<pre style="margin:0px;max-height:30em;overflow:auto;font-size:12px;orphans:2;widows:2;background-color:rgb(244, 245, 247)"><span style="color:rgb(145, 0, 145)">else</span> {
<span style="color: rgb(237, 92, 87);"><b> pthread_mutex_lock(&channel->lock);</b></span>
<span style="color:rgb(145, 0, 145)">if</span> (msg.u.snapshot_channel.metadata == 1) {
ret = lttng_kconsumer_snapshot_metadata(channel, key,
msg.u.snapshot_channel.pathname,
msg.u.snapshot_channel.relayd_id, ctx);
<span style="color:rgb(145, 0, 145)">if</span> (ret < 0)
{ ERR(<span style="color:rgb(0, 145, 0)">"Snapshot metadata failed"</span>); ret_code = LTTCOMM_CONSUMERD_SNAPSHOT_FAILED; }
} <span style="color:rgb(145, 0, 145)">else</span> {
ret = lttng_kconsumer_snapshot_channel(channel, key,
msg.u.snapshot_channel.pathname,
msg.u.snapshot_channel.relayd_id,
msg.u.snapshot_channel.nb_packets_per_stream,
ctx);
<span style="color:rgb(145, 0, 145)">if</span> (ret < 0)
{ ERR(<span style="color:rgb(0, 145, 0)">"Snapshot channel failed"</span>); ret_code = LTTCOMM_CONSUMERD_SNAPSHOT_FAILED; }
}
<span style="color: rgb(237, 92, 87);"><b> pthread_mutex_unlock(&channel->lock);</b></span></pre>
<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important">So,
my question is this: shouldn't be used also in <span style="orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important">
<span style="background-color:rgb(244, 245, 247);display:inline !important">lttng_ustconsumer_recv_cmd</span></span>
<b>a mutex lock for channel</b> like <br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important">it's
used in <span style="orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important">
<i><b><span style="background-color:rgb(244, 245, 247);display:inline !important;color:rgb(237, 92, 87)">lttng_kconsumer_recv_cmd</span></b></i></span> ?<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important">What's
your opinion on this issue ?<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(23, 43, 77);font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;font-size:14px;orphans:2;widows:2;background-color:rgb(255, 255, 255);display:inline !important"></span><br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<p class="MsoNormal" style="margin:0px; color:rgb(34,34,34); font-family:Arial,Helvetica,sans-serif; font-size:small; font-style:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; background-color:rgb(255,255,255)">
<span style="color:rgb(46,116,181)">Best Regards,</span></p>
<p class="MsoNormal" style="margin:0px; color:rgb(34,34,34); font-family:Arial,Helvetica,sans-serif; font-size:small; font-style:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; background-color:rgb(255,255,255)">
<span style="color:rgb(46,116,181)">Ph.D. eng. Bogdan Codres</span></p>
<p class="MsoNormal" style="margin:0px; color:rgb(34,34,34); font-family:Arial,Helvetica,sans-serif; font-size:small; font-style:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255)">
<span style="font-size:10pt; font-family:"Intel Clear",sans-serif; color:rgb(46,116,181)" lang="EN"><i><span style="font-size:10pt; font-family:"Intel Clear",sans-serif; color:rgb(46,116,181)" lang="EN">Senior Engineer</span> at
</i></span><b><span style="font-size:10pt; font-family:"Intel Clear",sans-serif; color:rgb(46,116,181)" lang="EN"><i></i></span><i><span style="font-size:10pt; font-family:"Intel Clear",sans-serif; color:rgb(46,116,181)" lang="EN"><span style="color:rgb(46,116,181); font-size:10pt; font-family:"intel clear",sans-serif; background-color:rgba(0,0,0,0)" lang="de">RDC-EMEA,
Professional Services, Wind River</span></span></i><span style="font-size:10pt; font-family:"Intel Clear",sans-serif; color:rgb(46,116,181)" lang="EN"><i></i></span></b></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>