<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:933053302;
mso-list-type:hybrid;
mso-list-template-ids:1501617316 945345500 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:SimSun;}
@list l0:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I built a trace.so as a wrapper of lttng-ust, which predefines some events and APIs based on lttng-ust.<o:p></o:p></p><p class=MsoNormal>And here is demo application linked to this share library. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Sometimes the demo hung at launch time. I did test with the demo and easy-ust of lttng-ust on both IA32 and X64 and got the same result.<o:p></o:p></p><p class=MsoNormal>Lttng-ust version is 2.02. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I collect some debuging info with gdb here:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Parent process:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>gdb) thread 3 (constructor of liblttng-ust.so)<o:p></o:p></p><p class=MsoNormal>[Switching to thread 3 (Thread 0x7f91d5487950 (LWP 21901))]#0 0x00007f91d82ad400 in wait () from /lib64/libpthread.so.0<o:p></o:p></p><p class=MsoNormal>(gdb) bt<o:p></o:p></p><p class=MsoNormal>#0 0x00007f91d82ad400 in wait () from /lib64/libpthread.so.0<o:p></o:p></p><p class=MsoNormal>#1 0x00007f91d64a8c4b in wait_for_sessiond (sock_info=0x7f91d66cc640) at lttng-ust-comm.c:481<o:p></o:p></p><p class=MsoNormal>#2 0x00007f91d64a9545 in ust_listener_thread (arg=<value optimized out>) at lttng-ust-comm.c:669<o:p></o:p></p><p class=MsoNormal>#3 0x00007f91d82a5650 in start_thread () from /lib64/libpthread.so.0<o:p></o:p></p><p class=MsoNormal>#4 0x00007f91d7b0315d in clone () from /lib64/libc.so.6<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>(gdb) thread 5 (constructor of demo)<o:p></o:p></p><p class=MsoNormal>[Switching to thread 5 (Thread 0x7f91d690e950 (LWP 21899))]#0 0x00007f91d82ac344 in __lll_lock_wait () from /lib64/libpthread.so.0<o:p></o:p></p><p class=MsoNormal>(gdb) bt<o:p></o:p></p><p class=MsoNormal>#0 0x00007f91d82ac344 in __lll_lock_wait () from /lib64/libpthread.so.0<o:p></o:p></p><p class=MsoNormal>#1 0x00007f91d82a72d0 in _L_lock_102 () from /lib64/libpthread.so.0<o:p></o:p></p><p class=MsoNormal>#2 0x00007f91d82a6bbe in pthread_mutex_lock () from /lib64/libpthread.so.0<o:p></o:p></p><p class=MsoNormal>#3 0x00007f91d64abf8b in ltt_probe_register (desc=0x7f91d66d0ce0) at ltt-probes.c:77<o:p></o:p></p><p class=MsoNormal>#4 0x00007f91d66e0cfb in __lttng_events_init__sample_component () at /usr/include/lttng/ust-tracepoint-event.h:550<o:p></o:p></p><p class=MsoNormal>#5 0x00007f91d66e71b6 in __do_global_ctors_aux () from mytrace.so<o:p></o:p></p><p class=MsoNormal>#6 0x00007f91d66e03c3 in _init () from mytrace.so<o:p></o:p></p><p class=MsoNormal>#7 0x00007f91d66df3d4 in ?? () from mytrace.so<o:p></o:p></p><p class=MsoNormal>#8 0x00007f91d94f78d8 in ?? () from /lib64/ld-linux-x86-64.so.2<o:p></o:p></p><p class=MsoNormal>#9 0x00007f91d94f7a07 in ?? () from /lib64/ld-linux-x86-64.so.2<o:p></o:p></p><p class=MsoNormal>#10 0x00007f91d94fbbde in ?? () from /lib64/ld-linux-x86-64.so.2<o:p></o:p></p><p class=MsoNormal>#11 0x00007f91d94f7566 in ?? () from /lib64/ld-linux-x86-64.so.2<o:p></o:p></p><p class=MsoNormal>#12 0x00007f91d94fb38b in ?? () from /lib64/ld-linux-x86-64.so.2<o:p></o:p></p><p class=MsoNormal>#13 0x00007f91d84bbf9b in ?? () from /lib64/libdl.so.2<o:p></o:p></p><p class=MsoNormal>#14 0x00007f91d94f7566 in ?? () from /lib64/ld-linux-x86-64.so.2<o:p></o:p></p><p class=MsoNormal>#15 0x00007f91d84bc34c in ?? () from /lib64/libdl.so.2<o:p></o:p></p><p class=MsoNormal>#16 0x00007f91d84bbf01 in dlopen () from /lib64/libdl.so.2<o:p></o:p></p><p class=MsoNormal>..............................................<o:p></o:p></p><p class=MsoNormal>#24 0x00007f91d82a5650 in start_thread () from /lib64/libpthread.so.0<o:p></o:p></p><p class=MsoNormal>#25 0x00007f91d7b0315d in clone () from /lib64/libc.so.6<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>(gdb) p sessions_mutex<o:p></o:p></p><p class=MsoNormal>$4 = {__data = {__lock = 2, __count = 0, __owner = 21901, __nusers = 1, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, <o:p></o:p></p><p class=MsoNormal> __size = "\002\000\000\000\000\000\000\000\215U\000\000\001", '\0' <repeats 26 times>, __align = 2}<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Child process:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>(gdb) info thread (forked from 21901)<o:p></o:p></p><p class=MsoNormal>* 1 Thread 0x7f91d5487950 (LWP 21902) 0x00007f91d82ac344 in __lll_lock_wait () from /lib64/libpthread.so.0<o:p></o:p></p><p class=MsoNormal>(gdb) bt<o:p></o:p></p><p class=MsoNormal>#0 0x00007f91d82ac344 in __lll_lock_wait () from /lib64/libpthread.so.0<o:p></o:p></p><p class=MsoNormal>#1 0x00007f91d82a72d0 in _L_lock_102 () from /lib64/libpthread.so.0<o:p></o:p></p><p class=MsoNormal>#2 0x00007f91d82a6bbe in pthread_mutex_lock () from /lib64/libpthread.so.0<o:p></o:p></p><p class=MsoNormal>#3 0x00007f91d64abe58 in ltt_probe_unregister (desc=0x7f91d66d0ce0) at ltt-probes.c:129<o:p></o:p></p><p class=MsoNormal>#4 0x00007f91d66e0d10 in __lttng_events_exit__sample_component () at /usr/include/lttng/ust-tracepoint-event.h:557<o:p></o:p></p><p class=MsoNormal>#5 0x00007f91d66e07cf in __do_global_dtors_aux () from mytrace.so<o:p></o:p></p><p class=MsoNormal>#6 0x00007f91d66cd664 in global_apps () from /usr/lib/liblttng-ust.so.0<o:p></o:p></p><p class=MsoNormal>#7 0x00007f91d5479df0 in ?? ()<o:p></o:p></p><p class=MsoNormal>#8 0x00007f91d66e71dd in _real_fini () from mytrace.so<o:p></o:p></p><p class=MsoNormal>#9 0x00007f91d66e71d2 in _fini () from mytrace.so<o:p></o:p></p><p class=MsoNormal>#10 0x00007f91d94f7f54 in ?? () from /lib64/ld-linux-x86-64.so.2<o:p></o:p></p><p class=MsoNormal>#11 0x00007f91d7a652ed in exit () from /lib64/libc.so.6<o:p></o:p></p><p class=MsoNormal>#12 0x00007f91d64a9207 in wait_for_sessiond (sock_info=0x7f91d66cc640) at lttng-ust-comm.c:542<o:p></o:p></p><p class=MsoNormal>#13 0x00007f91d64a9545 in ust_listener_thread (arg=<value optimized out>) at lttng-ust-comm.c:669<o:p></o:p></p><p class=MsoNormal>#14 0x00007f91d82a5650 in start_thread () from /lib64/libpthread.so.0<o:p></o:p></p><p class=MsoNormal>#15 0x00007f91d7b0315d in clone () from /lib64/libc.so.6<o:p></o:p></p><p class=MsoNormal>#16 0x0000000000000000 in ?? ()<o:p></o:p></p><p class=MsoNormal>(gdb) p sessions_mutex<o:p></o:p></p><p class=MsoNormal>$1 = {__data = {__lock = 2, __count = 0, __owner = 21901, __nusers = 1, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, <o:p></o:p></p><p class=MsoNormal> __size = "\002\000\000\000\000\000\000\000\215U\000\000\001", '\0' <repeats 26 times>, __align = 2}<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>What happened seems like this:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>lttng_ust_init<o:p></o:p></p><p class=MsoNormal> |<o:p></o:p></p><p class=MsoNormal> |-> ust_listener_thread <o:p></o:p></p><p class=MsoNormal> |<o:p></o:p></p><p class=MsoNormal> |-> wait_for_sessiond <o:p></o:p></p><p class=MsoNormal> |<o:p></o:p></p><p class=MsoNormal> |-> ust_lock <span style='color:#1F497D'> </span>(1)<o:p></o:p></p><p class=MsoNormal> |-> get_map_shm <o:p></o:p></p><p class=MsoNormal> | |<o:p></o:p></p><p class=MsoNormal> | |-> get_wait_shm <o:p></o:p></p><p class=MsoNormal> | |<o:p></o:p></p><p class=MsoNormal> | |-> fork<o:p></o:p></p><p class=MsoNormal> | parent -> wait<span style='color:#1F497D'> </span> (2)<o:p></o:p></p><p class=MsoNormal> | child -> exit -> _fini -> __do_global_dtors_aux -> ...... -> ltt_probe_unregister -> ust_lock (3)<o:p></o:p></p><p class=MsoNormal> |-> ust_unlock <span style='color:#1F497D'> </span>(4)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Deadlock happened at point (1) and (3). Parent waited for child’s termination and child waited for parent to release the lock.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Reproduction conditions:<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>First time to create share memory (/dev/shm/lttng-ust-apps-wait* don’t exist)<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Child process got delayed( I’m not quite sure with this, I used gdb to hold child process for a while and it happened either)<o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal>In normal case, child process didn’t call _fini when it exited so that no deadlock happened.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Is this a known issue?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks<o:p></o:p></p><p class=MsoNormal>-Zheng<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>