<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-CA">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">We have an init-like top-level application that starts a bunch of other daemons. This top-level daemon is started with LD_PRELOAD= /usr/lib/liblttng-ust-fork.so.0 so all the daemons that it starts inherit that. Sometimes
we want to debug one of these child applications, so we arrange that instead of starting it directly, the top-level daemon starts it like so: “gdbserver :23 <path/to/real/daemon>”. This arranges that the child daemon not actually run until a remote debugger
attaches. This scheme seemed to work reasonably well until recently. It still works maybe half the time, but the other half, gdbserver crashes in lttng related code. The backtrace is below. As a quick fix, I have tried starting gdbserver and the child
application like so: “/usr/bin/env -u LD_PRELOAD gdbserver :23 <path/to/real/daemon>”. In the limited testing I’ve done so far, this is looking good, but I’m wondering what other problems might turn up if I don’t preload liblttng-ust-fork into gdbserver and
the application it is debugging. Gdbserver does fork after all, and the application might as well. Does this seem like a reasonable long-term solution for using the remote debugger?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Here’s the crash backtrace:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">(gdb) bt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#0 0xb6e8c810 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#1 0xb6e9028c in __GI_abort () at abort.c:89<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#2 0xb6e08198 in lttng_ust_elf_destroy (elf=0x2, elf@entry=0xb6f9a088 <lock>)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> at /usr/src/debug/lttng-ust/2.8.2-r3.eriR0/git/liblttng-ust/lttng-ust-elf.c:342<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#3 0xb6e08bec in get_elf_info (bin_data=0xb6541308) at /usr/src/debug/lttng-ust/2.8.2-r3.eriR0/git/liblttng-ust/lttng-ust-statedump.c:165<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#4 trace_baddr (bin_data=0xb6541308) at /usr/src/debug/lttng-ust/2.8.2-r3.eriR0/git/liblttng-ust/lttng-ust-statedump.c:175<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#5 extract_bin_info_events (info=info@entry=0xb6542360, size=size@entry=40, _data=_data@entry=0xb65423b8)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> at /usr/src/debug/lttng-ust/2.8.2-r3.eriR0/git/liblttng-ust/lttng-ust-statedump.c:298<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#6 0xb6f65a78 in __GI___dl_iterate_phdr (callback=0xb6e08a34 <extract_bin_info_events>, data=data@entry=0xb65423b8) at dl-iteratephdr.c:76<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#7 0xb6e09c9c in do_baddr_statedump (owner=0xb6e5a618 <global_apps>)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> at /usr/src/debug/lttng-ust/2.8.2-r3.eriR0/git/liblttng-ust/lttng-ust-statedump.c:332<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#8 do_lttng_ust_statedump (owner=owner@entry=0xb6e5a618 <global_apps>)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> at /usr/src/debug/lttng-ust/2.8.2-r3.eriR0/git/liblttng-ust/lttng-ust-statedump.c:355<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#9 0xb6e007dc in lttng_handle_pending_statedump (owner=owner@entry=0xb6e5a618 <global_apps>)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> at /usr/src/debug/lttng-ust/2.8.2-r3.eriR0/git/liblttng-ust/lttng-events.c:857<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#10 0xb6df864c in handle_pending_statedump (sock_info=0xb6e5a618 <global_apps>)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> at /usr/src/debug/lttng-ust/2.8.2-r3.eriR0/git/liblttng-ust/lttng-ust-comm.c:599<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#11 handle_message (lum=0xb654286c, sock=<optimized out>, sock_info=<optimized out>)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> at /usr/src/debug/lttng-ust/2.8.2-r3.eriR0/git/liblttng-ust/lttng-ust-comm.c:957<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#12 ust_listener_thread (arg=0xb6e5a618 <global_apps>) at /usr/src/debug/lttng-ust/2.8.2-r3.eriR0/git/liblttng-ust/lttng-ust-comm.c:1520<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#13 0xb6d49f6c in start_thread (arg=0xb6543420) at pthread_create.c:339<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">I am not able to easily examine the data in lttng_ust_elf_destroy() because as you can see from the argument list to that function, gdb is confused about the value of the “elf” pointer (I find that gdb often gets confused
on ARM when unwinding from a call to abort()). But I can see that it aborted because a close() failed:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> if (close(elf->fd)) {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> abort();<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">The two other threads in the gdbserver application were here at the time of the crash:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#0 syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#1 0xb6df7cd0 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=0, op=0, uaddr=<optimized out>)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> at /repo/ngrbuilder/jenkins/ProductionBuild_0/xcs-build/build/xenon/bitbake_build/tmp/sysroots/xenon/usr/include/urcu/futex.h:65<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#2 futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0, uaddr=<optimized out>)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> at /repo/ngrbuilder/jenkins/ProductionBuild_0/xcs-build/build/xenon/bitbake_build/tmp/sysroots/xenon/usr/include/urcu/futex.h:97<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#3 wait_for_sessiond (sock_info=<optimized out>) at /usr/src/debug/lttng-ust/2.8.2-r3.eriR0/git/liblttng-ust/lttng-ust-comm.c:1276<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#4 ust_listener_thread (arg=0xb6e585ec <local_apps>) at /usr/src/debug/lttng-ust/2.8.2-r3.eriR0/git/liblttng-ust/lttng-ust-comm.c:1330<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#5 0xb6d49f6c in start_thread (arg=0xb6d43420) at pthread_create.c:339<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#0 0xb6efd7b8 in execve () at ../sysdeps/unix/syscall-template.S:84<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#1 0xb6efd8c8 in execv (path=<optimized out>, argv=<optimized out>) at execv.c:25<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#2 0x00035b40 in linux_create_inferior (program=0x6bc80 "/opt/radiosw/bin/ngr2mpw.elf", allargs=0x6bc70)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> at /usr/src/debug/gdb/7.10.1-r0/gdb-7.10.1/gdb/gdbserver/linux-low.c:889<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#3 0x00018e30 in start_inferior (argv=0x6bc70) at /usr/src/debug/gdb/7.10.1-r0/gdb-7.10.1/gdb/gdbserver/server.c:233<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#4 0x0001aa58 in captured_main (argc=argc@entry=-1225151608, argv=argv@entry=0x3fda8 <__libc_csu_init+32>)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> at /usr/src/debug/gdb/7.10.1-r0/gdb-7.10.1/gdb/gdbserver/server.c:3516<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">#5 0x0001250c in main (argc=-1225151608, argv=0x3fda8 <__libc_csu_init+32>)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Doug<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
</div>
</body>
</html>