<div dir="ltr">Hi LTTng folks<div><br></div><div>I have a strange problem using LTTng-UST on an ARM based platform. I have done some diagnosis but I am running low on ideas and was hoping for help from the experts. I am using lttng-tools 2.2.0, lttng-ust 2.2.0, liburcu 0.8.1. I know these are old but upgrading is easier said than done unfortunately. I didn't see anything related to this problem in relnotes, mailing list traffic, or master branch, but I could have missed something.</div>
<div><br></div><div>The problem showed up when I switched from GCC 4.6.4 to 4.7.2. Conceptually, the situation is that I have a single executable, call it MyProgram, with two plugins loaded at runtime with dlopen(), lets call them libPlugin1.so and libPlugin2.so. There are three different LTTng-UST tracepoint providers, one each for the executable and the two plugins. With GCC 4.7.2, tracepoints in libPlugin1 stopped working. The tracepoints in MyProgram and in libPlugin2 continue to work correctly.</div>
<div><br></div><div>I have established without a doubt that the toolchain upgrade is the cause of the regression.</div><div><br></div><div>In the debugger, I confirmed that the tracepoint for libPlugin1.so is being executed, but __tracepoint_##provider##___##name.state is always 0 even when I enable the tracepoint in lttng-tools. As a result the tracepoint callback is not being invoked when it should be. In MyProgram and libPlugin2.so, the .state variable correctly reflects whether the tracepoint is enabled, and if the tracepoint is enabled, the tracepoint callback is invoked.</div>
<div><br></div><div>Next I set a breakpoint in tracepoint_register_lib() and looked at tracepoints_start parameter.</div><div><br></div><div>1) With GCC 4.6.4 everything is as expected:</div><div> a) tracepoint_register_lib() for MyProgram called with MyProgramProvider's __start___tracepoints_ptrs.</div>
<div> b) tracepoint_register_lib() after libPlugin1 dlopen() called with libPlugin1Provider's __start___tracepoints_ptrs<br></div><div> c) tracepoint_register_lib() after libPlugin2 dlopen() called with libPlugin2Provider's __start___tracepoint_ptrs</div>
<div><br></div><div>2) With GCC 4.7.2 there is a problem:</div><div><div> a) tracepoint_register_lib() for MyProgram called with MyProgramProvider's __start___tracepoints_ptrs.</div><div> b) tracepoint_register_lib() after libPlugin1 dlopen() called with MyProgramProvider's __start___tracepoints_ptrs (!!!! THIS IS WRONG !!!!)<br>
</div><div><div> c) tracepoint_register_lib() after libPlugin2 dlopen() called with libPlugin2Provider's __start___tracepoint_ptrs</div></div></div><div><br></div><div>I looked at the symbol table for libPlugin1.so to see if it would shed some light on the problem.</div>
<div><br></div><div>1) With GCC 4.6.4:</div><div># objdump -t /usr/lib/.debug/libPlugin1.so | grep __start___tracepoints_ptrs<br></div><div><div>00025bb0 l *ABS*<span class="" style="white-space:pre"> </span>00000000 __start___tracepoints_ptrs<br>
</div></div><div><div># objdump -t /usr/lib/.debug/libPlugin2.so | grep __start___tracepoints_ptrs</div><div>00041eb4 l *ABS*<span class="" style="white-space:pre"> </span>00000000 __start___tracepoints_ptrs</div></div>
<div><br></div><div>2) With GCC 4.7.2:</div><div><div># objdump -t /usr/lib/.debug/libPlugin1.so | grep __start___tracepoints_ptrs</div></div><div><div>00025a90 g __tracepoints_ptrs<span class="" style="white-space:pre"> </span>00000000 __start___tracepoints_ptrs<br>
</div></div><div><div># objdump -t /usr/lib/.debug/libPlugin2.so | grep __start___tracepoints_ptrs</div><div>00041eb4 g __tracepoints_ptrs<span class="" style="white-space:pre"> </span>00000000 __start___tracepoints_ptrs</div>
</div><div><br></div><div>My hypothesis at this point is that since __start___tracepoints_ptrs changed from a local to a global symbol, the dynamic loader no longer knows how to select the correct weak symbol. I cannot explain why libPlugin2 still loads its provider correctly, perhaps it is just getting lucky.</div>
<div><br></div><div>A few questions come to mind...</div><div>1) Have you run into a problem like this? Is there a known fix/workaround?</div><div>2) __start____tracepoints_ptrs is declared as extern in tracepoint.h, but it is not defined. This appears to be some sort of undocumented linker magic. <a href="http://gcc.gnu.org/ml/gcc-help/2010-04/msg00120.html">http://gcc.gnu.org/ml/gcc-help/2010-04/msg00120.html</a> is the only reference I could find. Do you know where this behavior is documented or specified (if at all)?</div>
<div>3) Do you know why the symbol visibility for __start___tracepoints_ptrs changed between 4.6.4 to 4.7.2?</div><div><br></div><div>Thanks for any help. This is a real puzzler for me.</div><div><br></div><div>Martin</div>
<div><br></div></div>