[lttng-dev] dlopen, gcc -l and LD_PRELOAD

Mon Mar 18 16:58:08 EDT 2013

On Mon, Mar 18, 2013 at 3:07 PM, Thibault, Daniel
<Daniel.Thibault at drdc-rddc.gc.ca> wrote:
> -----Message d'origine-----
> Envoyé : 15 mars 2013 15:33
>
> On Fri, Mar 15, 2013 at 12:41 PM, Thibault, Daniel <Daniel.Thibault at drdc-rddc.gc.ca> wrote:
>>    In the lttng-ust man pages, there is a paragraph that reads:
>>
>> Note about dlopen() usage: Due to locking side-effects caused by the way libc lazily resolves Thread-Local Storage (TLS) symbols when a library is dlopened, linking the tracepoint probe or liblttng-ust with dlopen() is discouraged.  They should be linked with the application using "-llibname" or loaded with LD_PRELOAD.
>>
>>    I'm confused by the LD_PRELOAD bit: as far as I know, LD_PRELOAD serves only to substitute a dynamic library (the LD_PRELOAD argument) for another (the one the executable was linked against).  I don't see how one could compile an instrumented application and leave its tracepoint provider unlinked, which is what the stated -llibname <-> LD_PRELOAD opposition implies.
>>
>
> You may want to have a look at the comment at
> lttng-ust/include/lttng/tracepoint.h:229 which explains how this is accomplished. Basically, the tracepoint provider symbols are initialised and the probe is registered when the provider's shared object constructor is invoked. See ust-tracepoint-event.h:671 if you are interested in the actual implementation.
>
>>    (Also, shouldn't that paragraph drop 'lazily', assuming the locking
>> problems also arise if dlopen is used with RTLD_NOW?  Because if
>> RTLD_NOW fixed the locking problem, that's what would be recommended
>> here, no?)
>
> Good question. I think this problem has been fixed as part of the 2.1 release. However, I do know there are other issues related to "dlopen()"-ing probe providers; details on the bug tracker[1].
>
> Jérémie
> [1] https://bugs.lttng.org/issues/447
> -----Fin du message d'origine-----
>
>    Bug 447 concerns dlclose(), not dlopen().  It's a legitimate concern (that has yet to be addressed, it seems), but it has nothing to do with potential delays or deadlocks upon calling dlopen() on a tracepoint provider.
>

Indeed, hence why I have mentioned there were _other_ potential issues
with the dynamic linking loader interface of which you should be aware
before using it to load the providers.

>    I'm not concerned with how the tracepoint provider initialises its symbols and registers its probe(s) with the session daemons when its constructor is invoked, I'm just trying to figure out what "They [the tracepoint provider and liblttng-ust] should be linked with the application using '-llibname' or loaded with LD_PRELOAD" means.  The first part of the (apparently exclusive) or clearly refers to implementing the tracepoint provider as an .so but making the instrumented application statically aware of it (I'm dubious anyone would try to use liblttng-ust through dlopen(), though).  Is the second part saying that if the app explicitly loads the tracepoint provider .so through dlopen(), setting LD_PRELOAD when calling the app will fix (circumvent) the locking problems?
>

Typically, you would want to do one or the other [dlopen or
LD_PRELOAD]. But to answer your question, yes, this would workaround
the locking problem described in the man page that affected prior
versions of lttng-ust (before 2.1).

>    For that matter, I have yet to successfully divine what a tracepoint provider .so makefile should look like (the lttng-ust man pages don't give an understandable example, and my experience with dynamic libraries has mostly been under Windows).  If you could provide one, I'd be grateful.  Here's as far as I've got, using lttng-ust/doc/example/easy-ust:
>

Have a look at the Makefiles and sources in lttng-ust/tests/demo and
lttng-ust/tests/daemon which respectively demonstrate the use of
tracepoint providers that are dynamically linked and built as part of
the traced application.

> The normal Makefile:
>
> CC = gcc
> LIBS = -ldl -llttng-ust         # On Linux
> #LIBS = -lc -llttng-ust         # On BSD
> CFLAGS = -I.
> LDFLAGS = -L/usr/local/lib
>
> all: sample
>
> sample: sample.o tp.o
>         $(CC) -o $@ $^ $(LDFLAGS) $(LIBS)
>
> sample.o: sample.c sample_component_provider.h
>         $(CC) $(CFLAGS) -c -o $@ $(LDFLAGS) $<
>
> tp.o: tp.c sample_component_provider.h
>         $(CC) $(CFLAGS) -c -o $@ $(LDFLAGS) $<
> [...]
>
> Yields:
> gcc -I. -c -o sample.o -L/usr/local/lib sample.c
> gcc -I. -c -o tp.o -L/usr/local/lib tp.c
> gcc -o sample sample.o tp.o -L/usr/local/lib -ldl -llttng-ust
>
> I can make sample using a static library instead, by changing the Makefile thus:
>
> AR = ar
> CC = gcc
> [...]
> sample: sample.o libtp.a
>         $(CC) -o $@ $^ $(LDFLAGS) -L. $(LIBS)
>
> sample.o: sample.c sample_component_provider.h
>         $(CC) $(CFLAGS) -c -o $@ $(LDFLAGS) $<
>
> libtp.a: tp.o
>         $(AR) -cq $@ $<
>
> tp.o: tp.c sample_component_provider.h
>         $(CC) $(CFLAGS) -c -o $@ $(LDFLAGS) $<
> [...]
>
> Which yields:
> gcc -I. -c -o sample.o -L/usr/local/lib sample.c
> gcc -I. -c -o tp.o -L/usr/local/lib tp.c
> ar -cq libtp.a tp.o
> gcc -o sample sample.o libtp.a -L/usr/local/lib -L. -ldl -llttng-ust
>
> In switching to a statically aware dynamic library, I first modify sample.c by adding '#define TRACEPOINT_PROBE_DYNAMIC_LINKAGE' right after '#define TRACEPOINT_DEFINE'.
>
> Then I've tried to use this Makefile:
>
> CC = gcc
> LIBS = -ldl -llttng-ust         # On Linux
> #LIBS = -lc -llttng-ust         # On BSD
> CFLAGS = -I.
> SOFLAGS = -fPIC
> SOVERSION_MAJOR = 1
> SOVERSION_MINOR = 0
> LDFLAGS = -L/usr/local/lib
>
> all: libtp.so sample
>
> sample: sample.o
>         $(CC) -o $@ $< $(LDFLAGS) $(LIBS) -L. -ltp
>
> sample.o: sample_so.c sample_component_provider.h
>         $(CC) $(CFLAGS) -c -o $@ $(LDFLAGS) $(LIBS) -L. -ltp $<
>
> libtp.so: tp.o
>         $(CC) -shared -W1,-soname,$@.$(SOVERSION_MAJOR) -o $@.$(SOVERSION_MAJOR).$(SOVERSION_MINOR) $(LDFLAGS) $(LIBS) $<
>         ln -sf $@.$(SOVERSION_MAJOR).$(SOVERSION_MINOR) $@.$(SOVERSION_MAJOR)
>         ln -sf $@.$(SOVERSION_MAJOR) $@
>
> tp.o: tp.c sample_component_provider.h
>         $(CC) $(CFLAGS) $(SOFLAGS) $(LDFLAGS) $(LIBS) -c -o $@ $<
>
> Which yields:
> gcc -I. -fPIC -L/usr/local/lib -ldl -llttng-ust -c -o tp.o tp.c
> gcc -shared -W1,-soname,libtp.so.1 -o libtp.so.1.0 -L/usr/local/lib -ldl -llttng-ust tp.o
> ln -sf libtp.so.1.0 libtp.so.1
> ln -sf libtp.so.1 libtp.so
> gcc -I. -c -o sample.o -L/usr/local/lib -ldl -llttng-ust -L. -ltp sample.c
> gcc -o sample sample.o -L/usr/local/lib -ldl -llttng-ust -L. -ltp
> /usr/bin/ld: warning: hidden symbol 'tracepoint_dlopen' in sample.o is referenced by DSO ./libtp.so
>
>    I've tried a number of things to get rid of the warning.  For some reason the 'tracepoint_dlopen' struct symbol in tp.o gets linked (when building 'sample' in the last step) to the one in sample.o (which has the same source: the sample_component_provider.h) instead of the one in liblttng-ust.  If I try a command like 'LD_PRELOAD=./libtp.so ./sample', I get "symbol lookup error: ./libtp.so: undefined symbol: tracepoint_dlopen".
>
>    I'm sure this is a very simple, obvious, dumb mistake, so please point it out to me!
>
> lttng-ust/tests/demo/demo-trace, although built by lttng-ust's make, also has the tracepoint_dlopen problem.  This now looks like a bug introduced somewhen since the demos were written (and last tested, it seems).

I don't see this warning and error here. Which release are you using?

> Daniel U. Thibault
> R & D pour la défense Canada - Valcartier (RDDC Valcartier) / Defence R&D Canada - Valcartier (DRDC Valcartier)
> Cyber sécurité pour les missions essentielles (CME) / Mission Critical Cyber Security (MCCS)
> Protection des systèmes et contremesures (PSC) / Systems Protection & Countermeasures (SPC)
> 2459 route de la Bravoure
> Québec, QC  G3J 1X5
> CANADA
> Vox : (418) 844-4000 x4245
> Fax : (418) 844-4538
> NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
> Gouvernement du Canada / Government of Canada
> <http://www.valcartier.drdc-rddc.gc.ca/>