[lttng-dev] lttng-ust under the hood

[Thibault, Daniel]

 ----------------------------------------------------------------------
>> Date: Thu, 22 May 2014 18:21:29 +0200
>> From: Gerlando Falauto <gerlando.falauto at keymile.com>
>>
>>> I'm still struggling to figure out why, depending on how I compile 
>>> and link my application, I can get either custom tracepoint providers OR tracef() to work, but not both at the same time.
>>>
>>> Is there any documentation explaining how lttng-ust is roughly supposed to work?
>>> (i.e. linking trickeries, libraries needed in LD_PRELOAD, 
>>> _LGPL_SOURCE, tracepoints, probes and so on...)
>>>
>>> Gerlando
>>
>>     It's all a matter of choice.  Tracepoint providers can be in the app (statically linked in) or dynamically linked to it.  In the latter case, the linking can be mandatory (so the app will fail to launch if it cannot find the tracepoint provider .so) or optional.  With optional linking, failure to find the .so means the app isn't traceable but otherwise runs normally.
>
>Could you elaborate more on this optional linking? Perhaps it's the kind of problem I'm running into.

Assume tp.c to consist of just two lines:

----------------------------------------------------------------------
#define TRACEPOINT_CREATE_PROBES
#include "tp.h"
----------------------------------------------------------------------

tp.h is the tracepoint provider header, pretty much boilerplate:

----------------------------------------------------------------------
#undef TRACEPOINT_PROVIDER
#define TRACEPOINT_PROVIDER sample_component
#undef TRACEPOINT_INCLUDE
#define TRACEPOINT_INCLUDE "./tp.h"
#if !defined(_SAMPLE_COMPONENT_PROVIDER_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
#define _SAMPLE_COMPONENT_PROVIDER_H
#include <lttng/tracepoint.h>
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
TRACEPOINT_EVENT(
...
)
TRACEPOINT_LOGLEVEL(
...
)
#ifdef __cplusplus
}
#endif /* __cplusplus */
#include <lttng/tracepoint-event.h>
#endif /* _SAMPLE_COMPONENT_PROVIDER_H */
----------------------------------------------------------------------

   Now for some Makefile fragments.

----------------------------------------------------------------------
LIBDL = -ldl # On Linux
#LIBDL = -lc # On BSD
LIBUST = -llttng-ust
LDFLAGS += -L/usr/local/lib $(AM_LDFLAGS)
LOCAL_CPPFLAGS += -I. $(AM_CPPFLAGS)
SOFLAGS = -fPIC
SOVERSION_MAJOR = 1
SOVERSION_MINOR = 0

libtp.so: libtp.o
	@echo "~~~~~~Packaging $@:"
	$(CC) -shared -Wl,-soname,$@.$(SOVERSION_MAJOR) -Wl,-no-as-needed \
	   -o $@.$(SOVERSION_MAJOR).$(SOVERSION_MINOR) $(LDFLAGS) $(LIBUST) $<
	ln -sf $@.$(SOVERSION_MAJOR).$(SOVERSION_MINOR) $@.$(SOVERSION_MAJOR)
	ln -sf $@.$(SOVERSION_MAJOR) $@

libtp.o: tp.c tp.h
	@echo "~~~~~~Compiling $@:"
	$(CC) $(CPPFLAGS) $(LOCAL_CPPFLAGS) $(SOFLAGS) -c -o $@ $<
----------------------------------------------------------------------

The -as-needed linker flag means the libraries mentioned in the command line (LIBUST) generate a NEEDED tag in the executable only if actually satisfying an otherwise undefined symbol reference. If present, the resulting library won't NEED liblttng-ust, which breaks tracing.

Here is how the tracepoint provider (libtp) can be dynamically included as a dependency.  The sample application is statically aware of libtp and won't run without it (the inclusion of a libtp NEEDED tag requires "-no-as-needed -ltp").  Besides the recommended LD_PRELOAD technique, you could also use #LD_LIBRARY_PATH (which does not work in the 'dynamic' target case, below).  Note that "-L. -ltp", when used, should appear before $(LDFLAGS) $(LIBDL).

----------------------------------------------------------------------
TP_DEFINE = -D TRACEPOINT_DEFINE

static.o: sample.c tp.h
	@echo "~~~~~~Compiling $@:"
	$(CC) $(CPPFLAGS) $(LOCAL_CPPFLAGS) $(TP_DEFINE) -c -o $@ $<

static_aware: static.o libtp.so
	@echo "~~~~~~Linking sample_$@:"
	$(CC) -Wl,-no-as-needed -o sample_$@ -L. -ltp $< $(LDFLAGS) $(LIBDL)
	@echo "   Use 'LD_PRELOAD=./libtp.so ./sample_$@' to run sample_$@"
----------------------------------------------------------------------

static_aware could also be written with a "-Wl,-rpath,'$$ORIGIN'" linker call (see below).   This would add an RPATH tag to the application which makes it look for its dependencies in its starting directory ($ORIGIN). RPATH is, unlike RUNPATH, applicable to libraries as well: a library with an RPATH tag will look there for its dependencies; failing its own RPATH, it will use its executable's RPATH.

----------------------------------------------------------------------
static_aware: static.o libtp.so
	@echo "~~~~~~Linking sample_$@:"
	$(CC) -Wl,-no-as-needed -o sample_$@ -L. -ltp $< $(LDFLAGS) $(LIBDL) \
	   -Wl,-rpath,'$$ORIGIN',--enable-new-dtags
----------------------------------------------------------------------

Here is how the tracepoint provider (libtp) can be dynamically included without static awareness.  If not preloaded, the sample application runs but won't be traceable.  Putting libtp in the path won't work: the sample does not look for it.  This is indifferent to the -[no-]as-needed linker flag.

----------------------------------------------------------------------
TP_DEFINE_DYNAMIC = $(TP_DEFINE) -D TRACEPOINT_PROBE_DYNAMIC_LINKAGE

dynamic: dynamic.o libtp.so
	@echo "~~~~~~Linking sample_$@:"
	$(CC) -o sample_$@ $< $(LDFLAGS) $(LIBDL)
	@echo "   Use '[LD_PRELOAD=./libtp.so] ./sample_$@' to run sample_$@"

dynamic.o: sample.c tp.h
	@echo "~~~~~~Compiling $@:"
	$(CC) $(CPPFLAGS) $(LOCAL_CPPFLAGS) $(TP_DEFINE_DYNAMIC) \
	   -c -o $@ $<
----------------------------------------------------------------------

Compared with the 'static.o' target, 'dynamic.o' only adds -D TRACEPOINT_PROBE_DYNAMIC_LINKAGE.

>> LD_PRELOAD is generally used to provide the app with the tracepoint provider .so when tracing is desired.
>>
>>     Extensive documentation (in PDF) is currently undergoing editorial review, and the plan is to move said documentation to DocBook format so it can be included in the LTTng distribution.
>
>Would it be possible to have a copy of this doc?

   I'll be sending this to you tonight from my home e-address (to avoid attachment interference from our e-mail server).

>>    Since tracef() depends on liblttng-ust just like ordinary UST, I'm not sure which problem you're running into.  Could you give us a simplified makefile that showcases the issue?
>
>main.c:
>#include <lttng/tracef.h>
>
>int main() {
>    tracef("Hello world");
>    return 0;
>}
>
>gcc main.c -lttlng-ust -ldl -I.
>
>There, I don't get my tracef().
>This happens on top of an ELDK 5.4, on top of which I configured & compiled one of the latest releases of userspace-rc, lttng-ust, and lttng-tools.
>
>I tried either with native compiling on the target, or with cross-compiling. Same result.
>
>Notice how I used the same configure/compile/install script to compile on a pristine ch-rooted rootfs on the x86_64 host, and there everything works fine.
>
>I suspect my ELDK (so my rootfs) has some spurious LTTNG files (or other .so), since it already contains an older version of LTTNG, which confuse the linker. I tried rm'ing the existing /usr/lib/{liblttng-*,liburcu*} prior to configuring, but still no luck.
>
>Any clue?
>
>Gerlando

   You should definitely not mix LTTng bits and pieces across different versions.  Have you tried creating a clean ELDK on which to install LTTng?

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>