[lttng-dev] dlopen, gcc -l and LD_PRELOAD

[Thibault, Daniel]

> Message: 1
> Date: Tue, 19 Mar 2013 11:51:29 -0400
>
> >> I don't see this warning and error here. Which release are you using?
>>
> >    lttng-modules 2.1.1+ (74b9312, 2013-02-27)
> >    lttng-tools 2.1.1+ (e58727c, 2013-03-01)
> >    lttng-ust 2.1.1+ (009769c, 2013-03-05)
> >    userspace-rcu 0.7.6+ (108a92e, 2013-02-22)
> >
>
> I can confirm I'm able to reproduce the problem with these versions.
>
> >    (All are moderately dated by now, but none of the later commit 
> > comments seem pertinent)
>
> This revision fixed the "undefined symbol: tracepoint_dlopen" problem:
> https://bugs.lttng.org/projects/lttng-ust/repository/revisions/558b9d86247004f8e9bbaf8c982f3b2b182093d1

   Doh!  It had to be the revision right after the one I was using.

   Meanwhile I've been playing with some very simple examples of .so use, and found that the behaviour of LD_PRELOAD is incompletely documented.  For instance, if I run:

LD_LIBRARY_PATH=. ./sometest

   Where sometest is a simple application which uses a libwhatever.so built in the same directory, it works.  But:

LD_PRELOAD=./libwhatever.so ./sometest

   Fails with "No such file or directory".  LD_PRELOAD and LD_LIBRARY_PATH are NOT interchangeable.  The nitty-gritty (thanks to LD_DEBUG=libs) is as follows (my apologies if this is old-hat to fellow lttngeeks, but it was certainly new to me):

   Upon launching a process, the dynamic loader searches for the executable's requested libraries in the following ordered list of places, stopping as soon as a match is found:

. the rpath directory (or colon-separated directories) encoded in the executable, if any, but only if runpath is not also set (e.g. gcc -Wl,-rpath,'$ORIGIN/rpath' etc.) (instead of using the linker -rpath option, you can set LD_RUN_PATH before calling the linker);
. the (colon-separated) directories listed in the environment variable LD_LIBRARY_PATH, if any;
. the runpath directory (or colon-separated directories) encoded in the executable, if any (e.g. gcc -Wl,-rpath,'$ORIGIN/rpath',--enable-new-dtags etc.; although it is possible to modify the binary so rpath and runpath are different, gcc sets both values to the same string);
. the directories listed or included in /etc/ld.so.conf, if any;
. the trusted directory /lib; and finally
. the trusted directory /usr/lib.

   If the executable was linked with the -z nodeflib option ("no def lib"), then ld.so will not look in the last three locations for the requested shared library (that is to say, the search stops with the third bullet above).  The Linux /etc/ld.so.conf typically uses an explicit include instruction to add the collection of .conf files in the /etc/ld.so.conf.d directory.  The /etc/ld.so.conf.d files themselves are sorted alphabetically.  (I skip over /etc/ld.so.cache and the important role of the ldconfig command).  The found library is initialised, and then the dynamic loader checks in the following locations for the requested library:

. the libraries (not directories) listed in the environment variable LD_PRELOAD, if any (absolute or relative paths; LD_PRELOAD accepts colon- and/or space-separated entries); and
. the libraries (not directories) listed in the system file /etc/ld.so.preload, if any.

  If LD_PRELOAD or /etc/ld.so.preload supply a replacement library, the latter is initialised and its members pre-empt those from the previously found library.  Preloading thus serves to substitute replacements for some of the "normal" library methods.  Because full dynamicity (i.e. using dlopen to fetch the library) does not involve the dynamic loader, preloading does *not* work against it.  Also, if the requested library is not found in the "normal" places (rpath, LD_LIBRARY_PATH, runpath, /etc/ld.so.conf, /lib, /usr/lib) the dynamic loader issues an error ("cannot open shared object file: No such file or directory") and does *not* look into LD_PRELOAD and /etc/ld.so.preload.

   There are additional subtleties involving set-user-ID/set-group-ID binaries, but this is otherwise as complete as I can make it.

   The above should explain the trouble I've been having with lttng-ust's doc/examples/easy-ust and tests/demo.

Daniel U. Thibault
R & D pour la défense Canada - Valcartier (RDDC Valcartier) / Defence R&D Canada - Valcartier (DRDC Valcartier)
Cyber sécurité pour les missions essentielles (CME) / Mission Critical Cyber Security (MCCS)
Protection des systèmes et contremesures (PSC) / Systems Protection & Countermeasures (SPC)
2459 route de la Bravoure
Québec, QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada / Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>