[lttng-dev] [PATCH lttng-ust v2 4/6] Manually dlopen() liblttng-ust.so to prevent unloading

[Francis Deslauriers]

The support of probe provider dlclose() allows for the following problematic
scenario:
- Application is not linked against the liblttng-ust.so
- Application dlopen() a probe provider library that is linked against
	liblttng-ust.so
- Application dlclose() the probe provider

In this scenario, the probe provider has a dependency on liblttng-ust.so, so
when it's loaded by the application, liblttng-ust.so is loaded too. The probe
provider library now has the only reference to the liblttng-ust.so library.
When the application calls dlclose() on it, all its references are dropped,
thus triggering the unloading of both the probe provider library and
liblttng-ust.so.

This scenario is problematic because lttng ust_listener_threads are in DETACHED
state. We cannot join them and therefore we cannot unload the library
containing the code they run. Only the operating system can free those
resources.

The reason why those threads are in DETACHED state is to quickly teardown
applications on process exit.

A possible solution to investigate: if we can determine whether liblttng-ust.so
is being dlopen (directly or undirectly) or it's linked against the
application, we could set the detached state accordingly.

To prevent that unloading, we pin it in memory by grabbing an extra reference
on the library, with a RTLD_NODELETE flag. This will prevent the dynamic loader
from ever removing the liblttng-ust.so library from the process' address space.

Signed-off-by: Francis Deslauriers <francis.deslauriers at efficios.com>
---

v2:
Rephrase commit message and comment
---

 configure.ac                  |  1 +
 liblttng-ust/Makefile.am      |  2 ++
 liblttng-ust/lttng-ust-comm.c | 25 +++++++++++++++++++++++++
 3 files changed, 28 insertions(+)

diff --git a/configure.ac b/configure.ac
index b0b4157..4fc6f9c 100644
--- a/configure.ac
+++ b/configure.ac
@@ -25,6 +25,7 @@ m4_define([UST_LIB_V_MINOR], [0])
 m4_define([UST_LIB_V_PATCH], [0])
 
 AC_SUBST([LTTNG_UST_LIBRARY_VERSION], [UST_LIB_V_MAJOR:UST_LIB_V_MINOR:UST_LIB_V_PATCH])
+AC_SUBST([LTTNG_UST_LIBRARY_VERSION_MAJOR], [UST_LIB_V_MAJOR])
 # note: remember to update tracepoint.h dlopen() to match this version
 # number. TODO: eventually automate by exporting the major number.
 
diff --git a/liblttng-ust/Makefile.am b/liblttng-ust/Makefile.am
index 982be69..a7edfd5 100644
--- a/liblttng-ust/Makefile.am
+++ b/liblttng-ust/Makefile.am
@@ -60,6 +60,8 @@ liblttng_ust_runtime_la_SOURCES = \
 	string-utils.c \
 	string-utils.h
 
+liblttng_ust_runtime_la_CFLAGS = -DLTTNG_UST_LIBRARY_VERSION_MAJOR=\"$(LTTNG_UST_LIBRARY_VERSION_MAJOR)\"
+
 if HAVE_PERF_EVENT
 liblttng_ust_runtime_la_SOURCES += \
 	lttng-context-perf-counters.c \
diff --git a/liblttng-ust/lttng-ust-comm.c b/liblttng-ust/lttng-ust-comm.c
index 511b9cf..57e5ce6 100644
--- a/liblttng-ust/lttng-ust-comm.c
+++ b/liblttng-ust/lttng-ust-comm.c
@@ -27,6 +27,7 @@
 #include <sys/stat.h>
 #include <sys/types.h>
 #include <sys/wait.h>
+#include <dlfcn.h>
 #include <fcntl.h>
 #include <unistd.h>
 #include <errno.h>
@@ -59,6 +60,9 @@
 #include "../libringbuffer/getcpu.h"
 #include "getenv.h"
 
+/* Concatenate lttng ust shared library name with its major version number. */
+#define LTTNG_UST_LIB_SO_NAME "liblttng-ust.so." LTTNG_UST_LIBRARY_VERSION_MAJOR
+
 /*
  * Has lttng ust comm constructor been called ?
  */
@@ -1648,6 +1652,7 @@ void __attribute__((constructor)) lttng_ust_init(void)
 	pthread_attr_t thread_attr;
 	int timeout_mode;
 	int ret;
+	void *handle;
 
 	if (uatomic_xchg(&initialized, 1) == 1)
 		return;
@@ -1662,6 +1667,26 @@ void __attribute__((constructor)) lttng_ust_init(void)
 	lttng_ust_loaded = 1;
 
 	/*
+	 * We need to ensure that the liblttng-ust library is not unloaded to avoid
+	 * the unloading of code used by the ust_listener_threads as we can not
+	 * reliably know when they exited. To do that, manually load
+	 * liblttng-ust.so to increment the dynamic loader's internal refcount for
+	 * this library so it never becomes zero, thus never gets unloaded from the
+	 * address space of the process. Since we are already running in the
+	 * constructor of the LTTNG_UST_LIB_SO_NAME library, calling dlopen will
+	 * simply increment the refcount and no additionnal work is needed by the
+	 * dynamic loader as the shared library is already loaded in the address
+	 * space. As a safe guard, we use the RTLD_NODELETE flag to prevent
+	 * unloading of the UST library if its refcount becomes zero (which should
+	 * never happen). Do the return value check but discard the handle at the
+	 * end of the function as it's not needed.
+	 */
+	handle = dlopen(LTTNG_UST_LIB_SO_NAME, RTLD_LAZY | RTLD_NODELETE);
+	if (!handle) {
+		ERR("dlopen of liblttng-ust shared library (%s).", LTTNG_UST_LIB_SO_NAME);
+	}
+
+	/*
 	 * We want precise control over the order in which we construct
 	 * our sub-libraries vs starting to receive commands from
 	 * sessiond (otherwise leading to errors when trying to create
-- 
2.7.4