[lttng-dev] making liburcu and lttng coexist in a LGPL'ed program

Mon Dec 3 16:55:13 EST 2018

On Mon, 2018-12-03 at 14:13 -0500, Mathieu Desnoyers wrote:
> 
> ----- On Dec 3, 2018, at 1:43 PM, Jeff Layton jlayton at redhat.com wrote:
> 
> > On Mon, 2018-12-03 at 13:25 -0500, Mathieu Desnoyers wrote:
> > > ----- On Dec 3, 2018, at 12:58 PM, Jeff Layton jlayton at redhat.com wrote:
> > > 
> > > > On Wed, 2018-11-28 at 17:06 -0500, Mathieu Desnoyers wrote:
> > > > > ----- On Nov 27, 2018, at 1:17 PM, Jeff Layton jlayton at redhat.com wrote:
> > > > > 
> > > > > > The nfs-ganesha project has used lttng for quite some time to handle
> > > > > > tracing. Recently though, we decided to start building liburcu in as a
> > > > > > mandatory component, with an eye toward using it in certain areas.
> > > > > > 
> > > > > > Before this change, the code linked in liburcu-bp directly, but now we
> > > > > > just use liburcu. Unfortunately, when we enable tracepoints in the build
> > > > > > now, we get errors like this at link time:
> > > > > > 
> > > > > > --------------------8<----------------
> > > > > > [ 96%] Linking C executable ganesha.nfsd
> > > > > > /usr/bin/ld: libMainServices.a(nfs_worker_thread.c.o): undefined reference to
> > > > > > symbol 'rcu_gp_bp'
> > > > > > /usr/bin/ld: //usr/lib64/liburcu-bp.so.6: error adding symbols: DSO missing from
> > > > > > command line
> > > > > > collect2: error: ld returned 1 exit status
> > > > > > make[2]: *** [MainNFSD/CMakeFiles/ganesha.nfsd.dir/build.make:308:
> > > > > > MainNFSD/ganesha.nfsd] Error 1
> > > > > > make[1]: *** [CMakeFiles/Makefile2:2740:
> > > > > > MainNFSD/CMakeFiles/ganesha.nfsd.dir/all] Error 2
> > > > > > make: *** [Makefile:152: all] Error 2
> > > > > > --------------------8<----------------
> > > > > > 
> > > > > > nfs-ganesha defines _LGPL_SOURCE, and that makes lttng use the
> > > > > > redefinitions in tracepoint-rcu.h. If I disable _LGPL_SOURCE, it all
> > > > > > builds as expected.
> > > > > > 
> > > > > > I found a similar bug here:
> > > > > > 
> > > > > >    https://bugs.lttng.org/issues/1156
> > > > > > 
> > > > > > Any thoughts on the right fix for this? We'd like to eat our cake and
> > > > > > have it too, so that we can have _LGPL_SOURCE defined, lttng enabled,
> > > > > > and the urcu flavor be determined at runtime.
> > > > > 
> > > > > Hi!
> > > > > 
> > > > > It's great to hear that feedback. Indeed you are not the first one
> > > > > to hit this unfortunate limitation of liburcu API.
> > > > > 
> > > > > The use-case involves using many liburcu flavors within the same
> > > > > compile unit _with_ _LGPL_SOURCE defined. All the tricks done
> > > > > in urcu/map/*.h to map all flavors to a single API conflict
> > > > > with that use-case. And the fact that the "mb", memb", and
> > > > > signal flavors all share the same header prevents this
> > > > > even further. This becomes a real issue when trying to use
> > > > > the lttng-ust tracer instrumentation (which includes urcu-bp.h)
> > > > > along with other liburcu flavors in a compile unit.
> > > > > 
> > > > > So, I took the day to hack something out: beware, it's a complete
> > > > > overhaul of the liburcu tree. You can try it out in this private
> > > > > development branch:
> > > > > 
> > > > > https://github.com/compudj/userspace-rcu-dev/tree/multiflavor
> > > > > 
> > > > > Note: don't even try to follow the git commit history at the
> > > > > moment, this will need editing. ;)
> > > > > 
> > > > > A new unit test shows how it works:
> > > > > 
> > > > > https://github.com/compudj/userspace-rcu-dev/blob/multiflavor/tests/unit/test_urcu_multiflavor_single_unit.c
> > > > > 
> > > > > And while I was there, I did a lot of namespacing cleanups for the
> > > > > APIs, so we'll _definitely_ need to bump the major soname for this.
> > > > > However, I'm keeping the main use-cases of the API backward compatible
> > > > > (those which relied on the API mapping). Here is a dev branch of
> > > > > lttng-ust adapting to those API changes:
> > > > > 
> > > > > https://github.com/compudj/lttng-ust-dev/tree/urcu-multiflavor
> > > > > 
> > > > > Those who want to be able to use many liburcu flavors in a single
> > > > > compile unit would need to define URCU_NO_API_MAP, and use each flavor
> > > > > namespace directly, as done in the new unit test. I'm not particularly
> > > > > fond of the URCU_NO_API_MAP name. We could perhaps name it "URCU_MULTI_FLAVOR"
> > > > > or something else...
> > > 
> > > By the way, with the updated liburcu tree, there is no more need to
> > > define URCU_NO_API_MAP.
> > > 
> > > > > Please try it out and let me know how it goes.
> > > > > 
> > > > > Thanks,
> > > > > 
> > > > > Mathieu
> > > > > 
> > > > 
> > > > Thanks Mathieu,
> > > > 
> > > > Sorry for the delay, but this was rather difficult to test. I ended up
> > > > building a userspace-rcu package from this tree, but was unable to
> > > > build lttng-ust as a package from your tree, and so had to just do a
> > > > make/make install of it on the build host. Ditto for lttng-tools.
> > > 
> > > You will need lttng-ust from this tree:
> > > 
> > > https://github.com/compudj/lttng-ust-dev/tree/urcu-multiflavor
> > > 
> > > to build against the liburcu branch. This is causing your missing symbol
> > > error.
> > > 
> > > Thanks,
> > > 
> > > Mathieu
> > > 
> > 
> > That's the branch I'm using [1]. I basically checked out that branch,
> > and did:
> > 
> > configure --prefix=/usr
> > make
> > make install
> > 
> > ...and then did the same with lttng-tools (which ganesha also
> > requires). After that, I pulled down nfs-ganesha tree, checked out the
> > "next" branch and tried to build it, and got that same error.
> > 
> > It's certainly possible that we don't have something right in nfs-
> > ganesha that's causing this.
> > 
> > [1]: note that your lttng-ust branch also requires the attached patch.
> > Feel free to squash it in w/o attribution.
> > 
> > > With all of that, I reran the ganesha build and got the same error:
> > > > [ 97%] Linking C executable ganesha.nfsd
> > > > /usr/bin/ld: libMainServices.a(nfs_worker_thread.c.o): undefined reference to
> > > > symbol 'urcu_bp_register'
> > > > /usr/bin/ld: //usr/lib64/liburcu-bp.so.7: error adding symbols: DSO missing from
> > > > command line
> > > > collect2: error: ld returned 1 exit status
> > > > make[2]: *** [MainNFSD/CMakeFiles/ganesha.nfsd.dir/build.make:288:
> > > > MainNFSD/ganesha.nfsd] Error 1
> > > > make[1]: *** [CMakeFiles/Makefile2:2396:
> > > > MainNFSD/CMakeFiles/ganesha.nfsd.dir/all] Error 2
> > > > make: *** [Makefile:152: all] Error 2
> > > > 
> > > > Should we be passing in both -lurcu and -lurcu-bp on the link here?
> 
> If you are including lttng-ust headers with _LGPL_SOURCE defined, you will
> need to explicitly link against -lurcu-bp on distributions that do not preserve
> "needed" dependencies of liblttng-ust.
> 
> Does it work if you add the -lurcu-bp build dependency as well as
> -lurcu-memb ?
> 
> (note: liburcu.so still exists, but it's a duplicate of liburcu-memb.so
> which I introduce in 0.11 to cleanup the library namespace)
> 
> Thanks,
> 
> Mathieu
> 

Well...

Actually if I add in -lurcu-bp to the link, then it links correctly
even on the older code. I haven't yet tested it to see if lttng
actually still works, but it does link with that now. That may be the
best stopgap solution for the time being. I'll try to test it out soon,
but it may be a few days before I can get to it.

The other question is whether RCU will work properly if you have two
different flavors of urcu active at the same time? If the tracepoints
are using the bp RCU calls and the rest of the code is using a flavor
determined at runtime, is that going to cause problems?

Thanks,
-- 
Jeff Layton <jlayton at redhat.com>