[lttng-dev] New TLS usage in libgcc_s.so.1, compatibility impact

[Florian Weimer]

* Mathieu Desnoyers:

> On 2024-01-13 07:49, Florian Weimer via lttng-dev wrote:
>> This commit
>> commit 8abddb187b33480d8827f44ec655f45734a1749d
>> Author: Andrew Burgess <andrew.burgess at embecosm.com>
>> Date:   Sat Aug 5 14:31:06 2023 +0200
>>      libgcc: support heap-based trampolines
>>           Add support for heap-based trampolines on x86_64-linux,
>> aarch64-linux,
>>      and x86_64-darwin. Implement the __builtin_nested_func_ptr_created and
>>      __builtin_nested_func_ptr_deleted functions for these targets.
>>           Co-Authored-By: Maxim Blinov <maxim.blinov at embecosm.com>
>>      Co-Authored-By: Iain Sandoe <iain at sandoe.co.uk>
>>      Co-Authored-By: Francois-Xavier Coudert <fxcoudert at gcc.gnu.org>
>> added TLS usage to libgcc_s.so.1.  The way that libgcc_s is
>> currently
>> built, it ends up using a dynamic TLS variant on the Linux targets.
>> This means that there is no up-front TLS allocation with glibc (but
>> there would be one with musl).
>
> Trying to wrap my head around this:
>
> If I get this right, the previous behavior was that glibc did allocate
> global-dynamic variables from libraries which are preloaded and loaded
> on c startup as if they were initial-exec, but now that libgcc_s.so.1
> has a dynamic TLS variable, all those libraries loaded on c startup that
> have global-dynamic TLS do not get the initial allocation special
> treatment anymore. Is that more or less correct ?

Ahh.  I had forgotten about this aspect.  The allocation from the static
TLS area still happens as before.

> I've prepared a change for lttng-ust to move the lttng-ust libc wrapper
> "malloc nesting" guard variable from global-dynamic to initial-exec:
>
> https://review.lttng.org/c/lttng-ust/+/11677 Fix: libc wrapper: use initial-exec for malloc_nesting TLS

I don't know if this is completely sufficient if there are other TLS
variables in the lttng stack.

> This should help for the infinite recursion issue, but if my understanding
> is correct about the impact of effectively changing the behavior used
> for global-dynamic variables in preloaded and on-startup-loaded libraries
> introduced by this libgcc change, I suspect we have other new issues here,
> such as problems with async-signal safety of other global-dynamic variables
> within LTTng-UST.

This didn't change, and the allocation is not done lazily (contrary to
what I might have written before).  But even on the __tls_get_addr fast
path, we check the TLS generation counter, and if that has changed, we
do extra bookkeeping work.  TLS usage in libgcc_s.so.1 means that in the
now-failing test, the generation counter changed.  Before bug 19924

  TLS performance degradation after dlopen 
  <https://sourceware.org/bugzilla/show_bug.cgi?id=19924>

was fixed, we did not do this bookkeeping work, which is why the problem
didn't occur.

General use of lttng should be fine, I think, only the malloc wrapper
has this problem.

> But moving all TLS variables used by lttng-ust from global-dynamic to
> initial-exec is tricky, because a prior attempt to do so introduced
> regressions in use-cases where lttng-ust was dlopen'd by Java or
> Python, AFAIU situations where the runtimes were already using most of
> the extra memory pool for dlopen'd libraries initial-exec variables,
> causing dlopen of lttng-ust to fail.

Oh, right, that makes it quite difficult.  Could you link a private copy
of the libraries into the wrapper that uses initial-exec TLS?

Thanks,
Florian