Integrating QSBR with libuv

[Mathieu Desnoyers]

On 2025-07-16 00:09, Ondřej Surý via lttng-dev wrote:
> Hi,
> 
> currently, the BIND 9 design involves multiple threads with per-thread 
> event loop from libuv (uv_loop).
> 
> We are using the default rcu variant, but it seems like an ideal place 
> to switch to QSBR as we can call quiescent state periodically before 
> polling on the loop.
> 
> This comes with one trouble though. The incoming traffic is distributed 
> using SO_REUSEPORT load-balancing in the kernel and if the hashing 
> algorithm is too simple or the traffic just hashes only into 1..n-1 
> threads, the memory reclamation will never trigger (means indefinite 
> wait for synchronize_rcu() or indefinite queue in the call_rcu).
> 
> Now, I have basically two solutions I can think of:
> 
> 1. Bring the threads offline before polling and online after polling. 

If the threads block when polling, the QSBR URCU flavor would indeed
require them to be placed in "offline" state to prevent arbitrary
grace period delays.

> This has two problems: a) there’s overhead related to calling 
> thread_offline/thread_online and it is unclear to me how big overhead 
> this is;

Making a thread offline is cheap in qsbr. The overhead is somewhat
similar to marking a quiescent state. The offlining is pretty much
this:

static inline void _urcu_qsbr_thread_offline(void)
{
         urcu_assert_debug(URCU_TLS(urcu_qsbr_reader).registered);
         uatomic_store(&URCU_TLS(urcu_qsbr_reader).ctr, 0, CMM_SEQ_CST);
         /* write URCU_TLS(urcu_qsbr_reader).ctr before read futex */
         urcu_qsbr_wake_up_gp();
         cmm_barrier();  /* Ensure the compiler does not reorder us with mutex */
}

So it mostly stores to a TLS variable, and evaluates a "waiting" state
from another TLS variable to issue a futex(2) FUTEX_WAKE system call if
there is a grace period waiting for this thread.

> b) currently, there is no way to bring the thread online before 
> libuv callbacks are called - https://github.com/libuv/libuv/issues/4839 
> <https://github.com/libuv/libuv/issues/4839#issuecomment-3076493238>
> 

Figuring out a way to make the thread online again between the
return from polling and execution of user callbacks would seem
like a sane approach here.

I have not looked at libuv, but perhaps there a simple way to mark the
thread offline and the online again just around the call to poll or
epoll.

> 2. Add short per event loop periodic timer that would call the quiescent 
> state every <n> ms, but I have a vague recollection that unneeded 
> periodic timers mess with the power management as they wake up the 
> process even when it would be otherwise completely idle.

Indeed, this will likely hurt your power consumption. Also, you'll
end up with a quiescent state emitted from a signal handler, which can
nest over any code that does not have signals disabled, therefore
requiring RCU critical sections to disable signals. Probably not what
you are looking for.

> 
> The bonus third option is to give up on this because there are lower 
> hanging fruits to improve the performance than the membarrier vs QSBR. 
> That’s certainly true, but it is an interesting software engineering 
> problem, so I am not ready to give this up easily ;). And coming up with 
> a solution to this could lead to at least a blog post that more project 
> could benefit from a solution to this. (For a moment, pausing from the 
> grim world we live in now where every idea gets sucked into LLM and 
> venture capital.)

That's also a good option to keep in mind! :)

Thanks,

Mathieu

> 
> Ondrej
> --
> Ondřej Surý (He/Him)


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com