[lttng-dev] Can userspace RCU be used with shared memory?

[Mathieu Desnoyers]

* Devanjan Choudhury (devanjan.dev at gmail.com) wrote:
> Hello Mathieu,
> 
> We are interested in using userspace rcu for one of our projects.

Great!

> Our main requirement is that we have some hash tables stored in shared
> memory which will be accessed by multiple reader processes. We have one
> single writer process. Since our use case is more read intensive compared
> to write, we have choosen to use RCU to synchronize the shared data.
> 
> Now, when our writer process updates some nodes(lets say delete some node)
> in hash tables stored in shared memory, it must call synchronize_rcu(),
> before it can actually free up the node. However, one note in the
> discussion (
> http://lists.lttng.org/pipermail/lttng-dev/2012-December/019295.html)
> states that synchronize_rcu() (or call_rcu()) executing in the context of
> one process will not see the RCU read-side locks of another process. If
> this is the case, then how/when can the writer process delete the node
> permanently. Should we need to consider doing garbage collection on our own?

The currently existing flavors of userspace RCU don't support
multi-process use of shared data structures through shared memory.
Currently, we require that all threads of a process that can access data
under RCU read-side critical section be registered
(rcu_register_thread). This adds them to a linked list of threads
monitored by synchronize_rcu().

We could very well create a new flavor specifically for this use-case
though. It would rely on a shared memory area to store the linked list
of threads registered as RCU readers, so this list could be shared
between processes.

> 
> Also, if I undertsand correctly, shyncronize_rcu() does the following
> basically:
> 
> void synchronize_rcu(void) { int cpu; for_each_online_cpu(cpu) run_on(cpu);

Not exactly: it performs synchronization with each registered thread
(not cpu). The synchronization is not a "run on", but rather monitoring
variables (which updates are surrounded by memory barriers) that ensure
each monitored thread has passed through a quiescent state before
synchronize_rcu() is allowed to complete.

> }
> 
> So, when our writer process call synchronize_rcu(), it will basically be
> made to run on all the CPUs. Shouldn't this be sufficient to ensure that
> any other reader process which had started a read rcu critical section has
> completed?

No, because we cannot disable preemption from user-space, and therefore
even if you happen to run on every CPU, it does not mean that there are
no RCU critical sections still in progress within preempted threads.

Best regards,

Mathieu

> 
> 
> -- 
> Regards,
> Devanjan

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com