[lttng-dev] Userspace RCU: workqueue with batching, cheap wakeup, and work stealing

[Mathieu Desnoyers]

----- Original Message -----
> From: "Lai Jiangshan" <laijs at cn.fujitsu.com>
> To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
> Cc: "Ben Maurer" <bmaurer at fb.com>, "lttng-dev" <lttng-dev at lists.lttng.org>, "Yannick Brosseau" <scientist at fb.com>,
> "Paul E. McKenney" <paulmck at linux.vnet.ibm.com>, "Stephen Hemminger" <stephen at networkplumber.org>
> Sent: Thursday, October 23, 2014 9:45:59 PM
> Subject: Re: Userspace RCU: workqueue with batching, cheap wakeup, and work stealing
> 
> On 10/23/2014 09:06 PM, Mathieu Desnoyers wrote:
> > ----- Original Message -----
> >> From: "Lai Jiangshan" <laijs at cn.fujitsu.com>
> >> To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
> >> Cc: "Ben Maurer" <bmaurer at fb.com>, "lttng-dev"
> >> <lttng-dev at lists.lttng.org>, "Yannick Brosseau" <scientist at fb.com>,
> >> "Paul E. McKenney" <paulmck at linux.vnet.ibm.com>, "Stephen Hemminger"
> >> <stephen at networkplumber.org>
> >> Sent: Thursday, October 23, 2014 12:26:58 AM
> >> Subject: Re: Userspace RCU: workqueue with batching, cheap wakeup, and
> >> work stealing
> > 
[...]
> >>
> >> It is not a good idea for a worker to call synchronize_rcu() (in
> >> urcu_accept_work()).
> >> A new work may be coming soon.
> > 
> > The reason why the worker calls synchronize_rcu() there is to protect
> > the waitqueue lfstack "pop" operation from ABA. It matches the RCU
> > read-side critical sections encompassing urcu_dequeue_wake_single().
> > 
> > Another way to achieve this would be to grab a mutex across calls to
> > urcu_dequeue_wake_single(), but unfortunately this call needs to be
> > done by the dispatcher thread, and I'm trying to keep a low-latency
> > bias in favor of the dispatcher here. In this case, it is done at the
> > expense of overhead in the worker when the worker is about to busy-wait
> > and eventually block, so I thought the tradeoff was not so bad.
> 
> "single worker pattern" does be a frequent use-case.
> you can image the lag and jitter in this case.

I'd be happy to find a way to reduce unnecessary jitter, indeed.
However, we need to keep in mind that this is issued only when
the worker thread has no more work to do. And if there are few
registered RCU threads, synchronize_rcu() should be pretty quick,
especially now that multiple grace periods can run concurrently.

> 
> 
> How about a mutex&rcu protected idle-workers?
> 
> idle-worker is wokenup via rcu-protected scheme.
> and dequeued&requeued (in urcu_accept_work() or urcu_worker_unregister()) via
> mutex-protected scheme.

I'm not sure I understand your proposal. Can you give more
detail about the RCU protected scheme you refer to here,
along with how it is intended to interact with mutex ?

Note that if we use RCU read-side around lfstack pop, it
needs to be matched by a grace period between all other
"pop"/"pop all" and re-push to deal with ABA.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com