[lttng-dev] Alternative to signals/sys_membarrier() in liburcu
Paul E. McKenney
paulmck at linux.vnet.ibm.com
Fri Mar 13 10:18:53 EDT 2015
On Fri, Mar 13, 2015 at 09:07:43AM +0100, Ingo Molnar wrote:
> * Mathieu Desnoyers <mathieu.desnoyers at efficios.com> wrote:
> > ----- Original Message -----
> > > From: "Linus Torvalds" <torvalds at linux-foundation.org>
> > > To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
> > > Cc: "Michael Sullivan" <sully at msully.net>, lttng-dev at lists.lttng.org, "LKML" <linux-kernel at vger.kernel.org>, "Paul E.
> > > McKenney" <paulmck at linux.vnet.ibm.com>, "Peter Zijlstra" <peterz at infradead.org>, "Ingo Molnar" <mingo at kernel.org>,
> > > "Thomas Gleixner" <tglx at linutronix.de>, "Steven Rostedt" <rostedt at goodmis.org>
> > > Sent: Thursday, March 12, 2015 5:47:05 PM
> > > Subject: Re: Alternative to signals/sys_membarrier() in liburcu
> > >
> > > On Thu, Mar 12, 2015 at 1:53 PM, Mathieu Desnoyers
> > > <mathieu.desnoyers at efficios.com> wrote:
> > > >
> > > > So the question as it stands appears to be: would you be comfortable
> > > > having users abuse mprotect(), relying on its side-effect of issuing
> > > > a smp_mb() on each targeted CPU for the TLB shootdown, as
> > > > an effective implementation of process-wide memory barrier ?
> > >
> > > Be *very* careful.
> > >
> > > Just yesterday, in another thread (discussing the auto-numa TLB
> > > performance regression), we were discussing skipping the TLB
> > > invalidates entirely if the mprotect relaxes the protections.
> We have such code already in mm/mprotect.c, introduced in:
> 10c1045f28e8 mm: numa: avoid unnecessary TLB flushes when setting NUMA hinting entries
> which does:
> /* Avoid TLB flush if possible */
> if (pte_protnone(oldpte))
> > > Because if you *used* to be read-only, and them mprotect()
> > > something so that it is read-write, there really is no need to
> > > send a TLB invalidate, at least on x86. You can just change the
> > > page tables, and *if* any entries are stale in the TLB they'll
> > > take a microfault on access and then just reload the TLB.
> > >
> > > So mprotect() to a more permissive mode is not necessarily
> > > serializing.
> > The idea here is to always mprotect() to a more restrictive mode,
> > which should trigger the TLB shootdown.
> So what happens if a CPU comes around that integrates TLB shootdown
> management into its cache coherency protocol? In such a case IPI
> traffic can be skipped: the memory bus messages take care of TLB
> flushes in most cases.
> It's a natural optimization IMHO, because TLB flushes are conceptually
> pretty close to the synchronization mechanisms inherent in data cache
> coherency protocols:
> This could be implemented for example by a CPU that knows about ptes
> and handles their modification differently: when a pte is modified it
> will broadcast a MESI invalidation message not just for the cacheline
> belonging to the pte's physical address, but also an 'invalidate TLB'
> MESI message for the pte value's page.
> The TLB shootdown would either be guaranteed within the MESI
> transaction, or there would either be a deterministic timing
> guarantee, or some explicit synchronization mechanism (new
> instruction) to make sure the remote TLB(s) got shot down.
> Every form of this would be way faster than sending interrupts. New
> OSs could support this by the hardware telling them in which cases the
> TLBs are 'auto-flushed', while old OSs would still be compatible by
> sending (now pointless) TLB shootdown IPIs.
> So it's a relatively straightforward hardware optimization IMHO:
> assuming TLB flushes are considered important enough to complicate the
> cacheline state machine (which I think they currently aren't).
> So in this case there's no interrupt and no other interruption of the
> remote CPU's flow of execution in any fashion that could advance the
> RCU state machine.
> What do you think?
I agree -- there really have been systems able to flush remote TLBs
without interrupting the remote CPU.
So, given the fact that the userspace RCU library does now see
some real-world use, is it now time for Mathieu to resubmit his
More information about the lttng-dev