[ltt-dev] (forw) [rostedt at goodmis.org: Re: [PATCH 0/2] jump label: 2.6.38 updates]

Mon Feb 14 20:16:37 EST 2011

* Paul E. McKenney (paulmck at linux.vnet.ibm.com) wrote:
> On Mon, Feb 14, 2011 at 06:23:35PM -0500, Mathieu Desnoyers wrote:
> > What Steven and Matt are saying is that LL/SC vs cached read/writes
> > (standard load/store) don't seem to fit well together on some PowerPC:
> > LL/SC would be bypassing the cache and affecting memory directly, while
> > standard loads would be reading from cache, a possibly stale value.
> 
> News to me...
> 
> > So what they seem to say is that there are powerpc architectures out
> > there for which, given a combination of:
> > 
> > cpu 1:
> > 
> >   xchg(&memloc, newval);
> > 
> > cpu 2:
> > 
> >   while (memloc != newval)
> >     cpu_relax();
> > 
> > cpu 2 might end up in an infinite loop because the SC performed by
> > xchg on CPU 1 would never be seen by CPU 2's cache.
> > 
> > It's as if LL/SC are cache-coherent one wrt another, and standard
> > load/stores are cache-coherent on wrt another, but that they are not
> > when mixed together for the same memory locations.
> > 
> > Now I don't know if this can be verified, but it's worth investigating.
> 
> Again, stw is a simple store, not a read-modify-write instruction.
> The xchg macro would instead do an lwarx/stwcx sequence, but that should
> still play nicely with load.  The change in the patch would instead
> prevent the compiler from optimizing the loop into an infinite loop.

Yeah, I think I mistakenly thought Matt was talking about a powerpc
variant, when he was in fact talking some kind of top-secret
non-mainline architecture. ;)

No need to worry for now then.

Mathieu

> 
> 							Thanx, Paul
> 
> > Thanks,
> > 
> > Mathieu
> > 
> > * Paul E. McKenney (paulmck at linux.vnet.ibm.com) wrote:
> > > It is the memory barriers that must match, not LL/SC or cmpxchg.
> > > 
> > > Do we really want atomic ops to be used when initializing atomic
> > > variables?  Or is this a blackfin-ism?  In which case blackfin might need
> > > something special, but that should not carry over to cache-coherent CPUs.
> > > 
> > > 							Thanx, Paul
> > > 
> > > On Mon, Feb 14, 2011 at 05:31:48PM -0500, Mathieu Desnoyers wrote:
> > > > I'm also thinking that the combination of rcu_cmpxchg_pointer() and
> > > > rcu_dereference() are problematic, because we use ll/sc for the cmpxchg
> > > > without the matching lwz on the read-side. We should probably also use a
> > > > matching stw for rcu_assign_pointer if we want to support this case.
> > > > 
> > > > Mathieu
> > > > 
> > > > * Mathieu Desnoyers (mathieu.desnoyers at polymtl.ca) wrote:
> > > > > Hi Paul,
> > > > > 
> > > > > Please see the message below. It looks like the liburcu
> > > > > uatomic_read()/uatomic_set() implementations would need to be moved to
> > > > > lwz/stw if what Steven says below is true. It seems to be in sync with
> > > > > what is done in the libatomic ops implementation.
> > > > > 
> > > > > Thoughts ?
> > > > > 
> > > > > Mathieu
> > > > > 
> > > > > ----- Forwarded message from Steven Rostedt <rostedt at goodmis.org> -----
> > > > > 
> > > > > Date: Mon, 14 Feb 2011 16:39:36 -0500
> > > > > To: Peter Zijlstra <peterz at infradead.org>
> > > > > Cc: Will Newton <will.newton at gmail.com>, Jason Baron <jbaron at redhat.com>,
> > > > > 	Mathieu Desnoyers <mathieu.desnoyers at polymtl.ca>, hpa at zytor.com,
> > > > > 	mingo at elte.hu, tglx at linutronix.de, andi at firstfloor.org,
> > > > > 	roland at redhat.com, rth at redhat.com, masami.hiramatsu.pt at hitachi.com,
> > > > > 	fweisbec at gmail.com, avi at redhat.com, davem at davemloft.net,
> > > > > 	sam at ravnborg.org, ddaney at caviumnetworks.com, michael at ellerman.id.au,
> > > > > 	linux-kernel at vger.kernel.org, Mike Frysinger <vapier at gentoo.org>,
> > > > > 	Chris Metcalf <cmetcalf at tilera.com>, dhowells <dhowells at redhat.com>,
> > > > > 	Martin Schwidefsky <schwidefsky at de.ibm.com>,
> > > > > 	"heiko.carstens" <heiko.carstens at de.ibm.com>,
> > > > > 	benh <benh at kernel.crashing.org>
> > > > > X-Mailer: Evolution 2.30.3 
> > > > > From: Steven Rostedt <rostedt at goodmis.org>
> > > > > Subject: Re: [PATCH 0/2] jump label: 2.6.38 updates
> > > > > 
> > > > > On Mon, 2011-02-14 at 16:29 -0500, Steven Rostedt wrote:
> > > > > 
> > > > > > > while (atomic_read(&foo) != n)
> > > > > > >   cpu_relax();
> > > > > > > 
> > > > > > > and the problem is that cpu_relax() doesn't know which particular
> > > > > > > cacheline to flush in order to make things go faster, hm?
> > > > > > 
> > > > > > But what about any global variable? Can't we also just have:
> > > > > > 
> > > > > > 	while (global != n)
> > > > > > 		cpu_relax();
> > > > > > 
> > > > > > ?
> > > > > 
> > > > > Matt Fleming answered this for me on IRC, and I'll share the answer here
> > > > > (for those that are dying to know ;)
> > > > > 
> > > > > Seems that the atomic_inc() uses ll/sc operations that do not affect the
> > > > > cache. Thus the problem is only with atomic_read() as
> > > > > 
> > > > > 	while(atomic_read(&foo) != n)
> > > > > 		cpu_relax();
> > > > > 
> > > > > Will just check the cache version of foo. But because ll/sc skips the
> > > > > cache, the foo will never update. That is, atomic_inc() and friends do
> > > > > not touch the cache, and the CPU spinning in this loop will is only
> > > > > checking the cache, and will spin forever.
> > > > > 
> > > > > Thus it is not about global, as global is updated by normal means and
> > > > > will update the caches. atomic_t is updated via the ll/sc that ignores
> > > > > the cache and causes all this to break down. IOW... broken hardware ;)
> > > > > 
> > > > > Matt, feel free to correct this if it is wrong.
> > > > > 
> > > > > -- Steve
> > > > > 
> > > > > -- 
> > > > > Mathieu Desnoyers
> > > > > Operating System Efficiency R&D Consultant
> > > > > EfficiOS Inc.
> > > > > http://www.efficios.com
> > > > 
> > > > -- 
> > > > Mathieu Desnoyers
> > > > Operating System Efficiency R&D Consultant
> > > > EfficiOS Inc.
> > > > http://www.efficios.com
> > > 
> > 
> > -- 
> > Mathieu Desnoyers
> > Operating System Efficiency R&D Consultant
> > EfficiOS Inc.
> > http://www.efficios.com
> 

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com