[lttng-dev] userspace-rcu test scripts patches (was: ports/168339: [patch] sysutils/userspace-rcu update to 0.7.2

Mon May 28 11:00:36 EDT 2012

* Hirohisa Yamaguchi (umq at ueo.co.jp) wrote:
> Hi, Mathieu
> 
> Thanks for your comments.
> 
> At Sun, 27 May 2012 14:42:55 -0400,
> Mathieu Desnoyers wrote:
> > > The changes for FreeBSD in test scripts are mainly consists of
> > > following:
> 
> > This brings an interesting point: all the tests scripts start with
> > "#!/bin/sh", but use bash-isms. I guess the first question would be:
> > should be simply choose make test execution depend on bash (and push
> > #!/bin/bash prefix), since this is only needed for tests provided with
> > the package sources, and not needed for deployment of the library on
> > systems ? Of course, dependency on bash for in-source-tree tests would
> > have to be documented in the README.
> 
> I think there're two different purposes of tests:
>  1. for developers to ensure the implementations are correct
>  2. for users to see his/her environment is safe to use the feature
>     -- built without an error does not mean it runs okay
> 
> I think bashisms do not suit for latter purpose.

I agree that we should not depend on "bash" for any script installed
into the system.

However, the "tests/" subdirectory is not currently installed on the
user system. It's only used for in-tree testing, and "make check"
execution.

There might come a point where we want to deploy some tests into the
system so the library users can run them, I don't know. Maybe not yet
though, as we would need to clean them up first.

So at this stage, I would be fine with specifying in the README that
running tests depends on bash. However, if we can provide a wrapper for
all the commands we need, which would be sourced by all scripts in the
tests/ directory, which implements all wrappers into per-wrapper files,
sourced by one toplevel wrapper file, I would be open to that too, given
that it would make deployment of tests simpler for systems lacking bash
support. e.g.

tests/wrapper/all.sh
           . wrapper/seq.sh
           .....

tests/wrapper/seq.sh
tests/wrapper/....

each test script:

#!/bin/sh
. wrapper/all.sh

.....

> 
> > >  2. ash (i.e. /bin/sh in FreeBSD) does not have += operator:
> > >     replace a+="b" to a="${a}b"
> > The question on whether we specifically go for bash for in-source-tree
> > test scripts needs to be cleared before I proceed to this change.
> 
> If the test scripts are not written for all users, there's no problem
> when its shown in a document or somewhere.
> 
> 
> > >  3. BSD flavoured system have jot(1) instead of seq(1).
> > >     manual of jot(1) can be read at http://man.freebsd.org/jot/1
> 
> > For some reason, "seq" is installed on my FreeBSD 9.0 system, but not on
> > my 8.2. We could implement a seq-wrapper.sh in tests/ that can be
> > sourced by test shell scripts which would use the right seq
> > implementation underneath (and change the parameters accordingly).
> 
> > >  4. ash does not have PIPESTATUS:
> > Should decide for bash or sh first.
> 
> I agree, as I've mentioned above.
> 
> 
> > >  5. NUM_CPUS might be determined at runtime:
> > So I would recommend making this a
> > parameter, and autodetect only if no parameter is provided. The
> > autodetect would indeed have to be OS-specific. We could implement a
> > get_nr_cpu.sh script within tests/ that detects the number of CPUs on
> > the current OS.
> 
> That sounds good.

Can you prepare a patch for this ?

> 
> 
> At Sun, 27 May 2012 15:44:12 -0400,
> Mathieu Desnoyers wrote:
> > 
> > * Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote:
> > [...]
> > > >  #  btw, test_perthreadlock under FreeBSD 10-CURRENT always fails
> > > >  #  (releases prior to 9.0 are okay).
> > 
> > -> I guess you mean "releases up to, and including 9.0, are OK".
> 
> Ah, yes.
> 
> 
> > > >  #  The change in FreeBSD libthr might cause the error.
> > > >  #  I haven't looked into it yet.
> > > What are the details of your test invokation ? (how many reader/writer
> > > threads, how many cores do you have ? Is this on hardware or virtualized ?)
> 
> > Actually, the most interesting thing to know here would be your
> > architecture. Moreover, as a hint, the fact that the problem seems to
> > only arise in the perthreadlock test, which generates a lot of
> > contention when the writer takes all the reader locks, I would guess
> > that the bug would sit somewhere in the contention handling code, which
> > could be either at the user-space library or kernel-level side.
> 
> I've found the error first on a build farm called redports.org, which
> is opened for FreeBSD port maintainers to run build and tests.
> I'm not sure the architecture of the build farm, but some test shows
> it runs on Phenom II X6 1075T. There's no way to know it is on
> hardware or virtualized.
>  #  The test on the build farm ran with NUM_CPUS=6, I don't have logs
>  #  currently since they're expired.
> Any parameters in runall.sh lead to error.
> 
> My box that reproduce the error is a VMware guest on my MacBook.
>  #  Intel(R) Core(TM)2 Duo CPU     T8300  @ 2.40GHz
> It is a dual-core CPU, and I ran the tests with NUM_CPUS=2.

OK, so it seems to be reproducible on both AMD and Intel, that's good
news.

> 
> I attach a backtrace of a corefile when I ran:
>  test_perthreadlock 1 1 10 -d 0 -b 32768
> I only have stripped libthr in my environment, I haven't managed to
> build one.

You might want to compile userspace rcu with:

make CFLAGS=-g

to get debug symbols for the library. It will provide a more helpful
backtrace. We should consider that the issue can come from a race
between pthread_create and pthread_mutex_lock too.

Thanks!

Mathieu

> 
> 
> Regards,
> -- 
> 
> 	Hirohisa Yamaguchi
> 	  umq at ueo.co.jp

> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...
> Core was generated by `test_perthreadlock'.
> Program terminated with signal 10, Bus error.
> Reading symbols from /lib/libthr.so.3...done.
> Loaded symbols for /lib/libthr.so.3
> Reading symbols from /lib/libc.so.7...done.
> Loaded symbols for /lib/libc.so.7
> Reading symbols from /libexec/ld-elf.so.1...done.
> Loaded symbols for /libexec/ld-elf.so.1
> #0  0x000000080082d09b in pthread_mutex_lock () from /lib/libthr.so.3
> (gdb) info thread
>   3 process 100144  0x0000000800b3207c in nanosleep () from /lib/libc.so.7
>   2 process 101026  0x0000000800826b90 in pthread_create () from /lib/libthr.so.3
> * 1 process 101025  0x000000080082d09b in pthread_mutex_lock () from /lib/libthr.so.3
> (gdb) thread apply all bt full
> 
> Thread 3 (process 100144):
> #0  0x0000000800b3207c in nanosleep () from /lib/libc.so.7
> No symbol table info available.
> #1  0x0000000800a952e8 in sleep () from /lib/libc.so.7
> No symbol table info available.
> #2  0x0000000800828e18 in sleep () from /lib/libthr.so.3
> No symbol table info available.
> #3  0x00000000004011a8 in main (argc=1, argv=0x7fffffffda90) at test_perthreadlock.c:397
> 	err = 
> Thread 2 (process 101026):
> #0  0x0000000800826b90 in pthread_create () from /lib/libthr.so.3
> No symbol table info available.
> #1  0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> Thread 1 (process 101025):
> #0  0x000000080082d09b in pthread_mutex_lock () from /lib/libthr.so.3
> No symbol table info available.
> #1  0x00000000004016a3 in thr_reader (data=) at test_perthreadlock.c:211
> 	tidx = 0
> 	__func__ = "thr_reader"
> #2  0x0000000800826cdd in pthread_create () from /lib/libthr.so.3
> No symbol table info available.
> #3  0x0000000000000000 in ?? ()
> No symbol table info available.
> (gdb) q

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com