From david.goulet at polymtl.ca Wed Aug 1 10:02:30 2012 From: david.goulet at polymtl.ca (David Goulet) Date: Wed, 01 Aug 2012 10:02:30 -0400 Subject: [lttng-dev] [lttng-tools PATCH] lttng-tools python module In-Reply-To: <50182FEF.6070504@gmail.com> References: <1343326167-11522-1-git-send-email-danny.serres@efficios.com> <50182739.3030604@polymtl.ca> <50182FEF.6070504@gmail.com> Message-ID: <501936F6.7010800@polymtl.ca> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 31/07/12 03:20 PM, Yannick Brosseau wrote: > > On 2012-07-31 14:43, David Goulet wrote: > > >> Apart from those minor issues, I have two questions about this patch. > First, >> why is the python module in src/ and not in extras/ ? I consider src/ to >> be supported and stable code shipped with the installation and/or in >> packages. Also, the src/ directory is the core code of lttng-tools and >> this module is more an extra useful tool. > > I'm not sure it belongs to extras, since its an actual library that do > something and that is installed on the system. Yes but I see that as a very useful contribution to the project but not part of the core of lttng-tools which is lttng-ctl, sessiond and lttng cmdline. The point I'm getting to is that if it's in src/, the question is do we want to support it along side of liblttng-ctl. - From my point of view, I'm not sure it is, for now, a good idea since it adds a considerable amount work to fully support (at least on my side) a library that is an extra functionality and for that I see this in extras/ for now. > > If you do not want it in src, we could create a top level python directory > or a binding/python (which allows for other language later on). On the > other hand, its a binding of the liblttng-ctl, so having the binding live > with the lib directly can be a good idea. > > We should probably do a quick survey of other projects approach. Indeed. Cheers! David > > >> Second, _IF_ this goes into src/, please move the tests to the tests/ >> directory else we are going to make it self contained in extras/ which > is good >> right now. > > yes, good point. > > Yannick > > > > > > _______________________________________________ lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJQGTb2AAoJEELoaioR9I02Z1kH/1P5jyi8pmPkpYas2D1dHXYY ZIUwdvmFBaUCOOEk4onf5tOEBMt4QtJBlURa9OvgugEOJLCHCtm2ZgUV6d6YEOzK wyKmtel/LA7L2q4/zzpbzI2dr66zNCZ2mBW0tYP2/O7UgCPxgHA5WqJ8yeTMG/GU f1fjhArxevH31KYKKK0dKQzEj8ZUqQlrC70SyhOUIBZPaqCMPU2hrU0noh1sKTCx Ixd5XirGmcNIVjB9wF/M4gr2yi80aVYfNweBNuWpn8lvG1maiq7nGj/bLAM7wPo+ OGHlaknAg7LQz+7dcvZLZ4Yxx57KpMOVPyv++LApIj6/QYv3/hRoLGcHb89+Yhw= =0vV7 -----END PGP SIGNATURE----- From christian.babeux at efficios.com Wed Aug 1 11:13:27 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Wed, 1 Aug 2012 11:13:27 -0400 Subject: [lttng-dev] [lttng-tools PATCH] lttng-tools python module In-Reply-To: <501936F6.7010800@polymtl.ca> References: <1343326167-11522-1-git-send-email-danny.serres@efficios.com> <50182739.3030604@polymtl.ca> <50182FEF.6070504@gmail.com> <501936F6.7010800@polymtl.ca> Message-ID: > - From my point of view, I'm not sure it is, for now, a good idea since it adds > a considerable amount work to fully support (at least on my side) a library > that is an extra functionality and for that I see this in extras/ for now. +1 for extras. I suggest extras/bindings/swig/python. The Apache Subversion project has a similar tree [1]. Christian [1] - http://svn.apache.org/repos/asf/subversion/trunk/subversion/bindings/swig/ From david.goulet at polymtl.ca Wed Aug 1 12:02:30 2012 From: david.goulet at polymtl.ca (David Goulet) Date: Wed, 01 Aug 2012 12:02:30 -0400 Subject: [lttng-dev] [lttng-tools PATCH] lttng-tools python module In-Reply-To: References: <1343326167-11522-1-git-send-email-danny.serres@efficios.com> <50182739.3030604@polymtl.ca> <50182FEF.6070504@gmail.com> <501936F6.7010800@polymtl.ca> Message-ID: <50195316.6070606@polymtl.ca> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I agree with that. The main reason for extras/ is the fact that, for now, this lib would not be supported meaning that if liblttng-ctl changes, these bindings will NOT follow it unless someone submit a patch. If we decide at some point to fully support it, I have no problem seeing it in src/ Cheers! David On 01/08/12 11:13 AM, Christian Babeux wrote: >> - From my point of view, I'm not sure it is, for now, a good idea since >> it adds a considerable amount work to fully support (at least on my side) >> a library that is an extra functionality and for that I see this in >> extras/ for now. > > +1 for extras. I suggest extras/bindings/swig/python. The Apache Subversion > project has a similar tree [1]. > > Christian > > [1] - > http://svn.apache.org/repos/asf/subversion/trunk/subversion/bindings/swig/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJQGVMWAAoJEELoaioR9I02m7gH/2cnsQ4x4fmdmKKy422l7e9c 8mzsgkWR2NsJHF7d0E7lvg0Be8nsxq59FD0xvN1jqoh4I+aeMt+0Ulz/Z4Gr6c3U gXLRjDPGyIrkGdL/bFase6WZQKHtvI+6sDIgxDj0TbSjeZ0YOOaAASP6eFTOaWF1 DhyJBtkIIVb8pjU8dYMMWFSkFhlGpCYDangnx09OSY15+AiLQIYd1HJdZoDHwfjP M9V3cPEqvDFmc4H0qkgX4g5ACN8XlvSN2TZGimDo7KP8qMSRtBxTEy8GrOwXMi+O pdNVu1gk+f0tAwTFCPGSAXFCv2/B4/wDnBTvu/m06p/NB5sihsskdv3rXK3o4DQ= =VL+7 -----END PGP SIGNATURE----- From yannick.brosseau at gmail.com Wed Aug 1 12:12:15 2012 From: yannick.brosseau at gmail.com (Yannick Brosseau) Date: Wed, 01 Aug 2012 12:12:15 -0400 Subject: [lttng-dev] [lttng-tools PATCH] lttng-tools python module In-Reply-To: <50195316.6070606@polymtl.ca> References: <1343326167-11522-1-git-send-email-danny.serres@efficios.com> <50182739.3030604@polymtl.ca> <50182FEF.6070504@gmail.com> <501936F6.7010800@polymtl.ca> <50195316.6070606@polymtl.ca> Message-ID: <5019555F.9090108@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-08-01 12:02, David Goulet wrote: > I agree with that. > > The main reason for extras/ is the fact that, for now, this lib would not be > supported meaning that if liblttng-ctl changes, these bindings will NOT follow > it unless someone submit a patch. If we decide at some point to fully support If its not supported, maybe we should not put it upstream. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlAZVV8ACgkQFQrZ7GzHX2qbjQCgnpAkLTTBfsA5HVFuzfTnW+Bu FPgAoMSDLCMdqSE9ZpYgMbjiighqee1B =f7Xg -----END PGP SIGNATURE----- From david.goulet at polymtl.ca Wed Aug 1 12:15:45 2012 From: david.goulet at polymtl.ca (David Goulet) Date: Wed, 01 Aug 2012 12:15:45 -0400 Subject: [lttng-dev] [lttng-tools PATCH] lttng-tools python module In-Reply-To: <5019555F.9090108@gmail.com> References: <1343326167-11522-1-git-send-email-danny.serres@efficios.com> <50182739.3030604@polymtl.ca> <50182FEF.6070504@gmail.com> <501936F6.7010800@polymtl.ca> <50195316.6070606@polymtl.ca> <5019555F.9090108@gmail.com> Message-ID: <50195631.7000808@polymtl.ca> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/08/12 12:12 PM, Yannick Brosseau wrote: > > On 2012-08-01 12:02, David Goulet wrote: >> I agree with that. > >> The main reason for extras/ is the fact that, for now, this lib would > not be >> supported meaning that if liblttng-ctl changes, these bindings will NOT > follow >> it unless someone submit a patch. If we decide at some point to fully > support If its not supported, maybe we should not put it upstream. > Well I though extras/ was there for that?... the lttng bash completion file is not supported but upstream... -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJQGVYxAAoJEELoaioR9I02cuoH/3yyF/Kx9Dst3wgeZ3rCJm+L Mz2YFjju3+L7WmkyqY2uR0n2/+rIGC72rjt6sJmM8OQNTiI0EIhLl8TqVEDmpRut 0GrHPCCD6Si1NNr5kTJAsZ28i6N/zSR5/CoAik6ylo5euZRu2p55RZDqI8U5b3eT zc6547U+lTE2OU4Rm0f1Gt59pt+fGyTI5NrS7ZSW41/2hmAbAnkgGXX1sTV4emo4 epRdZQMxryngjxLY5Nk11qCAopXsKRJ8OBHw66G0a9RKPrQ7i5Ia73IlhTxyrwzV 0NLdGTW6CuYR3grF2dflcmUVFdKsWr7HITDw5IRYYN/ltuesSNm3HAanhH8uvGM= =Lh77 -----END PGP SIGNATURE----- From hhi08001 at student.mdh.se Thu Aug 2 08:06:29 2012 From: hhi08001 at student.mdh.se (Henrik Hautakoski) Date: Thu, 2 Aug 2012 14:06:29 +0200 Subject: [lttng-dev] [PATCH] Fix: unresolvable dependencies to liblttng-ust-tracepoint when cross-compiling. Message-ID: <1343909189-1338-1-git-send-email-hhi08001@student.mdh.se> Original patch by Christian Babeux that did not apply cleanly on upstream. Co-author: Stefan Karlsson Signed-off-by: Henrik Hautakoski --- tests/fork/Makefile.am | 12 ++++++++---- tests/hello.cxx/Makefile.am | 4 +++- tests/hello/Makefile.am | 4 +++- 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/tests/fork/Makefile.am b/tests/fork/Makefile.am index 0a649c7..b5830d7 100644 --- a/tests/fork/Makefile.am +++ b/tests/fork/Makefile.am @@ -2,11 +2,15 @@ AM_CPPFLAGS = -I$(top_srcdir)/include -Wsystem-headers noinst_PROGRAMS = fork fork2 fork_SOURCES = fork.c ust_tests_fork.h -fork_LDADD = $(top_builddir)/liblttng-ust/liblttng-ust.la \ - $(top_builddir)/liblttng-ust-fork/liblttng-ust-fork.la +fork_LDADD = \ + $(top_builddir)/liblttng-ust/liblttng-ust.la \ + $(top_builddir)/liblttng-ust/liblttng-ust-tracepoint.la \ + $(top_builddir)/liblttng-ust-fork/liblttng-ust-fork.la fork2_SOURCES = fork2.c -fork2_LDADD = $(top_builddir)/liblttng-ust/liblttng-ust.la \ - $(top_builddir)/liblttng-ust-fork/liblttng-ust-fork.la +fork2_LDADD = \ + $(top_builddir)/liblttng-ust/liblttng-ust.la \ + $(top_builddir)/liblttng-ust/liblttng-ust-tracepoint.la \ + $(top_builddir)/liblttng-ust-fork/liblttng-ust-fork.la if LTTNG_UST_BUILD_WITH_LIBDL fork_LDADD += -ldl diff --git a/tests/hello.cxx/Makefile.am b/tests/hello.cxx/Makefile.am index f56f431..ea034df 100644 --- a/tests/hello.cxx/Makefile.am +++ b/tests/hello.cxx/Makefile.am @@ -2,7 +2,9 @@ AM_CPPFLAGS = -I$(top_srcdir)/include -Wsystem-headers noinst_PROGRAMS = hello hello_SOURCES = hello.cpp tp.c ust_tests_hello.h -hello_LDADD = $(top_builddir)/liblttng-ust/liblttng-ust.la +hello_LDADD = \ + $(top_builddir)/liblttng-ust/liblttng-ust.la \ + $(top_builddir)/liblttng-ust/liblttng-ust-tracepoint.la if LTTNG_UST_BUILD_WITH_LIBDL hello_LDADD += -ldl diff --git a/tests/hello/Makefile.am b/tests/hello/Makefile.am index 0c4c311..c00ce8b 100644 --- a/tests/hello/Makefile.am +++ b/tests/hello/Makefile.am @@ -2,7 +2,9 @@ AM_CPPFLAGS = -I$(top_srcdir)/include -Wsystem-headers noinst_PROGRAMS = hello hello_SOURCES = hello.c tp.c ust_tests_hello.h -hello_LDADD = $(top_builddir)/liblttng-ust/liblttng-ust.la +hello_LDADD = \ + $(top_builddir)/liblttng-ust/liblttng-ust.la \ + $(top_builddir)/liblttng-ust/liblttng-ust-tracepoint.la hello_CFLAGS = -Werror=old-style-definition if LTTNG_UST_BUILD_WITH_LIBDL -- 1.7.0.4 From mathieu.desnoyers at efficios.com Thu Aug 2 09:00:14 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 2 Aug 2012 09:00:14 -0400 Subject: [lttng-dev] [PATCH] Fix: unresolvable dependencies to liblttng-ust-tracepoint when cross-compiling. In-Reply-To: <1343909189-1338-1-git-send-email-hhi08001@student.mdh.se> References: <1343909189-1338-1-git-send-email-hhi08001@student.mdh.se> Message-ID: <20120802130014.GA29924@Krystal> * Henrik Hautakoski (hhi08001 at student.mdh.se) wrote: > Original patch by Christian Babeux that did not apply cleanly on upstream. I will wait for the proper fix from Christian, which is to fix the libtool m4 file in the tree. Fix should be coming soon. Thanks, Mathieu > > Co-author: Stefan Karlsson > Signed-off-by: Henrik Hautakoski > --- > tests/fork/Makefile.am | 12 ++++++++---- > tests/hello.cxx/Makefile.am | 4 +++- > tests/hello/Makefile.am | 4 +++- > 3 files changed, 14 insertions(+), 6 deletions(-) > > diff --git a/tests/fork/Makefile.am b/tests/fork/Makefile.am > index 0a649c7..b5830d7 100644 > --- a/tests/fork/Makefile.am > +++ b/tests/fork/Makefile.am > @@ -2,11 +2,15 @@ AM_CPPFLAGS = -I$(top_srcdir)/include -Wsystem-headers > > noinst_PROGRAMS = fork fork2 > fork_SOURCES = fork.c ust_tests_fork.h > -fork_LDADD = $(top_builddir)/liblttng-ust/liblttng-ust.la \ > - $(top_builddir)/liblttng-ust-fork/liblttng-ust-fork.la > +fork_LDADD = \ > + $(top_builddir)/liblttng-ust/liblttng-ust.la \ > + $(top_builddir)/liblttng-ust/liblttng-ust-tracepoint.la \ > + $(top_builddir)/liblttng-ust-fork/liblttng-ust-fork.la > fork2_SOURCES = fork2.c > -fork2_LDADD = $(top_builddir)/liblttng-ust/liblttng-ust.la \ > - $(top_builddir)/liblttng-ust-fork/liblttng-ust-fork.la > +fork2_LDADD = \ > + $(top_builddir)/liblttng-ust/liblttng-ust.la \ > + $(top_builddir)/liblttng-ust/liblttng-ust-tracepoint.la \ > + $(top_builddir)/liblttng-ust-fork/liblttng-ust-fork.la > > if LTTNG_UST_BUILD_WITH_LIBDL > fork_LDADD += -ldl > diff --git a/tests/hello.cxx/Makefile.am b/tests/hello.cxx/Makefile.am > index f56f431..ea034df 100644 > --- a/tests/hello.cxx/Makefile.am > +++ b/tests/hello.cxx/Makefile.am > @@ -2,7 +2,9 @@ AM_CPPFLAGS = -I$(top_srcdir)/include -Wsystem-headers > > noinst_PROGRAMS = hello > hello_SOURCES = hello.cpp tp.c ust_tests_hello.h > -hello_LDADD = $(top_builddir)/liblttng-ust/liblttng-ust.la > +hello_LDADD = \ > + $(top_builddir)/liblttng-ust/liblttng-ust.la \ > + $(top_builddir)/liblttng-ust/liblttng-ust-tracepoint.la > > if LTTNG_UST_BUILD_WITH_LIBDL > hello_LDADD += -ldl > diff --git a/tests/hello/Makefile.am b/tests/hello/Makefile.am > index 0c4c311..c00ce8b 100644 > --- a/tests/hello/Makefile.am > +++ b/tests/hello/Makefile.am > @@ -2,7 +2,9 @@ AM_CPPFLAGS = -I$(top_srcdir)/include -Wsystem-headers > > noinst_PROGRAMS = hello > hello_SOURCES = hello.c tp.c ust_tests_hello.h > -hello_LDADD = $(top_builddir)/liblttng-ust/liblttng-ust.la > +hello_LDADD = \ > + $(top_builddir)/liblttng-ust/liblttng-ust.la \ > + $(top_builddir)/liblttng-ust/liblttng-ust-tracepoint.la > hello_CFLAGS = -Werror=old-style-definition > > if LTTNG_UST_BUILD_WITH_LIBDL > -- > 1.7.0.4 > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From yannick.brosseau at gmail.com Thu Aug 2 09:13:30 2012 From: yannick.brosseau at gmail.com (Yannick Brosseau) Date: Thu, 02 Aug 2012 09:13:30 -0400 Subject: [lttng-dev] [RFC] LTTng address API proposal In-Reply-To: <50180418.5020904@polymtl.ca> References: <50180418.5020904@polymtl.ca> Message-ID: <501A7CFA.8000005@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-07-31 12:13, David Goulet wrote: > This follows the discussion on IRC #lttng which aims at removing the lttng_uri > structure from the public API. > > Cheers! > David > --> lttng_create_session_addr(const char *name, const char *addr, Why not lttng_create_session_url ? > PROTO://[HOST|IP][:PORT][/PATH] > > Examples: > > * net://myhostname > * net://myhostname:9888 > * net://myhostname/foo/bar > * net://X.X.X.X:9888/foo/bar You should put the list of protocol available. Now it seems that we cannot do file:// > lttng_set_consumer_addr(struct lttng_handle *handle, const char *addr); Can we have examples for the set_consumer addr ? > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlAafOIACgkQFQrZ7GzHX2oKTQCeMZWzl27WI+qEkvik1TorHxtM RcQAoKZ6wsqNkfNb2WX3jj0Yc2TcPRZ3 =5pn9 -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.goulet at polymtl.ca Thu Aug 2 13:57:19 2012 From: david.goulet at polymtl.ca (David Goulet) Date: Thu, 02 Aug 2012 13:57:19 -0400 Subject: [lttng-dev] [RFC] LTTng address API proposal In-Reply-To: <501A7CFA.8000005@gmail.com> References: <50180418.5020904@polymtl.ca> <501A7CFA.8000005@gmail.com> Message-ID: <501ABF7F.7000603@polymtl.ca> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/08/12 09:13 AM, Yannick Brosseau wrote: > > On 2012-07-31 12:13, David Goulet wrote: >> This follows the discussion on IRC #lttng which aims at removing the > lttng_uri >> structure from the public API. > >> Cheers! David > >> --> lttng_create_session_addr(const char *name, const char *addr, > > Why not lttng_create_session_url ? No idea lol... you prefer that, I'm not against. > >> PROTO://[HOST|IP][:PORT][/PATH] > >> Examples: > >> * net://myhostname * net://myhostname:9888 * net://myhostname/foo/bar * >> net://X.X.X.X:9888/foo/bar > > > You should put the list of protocol available. Now it seems that we cannot > do file:// The protocols are listed in doc/proposals/0003-network.consumer.txt. However, I might move them in that one so we can clearly define the URL string format. > > >> lttng_set_consumer_addr(struct lttng_handle *handle, > const char *addr); > > > Can we have examples for the set_consumer addr ? Ofc. David > > > >> _______________________________________________ lttng-dev mailing list >> lttng-dev at lists.lttng.org >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > > > > > > _______________________________________________ lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJQGr9/AAoJEELoaioR9I02NSQH/iaf4/+vBwxkkW6+KuZiRW2o p/KkZssf1yWQPtxyspxr2JAueykKCH+PtIiB2s2QLDxWCYqOt7gSPI2XlTjdem4I e+N6h25zVpKvlV+rzW6Aw/sbNXRyX9Pw9ldkHYxIkNJjx67Y2ZnBGa9lxdAx7t/U LE+RMrzy9+lwdodZhx5MVJxs1JyVNmU+i+dbIsQlAPdNwUWTebxKP3JVwHz/OwwH 46++EnyFLPIeqvvVPCGJFOaBkBimMCDl0BFm+yviVWZXQKnYl44mnI41RFWa+JVF t1BLqkxExM7FzvnXuxUbBgbhRlRLlrXhRBFlNLK9kVSpPBBm49Y0g94tsB+hEjY= =t6Xs -----END PGP SIGNATURE----- From christian.babeux at efficios.com Thu Aug 2 16:53:56 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Thu, 2 Aug 2012 16:53:56 -0400 Subject: [lttng-dev] [PATCH lttng-ust] Fix: Libtool fails to find dependent libraries when cross-compiling lttng-ust Message-ID: <1343940836-21773-1-git-send-email-christian.babeux@efficios.com> This problem arise when cross compiling and linking libraries with indirect libraries dependencies (such as liblttng-ust). This "bug" is caused by an upstream modification in the libtool package on Debian system. The libtool "link_all_deplibs" flag is set to "no" by default on linux targets (AFAIK, other distros set it to "unknown"). The chosen solution is to detect such cases via the configure script and automagically patch the libtool.m4 by forcing the "link_all_deplibs" to "unknown". This fixup can be disabled with the appropriate configure flag: ./configure --disable-libtool-linkdep-fixup Sample configure output on affected systems: checking for occurence(s) of link_all_deplibs = no in ./config/libtool.m4... 3 configure: WARNING: the detected libtool will not link all dependencies, forcing link_all_deplibs = unknown Fixes: #321 Signed-off-by: Christian Babeux --- configure.ac | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/configure.ac b/configure.ac index 4d40f75..8a93896 100644 --- a/configure.ac +++ b/configure.ac @@ -39,6 +39,31 @@ version_description="New type of beer, 100% from Quebec, flavored with sapin bea AC_DEFINE_UNQUOTED([VERSION_NAME], ["$version_name"], [UST version name]) AC_DEFINE_UNQUOTED([VERSION_DESCRIPTION], ["$version_description"], [UST version description]) +AC_PROG_GREP +# libtool link_all_deplibs fixup. See http://bugs.lttng.org/issues/321. +AC_ARG_ENABLE(libtool-linkdep-fixup, + AS_HELP_STRING([--disable-libtool-linkdep-fixup], + [disable the libtool fixup for linking all dependent libraries (link_all_deplibs)]), + libtool_fixup=$enableval, + libtool_fixup=yes) + +AS_IF([test "x$libtool_fixup" = "xyes"], + [ + libtool_m4="$srcdir/config/libtool.m4" + libtool_flag_pattern=".*link_all_deplibs,.*\$1)" + AC_MSG_CHECKING([for occurence(s) of link_all_deplibs = no in $libtool_m4]) + libtool_flag_pattern_count=$(grep -c "$libtool_flag_pattern=no" $libtool_m4) + AS_IF([test $libtool_flag_pattern_count -ne 0], + [ + AC_MSG_RESULT([$libtool_flag_pattern_count]) + AC_MSG_WARN([the detected libtool will not link all dependencies, forcing link_all_deplibs = unknown]) + sed -i "s/\($libtool_flag_pattern\)=no/\1=unknown/g" $libtool_m4 + ], + [ + AC_MSG_RESULT([none]) + ]) + ]) + # Checks for programs. AC_PROG_CC AC_PROG_CXX -- 1.7.11.3 From mathieu.desnoyers at efficios.com Thu Aug 2 17:07:50 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 2 Aug 2012 17:07:50 -0400 Subject: [lttng-dev] [PATCH lttng-ust] Fix: Libtool fails to find dependent libraries when cross-compiling lttng-ust In-Reply-To: <1343940836-21773-1-git-send-email-christian.babeux@efficios.com> References: <1343940836-21773-1-git-send-email-christian.babeux@efficios.com> Message-ID: <20120802210750.GA24831@Krystal> * Christian Babeux (christian.babeux at efficios.com) wrote: > This problem arise when cross compiling and linking libraries with > indirect libraries dependencies (such as liblttng-ust). This "bug" is > caused by an upstream modification in the libtool package on Debian > system. The libtool "link_all_deplibs" flag is set to "no" by default > on linux targets (AFAIK, other distros set it to "unknown"). > > The chosen solution is to detect such cases via the configure script > and automagically patch the libtool.m4 by forcing the "link_all_deplibs" > to "unknown". > > This fixup can be disabled with the appropriate configure flag: > > ./configure --disable-libtool-linkdep-fixup > > Sample configure output on affected systems: > > checking for occurence(s) of link_all_deplibs = no in > ./config/libtool.m4... 3 > configure: WARNING: the detected libtool will not link all > dependencies, forcing link_all_deplibs = unknown > > Fixes: #321 > > Signed-off-by: Christian Babeux > --- > configure.ac | 25 +++++++++++++++++++++++++ > 1 file changed, 25 insertions(+) > > diff --git a/configure.ac b/configure.ac > index 4d40f75..8a93896 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -39,6 +39,31 @@ version_description="New type of beer, 100% from Quebec, flavored with sapin bea > AC_DEFINE_UNQUOTED([VERSION_NAME], ["$version_name"], [UST version name]) > AC_DEFINE_UNQUOTED([VERSION_DESCRIPTION], ["$version_description"], [UST version description]) > > +AC_PROG_GREP > +# libtool link_all_deplibs fixup. See http://bugs.lttng.org/issues/321. > +AC_ARG_ENABLE(libtool-linkdep-fixup, > + AS_HELP_STRING([--disable-libtool-linkdep-fixup], > + [disable the libtool fixup for linking all dependent libraries (link_all_deplibs)]), > + libtool_fixup=$enableval, > + libtool_fixup=yes) > + > +AS_IF([test "x$libtool_fixup" = "xyes"], > + [ > + libtool_m4="$srcdir/config/libtool.m4" > + libtool_flag_pattern=".*link_all_deplibs,.*\$1)" > + AC_MSG_CHECKING([for occurence(s) of link_all_deplibs = no in $libtool_m4]) > + libtool_flag_pattern_count=$(grep -c "$libtool_flag_pattern=no" $libtool_m4) > + AS_IF([test $libtool_flag_pattern_count -ne 0], > + [ > + AC_MSG_RESULT([$libtool_flag_pattern_count]) > + AC_MSG_WARN([the detected libtool will not link all dependencies, forcing link_all_deplibs = unknown]) > + sed -i "s/\($libtool_flag_pattern\)=no/\1=unknown/g" $libtool_m4 Could you modify your patch to tabs and spaces around "=" would be also accepted ? Thanks, Mathieu > + ], > + [ > + AC_MSG_RESULT([none]) > + ]) > + ]) > + > # Checks for programs. > AC_PROG_CC > AC_PROG_CXX > -- > 1.7.11.3 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 2 17:52:50 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 2 Aug 2012 17:52:50 -0400 Subject: [lttng-dev] LTTng / Android questions In-Reply-To: References: <20120724182031.GB2285@Krystal> Message-ID: <20120802215250.GB29960@Krystal> Hi Karim, Any word of advice about building LTTng 2.0 on Android ? See the question below. Thanks, Mathieu * Glossop, Kent (kent.glossop at intel.com) wrote: > Thanks. > > I've been trying to understand how to do the lttng cross build for > Android. Is there a particular approach that you would expect people > to most naturally use (e.g. if I were to write up a "how-to")? > It seems like one approach is to use a full Android build area, adding > the lttng components to the "external" directory, requiring Android.mk > files. If the files aren't generated automatically, this adds > maintenance. (There is a tool called androgenizer that can be used > assist, but that seems undesirable to include for a "general" > solution.) Another approach would be to build some of components with > options to configure, in conjunction with an Android build (or NDK?) > area. Would you view one of these (or something else), a "preferred" > way? > > Thanks, > Kent > > -----Original Message----- > From: Mathieu Desnoyers [mailto:mathieu.desnoyers at efficios.com] > Sent: Tuesday, July 24, 2012 2:21 PM > To: Glossop, Kent > Cc: christian.babeux at efficios.com > Subject: Re: LTTng / Android questions > > Hi Kent, > > It should work, theoretically. Testing would be welcome. We did port of the lttng tools (except for the kernel tracer) to NetBSD/FreeBSD recently, where we had to circumvent lack of TLS support, so this requirement on the glibc is now gone from the 2.0 lttng series. > > If Android kernel is close enough to mainline, lttng-modules should work too. > > I'm CCing christian, who is currently looking at the embedded aspect of the continuous integration heterogenous cluster we are currently building. > > Thanks, > > Mathieu > > > * Glossop, Kent (kent.glossop at intel.com) wrote: > > Mathieu, > > > > I'm interested in LTTng for Android. If you have time, a few > > questions... (Let me know if this would be better posted to > > lttng-dev.) > > > > From what I can tell from previous postings to lttng-dev and other places: > > > > - People apparently used a previous version with a 2.x kernel with some changes and building it in > > > > - There was/is an issue with glibc vs. bionic for lttng's use of shared memory > > > > - Back in Feb. there was a demo done of LTTng on Android done using a copy of glibc > > > > - There are at least some changes being made that mention Android (e.g. a TLS change that mentioned Android about 2 months ago) > > > > What I'm interested in: > > > > - Are there directions somewhere for building LTTng for use with Android? (e.g. how to configure component and reference an Android build area rather than things for the native host?) > > > > - How much is expected to work on Android? > > > > - Is glibc still needed? > > > > - Are there people actively working on Android support (if it doesn't already work)? > > > > - Do you have a rough idea what might be involved for an Android x86 version beyond arm? > > > > Ideally, I would like to be able to install a minimal set of pieces, preferably on to a stock phone, collect LTTng traces, and use the viewer on linux (or maybe even windows if linux-tools works there.) Then, to do that with x86 Android in addition to arm... > > > > Thanks, > > Kent Glossop, Intel > > > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant EfficiOS Inc. > http://www.efficios.com -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From christian.babeux at efficios.com Thu Aug 2 18:37:16 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Thu, 2 Aug 2012 18:37:16 -0400 Subject: [lttng-dev] [PATCH v2 lttng-ust] Fix: Libtool fails to find dependent libraries when cross-compiling lttng-ust In-Reply-To: <1343940836-21773-1-git-send-email-christian.babeux@efficios.com> References: <1343940836-21773-1-git-send-email-christian.babeux@efficios.com> Message-ID: <1343947036-29236-1-git-send-email-christian.babeux@efficios.com> This problem arise when cross compiling and linking libraries with indirect libraries dependencies (such as liblttng-ust). This "bug" is caused by an upstream modification in the libtool package on Debian system. The libtool "link_all_deplibs" flag is set to "no" by default on linux targets (AFAIK, other distros set it to "unknown"). The chosen solution is to detect such cases via the configure script and automagically patch the libtool.m4 by forcing the "link_all_deplibs" to "unknown". This fixup can be disabled with the appropriate configure flag: ./configure --disable-libtool-linkdep-fixup Sample configure output on affected systems: checking for occurence(s) of link_all_deplibs = no in ./config/libtool.m4... 3 configure: WARNING: the detected libtool will not link all dependencies, forcing link_all_deplibs = unknown Fixes: #321 Signed-off-by: Christian Babeux --- configure.ac | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/configure.ac b/configure.ac index 4d40f75..79c33ff 100644 --- a/configure.ac +++ b/configure.ac @@ -39,6 +39,31 @@ version_description="New type of beer, 100% from Quebec, flavored with sapin bea AC_DEFINE_UNQUOTED([VERSION_NAME], ["$version_name"], [UST version name]) AC_DEFINE_UNQUOTED([VERSION_DESCRIPTION], ["$version_description"], [UST version description]) +AC_PROG_GREP +# libtool link_all_deplibs fixup. See http://bugs.lttng.org/issues/321. +AC_ARG_ENABLE(libtool-linkdep-fixup, + AS_HELP_STRING([--disable-libtool-linkdep-fixup], + [disable the libtool fixup for linking all dependent libraries (link_all_deplibs)]), + libtool_fixup=$enableval, + libtool_fixup=yes) + +AS_IF([test "x$libtool_fixup" = "xyes"], + [ + libtool_m4="$srcdir/config/libtool.m4" + libtool_flag_pattern=".*link_all_deplibs\s*,\s*\$1\s*)" + AC_MSG_CHECKING([for occurence(s) of link_all_deplibs = no in $libtool_m4]) + libtool_flag_pattern_count=$(grep -c "$libtool_flag_pattern\s*=\s*no" $libtool_m4) + AS_IF([test $libtool_flag_pattern_count -ne 0], + [ + AC_MSG_RESULT([$libtool_flag_pattern_count]) + AC_MSG_WARN([the detected libtool will not link all dependencies, forcing link_all_deplibs = unknown]) + sed -i "s/\($libtool_flag_pattern\)\s*=\s*no/\1=unknown/g" $libtool_m4 + ], + [ + AC_MSG_RESULT([none]) + ]) + ]) + # Checks for programs. AC_PROG_CC AC_PROG_CXX -- 1.7.11.3 From mathieu.desnoyers at efficios.com Thu Aug 2 18:55:55 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 2 Aug 2012 18:55:55 -0400 Subject: [lttng-dev] [PATCH v2 lttng-ust] Fix: Libtool fails to find dependent libraries when cross-compiling lttng-ust In-Reply-To: <1343947036-29236-1-git-send-email-christian.babeux@efficios.com> References: <1343940836-21773-1-git-send-email-christian.babeux@efficios.com> <1343947036-29236-1-git-send-email-christian.babeux@efficios.com> Message-ID: <20120802225555.GA31856@Krystal> merged, thanks ! Mathieu * Christian Babeux (christian.babeux at efficios.com) wrote: > This problem arise when cross compiling and linking libraries with > indirect libraries dependencies (such as liblttng-ust). This "bug" is > caused by an upstream modification in the libtool package on Debian > system. The libtool "link_all_deplibs" flag is set to "no" by default > on linux targets (AFAIK, other distros set it to "unknown"). > > The chosen solution is to detect such cases via the configure script > and automagically patch the libtool.m4 by forcing the "link_all_deplibs" > to "unknown". > > This fixup can be disabled with the appropriate configure flag: > > ./configure --disable-libtool-linkdep-fixup > > Sample configure output on affected systems: > > checking for occurence(s) of link_all_deplibs = no in > ./config/libtool.m4... 3 > configure: WARNING: the detected libtool will not link all > dependencies, forcing link_all_deplibs = unknown > > Fixes: #321 > > Signed-off-by: Christian Babeux > --- > configure.ac | 25 +++++++++++++++++++++++++ > 1 file changed, 25 insertions(+) > > diff --git a/configure.ac b/configure.ac > index 4d40f75..79c33ff 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -39,6 +39,31 @@ version_description="New type of beer, 100% from Quebec, flavored with sapin bea > AC_DEFINE_UNQUOTED([VERSION_NAME], ["$version_name"], [UST version name]) > AC_DEFINE_UNQUOTED([VERSION_DESCRIPTION], ["$version_description"], [UST version description]) > > +AC_PROG_GREP > +# libtool link_all_deplibs fixup. See http://bugs.lttng.org/issues/321. > +AC_ARG_ENABLE(libtool-linkdep-fixup, > + AS_HELP_STRING([--disable-libtool-linkdep-fixup], > + [disable the libtool fixup for linking all dependent libraries (link_all_deplibs)]), > + libtool_fixup=$enableval, > + libtool_fixup=yes) > + > +AS_IF([test "x$libtool_fixup" = "xyes"], > + [ > + libtool_m4="$srcdir/config/libtool.m4" > + libtool_flag_pattern=".*link_all_deplibs\s*,\s*\$1\s*)" > + AC_MSG_CHECKING([for occurence(s) of link_all_deplibs = no in $libtool_m4]) > + libtool_flag_pattern_count=$(grep -c "$libtool_flag_pattern\s*=\s*no" $libtool_m4) > + AS_IF([test $libtool_flag_pattern_count -ne 0], > + [ > + AC_MSG_RESULT([$libtool_flag_pattern_count]) > + AC_MSG_WARN([the detected libtool will not link all dependencies, forcing link_all_deplibs = unknown]) > + sed -i "s/\($libtool_flag_pattern\)\s*=\s*no/\1=unknown/g" $libtool_m4 > + ], > + [ > + AC_MSG_RESULT([none]) > + ]) > + ]) > + > # Checks for programs. > AC_PROG_CC > AC_PROG_CXX > -- > 1.7.11.3 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From eikosaedron at gmail.com Thu Aug 2 23:56:16 2012 From: eikosaedron at gmail.com (eikosaedron at gmail.com) Date: Thu, 02 Aug 2012 20:56:16 -0700 Subject: [lttng-dev] Getting 64bit int from babeltrace library Message-ID: <138eaa0642b.-2681066139298325110.3245307989283453259@gmail.com> Hi all, I hope you can help me with a problem I've encountered using the babeltrace C libraries to write a trace parsing tool. I have fields which contain 64 bit integers. However, when I call bt_ctf_get_uint64 from the event callback on that field, it only returns the lower 32 bits. The upper bits are all 0's. I've called get_int_len to confirm they are 64 bits and tried accessing the values directly but I'm still facing the same problem. Running babeltrace on the same trace files prints out the full 64 bits in base 16. The top 32 bits should not be all 0's. Could there be something wrong with my build setup or anything else I've overlooked in getting the the full 64 bit integers into the babeltrace structures? Thanks, Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: From mathieu.desnoyers at efficios.com Fri Aug 3 08:35:07 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 3 Aug 2012 08:35:07 -0400 Subject: [lttng-dev] Getting 64bit int from babeltrace library In-Reply-To: <138eaa0642b.-2681066139298325110.3245307989283453259@gmail.com> References: <138eaa0642b.-2681066139298325110.3245307989283453259@gmail.com> Message-ID: <20120803123507.GA16463@Krystal> Hi Jon, Do you have a test trace and small test program you can provide that would allow us to reproduce your issue ? Please provide information about the build setup too (32-bit or 64-bit userspace ?) Thanks, Mathieu * eikosaedron at gmail.com (eikosaedron at gmail.com) wrote: > > Hi all, > > I hope you can help me with a problem I've encountered using the babeltrace C libraries to write a trace parsing tool. > > I have fields which contain 64 bit integers. However, when I call bt_ctf_get_uint64 from the event callback on that field, it only returns the lower 32 bits. The upper bits are all 0's. I've called get_int_len to confirm they are 64 bits and tried accessing the values directly but I'm still facing the same problem. > > Running babeltrace on the same trace files prints out the full 64 bits in base 16. The top 32 bits should not be all 0's. > > Could there be something wrong with my build setup or anything else I've overlooked in getting the the full 64 bit integers into the babeltrace structures? > > Thanks, > Jon > > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From jdesfossez at efficios.com Fri Aug 3 15:27:35 2012 From: jdesfossez at efficios.com (Julien Desfossez) Date: Fri, 03 Aug 2012 15:27:35 -0400 Subject: [lttng-dev] Babeltrace API timestamp manipulation changes Message-ID: <501C2627.6060401@efficios.com> Hi, Previously, the Babeltrace API forced the user to manipulate "raw timestamps" to seek, to get the timestamp begin and end of a trace, etc. This raw timestamp represented the cycle value scaled with the frequency but did not meant anything to the user and was likely to cause problem when dealing with traces recorded with different clock sources. The new patches commited in Babeltrace today removed the notion of the raw clock-source all over the code, and now the users can manipulate real timestamps (with the timezone offset applied) or cycle values if they want more fine-grained control on a specific trace. This means some API changes : - the "seek_time" member of the bt_iter_pos structure now takes real timestamp (with the offset applied) - the functions "bt_ctf_get_timestamp_raw" is removed - the funtion "bt_ctf_get_timestamp" is replaced by "bt_ctf_get_real_timestamp" - the function "bt_ctf_get_cycles_timestamp" is added - the functions "bt_trace_handle_get_timestamp_begin" and "bt_trace_handle_get_timestamp_end" now take a third parameter, either BT_CLOCK_CYCLES or BT_CLOCK_REAL. If you have any questions, feel free to ask. Thanks, Julien From yannick.brosseau at gmail.com Fri Aug 3 15:57:38 2012 From: yannick.brosseau at gmail.com (Yannick Brosseau) Date: Fri, 03 Aug 2012 15:57:38 -0400 Subject: [lttng-dev] Babeltrace API timestamp manipulation changes In-Reply-To: <501C2627.6060401@efficios.com> References: <501C2627.6060401@efficios.com> Message-ID: <501C2D32.5090907@gmail.com> On 2012-08-03 15:27, Julien Desfossez wrote: > Hi, > > Previously, the Babeltrace API forced the user to manipulate "raw > timestamps" to seek, to get the timestamp begin and end of a trace, etc. > > This raw timestamp represented the cycle value scaled with the frequency > but did not meant anything to the user and was likely to cause problem > when dealing with traces recorded with different clock sources. > > The new patches commited in Babeltrace today removed the notion of the > raw clock-source all over the code, and now the users can manipulate > real timestamps (with the timezone offset applied) or cycle values if > they want more fine-grained control on a specific trace. > > This means some API changes : > - the "seek_time" member of the bt_iter_pos structure now takes real > timestamp (with the offset applied) > - the functions "bt_ctf_get_timestamp_raw" is removed > - the funtion "bt_ctf_get_timestamp" is replaced by > "bt_ctf_get_real_timestamp" > - the function "bt_ctf_get_cycles_timestamp" is added > Why not simply calling them bt_ctf_get_timestamp and bt_ctf_get_cycles ? Yannick From christian.babeux at efficios.com Fri Aug 3 16:31:26 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Fri, 3 Aug 2012 16:31:26 -0400 Subject: [lttng-dev] [PATCH lttng-tools 0/2] Configure fixes for cross-compilation Message-ID: Hi all, Here are a few fixes to the configure.ac script to accomodate cross-compilation of lttng-tools. Patch 1: Same fix that was merged in lttng-ust to fix libtool.m4 on target where link_all_deplibs = no. Patch 2: Fix an issue encountered when cross-compiling. The configure script simply fails saying it cannot find the lttng-ust-ctl library even thought the proper "-L/path/to/ust" is specified in $LDFLAGS. The AC_CHECK_LIB test fails because it can't find the libraries that liblttng-ust-ctl depends on. The fix is to specify the dependents libraries (liburcu-{common,bp,cds}) in the other-lib-case of AC_CHECK_LIB. Thanks, Christian Christian Babeux (2): Fix: Libtool fails to find dependent libraries when cross-compiling lttng-tools Fix: Missing libs dependencies in configure check for lttng-ust-ctl configure.ac | 38 +++++++++++++++++++++++++++++++++----- 1 file changed, 33 insertions(+), 5 deletions(-) -- 1.7.11.3 From christian.babeux at efficios.com Fri Aug 3 16:31:27 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Fri, 3 Aug 2012 16:31:27 -0400 Subject: [lttng-dev] [PATCH lttng-tools 1/2] Fix: Libtool fails to find dependent libraries when cross-compiling lttng-tools In-Reply-To: References: Message-ID: This problem arise when cross compiling and linking libraries with indirect libraries dependencies (such as liblttng-ust). This "bug" is caused by an upstream modification in the libtool package on Debian system. The libtool "link_all_deplibs" flag is set to "no" by default on linux targets (AFAIK, other distros set it to "unknown"). The chosen solution is to detect such cases via the configure script and automagically patch the libtool.m4 by forcing the "link_all_deplibs" to "unknown". This fixup can be disabled with the appropriate configure flag: ./configure --disable-libtool-linkdep-fixup Sample configure output on affected systems: checking for occurence(s) of link_all_deplibs = no in ./config/libtool.m4... 3 configure: WARNING: the detected libtool will not link all dependencies, forcing link_all_deplibs = unknown Signed-off-by: Christian Babeux --- configure.ac | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/configure.ac b/configure.ac index 17e6b67..3a023cd 100644 --- a/configure.ac +++ b/configure.ac @@ -14,6 +14,31 @@ AC_DEFINE_UNQUOTED([VERSION_DESCRIPTION], ["$version_description"], "") AC_CONFIG_HEADERS([include/config.h]) +AC_PROG_GREP +# libtool link_all_deplibs fixup. See http://bugs.lttng.org/issues/321. +AC_ARG_ENABLE(libtool-linkdep-fixup, + AS_HELP_STRING([--disable-libtool-linkdep-fixup], + [disable the libtool fixup for linking all dependent libraries (link_all_deplibs)]), + libtool_fixup=$enableval, + libtool_fixup=yes) + +AS_IF([test "x$libtool_fixup" = "xyes"], + [ + libtool_m4="$srcdir/config/libtool.m4" + libtool_flag_pattern=".*link_all_deplibs\s*,\s*\$1\s*)" + AC_MSG_CHECKING([for occurence(s) of link_all_deplibs = no in $libtool_m4]) + libtool_flag_pattern_count=$(grep -c "$libtool_flag_pattern\s*=\s*no" $libtool_m4) + AS_IF([test $libtool_flag_pattern_count -ne 0], + [ + AC_MSG_RESULT([$libtool_flag_pattern_count]) + AC_MSG_WARN([the detected libtool will not link all dependencies, forcing link_all_deplibs = unknown]) + sed -i "s/\($libtool_flag_pattern\)\s*=\s*no/\1=unknown/g" $libtool_m4 + ], + [ + AC_MSG_RESULT([none]) + ]) + ]) + AC_CHECK_HEADERS([ \ sys/types.h unistd.h fcntl.h string.h pthread.h limits.h \ signal.h stdlib.h sys/un.h sys/socket.h stdlib.h stdio.h \ -- 1.7.11.3 From christian.babeux at efficios.com Fri Aug 3 16:31:28 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Fri, 3 Aug 2012 16:31:28 -0400 Subject: [lttng-dev] [PATCH lttng-tools 2/2] Fix: Missing libs dependencies in configure check for lttng-ust-ctl In-Reply-To: References: Message-ID: <884d75d90550f85492dfcaac2b0088f78ced542b.1344025572.git.christian.babeux@efficios.com> The lttng-ust-ctl library depends on liburcu-{common,bp,cds}. The AC_CHECK_LIBRARY macro can't automatically resolve dependents libraries (ala libtool), so any additionnals dependencies must be manually specified. Also, the AC_CHECK_LIB action-if-found case for the lttng-ust-ctl check is modified to have a similar behavior as the default action, which is to define HAVE_LIBxxx and append -lxxx to $LIBS, *except* for the later step. This is to ensure that any future addition of AC_CHECK_LIB after the one for lttng-ust-ctl will not need to append the liburcu dependencies or fail unexpectedly. Signed-off-by: Christian Babeux --- configure.ac | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/configure.ac b/configure.ac index 3a023cd..e4b9eb1 100644 --- a/configure.ac +++ b/configure.ac @@ -138,13 +138,16 @@ AC_ARG_ENABLE(lttng-ust, lttng_ust_support=$enableval, lttng_ust_support=yes) AS_IF([test "x$lttng_ust_support" = "xyes"], [ - AC_CHECK_LIB([lttng-ust-ctl], [ustctl_create_session], [], - [AC_MSG_ERROR([Cannot find LTTng-UST. Use [LDFLAGS]=-Ldir to specify its location, or specify --disable-lttng-ust to build lttng-tools without LTTng-UST support.])] + AC_CHECK_LIB([lttng-ust-ctl], [ustctl_create_session], + [ + AC_DEFINE([HAVE_LIBLTTNG_UST_CTL], [1], [has LTTng-UST control support]) + lttng_ust_ctl_found=yes + ], + [AC_MSG_ERROR([Cannot find LTTng-UST. Use [LDFLAGS]=-Ldir to specify its location, or specify --disable-lttng-ust to build lttng-tools without LTTng-UST support.])], + [-lurcu-common -lurcu-bp -lurcu-cds] ) ]) - -AM_CONDITIONAL([HAVE_LIBLTTNG_UST_CTL], [ test "x$ac_cv_lib_lttng_ust_ctl_ustctl_create_session" = "xyes" ]) - +AM_CONDITIONAL([HAVE_LIBLTTNG_UST_CTL], [test "x$lttng_ust_ctl_found" = xyes]) AC_CHECK_FUNCS([sched_getcpu sysconf]) # check for dlopen -- 1.7.11.3 From yannick.brosseau at gmail.com Fri Aug 3 18:00:29 2012 From: yannick.brosseau at gmail.com (Yannick Brosseau) Date: Fri, 03 Aug 2012 18:00:29 -0400 Subject: [lttng-dev] LTTng project server not available this Saturday august 4th Message-ID: <501C49FD.3040300@gmail.com> Hi All, Due to maintenance to the electrical installation, the servers of the LTTng project will be down this Saturday starting at 6AM EDT. Affected services: Website Mailing list GIT The services should be back during the night (not earliest than 6PM EDT) or sunday morning. Sorry for the inconvenience. Yannick Brosseau From r.han at umiami.edu Sun Aug 5 00:41:52 2012 From: r.han at umiami.edu (Rui Han) Date: Sun, 5 Aug 2012 00:41:52 -0400 Subject: [lttng-dev] ERROR: No buffer space available Message-ID: Hi, I try to modified the lttng2.0 source code for my research. For example, I try to log the "comm" for the write system call, I add the following filed in the file x86-64-syscalls-3.0.4_pointers.h located at the path lttng-modules-2.0.3/instrumentation/syscalls/headers. I modify the TRACE_EVENTS() macro from the original one: #ifndef OVERRIDE_64_sys_write SC_TRACE_EVENT(sys_write, TP_PROTO(unsigned int fd, const char * buf, size_t count), TP_ARGS(fd, buf, count), TP_STRUCT__entry(__field(unsigned int, fd) __field_hex(const char *, buf) __field(size_t, count)), TP_fast_assign(tp_assign(fd, fd) tp_assign(buf, buf) tp_assign(count, count)), TP_printk() ) #endif to the following one: #ifndef OVERRIDE_64_sys_write SC_TRACE_EVENT(sys_write, TP_PROTO(unsigned int fd, const char * buf, size_t count, struct task_struct *p), TP_ARGS(fd, buf, count, p), TP_STRUCT__entry( __field(unsigned int, fd) __field_hex(const char *, buf) __field(size_t, count) __array_text(char, comm, TASK_COMM_LEN)), TP_fast_assign( tp_assign(fd, fd) tp_assign(buf, buf) tp_assign(count, 2064) tp_memcpy(comm, p->comm, TASK_COMM_LEN)), TP_printk() ) #endif I add one argument *p and one filed "comm" for the macro. It was able to be compiled, however, after reload the lttng modules, It will show the following error messages when I start lttng. PERROR: ioctl start session: No buffer space available [in kernel_start_session() at kernel.c:397] Error: Starting kernel trace failed It won't work. Please help me out with this. My next step is try to add extra fields in the sys_connect macros to enable the logging of the connected IP/port from the socket. Is this method right? what are other possible solutions in order to logging the details of the socket information? Thank you very much. Regards, Rui -------------- next part -------------- An HTML attachment was scrubbed... URL: From mathieu.desnoyers at efficios.com Sun Aug 5 13:03:10 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Sun, 5 Aug 2012 13:03:10 -0400 Subject: [lttng-dev] ERROR: No buffer space available In-Reply-To: References: Message-ID: <20120805170310.GA22267@Krystal> * Rui Han (r.han at umiami.edu) wrote: > Hi, > > I try to modified the lttng2.0 source code for my research. For example, I > try to log the "comm" for the write system call, I add the following filed > in the file x86-64-syscalls-3.0.4_pointers.h located at the path > lttng-modules-2.0.3/instrumentation/syscalls/headers. I modify the > TRACE_EVENTS() macro from the original one: > > #ifndef OVERRIDE_64_sys_write > SC_TRACE_EVENT(sys_write, > TP_PROTO(unsigned int fd, const char * buf, size_t count), > TP_ARGS(fd, buf, count), > TP_STRUCT__entry(__field(unsigned int, fd) __field_hex(const char *, buf) > __field(size_t, count)), > TP_fast_assign(tp_assign(fd, fd) tp_assign(buf, buf) tp_assign(count, > count)), > TP_printk() > ) > #endif > > to the following one: > > #ifndef OVERRIDE_64_sys_write > SC_TRACE_EVENT(sys_write, > TP_PROTO(unsigned int fd, const char * buf, size_t count, struct > task_struct *p), > TP_ARGS(fd, buf, count, p), > TP_STRUCT__entry( __field(unsigned int, fd) __field_hex(const char *, > buf) __field(size_t, > count) __array_text(char, comm, TASK_COMM_LEN)), > TP_fast_assign( tp_assign(fd, fd) tp_assign(buf, buf) tp_assign(count, > 2064) tp_memcpy(comm, > p->comm, TASK_COMM_LEN)), > TP_printk() > ) > #endif > > I add one argument *p and one filed "comm" for the macro. It was able to > be compiled, however, after reload the lttng modules, It will show the > following error messages when I start lttng. > > PERROR: ioctl start session: No buffer space available [in > kernel_start_session() at kernel.c:397] > Error: Starting kernel trace failed I don't see how modifying the system call tracing macro would trigger this error at start. Try restarting lttng-sessiond. Try unloading the lttng modules, and, maybe, as last resort, rebooting your system. Thanks, Mathieu > > It won't work. Please help me out with this. My next step is try to add > extra fields in the sys_connect macros to enable the logging of the > connected IP/port from the socket. Is this method right? what are other > possible solutions in order to logging the details of the socket > information? > > Thank you very much. > > Regards, > Rui -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Sun Aug 5 13:46:35 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Sun, 5 Aug 2012 13:46:35 -0400 Subject: [lttng-dev] Babeltrace API timestamp manipulation changes In-Reply-To: <501C2D32.5090907@gmail.com> References: <501C2627.6060401@efficios.com> <501C2D32.5090907@gmail.com> Message-ID: <20120805174635.GA22981@Krystal> * Yannick Brosseau (yannick.brosseau at gmail.com) wrote: > On 2012-08-03 15:27, Julien Desfossez wrote: > > Hi, > > > > Previously, the Babeltrace API forced the user to manipulate "raw > > timestamps" to seek, to get the timestamp begin and end of a trace, etc. > > > > This raw timestamp represented the cycle value scaled with the frequency > > but did not meant anything to the user and was likely to cause problem > > when dealing with traces recorded with different clock sources. > > > > The new patches commited in Babeltrace today removed the notion of the > > raw clock-source all over the code, and now the users can manipulate > > real timestamps (with the timezone offset applied) or cycle values if > > they want more fine-grained control on a specific trace. > > > > This means some API changes : > > - the "seek_time" member of the bt_iter_pos structure now takes real > > timestamp (with the offset applied) > > - the functions "bt_ctf_get_timestamp_raw" is removed > > - the funtion "bt_ctf_get_timestamp" is replaced by > > "bt_ctf_get_real_timestamp" > > - the function "bt_ctf_get_cycles_timestamp" is added > > > Why not simply calling them > bt_ctf_get_timestamp and bt_ctf_get_cycles ? Yes, it makes sense. Julien, any thoughts on this ? Thanks, Mathieu > > Yannick > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From laijs at cn.fujitsu.com Tue Aug 7 05:12:31 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Tue, 07 Aug 2012 17:12:31 +0800 Subject: [lttng-dev] urcu rfc: privatize urcu/futex.h Message-ID: <5020DBFF.3020702@cn.fujitsu.com> futex.h is in urcu/, it means it can be used by users out of the library. But If the user's system has futex, he should use #include . If the user's system don't have futex, it is not good that if the user use this compat_futex. Because the compat_futex_async()'s behavior is different from the futex in linux. If the caller don't change the value of @uaddr, the futex(FUTEX_WAKE) in linux can also wake a thread, but compat_futex_async() don't. (I guess no one use this behavior, but if there are someone, we give them a wrong thing) So I suggest urcu/futex.h becomes a private thing for urcu. Alternative implements(only describe the alternative FUTEX_WAIT): 1) Like compat_futex_noasync(), but use mutex_lock_signal_save() + pthread_cond_wait(). Problem: the thread can't handle the signal when waiting on pthread_cond_wait(). (* mutex_lock_signal_save() is implemented in compat_arch_x86.c) 2) Like the linux kernel, use use mutex_lock_signal_save() + hash table + sleeping (use poll() + restore signal mask) Problem: very complex Thanks, Lai. (today I found a old patchset in my private tree, tried to rebase it and suddenly notice this urcu/futex.h but the patchset has nothing related with urcu/futex.h). From mathieu.desnoyers at efficios.com Tue Aug 7 10:06:16 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 7 Aug 2012 10:06:16 -0400 Subject: [lttng-dev] urcu rfc: privatize urcu/futex.h In-Reply-To: <5020DBFF.3020702@cn.fujitsu.com> References: <5020DBFF.3020702@cn.fujitsu.com> Message-ID: <20120807140616.GA29280@Krystal> Hi Lai, * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > futex.h is in urcu/, it means it can be used by users out of the library. > > But > > If the user's system has futex, he should use #include . > If the user's system don't have futex, it is not good that if the user > use this compat_futex. > > Because the compat_futex_async()'s behavior is different from the > futex in linux. If the caller don't change the value of @uaddr, the > futex(FUTEX_WAKE) in linux can also wake a thread, but > compat_futex_async() don't. (I guess no one use this behavior, but if > there are someone, we give them a wrong thing) Hrm, passing a NULL uaddr parameter is really not the targeted use-case. We could probably just document that uaddr should not be NULL when using compat_futex ? > So I suggest urcu/futex.h becomes a private thing for urcu. Well, if possible, I'd like to keep it public, since it allows us to do wait/wakeup schemes across many OSes. Do you have a use-case where you want to wake up all threads, even those which are not waiting on a specific uaddr ? > > Alternative implements(only describe the alternative FUTEX_WAIT): > 1) Like compat_futex_noasync(), but use mutex_lock_signal_save() + > pthread_cond_wait(). > Problem: the thread can't handle the signal when waiting on > pthread_cond_wait(). (* mutex_lock_signal_save() is implemented in > compat_arch_x86.c) The culprit of the problem here is not really on the "wait" side, but rather on the "wake" side. The wakeup, in urcu, is issued by rcu_read_unlock() to wake up synchronize_rcu() (if there is one waiting). However, we don't want to disable signals and take a mutex each time we do a rcu_read_unlock, because it would tremendously impact performance. AFAIK, pthread_cond_signal() is not signal-safe, so it could not be used without lock+signal save. Another site where the wait/wakeup is used: in call_rcu, waking up the worker threads. We also don't want to take a mutex there, to ensure that call_rcu is wait-free as far as userspace code is concerned. > > 2) Like the linux kernel, use use mutex_lock_signal_save() + hash table > + sleeping (use poll() + restore signal mask) > Problem: very complex Would this solution require a lock+signal save on the wake up site too ? > > Thanks, > Lai. > > (today I found a old patchset in my private tree, tried to rebase it > and suddenly notice this urcu/futex.h but the patchset has nothing > related with urcu/futex.h). I look forward to see that work! Thanks! Mathieu -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Tue Aug 7 10:43:18 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 7 Aug 2012 10:43:18 -0400 Subject: [lttng-dev] [PATCH] Cleanup: Remove misplaced config.h include Message-ID: <20120807144318.GA30155@Krystal> config.h should _always_ be included first, before system headers, to ensure coherent size_t across the file. We force the config.h inclusion by the compiler, so this is not an issue (configure.ac: DEFAULT_INCLUDES="-I\$(top_srcdir) -I\$(top_builddir) -I\$(top_builddir)/src -I\$(top_builddir)/include -include config.h"), but let's remove this misleading line of code. Signed-off-by: Mathieu Desnoyers --- diff --git a/src/common/hashtable/rculfhash.c b/src/common/hashtable/rculfhash.c index 776f9f3..423d186 100644 --- a/src/common/hashtable/rculfhash.c +++ b/src/common/hashtable/rculfhash.c @@ -157,7 +157,6 @@ #include #include -#include "config.h" #include #include #include -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From dgoulet at efficios.com Tue Aug 7 10:46:36 2012 From: dgoulet at efficios.com (David Goulet) Date: Tue, 07 Aug 2012 10:46:36 -0400 Subject: [lttng-dev] [PATCH] Cleanup: Remove misplaced config.h include In-Reply-To: <20120807144318.GA30155@Krystal> References: <20120807144318.GA30155@Krystal> Message-ID: <50212A4C.3030002@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Merged! Thx Mathieu Desnoyers: > config.h should _always_ be included first, before system headers, > to ensure coherent size_t across the file. > > We force the config.h inclusion by the compiler, so this is not an > issue (configure.ac: DEFAULT_INCLUDES="-I\$(top_srcdir) > -I\$(top_builddir) -I\$(top_builddir)/src > -I\$(top_builddir)/include -include config.h"), but let's remove > this misleading line of code. > > Signed-off-by: Mathieu Desnoyers > --- diff --git a/src/common/hashtable/rculfhash.c > b/src/common/hashtable/rculfhash.c index 776f9f3..423d186 100644 > --- a/src/common/hashtable/rculfhash.c +++ > b/src/common/hashtable/rculfhash.c @@ -157,7 +157,6 @@ #include > #include > > -#include "config.h" #include #include > #include > -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQISpMAAoJEELoaioR9I02IkMH/ilFqFAfZEUcKcheR6ZI0Nco DrL8gyfVIJYcxQFUyduqSsc1vSOtqzlyF3OvyyfHgJBo1HilODrqRBJP5/ZBflDz DgKkk46c2e3lrSm8pfjHgRDVPiPhTd+axyVjZ/dyGvrMvcPcjLH/QS/tEF1ziXP6 urVW4FT515fXUUYRPyd5K+ooWArlxwQTxExwpayHUqI8VmHHs9fqxjOB7YnkRhyl vpoQSzyiHsjCPvk1YJoTeC1p5BJmy8YckkwUhNgxgiuuPUMDfhWMoUC76oj2AAR0 TM7m9NgjpdAOEr5TAXFhFJVb8FNaH7nuymiXD/o0aF0/k9ko5vBP5Tdo6RV3xGI= =c0ji -----END PGP SIGNATURE----- From jdesfossez at efficios.com Tue Aug 7 11:15:49 2012 From: jdesfossez at efficios.com (Julien Desfossez) Date: Tue, 7 Aug 2012 11:15:49 -0400 Subject: [lttng-dev] [BABELTRACE PATCH] API cleanup name get_timestamp and get_cycles In-Reply-To: <20120805174635.GA22981@Krystal> References: <20120805174635.GA22981@Krystal> Message-ID: <1344352549-32224-1-git-send-email-jdesfossez@efficios.com> Rename bt_ctf_get_cycles_timestamp to bt_ctf_get_cycles, and bt_ctf_get_real_timestamp to bt_ctf_get_timestamp. Signed-off-by: Julien Desfossez --- formats/ctf/events.c | 4 ++-- include/babeltrace/ctf/events.h | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/formats/ctf/events.c b/formats/ctf/events.c index 06419a1..9693413 100644 --- a/formats/ctf/events.c +++ b/formats/ctf/events.c @@ -281,7 +281,7 @@ int bt_ctf_event_get_handle_id(const struct bt_ctf_event *ctf_event) return ret; } -uint64_t bt_ctf_get_real_timestamp(const struct bt_ctf_event *ctf_event) +uint64_t bt_ctf_get_timestamp(const struct bt_ctf_event *ctf_event) { struct ctf_event_definition *event = ctf_event->parent; if (event && event->stream->has_timestamp) @@ -290,7 +290,7 @@ uint64_t bt_ctf_get_real_timestamp(const struct bt_ctf_event *ctf_event) return -1ULL; } -uint64_t bt_ctf_get_cycles_timestamp(const struct bt_ctf_event *ctf_event) +uint64_t bt_ctf_get_cycles(const struct bt_ctf_event *ctf_event) { struct ctf_event_definition *event = ctf_event->parent; if (event && event->stream->has_timestamp) diff --git a/include/babeltrace/ctf/events.h b/include/babeltrace/ctf/events.h index 419c4c8..4232feb 100644 --- a/include/babeltrace/ctf/events.h +++ b/include/babeltrace/ctf/events.h @@ -90,16 +90,16 @@ const struct definition *bt_ctf_get_top_level_scope(const struct bt_ctf_event *e const char *bt_ctf_event_name(const struct bt_ctf_event *event); /* - * bt_ctf_get_cycles_timestamp: returns the timestamp of the event as written + * bt_ctf_get_cycles: returns the timestamp of the event as written * in the packet (in cycles) or -1ULL on error. */ -uint64_t bt_ctf_get_cycles_timestamp(const struct bt_ctf_event *event); +uint64_t bt_ctf_get_cycles(const struct bt_ctf_event *event); /* - * bt_ctf_get_real_timestamp: returns the timestamp of the event offsetted + * bt_ctf_get_timestamp: returns the timestamp of the event offsetted * with the system clock source (in ns) or -1ULL on error */ -uint64_t bt_ctf_get_real_timestamp(const struct bt_ctf_event *event); +uint64_t bt_ctf_get_timestamp(const struct bt_ctf_event *event); /* * bt_ctf_get_field_list: set list pointer to an array of definition -- 1.7.10.4 From mathieu.desnoyers at efficios.com Tue Aug 7 11:50:14 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 7 Aug 2012 11:50:14 -0400 Subject: [lttng-dev] [BABELTRACE PATCH] API cleanup name get_timestamp and get_cycles In-Reply-To: <1344352549-32224-1-git-send-email-jdesfossez@efficios.com> References: <20120805174635.GA22981@Krystal> <1344352549-32224-1-git-send-email-jdesfossez@efficios.com> Message-ID: <20120807155014.GA31008@Krystal> * Julien Desfossez (jdesfossez at efficios.com) wrote: > Rename bt_ctf_get_cycles_timestamp to bt_ctf_get_cycles, > and bt_ctf_get_real_timestamp to bt_ctf_get_timestamp. merged, thanks! Mathieu > > Signed-off-by: Julien Desfossez > --- > formats/ctf/events.c | 4 ++-- > include/babeltrace/ctf/events.h | 8 ++++---- > 2 files changed, 6 insertions(+), 6 deletions(-) > > diff --git a/formats/ctf/events.c b/formats/ctf/events.c > index 06419a1..9693413 100644 > --- a/formats/ctf/events.c > +++ b/formats/ctf/events.c > @@ -281,7 +281,7 @@ int bt_ctf_event_get_handle_id(const struct bt_ctf_event *ctf_event) > return ret; > } > > -uint64_t bt_ctf_get_real_timestamp(const struct bt_ctf_event *ctf_event) > +uint64_t bt_ctf_get_timestamp(const struct bt_ctf_event *ctf_event) > { > struct ctf_event_definition *event = ctf_event->parent; > if (event && event->stream->has_timestamp) > @@ -290,7 +290,7 @@ uint64_t bt_ctf_get_real_timestamp(const struct bt_ctf_event *ctf_event) > return -1ULL; > } > > -uint64_t bt_ctf_get_cycles_timestamp(const struct bt_ctf_event *ctf_event) > +uint64_t bt_ctf_get_cycles(const struct bt_ctf_event *ctf_event) > { > struct ctf_event_definition *event = ctf_event->parent; > if (event && event->stream->has_timestamp) > diff --git a/include/babeltrace/ctf/events.h b/include/babeltrace/ctf/events.h > index 419c4c8..4232feb 100644 > --- a/include/babeltrace/ctf/events.h > +++ b/include/babeltrace/ctf/events.h > @@ -90,16 +90,16 @@ const struct definition *bt_ctf_get_top_level_scope(const struct bt_ctf_event *e > const char *bt_ctf_event_name(const struct bt_ctf_event *event); > > /* > - * bt_ctf_get_cycles_timestamp: returns the timestamp of the event as written > + * bt_ctf_get_cycles: returns the timestamp of the event as written > * in the packet (in cycles) or -1ULL on error. > */ > -uint64_t bt_ctf_get_cycles_timestamp(const struct bt_ctf_event *event); > +uint64_t bt_ctf_get_cycles(const struct bt_ctf_event *event); > > /* > - * bt_ctf_get_real_timestamp: returns the timestamp of the event offsetted > + * bt_ctf_get_timestamp: returns the timestamp of the event offsetted > * with the system clock source (in ns) or -1ULL on error > */ > -uint64_t bt_ctf_get_real_timestamp(const struct bt_ctf_event *event); > +uint64_t bt_ctf_get_timestamp(const struct bt_ctf_event *event); > > /* > * bt_ctf_get_field_list: set list pointer to an array of definition > -- > 1.7.10.4 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From dgoulet at efficios.com Tue Aug 7 12:10:06 2012 From: dgoulet at efficios.com (David Goulet) Date: Tue, 07 Aug 2012 12:10:06 -0400 Subject: [lttng-dev] [RFC] LTTng address API proposal v2 Message-ID: <50213DDE.8070709@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Here is the second and updated version after Yannick comments. - v0.2: 07/08/0212 * Change lttng_create_session_addr to lttng_create_session_url * Describe URL string format * Add set_consumer_url examples * Add the HOP option to the set_consumer_url function Cheers! David -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQIT3aAAoJEELoaioR9I02vqIH/1YeAlh+ogt1wvRv3bkExZB6 vywtNVUVx/ys7KgaqkRolvGmbBKZEMPTwXaiGXDSfbij9hgkLvpjNhc26c9b443r 9GJ603Jn+PPEa/vOMBPY8sqxGKyKLoSL400sXNLhPo2t18+neWOLDhgcfXd4VIT7 FgmOQzUQZsImFNGJUTNSFKD7m/NM73F7pM8VG1OjXfOfNAb1KtoV6QYSApoAayQ+ g2XdPgsAkDQssh82xLQ0Ousg3Cpk72e/jyroUNNnT8Dy7rSTXLsLz7indBzfFoAw cIzTwREbarYT7cEbNnH0oFRxUK/94MEPjeWd2cSNJxsaidYj9mD07ST9UWfputU= =GYy5 -----END PGP SIGNATURE----- -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0004-lttng-address-api.txt URL: From mathieu.desnoyers at efficios.com Tue Aug 7 12:25:29 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 7 Aug 2012 12:25:29 -0400 Subject: [lttng-dev] [RFC] LTTng address API proposal v2 In-Reply-To: <50213DDE.8070709@efficios.com> References: <50213DDE.8070709@efficios.com> Message-ID: <20120807162529.GA31399@Krystal> * David Goulet (dgoulet at efficios.com) wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Here is the second and updated version after Yannick comments. > > - v0.2: 07/08/0212 > * Change lttng_create_session_addr to lttng_create_session_url > * Describe URL string format > * Add set_consumer_url examples > * Add the HOP option to the set_consumer_url function > > Cheers! > David > -----BEGIN PGP SIGNATURE----- > > iQEcBAEBCgAGBQJQIT3aAAoJEELoaioR9I02vqIH/1YeAlh+ogt1wvRv3bkExZB6 > vywtNVUVx/ys7KgaqkRolvGmbBKZEMPTwXaiGXDSfbij9hgkLvpjNhc26c9b443r > 9GJ603Jn+PPEa/vOMBPY8sqxGKyKLoSL400sXNLhPo2t18+neWOLDhgcfXd4VIT7 > FgmOQzUQZsImFNGJUTNSFKD7m/NM73F7pM8VG1OjXfOfNAb1KtoV6QYSApoAayQ+ > g2XdPgsAkDQssh82xLQ0Ousg3Cpk72e/jyroUNNnT8Dy7rSTXLsLz7indBzfFoAw > cIzTwREbarYT7cEbNnH0oFRxUK/94MEPjeWd2cSNJxsaidYj9mD07ST9UWfputU= > =GYy5 > -----END PGP SIGNATURE----- > RFC - LTTng address API proposal > > Author: David Goulet > > Contributors: > * Mathieu Desnoyers > * Yannick Brosseau > > Version: > - v0.1: 31/07/2012 > * Initial proposal > - v0.2: 07/08/0212 > * Change lttng_create_session_addr to lttng_create_session_url > * Describe URL string format > * Add set_consumer_url examples > * Add the HOP option to the set_consumer_url function > > Introduction > ----------------- > > This document proposes the use of string URLs to the command line interface and > API which will deprecate a function and propose new ones. > > The purpose of this proposal is to support network streaming using URL string > format that you can find in proposal doc/proposals/0003-network.consumer.txt, > remove the lttng_uri structure from the API and integrate the URL string to the > API. > > API > ----------------- > > In order not to expose the new lttng_uri structure used to identify trace > location for lttng consumer, the public API will only use string address where > it will be converted in a lttng_uri and sent to the session daemon. > > [*] Create session: > > With the introduction of the enable-consumer command used for network streaming, > the create session command has been modified so the user could define a consumer > location either on the network or local with the command. This change deprecates > the old API function and adds a new one. > > Current and will be deprecated: > --> lttng_create_session(const char *name, const char *path); > > Proposed: > --> lttng_create_session_url(const char *name, const char *url, > int enable_consumer); > > The _name_ argument is the session name and _url_ is a string representing the > URL specified by the user which is define like so: > > PROTO://[HOSTNAME|IP][:PORT][/PATH] > > The proto supported at this stage are (6 means IPv6): > > * net, net6, tcp, tcp6, file > > The PATH section is the destination path on the _remote_ host where the trace > data will be written. For example: > > * net://hostname/foo/bar > > Results in writing the trace data in $USER/lttng-traces/foo/bar where > "$USER/lttng-traces" is the default output directory set by the lttng-relayd. > It is possible to change this option. > > The protocol has a special case where the user can input two ports > respectively being the control and data port. > > * net://[HOSTNAME|IP][:CTRL_PORT][:DATA_PORT][/PATH] > > Finally, the enable_consumer option will disable the use of the consumer for enable_consumer option will disable -> this is backwards. We could say "allows controlling if enabled/disabled". > the tracing session. This will be useful with the upcoming snapshot feature. > The motivation behind this flag is to offer the same options as the > enable-consumer command where you can only set the URI for the consumer and not > enable it. > > Note that the hop feature CAN NOT be used with this function call since only > one URL can be define here. define -> defined I wonder if the url arg to lttng_create_session_url could be a "url_list" ? (comma separated) this would enable specifying hops+dest with create session. Thanks, Mathieu > > [*] Consumer: > > The 2.2+ roadmap will implement the concept of hop for network streaming. This > hop is basically a proxy daemon which will relay the information to a final > destination. It is possible to define as many hops as you want but once the > final destination is set, it's not possible anymore. > > Current and will be deprecated: > --> lttng_set_consumer_uri(...) > > Proposed: > --> lttng_set_consumer_url(struct lttng_handle *handle, > const char *url, enum lttng_url_type type); > > For both functions (consumer and create), the _url_ will be parsed into a > lttng_uri in the liblttng-ctl and sent to the session daemon. > > The _type_ argument is the URL type where the possibilities are: > > enum lttng_url_type { > LTTNG_URL_DST, /* Final destination */ > LTTNG_URL_HOP, /* HOP destination */ > }; > > Example: > > set_consumer_url(handle, "net://42.42.42.2", LTTNG_URL_HOP); > set_consumer_url(handle, "net://42.42.42.3", LTTNG_URL_HOP); > set_consumer_url(handle, "net://42.42.42.42", LTTNG_URL_DST); > > > With all this, the lttng_uri data structure will not be exposed to the public > API and the user command line interface. > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From dgoulet at efficios.com Tue Aug 7 12:43:06 2012 From: dgoulet at efficios.com (David Goulet) Date: Tue, 07 Aug 2012 12:43:06 -0400 Subject: [lttng-dev] [RFC] LTTng address API proposal v2 In-Reply-To: <20120807162529.GA31399@Krystal> References: <50213DDE.8070709@efficios.com> <20120807162529.GA31399@Krystal> Message-ID: <5021459A.1080304@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Mathieu Desnoyers: > * David Goulet (dgoulet at efficios.com) wrote: Here is the second and > updated version after Yannick comments. > > - v0.2: 07/08/0212 * Change lttng_create_session_addr to > lttng_create_session_url * Describe URL string format * Add > set_consumer_url examples * Add the HOP option to the > set_consumer_url function > > Cheers! David > >> RFC - LTTng address API proposal >> >> Author: David Goulet >> >> Contributors: * Mathieu Desnoyers >> * Yannick Brosseau >> >> >> Version: - v0.1: 31/07/2012 * Initial proposal - v0.2: >> 07/08/0212 * Change lttng_create_session_addr to >> lttng_create_session_url * Describe URL string format * Add >> set_consumer_url examples * Add the HOP option to the >> set_consumer_url function >> >> Introduction ----------------- >> >> This document proposes the use of string URLs to the command line >> interface and API which will deprecate a function and propose new >> ones. >> >> The purpose of this proposal is to support network streaming >> using URL string format that you can find in proposal >> doc/proposals/0003-network.consumer.txt, remove the lttng_uri >> structure from the API and integrate the URL string to the API. >> >> API ----------------- >> >> In order not to expose the new lttng_uri structure used to >> identify trace location for lttng consumer, the public API will >> only use string address where it will be converted in a lttng_uri >> and sent to the session daemon. >> >> [*] Create session: >> >> With the introduction of the enable-consumer command used for >> network streaming, the create session command has been modified >> so the user could define a consumer location either on the >> network or local with the command. This change deprecates the old >> API function and adds a new one. >> >> Current and will be deprecated: --> lttng_create_session(const >> char *name, const char *path); >> >> Proposed: --> lttng_create_session_url(const char *name, const >> char *url, int enable_consumer); >> >> The _name_ argument is the session name and _url_ is a string >> representing the URL specified by the user which is define like >> so: >> >> PROTO://[HOSTNAME|IP][:PORT][/PATH] >> >> The proto supported at this stage are (6 means IPv6): >> >> * net, net6, tcp, tcp6, file >> >> The PATH section is the destination path on the _remote_ host >> where the trace data will be written. For example: >> >> * net://hostname/foo/bar >> >> Results in writing the trace data in $USER/lttng-traces/foo/bar >> where "$USER/lttng-traces" is the default output directory set by >> the lttng-relayd. It is possible to change this option. >> >> The protocol has a special case where the user can input >> two ports respectively being the control and data port. >> >> * net://[HOSTNAME|IP][:CTRL_PORT][:DATA_PORT][/PATH] >> >> Finally, the enable_consumer option will disable the use of the >> consumer for > > enable_consumer option will disable -> this is backwards. We could > say "allows controlling if enabled/disabled". > >> the tracing session. This will be useful with the upcoming >> snapshot feature. The motivation behind this flag is to offer the >> same options as the enable-consumer command where you can only >> set the URI for the consumer and not enable it. >> >> Note that the hop feature CAN NOT be used with this function call >> since only one URL can be define here. > > define -> defined > > I wonder if the url arg to lttng_create_session_url could be a > "url_list" ? (comma separated) this would enable specifying > hops+dest with create session. I guess it could be and possible without too much overhead. One problem though I can see right now. Considering a comma separated list, this implies that an unknown number of call have to be made to set the consumer URL *unless* the list is parsed on the session daemon side. On the library side, here is the problem I see: lttng_create_session_url(name, "HOP1, HOP2, DST", ...) So this will create a session and set the consumer URL with a call *per* URL. Here is what could happen and be problematic: set_consumer_url(HOP1) --> GOOD set_consumer_url(HOP2) --> BAD [stop, return error] set_consumer_url(DST) On the session daemon side, do I keep HOP1 as valid or not knowing that the next call could change the HOP1 address thus keeping the original HOP1 on the daemon side ? Note that I have no way of knowing that I'm currently creating a session on the daemon side. Cheers! David > > Thanks, > > Mathieu > >> >> [*] Consumer: >> >> The 2.2+ roadmap will implement the concept of hop for network >> streaming. This hop is basically a proxy daemon which will relay >> the information to a final destination. It is possible to define >> as many hops as you want but once the final destination is set, >> it's not possible anymore. >> >> Current and will be deprecated: --> lttng_set_consumer_uri(...) >> >> Proposed: --> lttng_set_consumer_url(struct lttng_handle >> *handle, const char *url, enum lttng_url_type type); >> >> For both functions (consumer and create), the _url_ will be >> parsed into a lttng_uri in the liblttng-ctl and sent to the >> session daemon. >> >> The _type_ argument is the URL type where the possibilities are: >> >> enum lttng_url_type { LTTNG_URL_DST, /* Final destination */ >> LTTNG_URL_HOP, /* HOP destination */ }; >> >> Example: >> >> set_consumer_url(handle, "net://42.42.42.2", LTTNG_URL_HOP); >> set_consumer_url(handle, "net://42.42.42.3", LTTNG_URL_HOP); >> set_consumer_url(handle, "net://42.42.42.42", LTTNG_URL_DST); >> >> >> With all this, the lttng_uri data structure will not be exposed >> to the public API and the user command line interface. > >> _______________________________________________ lttng-dev mailing >> list lttng-dev at lists.lttng.org >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQIUWaAAoJEELoaioR9I02ihMIALDMYGflm4AiD8/K/yJZsMWK DGKQ6vRgrA2FKNWgSVQ0Wye8vf9ejSrhsLZEMgbnXw0suY2qume6ivh0cyJeBbC4 MYb1f1m1o50zB+NTOo2kgZqc1nMwjCyG/RPPTGtQxT5BxeRsDmhhmSdn5dDI8a/4 Aw+xYBKOKSRev3NXPs8jVxXM12Cb9LJUnJ2EGI4LeNWGuwsYNKDrZPhhS9oj3SJ0 +hWomnpHrw0nuLQ0qxYTNhSUzawNbaCeZ0QoFcHiFUT/g1vypPMDoKQauZn6RmFv 9MkynegFYxTs5y66KuVQxyx6vfIw8VGo8Tf3GW0bNGDSptfPSJ6pIh4iLy2VPYw= =1PcV -----END PGP SIGNATURE----- From danny.serres at efficios.com Tue Aug 7 14:43:06 2012 From: danny.serres at efficios.com (Danny Serres) Date: Tue, 7 Aug 2012 14:43:06 -0400 Subject: [lttng-dev] =?utf-8?q?=5Bbabeltrace_PATCH=5D_Babeltrace_python_mo?= =?utf-8?q?dule?= Message-ID: <1344364986-11257-1-git-send-email-danny.serres@efficios.com> The Babeltrace Python module can be used to directly control the Babeltrace API inside Python, using 'import babeltrace'. Therefore, it becomes possible to create a Context, add a trace to it, iterate on it, read events and so on from within Python. SWIG >= 2.0 is used to create the wrapper and its 'warning md variable unused' bug is patched in Makefile.am In the interface file, struct and enum are directly copied from the include files. All changes to struct bt_iter_pos and to enums in ctf/events.h must also be made in the interface file. Signed-off-by: Danny Serres Signed-off-by: Yannick Brosseau --- .gitignore | 4 + Makefile.am | 4 + README | 6 + bootstrap | 2 +- configure.ac | 36 + doc/python-howto.txt | 70 + m4/ax_pkg_swig.m4 | 135 ++ python/Makefile.am | 29 + python/babeltrace.i.in | 1043 +++++++++ python/examples/babeltrace_and_lttng.py | 107 + python/examples/eventcount.py | 66 + python/examples/eventcountlist.py | 65 + python/examples/events_per_cpu.py | 96 + python/examples/example-api-test.py | 59 + python/examples/histogram.py | 121 + python/examples/output_format_modules/cairoplot.py | 2336 ++++++++++++++++++++ .../examples/output_format_modules/pprint_table.py | 35 + python/examples/output_format_modules/series.py | 1140 ++++++++++ python/examples/sched_switch.py | 111 + python/examples/schedtimes.py | 92 + python/examples/softirqtimes.py | 120 + python/examples/syscalls_by_pid.py | 61 + python/examples/timestats.py | 116 + python/python-complements.c | 105 + python/python-complements.h | 36 + tests/tests-python.py | 115 + 26 files changed, 6109 insertions(+), 1 deletion(-) create mode 100644 doc/python-howto.txt create mode 100644 m4/ax_pkg_swig.m4 create mode 100644 python/Makefile.am create mode 100644 python/babeltrace.i.in create mode 100644 python/examples/babeltrace_and_lttng.py create mode 100644 python/examples/eventcount.py create mode 100644 python/examples/eventcountlist.py create mode 100644 python/examples/events_per_cpu.py create mode 100644 python/examples/example-api-test.py create mode 100644 python/examples/histogram.py create mode 100644 python/examples/output_format_modules/__init__.py create mode 100755 python/examples/output_format_modules/cairoplot.py create mode 100644 python/examples/output_format_modules/pprint_table.py create mode 100755 python/examples/output_format_modules/series.py create mode 100644 python/examples/sched_switch.py create mode 100644 python/examples/schedtimes.py create mode 100644 python/examples/softirqtimes.py create mode 100644 python/examples/syscalls_by_pid.py create mode 100644 python/examples/timestats.py create mode 100644 python/python-complements.c create mode 100644 python/python-complements.h create mode 100644 tests/tests-python.py diff --git a/.gitignore b/.gitignore index d6098ac..a347787 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ /tests/test-bitfield +*~ *.o *.a *.la @@ -26,3 +27,6 @@ converter/babeltrace-log core formats/ctf/metadata/ctf-parser.output stamp-h1 +python/babeltrace.i +python/babeltrace.py +python/babeltrace_wrap.c diff --git a/Makefile.am b/Makefile.am index 308ee16..b18c08a 100644 --- a/Makefile.am +++ b/Makefile.am @@ -4,6 +4,10 @@ ACLOCAL_AMFLAGS = -I m4 SUBDIRS = include types lib formats converter tests doc +if USE_PYTHON +SUBDIRS += python +endif + dist_doc_DATA = ChangeLog LICENSE mit-license.txt gpl-2.0.txt \ std-ext-lib.txt diff --git a/README b/README index 75bf0cf..1687075 100644 --- a/README +++ b/README @@ -25,6 +25,7 @@ BUILDING make install ldconfig + If you do not want Python bindings, run ./configure --disable-python. DEPENDENCIES ------------ @@ -44,6 +45,11 @@ To compile Babeltrace, you will need: libpopt >= 1.13 development libraries (Debian : libpopt-dev) (Fedora : popt) + python headers (optional) + (Debian/Ubuntu : python-dev) + swig >= 2.0 (optional) + (Debian/Ubuntu : swig2.0) + For developers using the git tree: diff --git a/bootstrap b/bootstrap index c507425..f6926ca 100755 --- a/bootstrap +++ b/bootstrap @@ -4,7 +4,7 @@ set -x if [ ! -e config ]; then mkdir config fi -aclocal +aclocal -I m4 libtoolize --force --copy autoheader automake --add-missing --copy diff --git a/configure.ac b/configure.ac index d90479d..062f5d5 100644 --- a/configure.ac +++ b/configure.ac @@ -74,6 +74,41 @@ AC_CHECK_LIB([popt], [poptGetContext], [], [AC_MSG_ERROR([Cannot find popt.])] ) + +# For Python +# SWIG version needed or newer: +swig_version=2.0.0 + +AC_ARG_ENABLE([python], + [AC_HELP_STRING([--disable-python], + [do not compile Python bindings])], + [], [enable_python=yes]) + +AM_CONDITIONAL([USE_PYTHON], [test "x${enable_python:-yes}" = xyes]) + +if test "x${enable_python:-yes}" = xyes; then + AC_MSG_NOTICE([You may configure with --disable-python ]dnl +[if you do not want Python bindings.]) + + AX_PKG_SWIG($swig_version, [], [ AC_MSG_ERROR([SWIG $swig_version or newer is needed]) ]) + AM_PATH_PYTHON + + AC_ARG_VAR([PYTHON_INCLUDE], [Include flags for python, bypassing python-config]) + AC_ARG_VAR([PYTHON_CONFIG], [Path to python-config]) + AS_IF([test -z "$PYTHON_INCLUDE"], [ + AS_IF([test -z "$PYTHON_CONFIG"], [ + AC_PATH_PROGS([PYTHON_CONFIG], + [python$PYTHON_VERSION-config python-config], + [no], + [`dirname $PYTHON`]) + AS_IF([test "$PYTHON_CONFIG" = no], [AC_MSG_ERROR([cannot find python-config for $PYTHON.])]) + ]) + AC_MSG_CHECKING([python include flags]) + PYTHON_INCLUDE=`$PYTHON_CONFIG --includes` + AC_MSG_RESULT([$PYTHON_INCLUDE]) + ]) +fi + pkg_modules="gmodule-2.0 >= 2.0.0" PKG_CHECK_MODULES(GMODULE, [$pkg_modules]) AC_SUBST(PACKAGE_LIBS) @@ -103,6 +138,7 @@ AC_CONFIG_FILES([ lib/Makefile lib/prio_heap/Makefile include/Makefile + python/Makefile tests/Makefile ]) AC_OUTPUT diff --git a/doc/python-howto.txt b/doc/python-howto.txt new file mode 100644 index 0000000..b490dac --- /dev/null +++ b/doc/python-howto.txt @@ -0,0 +1,70 @@ +PYTHON BINDINGS +---------------- + +This is a brief howto for using the Babeltrace Python module. + + +INSTALLATION: + +By default, the Python bindings are installed. +If you do not wish the Python bindings, you can configure with the +--disable-python option during the installation procedure: + + $ ./configure --disable-python + +The Python module is automatically generated using SWIG, therefore the +swig2.0 package on Debian/Ubuntu is requied. + + +USAGE: + +Once installed, the Python module can be used by importing it in Python. +In the Python interpreter: + + >>> import babeltrace + +Then the starting point is to create a context and add a trace to it. + + >>> ctx = babeltrace.Context() + >>> ctx.add_trace("path/to/trace", ) + +Where is a string containing the format name in which the trace +was produced. To print a list of available formats to the standard +output, it is possible to use the print_format_list function. + + >>> out = babeltrace.File(None) # This returns stdout + >>> babeltrace.print_format_list(out) + +When a trace is added to a context, it is opened and ready to read using +an iterator. While creating an iterator, optional starting and ending +position may be specified. So far, only ctf iterator are supported. + + >>> begin_pos = babeltrace.IterPos(babeltrace.SEEK_BEGIN) + >>> iterator = babeltrace.ctf.Iterator(ctx, begin_pos) + +From there, it is possible to read the events. + + >>> event = iterator.read_event() + +It is simple to obtain the timestamp of that event. + + >>> timestamp = event.get_timestamp() + +Let's say that we want to extract the prev_comm context info for a +sched_switch event. To do so, it is needed to set an event scope +with which we can obtain the field wanted. + + >>> if event.get_name == "sched_switch": + ... #prev_comm only for sched_switch events + ... scope = event.get_top_level_scope(babeltrace.ctf.scope.EVENT_FIELDS) + ... field = event.get_field(scope, "_prev_comm") + ... prev_comm = field.get_char_array() + +It is also possible to move on to the next event. + + >>> ret = iterator.next() # Move the iterator + >>> if ret == 0: # No error occured + ... event = iterator.read_event() # Read the next event + +For many usage script examples of the Babeltrace Python module, see the +python/examples directory. diff --git a/m4/ax_pkg_swig.m4 b/m4/ax_pkg_swig.m4 new file mode 100644 index 0000000..e112f3d --- /dev/null +++ b/m4/ax_pkg_swig.m4 @@ -0,0 +1,135 @@ +# =========================================================================== +# http://www.gnu.org/software/autoconf-archive/ax_pkg_swig.html +# =========================================================================== +# +# SYNOPSIS +# +# AX_PKG_SWIG([major.minor.micro], [action-if-found], [action-if-not-found]) +# +# DESCRIPTION +# +# This macro searches for a SWIG installation on your system. If found, +# then SWIG is AC_SUBST'd; if not found, then $SWIG is empty. If SWIG is +# found, then SWIG_LIB is set to the SWIG library path, and AC_SUBST'd. +# +# You can use the optional first argument to check if the version of the +# available SWIG is greater than or equal to the value of the argument. It +# should have the format: N[.N[.N]] (N is a number between 0 and 999. Only +# the first N is mandatory.) If the version argument is given (e.g. +# 1.3.17), AX_PKG_SWIG checks that the swig package is this version number +# or higher. +# +# As usual, action-if-found is executed if SWIG is found, otherwise +# action-if-not-found is executed. +# +# In configure.in, use as: +# +# AX_PKG_SWIG(1.3.17, [], [ AC_MSG_ERROR([SWIG is required to build..]) ]) +# AX_SWIG_ENABLE_CXX +# AX_SWIG_MULTI_MODULE_SUPPORT +# AX_SWIG_PYTHON +# +# LICENSE +# +# Copyright (c) 2008 Sebastian Huber +# Copyright (c) 2008 Alan W. Irwin +# Copyright (c) 2008 Rafael Laboissiere +# Copyright (c) 2008 Andrew Collier +# Copyright (c) 2011 Murray Cumming +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation; either version 2 of the License, or (at your +# option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program. If not, see . +# +# As a special exception, the respective Autoconf Macro's copyright owner +# gives unlimited permission to copy, distribute and modify the configure +# scripts that are the output of Autoconf when processing the Macro. You +# need not follow the terms of the GNU General Public License when using +# or distributing such scripts, even though portions of the text of the +# Macro appear in them. The GNU General Public License (GPL) does govern +# all other use of the material that constitutes the Autoconf Macro. +# +# This special exception to the GPL applies to versions of the Autoconf +# Macro released by the Autoconf Archive. When you make and distribute a +# modified version of the Autoconf Macro, you may extend this special +# exception to the GPL to apply to your modified version as well. + +#serial 8 + +AC_DEFUN([AX_PKG_SWIG],[ + # Ubuntu has swig 2.0 as /usr/bin/swig2.0 + AC_PATH_PROGS([SWIG],[swig swig2.0]) + if test -z "$SWIG" ; then + m4_ifval([$3],[$3],[:]) + elif test -n "$1" ; then + AC_MSG_CHECKING([SWIG version]) + [swig_version=`$SWIG -version 2>&1 | grep 'SWIG Version' | sed 's/.*\([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\).*/\1/g'`] + AC_MSG_RESULT([$swig_version]) + if test -n "$swig_version" ; then + # Calculate the required version number components + [required=$1] + [required_major=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_major" ; then + [required_major=0] + fi + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] + [required_minor=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_minor" ; then + [required_minor=0] + fi + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] + [required_patch=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_patch" ; then + [required_patch=0] + fi + # Calculate the available version number components + [available=$swig_version] + [available_major=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_major" ; then + [available_major=0] + fi + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] + [available_minor=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_minor" ; then + [available_minor=0] + fi + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] + [available_patch=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_patch" ; then + [available_patch=0] + fi + # Convert the version tuple into a single number for easier comparison. + # Using base 100 should be safe since SWIG internally uses BCD values + # to encode its version number. + required_swig_vernum=`expr $required_major \* 10000 \ + \+ $required_minor \* 100 \+ $required_patch` + available_swig_vernum=`expr $available_major \* 10000 \ + \+ $available_minor \* 100 \+ $available_patch` + + if test $available_swig_vernum -lt $required_swig_vernum; then + AC_MSG_WARN([SWIG version >= $1 is required. You have $swig_version.]) + SWIG='' + m4_ifval([$3],[$3],[]) + else + AC_MSG_CHECKING([for SWIG library]) + SWIG_LIB=`$SWIG -swiglib` + AC_MSG_RESULT([$SWIG_LIB]) + m4_ifval([$2],[$2],[]) + fi + else + AC_MSG_WARN([cannot determine SWIG version]) + SWIG='' + m4_ifval([$3],[$3],[]) + fi + fi + AC_SUBST([SWIG_LIB]) +]) diff --git a/python/Makefile.am b/python/Makefile.am new file mode 100644 index 0000000..9859196 --- /dev/null +++ b/python/Makefile.am @@ -0,0 +1,29 @@ +babeltrace.i: babeltrace.i.in + sed "s/BABELTRACE_VERSION_STR/Babeltrace $(PACKAGE_VERSION)/g" babeltrace.i + +AM_CFLAGS = -I$(PYTHON_INCLUDE) -I$(top_srcdir)/include/ + +EXTRA_DIST = babeltrace.i +python_PYTHON = babeltrace.py +pyexec_LTLIBRARIES = _babeltrace.la + +MAINTAINERCLEANFILES = babeltrace_wrap.c babeltrace.py + +_babeltrace_la_SOURCES = babeltrace_wrap.c python-complements.c \ + ../converter/babeltrace.c + +_babeltrace_la_LDFLAGS = -module + +_babeltrace_la_CFLAGS = $(GLIB_CFLAGS) $(AM_CFLAGS) + +_babeltrace_la_LIBS = $(GLIB_LIBS) + +_babeltrace_la_LIBADD = ../formats/ctf/libbabeltrace-ctf.la \ + ../formats/ctf-text/libbabeltrace-ctf-text.la + +# SWIG 'warning md variable unused' fixed after SWIG build: +babeltrace_wrap.c: babeltrace.i + $(SWIG) -python -Wall -I. -I$(top_srcdir)/include babeltrace.i + sed -i "s/PyObject \*m, \*d, \*md;/PyObject \*m, \*d;\n#if defined(SWIGPYTHON_BUILTIN)\nPyObject *md;\n#endif/g" babeltrace_wrap.c + sed -i "s/md = d/d/g" babeltrace_wrap.c + sed -i "s/(void)public_symbol;/(void)public_symbol;\n md = d;/g" babeltrace_wrap.c diff --git a/python/babeltrace.i.in b/python/babeltrace.i.in new file mode 100644 index 0000000..101b674 --- /dev/null +++ b/python/babeltrace.i.in @@ -0,0 +1,1043 @@ +/* BABELTRACE PYTHON MODULE interface file */ + +%define DOCSTRING +"BABELTRACE_VERSION_STR + +Babeltrace is a trace viewer and converter reading and writing the +Common Trace Format (CTF). Its main use is to pretty-print CTF +traces into a human-readable text output. + +To use this module, the first step is to create a Context and add a +trace to it." +%enddef + +%module(docstring=DOCSTRING) babeltrace + +%include "typemaps.i" +%{ +#define SWIG_FILE_WITH_INIT +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "python-complements.h" +%} + +typedef unsigned long long uint64_t; +typedef long long int64_t; +typedef int bt_intern_str; + +/* ================================================================= + CONTEXT.H, CONTEXT-INTERNAL.H + ????????????????????????????? +*/ + +%rename("_bt_context_create") bt_context_create(void); +%rename("_bt_context_add_trace") bt_context_add_trace( + struct bt_context *ctx, const char *path, const char *format, + void (*packet_seek)(struct stream_pos *pos, size_t index, int whence), + struct mmap_stream_list *stream_list, FILE *metadata); +%rename("_bt_context_remove_trace") bt_context_remove_trace( + struct bt_context *ctx, int trace_id); +%rename("_bt_context_get") bt_context_get(struct bt_context *ctx); +%rename("_bt_context_put") bt_context_put(struct bt_context *ctx); +%rename("_bt_ctf_event_get_context") bt_ctf_event_get_context( + const struct bt_ctf_event *event); + +struct bt_context *bt_context_create(void); +int bt_context_add_trace(struct bt_context *ctx, const char *path, const char *format, + void (*packet_seek)(struct stream_pos *pos, size_t index, int whence), + struct mmap_stream_list *stream_list, FILE *metadata); +void bt_context_remove_trace(struct bt_context *ctx, int trace_id); +void bt_context_get(struct bt_context *ctx); +void bt_context_put(struct bt_context *ctx); +struct bt_context *bt_ctf_event_get_context(const struct bt_ctf_event *event); + +// class Context to prevent direct access to struct bt_context +%pythoncode%{ +class Context: + """ + The context represents the object in which a trace_collection is + open. As long as this structure is allocated, the trace_collection + is open and the traces it contains can be read and seeked by the + iterators and callbacks. + """ + + def __init__(self): + self._c = _bt_context_create() + + def __del__(self): + _bt_context_put(self._c) + + def add_trace(self, path, format_str, + packet_seek=None, stream_list=None, metadata=None): + """ + Add a trace by path to the context. + + Open a trace. + + path is the path to the trace, it is not recursive. + If "path" is None, stream_list is used instead as a list + of mmap streams to open for the trace. + + format is a string containing the format name in which the trace was + produced. + + packet_seek is not implemented for Python. Should be left None to + use the default packet_seek handler provided by the trace format. + + stream_list is a linked list of streams, it is used to open a trace + where the trace data is located in memory mapped areas instead of + trace files, this argument should be None when path is not None. + + The metadata parameter acts as a metadata override when not None, + otherwise the format handles the metadata opening. + + Return: the corresponding TraceHandle on success or None on error. + """ + if metadata is not None: + metadata = metadata._file + + ret = _bt_context_add_trace(self._c, path, format_str, packet_seek, + stream_list, metadata) + if ret < 0: + return None + + th = TraceHandle.__new__(TraceHandle) + th._id = ret + return th + + def remove_trace(self, trace_handle): + """ + Remove a trace from the context. + Effectively closing the trace. + """ + try: + _bt_context_remove_trace(self._c, trace_handle._id) + except AttributeError: + raise TypeError("in remove_trace, " + "argument 2 must be a TraceHandle instance") +%} + + + +/* ================================================================= + FORMAT.H, REGISTRY + ?????????????????? +*/ + +%rename("lookup_format") bt_lookup_format(bt_intern_str qname); +%rename("_bt_print_format_list") bt_fprintf_format_list(FILE *fp); +%rename("register_format") bt_register_format(struct format *format); + +extern struct format *bt_lookup_format(bt_intern_str qname); +extern void bt_fprintf_format_list(FILE *fp); +extern int bt_register_format(struct format *format); + +void format_init(void); +void format_finalize(void); + +%pythoncode %{ + +def print_format_list(babeltrace_file): + """ + Print a list of available formats to file. + + babeltrace_file must be a File instance opened in write mode. + """ + try: + if babeltrace_file._file is not None: + _bt_print_format_list(babeltrace_file._file) + except AttributeError: + raise TypeError("in print_format_list, " + "argument 1 must be a File instance") + +%} + + +/* ================================================================= + ITERATOR.H, ITERATOR-INTERNAL.H + ??????????????????????????????? +*/ + +%rename("_bt_iter_create") bt_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, const struct bt_iter_pos *end_pos); +%rename("_bt_iter_destroy") bt_iter_destroy(struct bt_iter *iter); +%rename("_bt_iter_next") bt_iter_next(struct bt_iter *iter); +%rename("_bt_iter_get_pos") bt_iter_get_pos(struct bt_iter *iter); +%rename("_bt_iter_free_pos") bt_iter_free_pos(struct bt_iter_pos *pos); +%rename("_bt_iter_set_pos") bt_iter_set_pos(struct bt_iter *iter, + const struct bt_iter_pos *pos); +%rename("_bt_iter_create_time_pos") bt_iter_create_time_pos(struct bt_iter *iter, + uint64_t timestamp); + +struct bt_iter *bt_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, const struct bt_iter_pos *end_pos); +void bt_iter_destroy(struct bt_iter *iter); +int bt_iter_next(struct bt_iter *iter); +struct bt_iter_pos *bt_iter_get_pos(struct bt_iter *iter); +void bt_iter_free_pos(struct bt_iter_pos *pos); +int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *pos); +struct bt_iter_pos *bt_iter_create_time_pos(struct bt_iter *iter, uint64_t timestamp); + +%rename("_bt_iter_pos") bt_iter_pos; +%rename("SEEK_TIME") BT_SEEK_TIME; +%rename("SEEK_RESTORE") BT_SEEK_RESTORE; +%rename("SEEK_CUR") BT_SEEK_CUR; +%rename("SEEK_BEGIN") BT_SEEK_BEGIN; +%rename("SEEK_END") BT_SEEK_END; + + +// This struct is taken from iterator.h +// All changes to the struct must also be made here +struct bt_iter_pos { + enum { + BT_SEEK_TIME, /* uses u.seek_time */ + BT_SEEK_RESTORE, /* uses u.restore */ + BT_SEEK_CUR, + BT_SEEK_BEGIN, + BT_SEEK_END, + } type; + union { + uint64_t seek_time; + struct bt_saved_pos *restore; + } u; +}; + + +%pythoncode%{ + +class IterPos: + """This class represents the position where to set an iterator.""" + + __can_access = False + + def __init__(self, seek_type, seek_time = None): + """ + seek_type represents the type of seek to use. + seek_time is the timestamp to seek to when using SEEK_TIME, it + is expressed in nanoseconds + Only use SEEK_RESTORE on IterPos obtained from the get_pos function + in Iter class. + """ + + self._pos = _bt_iter_pos() + self._pos.type = seek_type + if seek_time and seek_type == SEEK_TIME: + self._pos.u.seek_time = seek_time + self.__can_access = True + + def __del__(self): + if not self.__can_access: + _bt_iter_free_pos(self._pos) + + def _get_type(self): + if not __can_access: + raise AttributeError("seek_type is not available") + return self._pos.type + + def _set_type(self, seek_type): + if not __can_access: + raise AttributeError("seek_type is not available") + self._pos.type = seek_type + + def _get_time(self): + if not __can_access: + raise AttributeError("seek_time is not available") + + elif self._pos.type is not SEEK_TIME: + raise TypeError("seek_type is not SEEK_TIME") + + return self._pos.u.seek_time + + def _set_time(self, time): + if not __can_access: + raise AttributeError("seek_time is not available") + + elif self._pos.type is not SEEK_TIME: + raise TypeError("seek_type is not SEEK_TIME") + + self._pos.u.seek_time = time + + def _get_pos(self): + return self._pos + + + seek_type = property(_get_type, _set_type) + seek_time = property(_get_time, _set_time) + + +class Iterator: + + __with_init = False + + def __init__(self, context, begin_pos = None, end_pos = None, _no_init = None): + """ + Allocate a trace collection iterator. + + begin_pos and end_pos are optional parameters to specify the + position at which the trace collection should be seeked upon + iterator creation, and the position at which iteration will + start returning "EOF". + + By default, if begin_pos is None, a BT_SEEK_CUR is performed at + creation. By default, if end_pos is None, a BT_SEEK_END (end of + trace) is the EOF criterion. + """ + if _no_init is None: + if begin_pos is None: + bp = None + else: + try: + bp = begin_pos._pos + except AttributeError: + raise TypeError("in __init__, " + "argument 3 must be a IterPos instance") + + if end_pos is None: + ep = None + else: + try: + ep = end_pos._pos + except AttributeError: + raise TypeError("in __init__, " + "argument 4 must be a IterPos instance") + + try: + self._bi = _bt_iter_create(context._c, bp, ep) + except AttributeError: + raise TypeError("in __init__, " + "argument 2 must be a Context instance") + + self.__with_init = True + + else: + self._bi = _no_init + + def __del__(self): + if self.__with_init: + _bt_iter_destroy(self._bi) + + def next(self): + """ + Move trace collection position to the next event. + Returns 0 on success, a negative value on error. + """ + return _bt_iter_next(self._bi) + + def get_pos(self): + """Return a IterPos class of the current iterator position.""" + ret = IterPos(0) + ret.__can_access = False + ret._pos = _bt_iter_get_pos(self._bi) + return ret + + def set_pos(self, pos): + """ + Move the iterator to a given position. + + On error, the stream_heap is reinitialized and returned empty. + Return 0 for success. + Return EOF if the position requested is after the last event of the + trace collection. + Return -EINVAL when called with invalid parameter. + Return -ENOMEM if the stream_heap could not be properly initialized. + """ + try: + return _bt_iter_set_pos(self._bi, pos._pos) + except AttributeError: + raise TypeError("in set_pos, " + "argument 2 must be a IterPos instance") + + def create_time_pos(self, timestamp): + """ + Create a position based on time + This function allocates and returns a new IterPos to be able to + restore an iterator position based on a timestamp. + """ + + if timestamp < 0: + raise TypeError("timestamp must be an unsigned int") + + ret = IterPos(0) + ret.__can_access = False + ret._pos = _bt_iter_create_time_pos(self._bi, timestamp) + return ret +%} + + +/* ================================================================= + CLOCK-TYPE.H + ???????????? + *** Enum copied from clock-type.h? + All changes must also be made here +*/ +%rename("CLOCK_CYCLES") BT_CLOCK_CYCLES; +%rename("CLOCK_REAL") BT_CLOCK_REAL; + +enum bt_clock_type { + BT_CLOCK_CYCLES = 0, + BT_CLOCK_REAL +}; + +/* ================================================================= + TRACE-HANDLE.H, TRACE-HANDLE-INTERNAL.H + ??????????????????????????????????????? +*/ + +%rename("_bt_trace_handle_create") bt_trace_handle_create(struct bt_context *ctx); +%rename("_bt_trace_handle_destroy") bt_trace_handle_destroy(struct bt_trace_handle *bt); +struct bt_trace_handle *bt_trace_handle_create(struct bt_context *ctx); +void bt_trace_handle_destroy(struct bt_trace_handle *bt); + +%rename("_bt_trace_handle_get_path") bt_trace_handle_get_path(struct bt_context *ctx, + int handle_id); +%rename("_bt_trace_handle_get_timestamp_begin") bt_trace_handle_get_timestamp_begin( + struct bt_context *ctx, int handle_id, enum bt_clock_type type); +%rename("_bt_trace_handle_get_timestamp_end") bt_trace_handle_get_timestamp_end( + struct bt_context *ctx, int handle_id, enum bt_clock_type type); +%rename("_bt_trace_handle_get_id") bt_trace_handle_get_id(struct bt_trace_handle *th); +int bt_trace_handle_get_id(struct bt_trace_handle *th); +const char *bt_trace_handle_get_path(struct bt_context *ctx, int handle_id); +uint64_t bt_trace_handle_get_timestamp_begin(struct bt_context *ctx, int handle_id, + enum bt_clock_type type); +uint64_t bt_trace_handle_get_timestamp_end(struct bt_context *ctx, int handle_id, + enum bt_clock_type type); + +%rename("_bt_ctf_event_get_handle_id") bt_ctf_event_get_handle_id( + const struct bt_ctf_event *event); +int bt_ctf_event_get_handle_id(const struct bt_ctf_event *event); + + +%pythoncode%{ + +class TraceHandle(object): + """ + The TraceHandle allows the user to manipulate a trace file directly. + It is a unique identifier representing a trace file. + Do not instantiate. + """ + + def __init__(self): + raise NotImplementedError("TraceHandle cannot be instantiated") + + def __repr__(self): + return "Babeltrace TraceHandle: trace_id('{}')".format(self._id) + + def get_id(self): + """Return the TraceHandle id.""" + return self._id + + def get_path(self, context): + """Return the path of a TraceHandle.""" + try: + return _bt_trace_handle_get_path(context._c, self._id) + except AttributeError: + raise TypeError("in get_path, " + "argument 2 must be a Context instance") + + def get_timestamp_begin(self, context, clock_type): + """Return the creation time of the buffers of a trace.""" + try: + return _bt_trace_handle_get_timestamp_begin( + context._c, self._id,clock_type) + except AttributeError: + raise TypeError("in get_timestamp_begin, " + "argument 2 must be a Context instance") + + def get_timestamp_end(self, context, clock_type): + """Return the destruction timestamp of the buffers of a trace.""" + try: + return _bt_trace_handle_get_timestamp_end( + context._c, self._id, clock_type) + except AttributeError: + raise TypeError("in get_timestamp_end, " + "argument 2 must be a Context instance") + +%} + + + +// ================================================================= +// CTF +// ================================================================= + +/* ================================================================= + ITERATOR.H, EVENTS.H + ???????????????????? +*/ + +//Iterator +%rename("_bt_ctf_iter_create") bt_ctf_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, + const struct bt_iter_pos *end_pos); +%rename("_bt_ctf_get_iter") bt_ctf_get_iter(struct bt_ctf_iter *iter); +%rename("_bt_ctf_iter_destroy") bt_ctf_iter_destroy(struct bt_ctf_iter *iter); +%rename("_bt_ctf_iter_read_event") bt_ctf_iter_read_event(struct bt_ctf_iter *iter); + +struct bt_ctf_iter *bt_ctf_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, + const struct bt_iter_pos *end_pos); +struct bt_iter *bt_ctf_get_iter(struct bt_ctf_iter *iter); +void bt_ctf_iter_destroy(struct bt_ctf_iter *iter); +struct bt_ctf_event *bt_ctf_iter_read_event(struct bt_ctf_iter *iter); + + +//Events + +%rename("_bt_ctf_get_top_level_scope") bt_ctf_get_top_level_scope(const struct + bt_ctf_event *event, enum bt_ctf_scope scope); +%rename("_bt_ctf_event_name") bt_ctf_event_name(const struct bt_ctf_event *ctf_event); +%rename("_bt_ctf_get_timestamp") bt_ctf_get_timestamp( + const struct bt_ctf_event *ctf_event); +%rename("_bt_ctf_get_cycles") bt_ctf_get_cycles( + const struct bt_ctf_event *ctf_event); + +%rename("_bt_ctf_get_field") bt_ctf_get_field(const struct bt_ctf_event *ctf_event, + const struct definition *scope, const char *field); +%rename("_bt_ctf_get_index") bt_ctf_get_index(const struct bt_ctf_event *ctf_event, + const struct definition *field, unsigned int index); +%rename("_bt_ctf_field_name") bt_ctf_field_name(const struct definition *def); +%rename("_bt_ctf_field_type") bt_ctf_field_type(const struct definition *def); +%rename("_bt_ctf_get_int_signedness") bt_ctf_get_int_signedness( + const struct definition *field); +%rename("_bt_ctf_get_int_base") bt_ctf_get_int_base(const struct definition *field); +%rename("_bt_ctf_get_int_byte_order") bt_ctf_get_int_byte_order( + const struct definition *field); +%rename("_bt_ctf_get_int_len") bt_ctf_get_int_len(const struct definition *field); +%rename("_bt_ctf_get_encoding") bt_ctf_get_encoding(const struct definition *field); +%rename("_bt_ctf_get_array_len") bt_ctf_get_array_len(const struct definition *field); +%rename("_bt_ctf_get_uint64") bt_ctf_get_uint64(const struct definition *field); +%rename("_bt_ctf_get_int64") bt_ctf_get_int64(const struct definition *field); +%rename("_bt_ctf_get_char_array") bt_ctf_get_char_array(const struct definition *field); +%rename("_bt_ctf_get_string") bt_ctf_get_string(const struct definition *field); +%rename("_bt_ctf_field_get_error") bt_ctf_field_get_error(void); +%rename("_bt_ctf_get_decl_event_name") bt_ctf_get_decl_event_name(const struct + bt_ctf_event_decl *event); +%rename("_bt_ctf_get_decl_field_name") bt_ctf_get_decl_field_name( + const struct bt_ctf_field_decl *field); + +const struct definition *bt_ctf_get_top_level_scope(const struct bt_ctf_event *ctf_event, + enum bt_ctf_scope scope); +const char *bt_ctf_event_name(const struct bt_ctf_event *ctf_event); +uint64_t bt_ctf_get_timestamp(const struct bt_ctf_event *ctf_event); +uint64_t bt_ctf_get_cycles(const struct bt_ctf_event *ctf_event); +const struct definition *bt_ctf_get_field(const struct bt_ctf_event *ctf_event, + const struct definition *scope, + const char *field); +const struct definition *bt_ctf_get_index(const struct bt_ctf_event *ctf_event, + const struct definition *field, + unsigned int index); +const char *bt_ctf_field_name(const struct definition *def); +enum ctf_type_id bt_ctf_field_type(const struct definition *def); +int bt_ctf_get_int_signedness(const struct definition *field); +int bt_ctf_get_int_base(const struct definition *field); +int bt_ctf_get_int_byte_order(const struct definition *field); +ssize_t bt_ctf_get_int_len(const struct definition *field); +enum ctf_string_encoding bt_ctf_get_encoding(const struct definition *field); +int bt_ctf_get_array_len(const struct definition *field); +uint64_t bt_ctf_get_uint64(const struct definition *field); +int64_t bt_ctf_get_int64(const struct definition *field); +char *bt_ctf_get_char_array(const struct definition *field); +char *bt_ctf_get_string(const struct definition *field); +int bt_ctf_field_get_error(void); +const char *bt_ctf_get_decl_event_name(const struct bt_ctf_event_decl *event); +const char *bt_ctf_get_decl_field_name(const struct bt_ctf_field_decl *field); + +%pythoncode%{ + +class ctf: + + #enum equivalent, accessible constants + #These are taken directly from ctf/events.h + #All changes to enums must also be made here + class type_id: + UNKNOWN = 0 + INTEGER = 1 + FLOAT = 2 + ENUM = 3 + STRING = 4 + STRUCT = 5 + UNTAGGED_VARIANT = 6 + VARIANT = 7 + ARRAY = 8 + SEQUENCE = 9 + NR_CTF_TYPES = 10 + + class scope: + TRACE_PACKET_HEADER = 0 + STREAM_PACKET_CONTEXT = 1 + STREAM_EVENT_HEADER = 2 + STREAM_EVENT_CONTEXT = 3 + EVENT_CONTEXT = 4 + EVENT_FIELDS = 5 + + class string_encoding: + NONE = 0 + UTF8 = 1 + ASCII = 2 + UNKNOWN = 3 + + class Iterator(Iterator, object): + """ + Allocate a CTF trace collection iterator. + + begin_pos and end_pos are optional parameters to specify the + position at which the trace collection should be seeked upon + iterator creation, and the position at which iteration will + start returning "EOF". + + By default, if begin_pos is None, a SEEK_CUR is performed at + creation. By default, if end_pos is None, a SEEK_END (end of + trace) is the EOF criterion. + + Only one iterator can be created against a context. If more than one + iterator is being created for the same context, the second creation + will return None. The previous iterator must be destroyed before + creation of the new iterator for this function to succeed. + """ + + def __new__(cls, context, begin_pos = None, end_pos = None): + # __new__ is used to control the return value + # as the ctf.Iterator class should return None + # if bt_ctf_iter_create returns NULL + + if begin_pos is None: + bp = None + else: + bp = begin_pos._pos + if end_pos is None: + ep = None + else: + ep = end_pos._pos + try: + it = _bt_ctf_iter_create(context._c, bp, ep) + except AttributeError: + raise TypeError("in __init__, " + "argument 2 must be a Context instance") + if it is None: + return None + + ret_class = super(ctf.Iterator, cls).__new__(cls) + ret_class._i = it + return ret_class + + def __init__(self, context, begin_pos = None, end_pos = None): + Iterator.__init__(self, None, None, None, + _bt_ctf_get_iter(self._i)) + + def __del__(self): + _bt_ctf_iter_destroy(self._i) + + def read_event(self): + """ + Read the iterator's current event data. + Return current event on success, None on end of trace. + """ + ret = _bt_ctf_iter_read_event(self._i) + if ret is None: + return ret + ev = ctf.Event.__new__(ctf.Event) + ev._e = ret + return ev + + + class Event(object): + """ + This class represents an event from the trace. + It is obtained with read_event() from ctf.Iterator. + Do not instantiate. + """ + + def __init__(self): + raise NotImplementedError("ctf.Event cannot be instantiated") + + def get_top_level_scope(self, scope): + """ + Return a definition of the top-level scope + Top-level scopes are defined in ctf.scope. + In order to get a field or a field list, the user needs to pass a + scope as argument, this scope can be a top-level scope or a scope + relative to an arbitrary field. This function provides the mapping + between the scope and the actual definition of top-level scopes. + On error return None. + """ + evDef = ctf.Definition.__new__(ctf.Definition) + evDef._d = _bt_ctf_get_top_level_scope(self._e, scope) + if evDef._d is None: + return None + return evDef + + def get_name(self): + """Return the name of the event or None on error.""" + return _bt_ctf_event_name(self._e) + + def get_cycles(self): + """ + Return the timestamp of the event as written in + the packet (in cycles) or -1ULL on error. + """ + return _bt_ctf_get_cycles(self._e) + + def get_timestamp(self): + """ + Return the timestamp of the event offsetted with the + system clock source or -1ULL on error. + """ + return _bt_ctf_get_timestamp(self._e) + + def get_field(self, scope, field): + """Return the definition of a specific field.""" + evDef = ctf.Definition.__new__(ctf.Definition) + try: + evDef._d = _bt_ctf_get_field(self._e, scope._d, field) + except AttributeError: + raise TypeError("in get_field, argument 2 must be a " + "Definition (scope) instance") + return evDef + + def get_field_list(self, scope): + """ + Return a list of Definitions + Return None on error. + """ + try: + field_lc = _bt_python_field_listcaller(self._e, scope._d) + except AttributeError: + raise TypeError("in get_field_list, argument 2 must be a " + "Definition (scope) instance") + + if field_lc is None: + return None + + def_list = [] + i = 0 + while True: + tmp = ctf.Definition.__new__(ctf.Definition) + tmp._d = _bt_python_field_one_from_list(field_lc, i) + + if tmp._d is None: + #Last item of list is None, assured in + #_bt_python_field_listcaller + break + + def_list.append(tmp) + i += 1 + return def_list + + def get_index(self, field, index): + """ + If the field is an array or a sequence, return the element + at position index, otherwise return None + """ + evDef = ctf.Definition.__new__(ctf.Definition) + try: + evDef._d = _bt_ctf_get_index(self._e, field._d, index) + except AttributeError: + raise TypeError("in get_index, argument 2 must be a " + "Definition (field) instance") + + if evDef._d is None: + return None + return evDef + + def get_handle(self): + """ + Get the TraceHandle associated with an event + Return None on error + """ + ret = _bt_ctf_event_get_handle_id(self._e) + if ret < 0: + return None + + th = TraceHandle.__new__(TraceHandle) + th._id = ret + return th + + def get_context(self): + """ + Get the context associated with an event. + Return None on error. + """ + ctx = Context() + ctx._c = _bt_ctf_event_get_context(self._e); + if ctx._c is None: + return None + else: + return ctx + + + class Definition(object): + """Definition class. Do not instantiate.""" + + def __init__(self): + raise NotImplementedError("ctf.Definition cannot be instantiated") + + def field_name(self): + """Return the name of a field or None on error.""" + return _bt_ctf_field_name(self._d) + + def field_type(self): + """Return the type of a field or -1 if unknown.""" + return _bt_ctf_field_type(self._d) + + def get_int_signedness(self): + """ + Return the signedness of an integer: + 0 if unsigned; 1 if signed; -1 on error. + """ + return _bt_ctf_get_int_signedness(self._d) + + def get_int_base(self): + """Return the base of an int or a negative value on error.""" + return _bt_ctf_get_int_base(self._d) + + def get_int_byte_order(self): + """ + Return the byte order of an int or a negative + value on error. + """ + return _bt_ctf_get_int_byte_order(self._d) + + def get_int_len(self): + """ + Return the size, in bits, of an int or a negative + value on error. + """ + return _bt_ctf_get_int_len(self._d) + + def get_encoding(self): + """ + Return the encoding of an int or a string. + Return a negative value on error. + """ + return _bt_ctf_get_encoding(self._d) + + def get_array_len(self): + """ + Return the len of an array or a negative + value on error. + """ + return _bt_ctf_get_array_len(self._d) + + def get_uint64(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_uint64(self._d) + + def get_int64(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_int64(self._d) + + def get_char_array(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_char_array(self._d) + + def get_str(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_string(self._d) + + + class EventDecl(object): + """Event declaration class. Do not instantiate.""" + + def __init__(self): + raise NotImplementedError("ctf.EventDecl cannot be instantiated") + + def __repr__(self): + return "Babeltrace EventDecl: name {}".format(self.get_name()) + + def get_name(self): + """Return the name of the event or None on error""" + return _bt_ctf_get_decl_event_name(self._d) + + def get_decl_fields(self, scope): + """ + Return a list of ctf.FieldDecl + Return None on error. + """ + ptr_list = _by_python_field_decl_listcaller(self._d, scope) + + if ptr_list is None: + return None + + decl_list = [] + i = 0 + while True: + tmp = ctf.FieldDecl.__new__(ctf.FieldDecl) + tmp._d = _bt_python_field_decl_one_from_list( + ptr_list, i) + + if tmp._d is None: + #Last item of list is None + break + + decl_list.append(tmp) + i += 1 + return decl_list + + + class FieldDecl(object): + """Field declaration class. Do not instantiate.""" + + def __init__(self): + raise NotImplementedError("ctf.FieldDecl cannot be instantiated") + + def __repr__(self): + return "Babeltrace FieldDecl: name {}".format(self.get_name()) + + def get_name(self): + """Return the name of a FieldDecl or None on error""" + return _bt_ctf_get_decl_field_name(self._d) + + + @staticmethod + def field_error(): + """ + Return the last error code encountered while + accessing a field and reset the error flag. + Return 0 if no error, a negative value otherwise. + """ + return _bt_ctf_field_get_error() + + @staticmethod + def get_event_decl_list(trace_handle, context): + """ + Return a list of ctf.EventDecl + Return None on error. + """ + try: + handle_id = trace_handle._id + except AttributeError: + raise TypeError("in get_event_decl_list, " + "argument 1 must be a TraceHandle instance") + try: + ptr_list = _bt_python_event_decl_listcaller(handle_id, context._c) + except AttributeError: + raise TypeError("in get_event_decl_list, " + "argument 2 must be a Context instance") + + if ptr_list is None: + return None + + decl_list = [] + i = 0 + while True: + tmp = ctf.EventDecl.__new__(ctf.EventDecl) + tmp._d = _bt_python_decl_one_from_list(ptr_list, i) + + if tmp._d is None: + #Last item of list is None + break + + decl_list.append(tmp) + i += 1 + return decl_list + +%} + + + +// ================================================================= +// NEW FUNCTIONS +// File and list-related +// python-complements.h +// ================================================================= + +%include python-complements.c + +%pythoncode %{ + +class File(object): + """ + Open a file for babeltrace. + + file_path is a string containing the path or None to use the + standard output in writing mode. + + The mode can be 'r', 'w' or 'a' for reading (default), writing or + appending. The file will be created if it doesn't exist when + opened for writing or appending; it will be truncated when opened + for writing. Add a 'b' to the mode for binary files. Add a '+' + to the mode to allow simultaneous reading and writing. + """ + + def __new__(cls, file_path, mode='r'): + # __new__ is used to control the return value + # as the File class should return None + # if _bt_file_open returns NULL + + # Type check + if file_path is not None and type(file_path) is not str: + raise TypeError("in method __init__, argument 2 of type 'str'") + if type(mode) is not str: + raise TypeError("in method __init__, argument 3 of type 'str'") + + # Opening file + file_ptr = _bt_file_open(file_path, mode) + if file_ptr is None: + return None + + # Class instantiation + file_inst = super(File, cls).__new__(cls) + file_inst._file = file_ptr + return file_inst + + def __init__(self, file_path, mode='r'): + self._opened = True + self._use_stdout = False + + if file_path is None: + # use stdout + file_path = "stdout" + mode = 'w' + self._use_stdout = True + + self._file_path = file_path + self._mode = mode + + def __del__(self): + self.close() + + def __repr__(self): + if self._opened: + stat = 'opened' + else: + stat = 'closed' + return "{} babeltrace File; file_path('{}'), mode('{}')".format( + stat, self._file_path, self._mode) + + def close(self): + """Close the file. Is also called using del.""" + if self._opened and not self._use_stdout: + _bt_file_close(self._file) + self._opened = False +%} diff --git a/python/examples/babeltrace_and_lttng.py b/python/examples/babeltrace_and_lttng.py new file mode 100644 index 0000000..ef0e35c --- /dev/null +++ b/python/examples/babeltrace_and_lttng.py @@ -0,0 +1,107 @@ +# This script uses both lttng-tools and babeltrace +# python modules. It creates a session, enables +# events, starts tracing for 2 seconds, stops tracing, +# destroys the session and outputs the trace in the +# specified output file. +# +# WARNING: will destroy any existing trace having +# the same name as ses_name + + +# ------------------------------------------------------ +ses_name = "babeltrace-lttng-test" +trace_path = "/lttng-traces/babeltrace-lttng-trace/" +out_file = "babeltrace-lttng-trace-text-output.txt" +# ------------------------------------------------------ + + +import time +try: + import babeltrace, lttng +except ImportError: + raise ImportError( "both babeltrace and lttng-tools " + "python modules must be installed" ) + + +# Errors to raise if something goes wrong +class LTTngError(Exception): + pass +class BabeltraceError(Exception): + pass + + +# LTTNG-TOOLS + +# Making sure session does not already exist +lttng.destroy(ses_name) + +# Creating a new session and handle +ret = lttng.create(ses_name,trace_path) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + +han = None +han = lttng.Handle(ses_name, lttng.Domain()) +if han is None: + raise LTTngError("Handle not created") + + +# Enabling all events +ret = lttng.enable_event(han, lttng.Event(), None) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + + +# Start, wait, stop +ret = lttng.start(ses_name) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) +print("Tracing...") +time.sleep(2) +print("Stopped.") +ret = lttng.stop(ses_name) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + + +# Destroying tracing session +ret = lttng.destroy(ses_name) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + + +# BABELTRACE + +# Create context and add trace: +ctx = babeltrace.Context() +ret = ctx.add_trace(trace_path + "/kernel", "ctf") +if ret is None: + raise BabeltraceError("Error adding trace") + +# Iterator setup +bp = babeltrace.IterPos(babeltrace.SEEK_BEGIN) +ctf_it = babeltrace.ctf.Iterator(ctx,bp) + +# Reading events from trace +# and outputting timestamps and event names +# in out_file +print("Writing trace file...") +output = open(out_file, "wt") + +event = ctf_it.read_event() +while(event is not None): + output.write("TS: {}, {} : {}\n".format(event.get_timestamp(), + event.get_cycles(), event.get_name())) + + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +# Closing file +output.close() + +# Destroying dynamic elements +del ctf_it, han +print("Done.") diff --git a/python/examples/eventcount.py b/python/examples/eventcount.py new file mode 100644 index 0000000..2a63b74 --- /dev/null +++ b/python/examples/eventcount.py @@ -0,0 +1,66 @@ +# The script prints a count of specified events and +# their related tid's in a given trace. +# The trace needs TID context (lttng add-context -k -t tid) + +import sys +from babeltrace import * +from output_format_modules.pprint_table import pprint_table as pprint + +if len(sys.argv) < 3: + raise TypeError("Usage: python eventcount.py event1 [event2 ...] path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +counts = {} + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +while(event is not None): + for event_type in sys.argv[1:len(sys.argv)-1]: + if event_type == event.get_name(): + + # Getting scope definition + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) + if sco is None: + print("ERROR: Cannot get definition scope for {}".format( + event.get_name())) + continue + + # Getting TID + tid_field = event.get_field(sco, "_tid") + tid = tid_field.get_int64() + + if ctf.field_error(): + print("ERROR: Missing TID info for {}".format( + event.get_name())) + continue + + tmp = (tid, event.get_name()) + + if tmp in counts: + counts[tmp] += 1 + else: + counts[tmp] = 1 + + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it + +# Appending data to table for output +table = [] +for item in counts: + table.append([item[0], item[1], counts[item]]) +table = sorted(table) +table.insert(0,["TID", "EVENT", "COUNT"]) +pprint(table, 2) diff --git a/python/examples/eventcountlist.py b/python/examples/eventcountlist.py new file mode 100644 index 0000000..800996c --- /dev/null +++ b/python/examples/eventcountlist.py @@ -0,0 +1,65 @@ +# The script prints a count and rate of events. +# It also outputs a bar graph of count per event, using the cairoplot module. + +import sys +from babeltrace import * +from output_format_modules import cairoplot +from output_format_modules.pprint_table import pprint_table as pprint + +# Check for path arg: +if len(sys.argv) < 2: + raise TypeError("Usage: python eventcountlist.py path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +# Events and their assossiated count +# will be stored as a dict: +events_count = {} + +# Setting iterator: +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx,bp) + +prev_event = None +event = ctf_it.read_event() + +start_time = event.get_timestamp() + +# Reading events: +while(event is not None): + if event.get_name() in events_count: + events_count[event.get_name()] += 1 + else: + events_count[event.get_name()] = 1 + + ret = ctf_it.next() + if ret < 0: + break + else: + prev_event = event + event = ctf_it.read_event() + +if event: + total_time = event.get_timestamp() - start_time +else: + total_time = prev_event.get_timestamp() - start_time + +del ctf_it + +# Printing encountered events with respective count and rate: +print("Total time: {} ns".format(total_time)) +table = [["EVENT", "COUNT", "RATE (Hz)"]] +for item in sorted(events_count.iterkeys()): + tmp = [item, events_count[item], + events_count[item]/(total_time/1000000000.0)] + table.append(tmp) +pprint(table) + +# Exporting data as bar graph +cairoplot.vertical_bar_plot ( 'eventcountlist.svg', events_count, 50+85*len(events_count), + 800, border = 20, display_values = True, grid = True, + rounded_corners = True, + x_labels = sorted(events_count.keys()) ) diff --git a/python/examples/events_per_cpu.py b/python/examples/events_per_cpu.py new file mode 100644 index 0000000..b464735 --- /dev/null +++ b/python/examples/events_per_cpu.py @@ -0,0 +1,96 @@ +# The script opens a trace and prints out CPU statistics +# for the given trace (event count per CPU, total active +# time and % of time processing events). +# It also outputs a .txt file showing each time interval +# (since the beginning of the trace) in which each CPU +# was active and the corresponding event. + +import sys, multiprocessing +from output_format_modules.pprint_table import pprint_table as pprint +from babeltrace import * + +if len(sys.argv) < 2: + raise TypeError("Usage: python events_per_cpu.py path/to/trace") + +# Adding trace +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +cpu_usage = [] +usage_time = [] +nbEvents = 0 +i = 0 +while i < multiprocessing.cpu_count(): + cpu_usage.append([]) + usage_time.append(0) + i += 1 + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +start_time = event.get_timestamp() +old_ts = start_time + +while(event is not None): + + skip = False + event_name = event.get_name() + + # Getting cpu_id + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) + field = event.get_field(scope, "cpu_id") + cpu_id = field.get_uint64() + if ctf.field_error(): + print("ERROR: Missing cpu_id info for {}".format(event.get_name())) + skip = True + + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + if event is None: + break + + ts = event.get_timestamp() + + if not skip: + cpu_usage[cpu_id].append( (old_ts - start_time, ts - start_time, event_name) ) + usage_time[cpu_id] += ts - old_ts + nbEvents += 1 + + old_ts = ts + +trace_time = old_ts - start_time + +# Outputting +table = [] +output = open("events_per_cpu.txt", "wt") +output.write("(start time (ns since beginning of trace), end time (ns), event)\n") + +for cpu in range(len(usage_time)): + # Setting table + time_str = str(100.0 * usage_time[cpu] / trace_time) + '000' + event_str = str(100.0 * len(cpu_usage[cpu]) / nbEvents) + '000' + # % is printed with 2 decimals + table.append([cpu, len(cpu_usage[cpu]), event_str[0:event_str.find('.') + 3] + ' %', + usage_time[cpu], time_str[0:time_str.find('.') + 3] + ' %']) + + # Writing to file + output.write("\n\n\n----------------------\n") + output.write("CPU {}\n\n".format(cpu)) + for event in cpu_usage[cpu]: + output.write(str(event) + '\n') + +# Printing table +table.insert(0, ["CPU ID", "EVENT COUNT", "TRACE EVENT %", "TOTAL ACTIVE TIME (ns)", "TRACE TIME %"]) +pprint(table) +print("Total event count: {}".format(nbEvents)) +print("Total trace time: {} ns".format(trace_time)) + +output.close() diff --git a/python/examples/example-api-test.py b/python/examples/example-api-test.py new file mode 100644 index 0000000..0cfc3ed --- /dev/null +++ b/python/examples/example-api-test.py @@ -0,0 +1,59 @@ +# This example uses the babeltrace python module +# to partially test the api. + +import sys +from babeltrace import * + +# Check for path arg: +if len(sys.argv) < 2: + raise TypeError("Usage: python example-api-test.py path/to/file") + +# Create context and add trace: +ctx = Context() +trace_handle = ctx.add_trace(sys.argv[1], "ctf") +if trace_handle is None: + raise IOError("Error adding trace") + +# Listing events +lst = ctf.get_event_decl_list(trace_handle, ctx) +print("--- Event list ---") +for item in lst: + print("event : {}".format(item.get_name())) +print("--- Done ---") + +# Iter trace +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx,bp) +event = ctf_it.read_event() + +while(event is not None): + print("TS: {}, {} : {}".format(event.get_timestamp(), + event.get_cycles(), event.get_name())) + + if event.get_name() == "sched_switch": + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + prev_field = event.get_field(sco, "_prev_comm") + prev_comm = prev_field.get_char_array() + + if ctf.field_error(): + print("ERROR: Missing prev_comm context info") + else: + print("sched_switch prev_comm: {}".format(prev_comm)) + + if event.get_name() == "exit_syscall": + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + ret_field = event.get_field(sco, "_ret") + ret_code = ret_field.get_int64() + + if ctf.field_error(): + print("ERROR: Unable to extract ret") + else: + print("exit_syscall ret: {}".format(ret_code)) + + ret = ctf_it.next() + if ret < 0: + break + else: + event = ctf_it.read_event() + +del ctf_it diff --git a/python/examples/histogram.py b/python/examples/histogram.py new file mode 100644 index 0000000..a24c00d --- /dev/null +++ b/python/examples/histogram.py @@ -0,0 +1,121 @@ +# The script checks the number of events in the trace +# and outputs a table and a .svg histogram for the specified +# range (microseconds) or the total trace if no range specified. +# The graph is generated using the cairoplot module. + +import sys +from babeltrace import * +from output_format_modules import cairoplot +from output_format_modules.pprint_table import pprint_table as pprint + +# ------------------------------------------------ +# Output settings + +# number of intervals: +nbDiv = 25 # Should not be over 150 + # for usable graph output + +# table output stream (file-like object): +out = sys.stdout +# ------------------------------------------------- + +if len(sys.argv) < 2 or len(sys.argv) > 4: + raise TypeError("Usage: python histogram.py [ start_time [end_time] ] path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +# Check when to start/stop graphing +sinceBegin = True +beginTime = 0.0 +if len(sys.argv) > 2: + sinceBegin = False + beginTime = float(sys.argv[1]) +untilEnd = True +if len(sys.argv) == 4: + untilEnd = False + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +start_time = event.get_timestamp() +time = 0 +count = {} + +while(event is not None): + # Microsec. + time = (event.get_timestamp() - start_time)/1000.0 + + # Check if in range + if not sinceBegin: + if time < beginTime: + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + continue + if not untilEnd: + if time > float(sys.argv[2]): + break + + # Counting events per timestamp: + if time in count: + count[time] += 1 + else: + count[time] = 1 + + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it + +# Setting data for output +interval = (time - beginTime)/nbDiv +div_begin_time = beginTime +div_end_time = beginTime + interval +data = {} + +# Prefix for string sorting, considering +# there should not be over 150 intervals. +# This would work up to 9999 intervals. +# If needed, add zeros. +prefix = 0.0001 + +while div_end_time <= time: + key = str(prefix) + '[' + str(div_begin_time) + ';' + str(div_end_time) + '[' + for tmp in count: + if tmp >= div_begin_time and tmp < div_end_time: + if key in data: + data[key] += count[tmp] + else: + data[key] = count[tmp] + if not key in data: + data[key] = 0 + div_begin_time = div_end_time + div_end_time += interval + # Prefix increment + prefix += 0.001 + +table = [] +x_labels = [] +for key in sorted(data): + table.append([key[key.find('['):], data[key]]) + x_labels.append(key[key.find('['):]) + +# Table output +table.insert(0, ["INTERVAL (us)", "COUNT"]) +pprint(table, 1, out) + +# Graph output +cairoplot.vertical_bar_plot ( 'histogram.svg', data, 50 + 150*nbDiv, 50*nbDiv, + border = 20, display_values = True, grid = True, + x_labels = x_labels, rounded_corners = True ) diff --git a/python/examples/output_format_modules/__init__.py b/python/examples/output_format_modules/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/python/examples/output_format_modules/cairoplot.py b/python/examples/output_format_modules/cairoplot.py new file mode 100755 index 0000000..a27113f --- /dev/null +++ b/python/examples/output_format_modules/cairoplot.py @@ -0,0 +1,2336 @@ +?#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# CairoPlot.py +# +# Copyright (c) 2008 Rodrigo Moreira Ara?jo +# +# Author: Rodrigo Moreiro Araujo +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public License +# as published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 +# USA + +#Contributor: Jo?o S. O. Bueno + +#TODO: review BarPlot Code +#TODO: x_label colision problem on Horizontal Bar Plot +#TODO: y_label's eat too much space on HBP + + +__version__ = 1.2 + +import cairo +import math +import random +from series import Series, Group, Data + +HORZ = 0 +VERT = 1 +NORM = 2 + +COLORS = {"red" : (1.0,0.0,0.0,1.0), "lime" : (0.0,1.0,0.0,1.0), "blue" : (0.0,0.0,1.0,1.0), + "maroon" : (0.5,0.0,0.0,1.0), "green" : (0.0,0.5,0.0,1.0), "navy" : (0.0,0.0,0.5,1.0), + "yellow" : (1.0,1.0,0.0,1.0), "magenta" : (1.0,0.0,1.0,1.0), "cyan" : (0.0,1.0,1.0,1.0), + "orange" : (1.0,0.5,0.0,1.0), "white" : (1.0,1.0,1.0,1.0), "black" : (0.0,0.0,0.0,1.0), + "gray" : (0.5,0.5,0.5,1.0), "light_gray" : (0.9,0.9,0.9,1.0), + "transparent" : (0.0,0.0,0.0,0.0)} + +THEMES = {"black_red" : [(0.0,0.0,0.0,1.0), (1.0,0.0,0.0,1.0)], + "red_green_blue" : [(1.0,0.0,0.0,1.0), (0.0,1.0,0.0,1.0), (0.0,0.0,1.0,1.0)], + "red_orange_yellow" : [(1.0,0.2,0.0,1.0), (1.0,0.7,0.0,1.0), (1.0,1.0,0.0,1.0)], + "yellow_orange_red" : [(1.0,1.0,0.0,1.0), (1.0,0.7,0.0,1.0), (1.0,0.2,0.0,1.0)], + "rainbow" : [(1.0,0.0,0.0,1.0), (1.0,0.5,0.0,1.0), (1.0,1.0,0.0,1.0), (0.0,1.0,0.0,1.0), (0.0,0.0,1.0,1.0), (0.3, 0.0, 0.5,1.0), (0.5, 0.0, 1.0, 1.0)]} + +def colors_from_theme( theme, series_length, mode = 'solid' ): + colors = [] + if theme not in THEMES.keys() : + raise Exception, "Theme not defined" + color_steps = THEMES[theme] + n_colors = len(color_steps) + if series_length <= n_colors: + colors = [color + tuple([mode]) for color in color_steps[0:n_colors]] + else: + iterations = [(series_length - n_colors)/(n_colors - 1) for i in color_steps[:-1]] + over_iterations = (series_length - n_colors) % (n_colors - 1) + for i in range(n_colors - 1): + if over_iterations <= 0: + break + iterations[i] += 1 + over_iterations -= 1 + for index,color in enumerate(color_steps[:-1]): + colors.append(color + tuple([mode])) + if iterations[index] == 0: + continue + next_color = color_steps[index+1] + color_step = ((next_color[0] - color[0])/(iterations[index] + 1), + (next_color[1] - color[1])/(iterations[index] + 1), + (next_color[2] - color[2])/(iterations[index] + 1), + (next_color[3] - color[3])/(iterations[index] + 1)) + for i in range( iterations[index] ): + colors.append((color[0] + color_step[0]*(i+1), + color[1] + color_step[1]*(i+1), + color[2] + color_step[2]*(i+1), + color[3] + color_step[3]*(i+1), + mode)) + colors.append(color_steps[-1] + tuple([mode])) + return colors + + +def other_direction(direction): + "explicit is better than implicit" + if direction == HORZ: + return VERT + else: + return HORZ + +#Class definition + +class Plot(object): + def __init__(self, + surface=None, + data=None, + width=640, + height=480, + background=None, + border = 0, + x_labels = None, + y_labels = None, + series_colors = None): + random.seed(2) + self.create_surface(surface, width, height) + self.dimensions = {} + self.dimensions[HORZ] = width + self.dimensions[VERT] = height + self.context = cairo.Context(self.surface) + self.labels={} + self.labels[HORZ] = x_labels + self.labels[VERT] = y_labels + self.load_series(data, x_labels, y_labels, series_colors) + self.font_size = 10 + self.set_background (background) + self.border = border + self.borders = {} + self.line_color = (0.5, 0.5, 0.5) + self.line_width = 0.5 + self.label_color = (0.0, 0.0, 0.0) + self.grid_color = (0.8, 0.8, 0.8) + + def create_surface(self, surface, width=None, height=None): + self.filename = None + if isinstance(surface, cairo.Surface): + self.surface = surface + return + if not type(surface) in (str, unicode): + raise TypeError("Surface should be either a Cairo surface or a filename, not %s" % surface) + sufix = surface.rsplit(".")[-1].lower() + self.filename = surface + if sufix == "png": + self.surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, width, height) + elif sufix == "ps": + self.surface = cairo.PSSurface(surface, width, height) + elif sufix == "pdf": + self.surface = cairo.PSSurface(surface, width, height) + else: + if sufix != "svg": + self.filename += ".svg" + self.surface = cairo.SVGSurface(self.filename, width, height) + + def commit(self): + try: + self.context.show_page() + if self.filename and self.filename.endswith(".png"): + self.surface.write_to_png(self.filename) + else: + self.surface.finish() + except cairo.Error: + pass + + def load_series (self, data, x_labels=None, y_labels=None, series_colors=None): + self.series_labels = [] + self.series = None + + #The pretty way + #if not isinstance(data, Series): + # # Not an instance of Series + # self.series = Series(data) + #else: + # self.series = data + # + #self.series_labels = self.series.get_names() + + #TODO: Remove on next version + # The ugly way, keeping retrocompatibility... + if callable(data) or type(data) is list and callable(data[0]): # Lambda or List of lambdas + self.series = data + self.series_labels = None + elif isinstance(data, Series): # Instance of Series + self.series = data + self.series_labels = data.get_names() + else: # Anything else + self.series = Series(data) + self.series_labels = self.series.get_names() + + #TODO: allow user passed series_widths + self.series_widths = [1.0 for group in self.series] + + #TODO: Remove on next version + self.process_colors( series_colors ) + + def process_colors( self, series_colors, length = None, mode = 'solid' ): + #series_colors might be None, a theme, a string of colors names or a list of color tuples + if length is None : + length = len( self.series.to_list() ) + + #no colors passed + if not series_colors: + #Randomize colors + self.series_colors = [ [random.random() for i in range(3)] + [1.0, mode] for series in range( length ) ] + else: + #Just theme pattern + if not hasattr( series_colors, "__iter__" ): + theme = series_colors + self.series_colors = colors_from_theme( theme.lower(), length ) + + #Theme pattern and mode + elif not hasattr(series_colors, '__delitem__') and not hasattr( series_colors[0], "__iter__" ): + theme = series_colors[0] + mode = series_colors[1] + self.series_colors = colors_from_theme( theme.lower(), length, mode ) + + #List + else: + self.series_colors = series_colors + for index, color in enumerate( self.series_colors ): + #element is a color name + if not hasattr(color, "__iter__"): + self.series_colors[index] = COLORS[color.lower()] + tuple([mode]) + #element is rgb tuple instead of rgba + elif len( color ) == 3 : + self.series_colors[index] += (1.0,mode) + #element has 4 elements, might be rgba tuple or rgb tuple with mode + elif len( color ) == 4 : + #last element is mode + if not hasattr(color[3], "__iter__"): + self.series_colors[index] += tuple([color[3]]) + self.series_colors[index][3] = 1.0 + #last element is alpha + else: + self.series_colors[index] += tuple([mode]) + + def get_width(self): + return self.surface.get_width() + + def get_height(self): + return self.surface.get_height() + + def set_background(self, background): + if background is None: + self.background = (0.0,0.0,0.0,0.0) + elif type(background) in (cairo.LinearGradient, tuple): + self.background = background + elif not hasattr(background,"__iter__"): + colors = background.split(" ") + if len(colors) == 1 and colors[0] in COLORS: + self.background = COLORS[background] + elif len(colors) > 1: + self.background = cairo.LinearGradient(self.dimensions[HORZ] / 2, 0, self.dimensions[HORZ] / 2, self.dimensions[VERT]) + for index,color in enumerate(colors): + self.background.add_color_stop_rgba(float(index)/(len(colors)-1),*COLORS[color]) + else: + raise TypeError ("Background should be either cairo.LinearGradient or a 3/4-tuple, not %s" % type(background)) + + def render_background(self): + if isinstance(self.background, cairo.LinearGradient): + self.context.set_source(self.background) + else: + self.context.set_source_rgba(*self.background) + self.context.rectangle(0,0, self.dimensions[HORZ], self.dimensions[VERT]) + self.context.fill() + + def render_bounding_box(self): + self.context.set_source_rgba(*self.line_color) + self.context.set_line_width(self.line_width) + self.context.rectangle(self.border, self.border, + self.dimensions[HORZ] - 2 * self.border, + self.dimensions[VERT] - 2 * self.border) + self.context.stroke() + + def render(self): + pass + +class ScatterPlot( Plot ): + def __init__(self, + surface=None, + data=None, + errorx=None, + errory=None, + width=640, + height=480, + background=None, + border=0, + axis = False, + dash = False, + discrete = False, + dots = 0, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + z_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + circle_colors = None ): + + self.bounds = {} + self.bounds[HORZ] = x_bounds + self.bounds[VERT] = y_bounds + self.bounds[NORM] = z_bounds + self.titles = {} + self.titles[HORZ] = x_title + self.titles[VERT] = y_title + self.max_value = {} + self.axis = axis + self.discrete = discrete + self.dots = dots + self.grid = grid + self.series_legend = series_legend + self.variable_radius = False + self.x_label_angle = math.pi / 2.5 + self.circle_colors = circle_colors + + Plot.__init__(self, surface, data, width, height, background, border, x_labels, y_labels, series_colors) + + self.dash = None + if dash: + if hasattr(dash, "keys"): + self.dash = [dash[key] for key in self.series_labels] + elif max([hasattr(item,'__delitem__') for item in data]) : + self.dash = dash + else: + self.dash = [dash] + + self.load_errors(errorx, errory) + + def convert_list_to_tuple(self, data): + #Data must be converted from lists of coordinates to a single + # list of tuples + out_data = zip(*data) + if len(data) == 3: + self.variable_radius = True + return out_data + + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): + #TODO: In cairoplot 2.0 keep only the Series instances + + # Convert Data and Group to Series + if isinstance(data, Data) or isinstance(data, Group): + data = Series(data) + + # Series + if isinstance(data, Series): + for group in data: + for item in group: + if len(item) is 3: + self.variable_radius = True + + #Dictionary with lists + if hasattr(data, "keys") : + if hasattr( data.values()[0][0], "__delitem__" ) : + for key in data.keys() : + data[key] = self.convert_list_to_tuple(data[key]) + elif len(data.values()[0][0]) == 3: + self.variable_radius = True + #List + elif hasattr(data[0], "__delitem__") : + #List of lists + if hasattr(data[0][0], "__delitem__") : + for index,value in enumerate(data) : + data[index] = self.convert_list_to_tuple(value) + #List + elif type(data[0][0]) != type((0,0)): + data = self.convert_list_to_tuple(data) + #Three dimensional data + elif len(data[0][0]) == 3: + self.variable_radius = True + + #List with three dimensional tuples + elif len(data[0]) == 3: + self.variable_radius = True + Plot.load_series(self, data, x_labels, y_labels, series_colors) + self.calc_boundaries() + self.calc_labels() + + def load_errors(self, errorx, errory): + self.errors = None + if errorx == None and errory == None: + return + self.errors = {} + self.errors[HORZ] = None + self.errors[VERT] = None + #asimetric errors + if errorx and hasattr(errorx[0], "__delitem__"): + self.errors[HORZ] = errorx + #simetric errors + elif errorx: + self.errors[HORZ] = [errorx] + #asimetric errors + if errory and hasattr(errory[0], "__delitem__"): + self.errors[VERT] = errory + #simetric errors + elif errory: + self.errors[VERT] = [errory] + + def calc_labels(self): + if not self.labels[HORZ]: + amplitude = self.bounds[HORZ][1] - self.bounds[HORZ][0] + if amplitude % 10: #if horizontal labels need floating points + self.labels[HORZ] = ["%.2lf" % (float(self.bounds[HORZ][0] + (amplitude * i / 10.0))) for i in range(11) ] + else: + self.labels[HORZ] = ["%d" % (int(self.bounds[HORZ][0] + (amplitude * i / 10.0))) for i in range(11) ] + if not self.labels[VERT]: + amplitude = self.bounds[VERT][1] - self.bounds[VERT][0] + if amplitude % 10: #if vertical labels need floating points + self.labels[VERT] = ["%.2lf" % (float(self.bounds[VERT][0] + (amplitude * i / 10.0))) for i in range(11) ] + else: + self.labels[VERT] = ["%d" % (int(self.bounds[VERT][0] + (amplitude * i / 10.0))) for i in range(11) ] + + def calc_extents(self, direction): + self.context.set_font_size(self.font_size * 0.8) + self.max_value[direction] = max(self.context.text_extents(item)[2] for item in self.labels[direction]) + self.borders[other_direction(direction)] = self.max_value[direction] + self.border + 20 + + def calc_boundaries(self): + #HORZ = 0, VERT = 1, NORM = 2 + min_data_value = [0,0,0] + max_data_value = [0,0,0] + + for group in self.series: + if type(group[0].content) in (int, float, long): + group = [Data((index, item.content)) for index,item in enumerate(group)] + + for point in group: + for index, item in enumerate(point.content): + if item > max_data_value[index]: + max_data_value[index] = item + elif item < min_data_value[index]: + min_data_value[index] = item + + if not self.bounds[HORZ]: + self.bounds[HORZ] = (min_data_value[HORZ], max_data_value[HORZ]) + if not self.bounds[VERT]: + self.bounds[VERT] = (min_data_value[VERT], max_data_value[VERT]) + if not self.bounds[NORM]: + self.bounds[NORM] = (min_data_value[NORM], max_data_value[NORM]) + + def calc_all_extents(self): + self.calc_extents(HORZ) + self.calc_extents(VERT) + + self.plot_height = self.dimensions[VERT] - 2 * self.borders[VERT] + self.plot_width = self.dimensions[HORZ] - 2* self.borders[HORZ] + + self.plot_top = self.dimensions[VERT] - self.borders[VERT] + + def calc_steps(self): + #Calculates all the x, y, z and color steps + series_amplitude = [self.bounds[index][1] - self.bounds[index][0] for index in range(3)] + + if series_amplitude[HORZ]: + self.horizontal_step = float (self.plot_width) / series_amplitude[HORZ] + else: + self.horizontal_step = 0.00 + + if series_amplitude[VERT]: + self.vertical_step = float (self.plot_height) / series_amplitude[VERT] + else: + self.vertical_step = 0.00 + + if series_amplitude[NORM]: + if self.variable_radius: + self.z_step = float (self.bounds[NORM][1]) / series_amplitude[NORM] + if self.circle_colors: + self.circle_color_step = tuple([float(self.circle_colors[1][i]-self.circle_colors[0][i])/series_amplitude[NORM] for i in range(4)]) + else: + self.z_step = 0.00 + self.circle_color_step = ( 0.0, 0.0, 0.0, 0.0 ) + + def get_circle_color(self, value): + return tuple( [self.circle_colors[0][i] + value*self.circle_color_step[i] for i in range(4)] ) + + def render(self): + self.calc_all_extents() + self.calc_steps() + self.render_background() + self.render_bounding_box() + if self.axis: + self.render_axis() + if self.grid: + self.render_grid() + self.render_labels() + self.render_plot() + if self.errors: + self.render_errors() + if self.series_legend and self.series_labels: + self.render_legend() + + def render_axis(self): + #Draws both the axis lines and their titles + cr = self.context + cr.set_source_rgba(*self.line_color) + cr.move_to(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) + cr.line_to(self.borders[HORZ], self.borders[VERT]) + cr.stroke() + + cr.move_to(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) + cr.line_to(self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) + cr.stroke() + + cr.set_source_rgba(*self.label_color) + self.context.set_font_size( 1.2 * self.font_size ) + if self.titles[HORZ]: + title_width,title_height = cr.text_extents(self.titles[HORZ])[2:4] + cr.move_to( self.dimensions[HORZ]/2 - title_width/2, self.borders[VERT] - title_height/2 ) + cr.show_text( self.titles[HORZ] ) + + if self.titles[VERT]: + title_width,title_height = cr.text_extents(self.titles[VERT])[2:4] + cr.move_to( self.dimensions[HORZ] - self.borders[HORZ] + title_height/2, self.dimensions[VERT]/2 - title_width/2) + cr.save() + cr.rotate( math.pi/2 ) + cr.show_text( self.titles[VERT] ) + cr.restore() + + def render_grid(self): + cr = self.context + horizontal_step = float( self.plot_height ) / ( len( self.labels[VERT] ) - 1 ) + vertical_step = float( self.plot_width ) / ( len( self.labels[HORZ] ) - 1 ) + + x = self.borders[HORZ] + vertical_step + y = self.plot_top - horizontal_step + + for label in self.labels[HORZ][:-1]: + cr.set_source_rgba(*self.grid_color) + cr.move_to(x, self.dimensions[VERT] - self.borders[VERT]) + cr.line_to(x, self.borders[VERT]) + cr.stroke() + x += vertical_step + for label in self.labels[VERT][:-1]: + cr.set_source_rgba(*self.grid_color) + cr.move_to(self.borders[HORZ], y) + cr.line_to(self.dimensions[HORZ] - self.borders[HORZ], y) + cr.stroke() + y -= horizontal_step + + def render_labels(self): + self.context.set_font_size(self.font_size * 0.8) + self.render_horz_labels() + self.render_vert_labels() + + def render_horz_labels(self): + cr = self.context + step = float( self.plot_width ) / ( len( self.labels[HORZ] ) - 1 ) + x = self.borders[HORZ] + y = self.dimensions[VERT] - self.borders[VERT] + 5 + + # store rotation matrix from the initial state + rotation_matrix = cr.get_matrix() + rotation_matrix.rotate(self.x_label_angle) + + cr.set_source_rgba(*self.label_color) + + for item in self.labels[HORZ]: + width = cr.text_extents(item)[2] + cr.move_to(x, y) + cr.save() + cr.set_matrix(rotation_matrix) + cr.show_text(item) + cr.restore() + x += step + + def render_vert_labels(self): + cr = self.context + step = ( self.plot_height ) / ( len( self.labels[VERT] ) - 1 ) + y = self.plot_top + cr.set_source_rgba(*self.label_color) + for item in self.labels[VERT]: + width = cr.text_extents(item)[2] + cr.move_to(self.borders[HORZ] - width - 5,y) + cr.show_text(item) + y -= step + + def render_legend(self): + cr = self.context + cr.set_font_size(self.font_size) + cr.set_line_width(self.line_width) + + widest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[2]) + tallest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[3]) + max_width = self.context.text_extents(widest_word)[2] + max_height = self.context.text_extents(tallest_word)[3] * 1.1 + + color_box_height = max_height / 2 + color_box_width = color_box_height * 2 + + #Draw a bounding box + bounding_box_width = max_width + color_box_width + 15 + bounding_box_height = (len(self.series_labels)+0.5) * max_height + cr.set_source_rgba(1,1,1) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - bounding_box_width, self.borders[VERT], + bounding_box_width, bounding_box_height) + cr.fill() + + cr.set_source_rgba(*self.line_color) + cr.set_line_width(self.line_width) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - bounding_box_width, self.borders[VERT], + bounding_box_width, bounding_box_height) + cr.stroke() + + for idx,key in enumerate(self.series_labels): + #Draw color box + cr.set_source_rgba(*self.series_colors[idx][:4]) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - max_width - color_box_width - 10, + self.borders[VERT] + color_box_height + (idx*max_height) , + color_box_width, color_box_height) + cr.fill() + + cr.set_source_rgba(0, 0, 0) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - max_width - color_box_width - 10, + self.borders[VERT] + color_box_height + (idx*max_height), + color_box_width, color_box_height) + cr.stroke() + + #Draw series labels + cr.set_source_rgba(0, 0, 0) + cr.move_to(self.dimensions[HORZ] - self.borders[HORZ] - max_width - 5, self.borders[VERT] + ((idx+1)*max_height)) + cr.show_text(key) + + def render_errors(self): + cr = self.context + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) + cr.clip() + radius = self.dots + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + for index, group in enumerate(self.series): + cr.set_source_rgba(*self.series_colors[index][:4]) + for number, data in enumerate(group): + x = x0 + self.horizontal_step * data.content[0] + y = self.dimensions[VERT] - y0 - self.vertical_step * data.content[1] + if self.errors[HORZ]: + cr.move_to(x, y) + x1 = x - self.horizontal_step * self.errors[HORZ][0][number] + cr.line_to(x1, y) + cr.line_to(x1, y - radius) + cr.line_to(x1, y + radius) + cr.stroke() + if self.errors[HORZ] and len(self.errors[HORZ]) == 2: + cr.move_to(x, y) + x1 = x + self.horizontal_step * self.errors[HORZ][1][number] + cr.line_to(x1, y) + cr.line_to(x1, y - radius) + cr.line_to(x1, y + radius) + cr.stroke() + if self.errors[VERT]: + cr.move_to(x, y) + y1 = y + self.vertical_step * self.errors[VERT][0][number] + cr.line_to(x, y1) + cr.line_to(x - radius, y1) + cr.line_to(x + radius, y1) + cr.stroke() + if self.errors[VERT] and len(self.errors[VERT]) == 2: + cr.move_to(x, y) + y1 = y - self.vertical_step * self.errors[VERT][1][number] + cr.line_to(x, y1) + cr.line_to(x - radius, y1) + cr.line_to(x + radius, y1) + cr.stroke() + + + def render_plot(self): + cr = self.context + if self.discrete: + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) + cr.clip() + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + radius = self.dots + for number, group in enumerate (self.series): + cr.set_source_rgba(*self.series_colors[number][:4]) + for data in group : + if self.variable_radius: + radius = data.content[2]*self.z_step + if self.circle_colors: + cr.set_source_rgba( *self.get_circle_color( data.content[2]) ) + x = x0 + self.horizontal_step*data.content[0] + y = y0 + self.vertical_step*data.content[1] + cr.arc(x, self.dimensions[VERT] - y, radius, 0, 2*math.pi) + cr.fill() + else: + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) + cr.clip() + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + radius = self.dots + for number, group in enumerate (self.series): + last_data = None + cr.set_source_rgba(*self.series_colors[number][:4]) + for data in group : + x = x0 + self.horizontal_step*data.content[0] + y = y0 + self.vertical_step*data.content[1] + if self.dots: + if self.variable_radius: + radius = data.content[2]*self.z_step + cr.arc(x, self.dimensions[VERT] - y, radius, 0, 2*math.pi) + cr.fill() + if last_data : + old_x = x0 + self.horizontal_step*last_data.content[0] + old_y = y0 + self.vertical_step*last_data.content[1] + cr.move_to( old_x, self.dimensions[VERT] - old_y ) + cr.line_to( x, self.dimensions[VERT] - y) + cr.set_line_width(self.series_widths[number]) + + #?Display line as dash line + if self.dash and self.dash[number]: + s = self.series_widths[number] + cr.set_dash([s*3, s*3], 0) + + cr.stroke() + cr.set_dash([]) + last_data = data + +class DotLinePlot(ScatterPlot): + def __init__(self, + surface=None, + data=None, + width=640, + height=480, + background=None, + border=0, + axis = False, + dash = False, + dots = 0, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None): + + ScatterPlot.__init__(self, surface, data, None, None, width, height, background, border, + axis, dash, False, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, None, x_title, y_title, series_colors, None ) + + + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + for group in self.series : + for index,data in enumerate(group): + group[index].content = (index, data.content) + + self.calc_boundaries() + self.calc_labels() + +class FunctionPlot(ScatterPlot): + def __init__(self, + surface=None, + data=None, + width=640, + height=480, + background=None, + border=0, + axis = False, + discrete = False, + dots = 0, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + step = 1): + + self.function = data + self.step = step + self.discrete = discrete + + data, x_bounds = self.load_series_from_function( self.function, x_bounds ) + + ScatterPlot.__init__(self, surface, data, None, None, width, height, background, border, + axis, False, discrete, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, None, x_title, y_title, series_colors, None ) + + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + + if len(self.series[0][0]) is 1: + for group_id, group in enumerate(self.series) : + for index,data in enumerate(group): + group[index].content = (self.bounds[HORZ][0] + self.step*index, data.content) + + self.calc_boundaries() + self.calc_labels() + + def load_series_from_function( self, function, x_bounds ): + #TODO: Add the possibility for the user to define multiple functions with different discretization parameters + + #This function converts a function, a list of functions or a dictionary + #of functions into its corresponding array of data + series = Series() + + if isinstance(function, Group) or isinstance(function, Data): + function = Series(function) + + # If is instance of Series + if isinstance(function, Series): + # Overwrite any bounds passed by the function + x_bounds = (function.range[0],function.range[-1]) + + #if no bounds are provided + if x_bounds == None: + x_bounds = (0,10) + + + #TODO: Finish the dict translation + if hasattr(function, "keys"): #dictionary: + for key in function.keys(): + group = Group(name=key) + #data[ key ] = [] + i = x_bounds[0] + while i <= x_bounds[1] : + group.add_data(function[ key ](i)) + #data[ key ].append( function[ key ](i) ) + i += self.step + series.add_group(group) + + elif hasattr(function, "__delitem__"): #list of functions + for index,f in enumerate( function ) : + group = Group() + #data.append( [] ) + i = x_bounds[0] + while i <= x_bounds[1] : + group.add_data(f(i)) + #data[ index ].append( f(i) ) + i += self.step + series.add_group(group) + + elif isinstance(function, Series): # instance of Series + series = function + + else: #function + group = Group() + i = x_bounds[0] + while i <= x_bounds[1] : + group.add_data(function(i)) + i += self.step + series.add_group(group) + + + return series, x_bounds + + def calc_labels(self): + if not self.labels[HORZ]: + self.labels[HORZ] = [] + i = self.bounds[HORZ][0] + while i<=self.bounds[HORZ][1]: + self.labels[HORZ].append(str(i)) + i += float(self.bounds[HORZ][1] - self.bounds[HORZ][0])/10 + ScatterPlot.calc_labels(self) + + def render_plot(self): + if not self.discrete: + ScatterPlot.render_plot(self) + else: + last = None + cr = self.context + for number, group in enumerate (self.series): + cr.set_source_rgba(*self.series_colors[number][:4]) + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + for data in group: + x = x0 + self.horizontal_step * data.content[0] + y = y0 + self.vertical_step * data.content[1] + cr.move_to(x, self.dimensions[VERT] - y) + cr.line_to(x, self.plot_top) + cr.set_line_width(self.series_widths[number]) + cr.stroke() + if self.dots: + cr.new_path() + cr.arc(x, self.dimensions[VERT] - y, 3, 0, 2.1 * math.pi) + cr.close_path() + cr.fill() + +class BarPlot(Plot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None, + main_dir = None): + + self.bounds = {} + self.bounds[HORZ] = x_bounds + self.bounds[VERT] = y_bounds + self.display_values = display_values + self.grid = grid + self.rounded_corners = rounded_corners + self.stack = stack + self.three_dimension = three_dimension + self.x_label_angle = math.pi / 2.5 + self.main_dir = main_dir + self.max_value = {} + self.plot_dimensions = {} + self.steps = {} + self.value_label_color = (0.5,0.5,0.5,1.0) + + Plot.__init__(self, surface, data, width, height, background, border, x_labels, y_labels, series_colors) + + def load_series(self, data, x_labels = None, y_labels = None, series_colors = None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + self.calc_boundaries() + + def process_colors(self, series_colors): + #Data for a BarPlot might be a List or a List of Lists. + #On the first case, colors must be generated for all bars, + #On the second, colors must be generated for each of the inner lists. + + #TODO: Didn't get it... + #if hasattr(self.data[0], '__getitem__'): + # length = max(len(series) for series in self.data) + #else: + # length = len( self.data ) + + length = max(len(group) for group in self.series) + + Plot.process_colors( self, series_colors, length, 'linear') + + def calc_boundaries(self): + if not self.bounds[self.main_dir]: + if self.stack: + max_data_value = max(sum(group.to_list()) for group in self.series) + else: + max_data_value = max(max(group.to_list()) for group in self.series) + self.bounds[self.main_dir] = (0, max_data_value) + if not self.bounds[other_direction(self.main_dir)]: + self.bounds[other_direction(self.main_dir)] = (0, len(self.series)) + + def calc_extents(self, direction): + self.max_value[direction] = 0 + if self.labels[direction]: + widest_word = max(self.labels[direction], key = lambda item: self.context.text_extents(item)[2]) + self.max_value[direction] = self.context.text_extents(widest_word)[3 - direction] + self.borders[other_direction(direction)] = (2-direction)*self.max_value[direction] + self.border + direction*(5) + else: + self.borders[other_direction(direction)] = self.border + + def calc_horz_extents(self): + self.calc_extents(HORZ) + + def calc_vert_extents(self): + self.calc_extents(VERT) + + def calc_all_extents(self): + self.calc_horz_extents() + self.calc_vert_extents() + other_dir = other_direction(self.main_dir) + self.value_label = 0 + if self.display_values: + if self.stack: + self.value_label = self.context.text_extents(str(max(sum(group.to_list()) for group in self.series)))[2 + self.main_dir] + else: + self.value_label = self.context.text_extents(str(max(max(group.to_list()) for group in self.series)))[2 + self.main_dir] + if self.labels[self.main_dir]: + self.plot_dimensions[self.main_dir] = self.dimensions[self.main_dir] - 2*self.borders[self.main_dir] - self.value_label + else: + self.plot_dimensions[self.main_dir] = self.dimensions[self.main_dir] - self.borders[self.main_dir] - 1.2*self.border - self.value_label + self.plot_dimensions[other_dir] = self.dimensions[other_dir] - self.borders[other_dir] - self.border + self.plot_top = self.dimensions[VERT] - self.borders[VERT] + + def calc_steps(self): + other_dir = other_direction(self.main_dir) + self.series_amplitude = self.bounds[self.main_dir][1] - self.bounds[self.main_dir][0] + if self.series_amplitude: + self.steps[self.main_dir] = float(self.plot_dimensions[self.main_dir])/self.series_amplitude + else: + self.steps[self.main_dir] = 0.00 + series_length = len(self.series) + self.steps[other_dir] = float(self.plot_dimensions[other_dir])/(series_length + 0.1*(series_length + 1)) + self.space = 0.1*self.steps[other_dir] + + def render(self): + self.calc_all_extents() + self.calc_steps() + self.render_background() + self.render_bounding_box() + if self.grid: + self.render_grid() + if self.three_dimension: + self.render_ground() + if self.display_values: + self.render_values() + self.render_labels() + self.render_plot() + if self.series_labels: + self.render_legend() + + def draw_3d_rectangle_front(self, x0, y0, x1, y1, shift): + self.context.rectangle(x0-shift, y0+shift, x1-x0, y1-y0) + + def draw_3d_rectangle_side(self, x0, y0, x1, y1, shift): + self.context.move_to(x1-shift,y0+shift) + self.context.line_to(x1, y0) + self.context.line_to(x1, y1) + self.context.line_to(x1-shift, y1+shift) + self.context.line_to(x1-shift, y0+shift) + self.context.close_path() + + def draw_3d_rectangle_top(self, x0, y0, x1, y1, shift): + self.context.move_to(x0-shift,y0+shift) + self.context.line_to(x0, y0) + self.context.line_to(x1, y0) + self.context.line_to(x1-shift, y0+shift) + self.context.line_to(x0-shift, y0+shift) + self.context.close_path() + + def draw_round_rectangle(self, x0, y0, x1, y1): + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) + self.context.line_to(x1-5, y0) + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) + self.context.line_to(x1, y1-5) + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) + self.context.line_to(x0+5, y1) + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) + self.context.line_to(x0, y0+5) + self.context.close_path() + + def render_ground(self): + self.draw_3d_rectangle_front(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_side (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_top (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + def render_labels(self): + self.context.set_font_size(self.font_size * 0.8) + if self.labels[HORZ]: + self.render_horz_labels() + if self.labels[VERT]: + self.render_vert_labels() + + def render_legend(self): + cr = self.context + cr.set_font_size(self.font_size) + cr.set_line_width(self.line_width) + + widest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[2]) + tallest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[3]) + max_width = self.context.text_extents(widest_word)[2] + max_height = self.context.text_extents(tallest_word)[3] * 1.1 + 5 + + color_box_height = max_height / 2 + color_box_width = color_box_height * 2 + + #Draw a bounding box + bounding_box_width = max_width + color_box_width + 15 + bounding_box_height = (len(self.series_labels)+0.5) * max_height + cr.set_source_rgba(1,1,1) + cr.rectangle(self.dimensions[HORZ] - self.border - bounding_box_width, self.border, + bounding_box_width, bounding_box_height) + cr.fill() + + cr.set_source_rgba(*self.line_color) + cr.set_line_width(self.line_width) + cr.rectangle(self.dimensions[HORZ] - self.border - bounding_box_width, self.border, + bounding_box_width, bounding_box_height) + cr.stroke() + + for idx,key in enumerate(self.series_labels): + #Draw color box + cr.set_source_rgba(*self.series_colors[idx][:4]) + cr.rectangle(self.dimensions[HORZ] - self.border - max_width - color_box_width - 10, + self.border + color_box_height + (idx*max_height) , + color_box_width, color_box_height) + cr.fill() + + cr.set_source_rgba(0, 0, 0) + cr.rectangle(self.dimensions[HORZ] - self.border - max_width - color_box_width - 10, + self.border + color_box_height + (idx*max_height), + color_box_width, color_box_height) + cr.stroke() + + #Draw series labels + cr.set_source_rgba(0, 0, 0) + cr.move_to(self.dimensions[HORZ] - self.border - max_width - 5, self.border + ((idx+1)*max_height)) + cr.show_text(key) + + +class HorizontalBarPlot(BarPlot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None): + + BarPlot.__init__(self, surface, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, series_colors, HORZ) + self.series_labels = series_labels + + def calc_vert_extents(self): + self.calc_extents(VERT) + if self.labels[HORZ] and not self.labels[VERT]: + self.borders[HORZ] += 10 + + def draw_rectangle_bottom(self, x0, y0, x1, y1): + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) + self.context.line_to(x0, y0+5) + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) + self.context.line_to(x1, y0) + self.context.line_to(x1, y1) + self.context.line_to(x0+5, y1) + self.context.close_path() + + def draw_rectangle_top(self, x0, y0, x1, y1): + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) + self.context.line_to(x1, y1-5) + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) + self.context.line_to(x0, y1) + self.context.line_to(x0, y0) + self.context.line_to(x1, y0) + self.context.close_path() + + def draw_rectangle(self, index, length, x0, y0, x1, y1): + if length == 1: + BarPlot.draw_rectangle(self, x0, y0, x1, y1) + elif index == 0: + self.draw_rectangle_bottom(x0, y0, x1, y1) + elif index == length-1: + self.draw_rectangle_top(x0, y0, x1, y1) + else: + self.context.rectangle(x0, y0, x1-x0, y1-y0) + + #TODO: Review BarPlot.render_grid code + def render_grid(self): + self.context.set_source_rgba(0.8, 0.8, 0.8) + if self.labels[HORZ]: + self.context.set_font_size(self.font_size * 0.8) + step = (self.dimensions[HORZ] - 2*self.borders[HORZ] - self.value_label)/(len(self.labels[HORZ])-1) + x = self.borders[HORZ] + next_x = 0 + for item in self.labels[HORZ]: + width = self.context.text_extents(item)[2] + if x - width/2 > next_x and x - width/2 > self.border: + self.context.move_to(x, self.border) + self.context.line_to(x, self.dimensions[VERT] - self.borders[VERT]) + self.context.stroke() + next_x = x + width/2 + x += step + else: + lines = 11 + horizontal_step = float(self.plot_dimensions[HORZ])/(lines-1) + x = self.borders[HORZ] + for y in xrange(0, lines): + self.context.move_to(x, self.border) + self.context.line_to(x, self.dimensions[VERT] - self.borders[VERT]) + self.context.stroke() + x += horizontal_step + + def render_horz_labels(self): + step = (self.dimensions[HORZ] - 2*self.borders[HORZ])/(len(self.labels[HORZ])-1) + x = self.borders[HORZ] + next_x = 0 + + for item in self.labels[HORZ]: + self.context.set_source_rgba(*self.label_color) + width = self.context.text_extents(item)[2] + if x - width/2 > next_x and x - width/2 > self.border: + self.context.move_to(x - width/2, self.dimensions[VERT] - self.borders[VERT] + self.max_value[HORZ] + 3) + self.context.show_text(item) + next_x = x + width/2 + x += step + + def render_vert_labels(self): + series_length = len(self.labels[VERT]) + step = (self.plot_dimensions[VERT] - (series_length + 1)*self.space)/(len(self.labels[VERT])) + y = self.border + step/2 + self.space + + for item in self.labels[VERT]: + self.context.set_source_rgba(*self.label_color) + width, height = self.context.text_extents(item)[2:4] + self.context.move_to(self.borders[HORZ] - width - 5, y + height/2) + self.context.show_text(item) + y += step + self.space + self.labels[VERT].reverse() + + def render_values(self): + self.context.set_source_rgba(*self.value_label_color) + self.context.set_font_size(self.font_size * 0.8) + if self.stack: + for i,group in enumerate(self.series): + value = sum(group.to_list()) + height = self.context.text_extents(str(value))[3] + x = self.borders[HORZ] + value*self.steps[HORZ] + 2 + y = self.borders[VERT] + (i+0.5)*self.steps[VERT] + (i+1)*self.space + height/2 + self.context.move_to(x, y) + self.context.show_text(str(value)) + else: + for i,group in enumerate(self.series): + inner_step = self.steps[VERT]/len(group) + y0 = self.border + i*self.steps[VERT] + (i+1)*self.space + for number,data in enumerate(group): + height = self.context.text_extents(str(data.content))[3] + self.context.move_to(self.borders[HORZ] + data.content*self.steps[HORZ] + 2, y0 + 0.5*inner_step + height/2, ) + self.context.show_text(str(data.content)) + y0 += inner_step + + def render_plot(self): + if self.stack: + for i,group in enumerate(self.series): + x0 = self.borders[HORZ] + y0 = self.borders[VERT] + i*self.steps[VERT] + (i+1)*self.space + for number,data in enumerate(group): + if self.series_colors[number][4] in ('radial','linear') : + linear = cairo.LinearGradient( data.content*self.steps[HORZ]/2, y0, data.content*self.steps[HORZ]/2, y0 + self.steps[VERT] ) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + elif self.series_colors[number][4] == 'solid': + self.context.set_source_rgba(*self.series_colors[number][:4]) + if self.rounded_corners: + self.draw_rectangle(number, len(group), x0, y0, x0+data.content*self.steps[HORZ], y0+self.steps[VERT]) + self.context.fill() + else: + self.context.rectangle(x0, y0, data.content*self.steps[HORZ], self.steps[VERT]) + self.context.fill() + x0 += data.content*self.steps[HORZ] + else: + for i,group in enumerate(self.series): + inner_step = self.steps[VERT]/len(group) + x0 = self.borders[HORZ] + y0 = self.border + i*self.steps[VERT] + (i+1)*self.space + for number,data in enumerate(group): + linear = cairo.LinearGradient(data.content*self.steps[HORZ]/2, y0, data.content*self.steps[HORZ]/2, y0 + inner_step) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + if self.rounded_corners and data.content != 0: + BarPlot.draw_round_rectangle(self,x0, y0, x0 + data.content*self.steps[HORZ], y0 + inner_step) + self.context.fill() + else: + self.context.rectangle(x0, y0, data.content*self.steps[HORZ], inner_step) + self.context.fill() + y0 += inner_step + +class VerticalBarPlot(BarPlot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None): + + BarPlot.__init__(self, surface, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, series_colors, VERT) + self.series_labels = series_labels + + def calc_vert_extents(self): + self.calc_extents(VERT) + if self.labels[VERT] and not self.labels[HORZ]: + self.borders[VERT] += 10 + + def draw_rectangle_bottom(self, x0, y0, x1, y1): + self.context.move_to(x1,y1) + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) + self.context.line_to(x0+5, y1) + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) + self.context.line_to(x0, y0) + self.context.line_to(x1, y0) + self.context.line_to(x1, y1) + self.context.close_path() + + def draw_rectangle_top(self, x0, y0, x1, y1): + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) + self.context.line_to(x1-5, y0) + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) + self.context.line_to(x1, y1) + self.context.line_to(x0, y1) + self.context.line_to(x0, y0) + self.context.close_path() + + def draw_rectangle(self, index, length, x0, y0, x1, y1): + if length == 1: + BarPlot.draw_rectangle(self, x0, y0, x1, y1) + elif index == 0: + self.draw_rectangle_bottom(x0, y0, x1, y1) + elif index == length-1: + self.draw_rectangle_top(x0, y0, x1, y1) + else: + self.context.rectangle(x0, y0, x1-x0, y1-y0) + + def render_grid(self): + self.context.set_source_rgba(0.8, 0.8, 0.8) + if self.labels[VERT]: + lines = len(self.labels[VERT]) + vertical_step = float(self.plot_dimensions[self.main_dir])/(lines-1) + y = self.borders[VERT] + self.value_label + else: + lines = 11 + vertical_step = float(self.plot_dimensions[self.main_dir])/(lines-1) + y = 1.2*self.border + self.value_label + for x in xrange(0, lines): + self.context.move_to(self.borders[HORZ], y) + self.context.line_to(self.dimensions[HORZ] - self.border, y) + self.context.stroke() + y += vertical_step + + def render_ground(self): + self.draw_3d_rectangle_front(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_side (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_top (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + def render_horz_labels(self): + series_length = len(self.labels[HORZ]) + step = float (self.plot_dimensions[HORZ] - (series_length + 1)*self.space)/len(self.labels[HORZ]) + x = self.borders[HORZ] + step/2 + self.space + next_x = 0 + + for item in self.labels[HORZ]: + self.context.set_source_rgba(*self.label_color) + width = self.context.text_extents(item)[2] + if x - width/2 > next_x and x - width/2 > self.borders[HORZ]: + self.context.move_to(x - width/2, self.dimensions[VERT] - self.borders[VERT] + self.max_value[HORZ] + 3) + self.context.show_text(item) + next_x = x + width/2 + x += step + self.space + + def render_vert_labels(self): + self.context.set_source_rgba(*self.label_color) + y = self.borders[VERT] + self.value_label + step = (self.dimensions[VERT] - 2*self.borders[VERT] - self.value_label)/(len(self.labels[VERT]) - 1) + self.labels[VERT].reverse() + for item in self.labels[VERT]: + width, height = self.context.text_extents(item)[2:4] + self.context.move_to(self.borders[HORZ] - width - 5, y + height/2) + self.context.show_text(item) + y += step + self.labels[VERT].reverse() + + def render_values(self): + self.context.set_source_rgba(*self.value_label_color) + self.context.set_font_size(self.font_size * 0.8) + if self.stack: + for i,group in enumerate(self.series): + value = sum(group.to_list()) + width = self.context.text_extents(str(value))[2] + x = self.borders[HORZ] + (i+0.5)*self.steps[HORZ] + (i+1)*self.space - width/2 + y = value*self.steps[VERT] + 2 + self.context.move_to(x, self.plot_top-y) + self.context.show_text(str(value)) + else: + for i,group in enumerate(self.series): + inner_step = self.steps[HORZ]/len(group) + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space + for number,data in enumerate(group): + width = self.context.text_extents(str(data.content))[2] + self.context.move_to(x0 + 0.5*inner_step - width/2, self.plot_top - data.content*self.steps[VERT] - 2) + self.context.show_text(str(data.content)) + x0 += inner_step + + def render_plot(self): + if self.stack: + for i,group in enumerate(self.series): + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space + y0 = 0 + for number,data in enumerate(group): + if self.series_colors[number][4] in ('linear','radial'): + linear = cairo.LinearGradient( x0, data.content*self.steps[VERT]/2, x0 + self.steps[HORZ], data.content*self.steps[VERT]/2 ) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + elif self.series_colors[number][4] == 'solid': + self.context.set_source_rgba(*self.series_colors[number][:4]) + if self.rounded_corners: + self.draw_rectangle(number, len(group), x0, self.plot_top - y0 - data.content*self.steps[VERT], x0 + self.steps[HORZ], self.plot_top - y0) + self.context.fill() + else: + self.context.rectangle(x0, self.plot_top - y0 - data.content*self.steps[VERT], self.steps[HORZ], data.content*self.steps[VERT]) + self.context.fill() + y0 += data.content*self.steps[VERT] + else: + for i,group in enumerate(self.series): + inner_step = self.steps[HORZ]/len(group) + y0 = self.borders[VERT] + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space + for number,data in enumerate(group): + if self.series_colors[number][4] == 'linear': + linear = cairo.LinearGradient( x0, data.content*self.steps[VERT]/2, x0 + inner_step, data.content*self.steps[VERT]/2 ) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + elif self.series_colors[number][4] == 'solid': + self.context.set_source_rgba(*self.series_colors[number][:4]) + if self.rounded_corners and data.content != 0: + BarPlot.draw_round_rectangle(self, x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top) + self.context.fill() + elif self.three_dimension: + self.draw_3d_rectangle_front(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) + self.context.fill() + self.draw_3d_rectangle_side(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) + self.context.fill() + self.draw_3d_rectangle_top(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) + self.context.fill() + else: + self.context.rectangle(x0, self.plot_top - data.content*self.steps[VERT], inner_step, data.content*self.steps[VERT]) + self.context.fill() + + x0 += inner_step + +class StreamChart(VerticalBarPlot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + grid = False, + series_legend = None, + x_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None): + + VerticalBarPlot.__init__(self, surface, data, width, height, background, border, + False, grid, False, True, False, + None, x_labels, None, x_bounds, y_bounds, series_colors) + + def calc_steps(self): + other_dir = other_direction(self.main_dir) + self.series_amplitude = self.bounds[self.main_dir][1] - self.bounds[self.main_dir][0] + if self.series_amplitude: + self.steps[self.main_dir] = float(self.plot_dimensions[self.main_dir])/self.series_amplitude + else: + self.steps[self.main_dir] = 0.00 + series_length = len(self.data) + self.steps[other_dir] = float(self.plot_dimensions[other_dir])/series_length + + def render_legend(self): + pass + + def ground(self, index): + sum_values = sum(self.data[index]) + return -0.5*sum_values + + def calc_angles(self): + middle = self.plot_top - self.plot_dimensions[VERT]/2.0 + self.angles = [tuple([0.0 for x in range(len(self.data)+1)])] + for x_index in range(1, len(self.data)-1): + t = [] + x0 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y0 = middle - self.ground(x_index-1)*self.steps[VERT] + y2 = middle - self.ground(x_index+1)*self.steps[VERT] + t.append(math.atan(float(y0-y2)/(x0-x2))) + for data_index in range(len(self.data[x_index])): + x0 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y0 = middle - self.ground(x_index-1)*self.steps[VERT] - self.data[x_index-1][data_index]*self.steps[VERT] + y2 = middle - self.ground(x_index+1)*self.steps[VERT] - self.data[x_index+1][data_index]*self.steps[VERT] + + for i in range(0,data_index): + y0 -= self.data[x_index-1][i]*self.steps[VERT] + y2 -= self.data[x_index+1][i]*self.steps[VERT] + + if data_index == len(self.data[0])-1 and False: + self.context.set_source_rgba(0.0,0.0,0.0,0.3) + self.context.move_to(x0,y0) + self.context.line_to(x2,y2) + self.context.stroke() + self.context.arc(x0,y0,2,0,2*math.pi) + self.context.fill() + t.append(math.atan(float(y0-y2)/(x0-x2))) + self.angles.append(tuple(t)) + self.angles.append(tuple([0.0 for x in range(len(self.data)+1)])) + + def render_plot(self): + self.calc_angles() + middle = self.plot_top - self.plot_dimensions[VERT]/2.0 + p = 0.4*self.steps[HORZ] + for data_index in range(len(self.data[0])-1,-1,-1): + self.context.set_source_rgba(*self.series_colors[data_index][:4]) + + #draw the upper line + for x_index in range(len(self.data)-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] - self.data[x_index][data_index]*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index + 1)*self.steps[VERT] - self.data[x_index + 1][data_index]*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index+1][i]*self.steps[VERT] + + if x_index == 0: + self.context.move_to(x1,y1) + + ang1 = self.angles[x_index][data_index+1] + ang2 = self.angles[x_index+1][data_index+1] + math.pi + self.context.curve_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1), + x2+p*math.cos(ang2),y2+p*math.sin(ang2), + x2,y2) + + for x_index in range(len(self.data)-1,0,-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index - 1)*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index-1][i]*self.steps[VERT] + + if x_index == len(self.data)-1: + self.context.line_to(x1,y1+2) + + #revert angles by pi degrees to take the turn back + ang1 = self.angles[x_index][data_index] + math.pi + ang2 = self.angles[x_index-1][data_index] + self.context.curve_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1), + x2+p*math.cos(ang2),y2+p*math.sin(ang2), + x2,y2+2) + + self.context.close_path() + self.context.fill() + + if False: + self.context.move_to(self.borders[HORZ] + 0.5*self.steps[HORZ], middle) + for x_index in range(len(self.data)-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] - self.data[x_index][data_index]*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index + 1)*self.steps[VERT] - self.data[x_index + 1][data_index]*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index+1][i]*self.steps[VERT] + + ang1 = self.angles[x_index][data_index+1] + ang2 = self.angles[x_index+1][data_index+1] + math.pi + self.context.set_source_rgba(1.0,0.0,0.0) + self.context.arc(x1+p*math.cos(ang1),y1+p*math.sin(ang1),2,0,2*math.pi) + self.context.fill() + self.context.set_source_rgba(0.0,0.0,0.0) + self.context.arc(x2+p*math.cos(ang2),y2+p*math.sin(ang2),2,0,2*math.pi) + self.context.fill() + '''self.context.set_source_rgba(0.0,0.0,0.0,0.3) + self.context.arc(x2,y2,2,0,2*math.pi) + self.context.fill()''' + self.context.move_to(x1,y1) + self.context.line_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1)) + self.context.stroke() + self.context.move_to(x2,y2) + self.context.line_to(x2+p*math.cos(ang2),y2+p*math.sin(ang2)) + self.context.stroke() + if False: + for x_index in range(len(self.data)-1,0,-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index - 1)*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index-1][i]*self.steps[VERT] + + #revert angles by pi degrees to take the turn back + ang1 = self.angles[x_index][data_index] + math.pi + ang2 = self.angles[x_index-1][data_index] + self.context.set_source_rgba(0.0,1.0,0.0) + self.context.arc(x1+p*math.cos(ang1),y1+p*math.sin(ang1),2,0,2*math.pi) + self.context.fill() + self.context.set_source_rgba(0.0,0.0,1.0) + self.context.arc(x2+p*math.cos(ang2),y2+p*math.sin(ang2),2,0,2*math.pi) + self.context.fill() + '''self.context.set_source_rgba(0.0,0.0,0.0,0.3) + self.context.arc(x2,y2,2,0,2*math.pi) + self.context.fill()''' + self.context.move_to(x1,y1) + self.context.line_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1)) + self.context.stroke() + self.context.move_to(x2,y2) + self.context.line_to(x2+p*math.cos(ang2),y2+p*math.sin(ang2)) + self.context.stroke() + #break + + #self.context.arc(self.dimensions[HORZ]/2, self.dimensions[VERT]/2,50,0,3*math.pi/2) + #self.context.fill() + + +class PiePlot(Plot): + #TODO: Check the old cairoplot, graphs aren't matching + def __init__ (self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + gradient = False, + shadow = False, + colors = None): + + Plot.__init__( self, surface, data, width, height, background, series_colors = colors ) + self.center = (self.dimensions[HORZ]/2, self.dimensions[VERT]/2) + self.total = sum( self.series.to_list() ) + self.radius = min(self.dimensions[HORZ]/3,self.dimensions[VERT]/3) + self.gradient = gradient + self.shadow = shadow + + def sort_function(x,y): + return x.content - y.content + + def load_series(self, data, x_labels=None, y_labels=None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + # Already done inside series + #self.data = sorted(self.data) + + def draw_piece(self, angle, next_angle): + self.context.move_to(self.center[0],self.center[1]) + self.context.line_to(self.center[0] + self.radius*math.cos(angle), self.center[1] + self.radius*math.sin(angle)) + self.context.arc(self.center[0], self.center[1], self.radius, angle, next_angle) + self.context.line_to(self.center[0], self.center[1]) + self.context.close_path() + + def render(self): + self.render_background() + self.render_bounding_box() + if self.shadow: + self.render_shadow() + self.render_plot() + self.render_series_labels() + + def render_shadow(self): + horizontal_shift = 3 + vertical_shift = 3 + self.context.set_source_rgba(0, 0, 0, 0.5) + self.context.arc(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.radius, 0, 2*math.pi) + self.context.fill() + + def render_series_labels(self): + angle = 0 + next_angle = 0 + x0,y0 = self.center + cr = self.context + for number,key in enumerate(self.series_labels): + # self.data[number] should be just a number + data = sum(self.series[number].to_list()) + + next_angle = angle + 2.0*math.pi*data/self.total + cr.set_source_rgba(*self.series_colors[number][:4]) + w = cr.text_extents(key)[2] + if (angle + next_angle)/2 < math.pi/2 or (angle + next_angle)/2 > 3*math.pi/2: + cr.move_to(x0 + (self.radius+10)*math.cos((angle+next_angle)/2), y0 + (self.radius+10)*math.sin((angle+next_angle)/2) ) + else: + cr.move_to(x0 + (self.radius+10)*math.cos((angle+next_angle)/2) - w, y0 + (self.radius+10)*math.sin((angle+next_angle)/2) ) + cr.show_text(key) + angle = next_angle + + def render_plot(self): + angle = 0 + next_angle = 0 + x0,y0 = self.center + cr = self.context + for number,group in enumerate(self.series): + # Group should be just a number + data = sum(group.to_list()) + next_angle = angle + 2.0*math.pi*data/self.total + if self.gradient or self.series_colors[number][4] in ('linear','radial'): + gradient_color = cairo.RadialGradient(self.center[0], self.center[1], 0, self.center[0], self.center[1], self.radius) + gradient_color.add_color_stop_rgba(0.3, *self.series_colors[number][:4]) + gradient_color.add_color_stop_rgba(1, self.series_colors[number][0]*0.7, + self.series_colors[number][1]*0.7, + self.series_colors[number][2]*0.7, + self.series_colors[number][3]) + cr.set_source(gradient_color) + else: + cr.set_source_rgba(*self.series_colors[number][:4]) + + self.draw_piece(angle, next_angle) + cr.fill() + + cr.set_source_rgba(1.0, 1.0, 1.0) + self.draw_piece(angle, next_angle) + cr.stroke() + + angle = next_angle + +class DonutPlot(PiePlot): + def __init__ (self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + gradient = False, + shadow = False, + colors = None, + inner_radius=-1): + + Plot.__init__( self, surface, data, width, height, background, series_colors = colors ) + + self.center = ( self.dimensions[HORZ]/2, self.dimensions[VERT]/2 ) + self.total = sum( self.series.to_list() ) + self.radius = min( self.dimensions[HORZ]/3,self.dimensions[VERT]/3 ) + self.inner_radius = inner_radius*self.radius + + if inner_radius == -1: + self.inner_radius = self.radius/3 + + self.gradient = gradient + self.shadow = shadow + + def draw_piece(self, angle, next_angle): + self.context.move_to(self.center[0] + (self.inner_radius)*math.cos(angle), self.center[1] + (self.inner_radius)*math.sin(angle)) + self.context.line_to(self.center[0] + self.radius*math.cos(angle), self.center[1] + self.radius*math.sin(angle)) + self.context.arc(self.center[0], self.center[1], self.radius, angle, next_angle) + self.context.line_to(self.center[0] + (self.inner_radius)*math.cos(next_angle), self.center[1] + (self.inner_radius)*math.sin(next_angle)) + self.context.arc_negative(self.center[0], self.center[1], self.inner_radius, next_angle, angle) + self.context.close_path() + + def render_shadow(self): + horizontal_shift = 3 + vertical_shift = 3 + self.context.set_source_rgba(0, 0, 0, 0.5) + self.context.arc(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.inner_radius, 0, 2*math.pi) + self.context.arc_negative(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.radius, 0, -2*math.pi) + self.context.fill() + +class GanttChart (Plot) : + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + x_labels = None, + y_labels = None, + colors = None): + self.bounds = {} + self.max_value = {} + Plot.__init__(self, surface, data, width, height, x_labels = x_labels, y_labels = y_labels, series_colors = colors) + + def load_series(self, data, x_labels=None, y_labels=None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + self.calc_boundaries() + + def calc_boundaries(self): + self.bounds[HORZ] = (0,len(self.series)) + end_pos = max(self.series.to_list()) + + #for group in self.series: + # if hasattr(item, "__delitem__"): + # for sub_item in item: + # end_pos = max(sub_item) + # else: + # end_pos = max(item) + self.bounds[VERT] = (0,end_pos) + + def calc_extents(self, direction): + self.max_value[direction] = 0 + if self.labels[direction]: + self.max_value[direction] = max(self.context.text_extents(item)[2] for item in self.labels[direction]) + else: + self.max_value[direction] = self.context.text_extents( str(self.bounds[direction][1] + 1) )[2] + + def calc_horz_extents(self): + self.calc_extents(HORZ) + self.borders[HORZ] = 100 + self.max_value[HORZ] + + def calc_vert_extents(self): + self.calc_extents(VERT) + self.borders[VERT] = self.dimensions[VERT]/(self.bounds[HORZ][1] + 1) + + def calc_steps(self): + self.horizontal_step = (self.dimensions[HORZ] - self.borders[HORZ])/(len(self.labels[VERT])) + self.vertical_step = self.borders[VERT] + + def render(self): + self.calc_horz_extents() + self.calc_vert_extents() + self.calc_steps() + self.render_background() + + self.render_labels() + self.render_grid() + self.render_plot() + + def render_background(self): + cr = self.context + cr.set_source_rgba(255,255,255) + cr.rectangle(0,0,self.dimensions[HORZ], self.dimensions[VERT]) + cr.fill() + for number,group in enumerate(self.series): + linear = cairo.LinearGradient(self.dimensions[HORZ]/2, self.borders[VERT] + number*self.vertical_step, + self.dimensions[HORZ]/2, self.borders[VERT] + (number+1)*self.vertical_step) + linear.add_color_stop_rgba(0,1.0,1.0,1.0,1.0) + linear.add_color_stop_rgba(1.0,0.9,0.9,0.9,1.0) + cr.set_source(linear) + cr.rectangle(0,self.borders[VERT] + number*self.vertical_step,self.dimensions[HORZ],self.vertical_step) + cr.fill() + + def render_grid(self): + cr = self.context + cr.set_source_rgba(0.7, 0.7, 0.7) + cr.set_dash((1,0,0,0,0,0,1)) + cr.set_line_width(0.5) + for number,label in enumerate(self.labels[VERT]): + h = cr.text_extents(label)[3] + cr.move_to(self.borders[HORZ] + number*self.horizontal_step, self.vertical_step/2 + h) + cr.line_to(self.borders[HORZ] + number*self.horizontal_step, self.dimensions[VERT]) + cr.stroke() + + def render_labels(self): + self.context.set_font_size(0.02 * self.dimensions[HORZ]) + + self.render_horz_labels() + self.render_vert_labels() + + def render_horz_labels(self): + cr = self.context + labels = self.labels[HORZ] + if not labels: + labels = [str(i) for i in range(1, self.bounds[HORZ][1] + 1) ] + for number,label in enumerate(labels): + if label != None: + cr.set_source_rgba(0.5, 0.5, 0.5) + w,h = cr.text_extents(label)[2], cr.text_extents(label)[3] + cr.move_to(40,self.borders[VERT] + number*self.vertical_step + self.vertical_step/2 + h/2) + cr.show_text(label) + + def render_vert_labels(self): + cr = self.context + labels = self.labels[VERT] + if not labels: + labels = [str(i) for i in range(1, self.bounds[VERT][1] + 1) ] + for number,label in enumerate(labels): + w,h = cr.text_extents(label)[2], cr.text_extents(label)[3] + cr.move_to(self.borders[HORZ] + number*self.horizontal_step - w/2, self.vertical_step/2) + cr.show_text(label) + + def render_rectangle(self, x0, y0, x1, y1, color): + self.draw_shadow(x0, y0, x1, y1) + self.draw_rectangle(x0, y0, x1, y1, color) + + def draw_rectangular_shadow(self, gradient, x0, y0, w, h): + self.context.set_source(gradient) + self.context.rectangle(x0,y0,w,h) + self.context.fill() + + def draw_circular_shadow(self, x, y, radius, ang_start, ang_end, mult, shadow): + gradient = cairo.RadialGradient(x, y, 0, x, y, 2*radius) + gradient.add_color_stop_rgba(0, 0, 0, 0, shadow) + gradient.add_color_stop_rgba(1, 0, 0, 0, 0) + self.context.set_source(gradient) + self.context.move_to(x,y) + self.context.line_to(x + mult[0]*radius,y + mult[1]*radius) + self.context.arc(x, y, 8, ang_start, ang_end) + self.context.line_to(x,y) + self.context.close_path() + self.context.fill() + + def draw_rectangle(self, x0, y0, x1, y1, color): + cr = self.context + middle = (x0+x1)/2 + linear = cairo.LinearGradient(middle,y0,middle,y1) + linear.add_color_stop_rgba(0,3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1,*color[:4]) + cr.set_source(linear) + + cr.arc(x0+5, y0+5, 5, 0, 2*math.pi) + cr.arc(x1-5, y0+5, 5, 0, 2*math.pi) + cr.arc(x0+5, y1-5, 5, 0, 2*math.pi) + cr.arc(x1-5, y1-5, 5, 0, 2*math.pi) + cr.rectangle(x0+5,y0,x1-x0-10,y1-y0) + cr.rectangle(x0,y0+5,x1-x0,y1-y0-10) + cr.fill() + + def draw_shadow(self, x0, y0, x1, y1): + shadow = 0.4 + h_mid = (x0+x1)/2 + v_mid = (y0+y1)/2 + h_linear_1 = cairo.LinearGradient(h_mid,y0-4,h_mid,y0+4) + h_linear_2 = cairo.LinearGradient(h_mid,y1-4,h_mid,y1+4) + v_linear_1 = cairo.LinearGradient(x0-4,v_mid,x0+4,v_mid) + v_linear_2 = cairo.LinearGradient(x1-4,v_mid,x1+4,v_mid) + + h_linear_1.add_color_stop_rgba( 0, 0, 0, 0, 0) + h_linear_1.add_color_stop_rgba( 1, 0, 0, 0, shadow) + h_linear_2.add_color_stop_rgba( 0, 0, 0, 0, shadow) + h_linear_2.add_color_stop_rgba( 1, 0, 0, 0, 0) + v_linear_1.add_color_stop_rgba( 0, 0, 0, 0, 0) + v_linear_1.add_color_stop_rgba( 1, 0, 0, 0, shadow) + v_linear_2.add_color_stop_rgba( 0, 0, 0, 0, shadow) + v_linear_2.add_color_stop_rgba( 1, 0, 0, 0, 0) + + self.draw_rectangular_shadow(h_linear_1,x0+4,y0-4,x1-x0-8,8) + self.draw_rectangular_shadow(h_linear_2,x0+4,y1-4,x1-x0-8,8) + self.draw_rectangular_shadow(v_linear_1,x0-4,y0+4,8,y1-y0-8) + self.draw_rectangular_shadow(v_linear_2,x1-4,y0+4,8,y1-y0-8) + + self.draw_circular_shadow(x0+4, y0+4, 4, math.pi, 3*math.pi/2, (-1,0), shadow) + self.draw_circular_shadow(x1-4, y0+4, 4, 3*math.pi/2, 2*math.pi, (0,-1), shadow) + self.draw_circular_shadow(x0+4, y1-4, 4, math.pi/2, math.pi, (0,1), shadow) + self.draw_circular_shadow(x1-4, y1-4, 4, 0, math.pi/2, (1,0), shadow) + + def render_plot(self): + for index,group in enumerate(self.series): + for data in group: + self.render_rectangle(self.borders[HORZ] + data.content[0]*self.horizontal_step, + self.borders[VERT] + index*self.vertical_step + self.vertical_step/4.0, + self.borders[HORZ] + data.content[1]*self.horizontal_step, + self.borders[VERT] + index*self.vertical_step + 3.0*self.vertical_step/4.0, + self.series_colors[index]) + +# Function definition + +def scatter_plot(name, + data = None, + errorx = None, + errory = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + axis = False, + dash = False, + discrete = False, + dots = False, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + z_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + circle_colors = None): + + ''' + - Function to plot scatter data. + + - Parameters + + data - The values to be ploted might be passed in a two basic: + list of points: [(0,0), (0,1), (0,2)] or [(0,0,1), (0,1,4), (0,2,1)] + lists of coordinates: [ [0,0,0] , [0,1,2] ] or [ [0,0,0] , [0,1,2] , [1,4,1] ] + Notice that these kinds of that can be grouped in order to form more complex data + using lists of lists or dictionaries; + series_colors - Define color values for each of the series + circle_colors - Define a lower and an upper bound for the circle colors for variable radius + (3 dimensions) series + ''' + + plot = ScatterPlot( name, data, errorx, errory, width, height, background, border, + axis, dash, discrete, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, z_bounds, x_title, y_title, series_colors, circle_colors ) + plot.render() + plot.commit() + +def dot_line_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + axis = False, + dash = False, + dots = False, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None): + ''' + - Function to plot graphics using dots and lines. + + dot_line_plot (name, data, width, height, background = "white light_gray", border = 0, axis = False, grid = False, x_labels = None, y_labels = None, x_bounds = None, y_bounds = None) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + axis - Whether or not the axis are to be drawn; + dash - Boolean or a list or a dictionary of booleans indicating whether or not the associated series should be drawn in dashed mode; + dots - Whether or not dots should be drawn on each point; + grid - Whether or not the gris is to be drawn; + series_legend - Whether or not the legend is to be drawn; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + x_title - Whether or not to plot a title over the x axis. + y_title - Whether or not to plot a title over the y axis. + + - Examples of use + + data = [0, 1, 3, 8, 9, 0, 10, 10, 2, 1] + CairoPlot.dot_line_plot('teste', data, 400, 300) + + data = { "john" : [10, 10, 10, 10, 30], "mary" : [0, 0, 3, 5, 15], "philip" : [13, 32, 11, 25, 2] } + x_labels = ["jan/2008", "feb/2008", "mar/2008", "apr/2008", "may/2008" ] + CairoPlot.dot_line_plot( 'test', data, 400, 300, axis = True, grid = True, + series_legend = True, x_labels = x_labels ) + ''' + plot = DotLinePlot( name, data, width, height, background, border, + axis, dash, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, x_title, y_title, series_colors ) + plot.render() + plot.commit() + +def function_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + axis = True, + dots = False, + discrete = False, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + step = 1): + + ''' + - Function to plot functions. + + function_plot(name, data, width, height, background = "white light_gray", border = 0, axis = True, grid = False, dots = False, x_labels = None, y_labels = None, x_bounds = None, y_bounds = None, step = 1, discrete = False) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + axis - Whether or not the axis are to be drawn; + grid - Whether or not the gris is to be drawn; + dots - Whether or not dots should be shown at each point; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + step - the horizontal distance from one point to the other. The smaller, the smoother the curve will be; + discrete - whether or not the function should be plotted in discrete format. + + - Example of use + + data = lambda x : x**2 + CairoPlot.function_plot('function4', data, 400, 300, grid = True, x_bounds=(-10,10), step = 0.1) + ''' + + plot = FunctionPlot( name, data, width, height, background, border, + axis, discrete, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, x_title, y_title, series_colors, step ) + plot.render() + plot.commit() + +def pie_plot( name, data, width, height, background = "white light_gray", gradient = False, shadow = False, colors = None ): + + ''' + - Function to plot pie graphics. + + pie_plot(name, data, width, height, background = "white light_gray", gradient = False, colors = None) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + gradient - Whether or not the pie color will be painted with a gradient; + shadow - Whether or not there will be a shadow behind the pie; + colors - List of slices colors. + + - Example of use + + teste_data = {"john" : 123, "mary" : 489, "philip" : 890 , "suzy" : 235} + CairoPlot.pie_plot("pie_teste", teste_data, 500, 500) + ''' + + plot = PiePlot( name, data, width, height, background, gradient, shadow, colors ) + plot.render() + plot.commit() + +def donut_plot(name, data, width, height, background = "white light_gray", gradient = False, shadow = False, colors = None, inner_radius = -1): + + ''' + - Function to plot donut graphics. + + donut_plot(name, data, width, height, background = "white light_gray", gradient = False, inner_radius = -1) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + shadow - Whether or not there will be a shadow behind the donut; + gradient - Whether or not the donut color will be painted with a gradient; + colors - List of slices colors; + inner_radius - The radius of the donut's inner circle. + + - Example of use + + teste_data = {"john" : 123, "mary" : 489, "philip" : 890 , "suzy" : 235} + CairoPlot.donut_plot("donut_teste", teste_data, 500, 500) + ''' + + plot = DonutPlot(name, data, width, height, background, gradient, shadow, colors, inner_radius) + plot.render() + plot.commit() + +def gantt_chart(name, pieces, width, height, x_labels, y_labels, colors): + + ''' + - Function to generate Gantt Charts. + + gantt_chart(name, pieces, width, height, x_labels, y_labels, colors): + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + pieces - A list defining the spaces to be drawn. The user must pass, for each line, the index of its start and the index of its end. If a line must have two or more spaces, they must be passed inside a list; + width, height - Dimensions of the output image; + x_labels - A list of names for each of the vertical lines; + y_labels - A list of names for each of the horizontal spaces; + colors - List containing the colors expected for each of the horizontal spaces + + - Example of use + + pieces = [ (0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,8)] + x_labels = [ 'teste01', 'teste02', 'teste03', 'teste04'] + y_labels = [ '0001', '0002', '0003', '0004', '0005', '0006', '0007', '0008', '0009', '0010' ] + colors = [ (1.0, 0.0, 0.0), (1.0, 0.7, 0.0), (1.0, 1.0, 0.0), (0.0, 1.0, 0.0) ] + CairoPlot.gantt_chart('gantt_teste', pieces, 600, 300, x_labels, y_labels, colors) + ''' + + plot = GanttChart(name, pieces, width, height, x_labels, y_labels, colors) + plot.render() + plot.commit() + +def vertical_bar_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + colors = None): + #TODO: Fix docstring for vertical_bar_plot + ''' + - Function to generate vertical Bar Plot Charts. + + bar_plot(name, data, width, height, background, border, grid, rounded_corners, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, colors): + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtime; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + grid - Whether or not the gris is to be drawn; + rounded_corners - Whether or not the bars should have rounded corners; + three_dimension - Whether or not the bars should be drawn in pseudo 3D; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + colors - List containing the colors expected for each of the bars. + + - Example of use + + data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] + CairoPlot.vertical_bar_plot ('bar2', data, 400, 300, border = 20, grid = True, rounded_corners = False) + ''' + + plot = VerticalBarPlot(name, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + series_labels, x_labels, y_labels, x_bounds, y_bounds, colors) + plot.render() + plot.commit() + +def horizontal_bar_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + colors = None): + + #TODO: Fix docstring for horizontal_bar_plot + ''' + - Function to generate Horizontal Bar Plot Charts. + + bar_plot(name, data, width, height, background, border, grid, rounded_corners, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, colors): + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtime; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + grid - Whether or not the gris is to be drawn; + rounded_corners - Whether or not the bars should have rounded corners; + three_dimension - Whether or not the bars should be drawn in pseudo 3D; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + colors - List containing the colors expected for each of the bars. + + - Example of use + + data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] + CairoPlot.bar_plot ('bar2', data, 400, 300, border = 20, grid = True, rounded_corners = False) + ''' + + plot = HorizontalBarPlot(name, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + series_labels, x_labels, y_labels, x_bounds, y_bounds, colors) + plot.render() + plot.commit() + +def stream_chart(name, + data, + width, + height, + background = "white light_gray", + border = 0, + grid = False, + series_legend = None, + x_labels = None, + x_bounds = None, + y_bounds = None, + colors = None): + + #TODO: Fix docstring for horizontal_bar_plot + plot = StreamChart(name, data, width, height, background, border, + grid, series_legend, x_labels, x_bounds, y_bounds, colors) + plot.render() + plot.commit() + + +if __name__ == "__main__": + import tests + import seriestests diff --git a/python/examples/output_format_modules/pprint_table.py b/python/examples/output_format_modules/pprint_table.py new file mode 100644 index 0000000..1cd8620 --- /dev/null +++ b/python/examples/output_format_modules/pprint_table.py @@ -0,0 +1,35 @@ +# This module is used to pretty-print a table +# Adapted from +# http://ginstrom.com/scribbles/2007/09/04/pretty-printing-a-table-in-python/ + +import sys + +def get_max_width(table, index): + """Get the maximum width of the given column index""" + + return max([len(str(row[index])) for row in table]) + + +def pprint_table(table, nbLeft=1, out=sys.stdout): + """ + Prints out a table of data, padded for alignment + @param table: The table to print. A list of lists. + Each row must have the same number of columns. + @param nbLeft: The number of columns aligned left + @param out: Output stream (file-like object) + """ + + col_paddings = [] + + for i in range(len(table[0])): + col_paddings.append(get_max_width(table, i)) + + for row in table: + # left cols + for i in range(nbLeft): + print >> out, str(row[i]).ljust(col_paddings[i] + 1), + # rest of the cols + for i in range(nbLeft, len(row)): + col = str(row[i]).rjust(col_paddings[i] + 2) + print >> out, col, + print >> out diff --git a/python/examples/output_format_modules/series.py b/python/examples/output_format_modules/series.py new file mode 100755 index 0000000..8e8b236 --- /dev/null +++ b/python/examples/output_format_modules/series.py @@ -0,0 +1,1140 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# Serie.py +# +# Copyright (c) 2008 Magnun Leno da Silva +# +# Author: Magnun Leno da Silva +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public License +# as published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 +# USA + +# Contributor: Rodrigo Moreiro Araujo + +#import cairoplot +import doctest + +NUMTYPES = (int, float, long) +LISTTYPES = (list, tuple) +STRTYPES = (str, unicode) +FILLING_TYPES = ['linear', 'solid', 'gradient'] +DEFAULT_COLOR_FILLING = 'solid' +#TODO: Define default color list +DEFAULT_COLOR_LIST = None + +class Data(object): + ''' + Class that models the main data structure. + It can hold: + - a number type (int, float or long) + - a tuple, witch represents a point and can have 2 or 3 items (x,y,z) + - if a list is passed it will be converted to a tuple. + + obs: In case a tuple is passed it will convert to tuple + ''' + def __init__(self, data=None, name=None, parent=None): + ''' + Starts main atributes from the Data class + @name - Name for each point; + @content - The real data, can be an int, float, long or tuple, which + represents a point (x,y) or (x,y,z); + @parent - A pointer that give the data access to it's parent. + + Usage: + >>> d = Data(name='empty'); print d + empty: () + >>> d = Data((1,1),'point a'); print d + point a: (1, 1) + >>> d = Data((1,2,3),'point b'); print d + point b: (1, 2, 3) + >>> d = Data([2,3],'point c'); print d + point c: (2, 3) + >>> d = Data(12, 'simple value'); print d + simple value: 12 + ''' + # Initial values + self.__content = None + self.__name = None + + # Setting passed values + self.parent = parent + self.name = name + self.content = data + + # Name property + @apply + def name(): + doc = ''' + Name is a read/write property that controls the input of name. + - If passed an invalid value it cleans the name with None + + Usage: + >>> d = Data(13); d.name = 'name_test'; print d + name_test: 13 + >>> d.name = 11; print d + 13 + >>> d.name = 'other_name'; print d + other_name: 13 + >>> d.name = None; print d + 13 + >>> d.name = 'last_name'; print d + last_name: 13 + >>> d.name = ''; print d + 13 + ''' + def fget(self): + ''' + returns the name as a string + ''' + return self.__name + + def fset(self, name): + ''' + Sets the name of the Data + ''' + if type(name) in STRTYPES and len(name) > 0: + self.__name = name + else: + self.__name = None + + + + return property(**locals()) + + # Content property + @apply + def content(): + doc = ''' + Content is a read/write property that validate the data passed + and return it. + + Usage: + >>> d = Data(); d.content = 13; d.content + 13 + >>> d = Data(); d.content = (1,2); d.content + (1, 2) + >>> d = Data(); d.content = (1,2,3); d.content + (1, 2, 3) + >>> d = Data(); d.content = [1,2,3]; d.content + (1, 2, 3) + >>> d = Data(); d.content = [1.5,.2,3.3]; d.content + (1.5, 0.20000000000000001, 3.2999999999999998) + ''' + def fget(self): + ''' + Return the content of Data + ''' + return self.__content + + def fset(self, data): + ''' + Ensures that data is a valid tuple/list or a number (int, float + or long) + ''' + # Type: None + if data is None: + self.__content = None + return + + # Type: Int or Float + elif type(data) in NUMTYPES: + self.__content = data + + # Type: List or Tuple + elif type(data) in LISTTYPES: + # Ensures the correct size + if len(data) not in (2, 3): + raise TypeError, "Data (as list/tuple) must have 2 or 3 items" + return + + # Ensures that all items in list/tuple is a number + isnum = lambda x : type(x) not in NUMTYPES + + if max(map(isnum, data)): + # An item in data isn't an int or a float + raise TypeError, "All content of data must be a number (int or float)" + + # Convert the tuple to list + if type(data) is list: + data = tuple(data) + + # Append a copy and sets the type + self.__content = data[:] + + # Unknown type! + else: + self.__content = None + raise TypeError, "Data must be an int, float or a tuple with two or three items" + return + + return property(**locals()) + + + def clear(self): + ''' + Clear the all Data (content, name and parent) + ''' + self.content = None + self.name = None + self.parent = None + + def copy(self): + ''' + Returns a copy of the Data structure + ''' + # The copy + new_data = Data() + if self.content is not None: + # If content is a point + if type(self.content) is tuple: + new_data.__content = self.content[:] + + # If content is a number + else: + new_data.__content = self.content + + # If it has a name + if self.name is not None: + new_data.__name = self.name + + return new_data + + def __str__(self): + ''' + Return a string representation of the Data structure + ''' + if self.name is None: + if self.content is None: + return '' + return str(self.content) + else: + if self.content is None: + return self.name+": ()" + return self.name+": "+str(self.content) + + def __len__(self): + ''' + Return the length of the Data. + - If it's a number return 1; + - If it's a list return it's length; + - If its None return 0. + ''' + if self.content is None: + return 0 + elif type(self.content) in NUMTYPES: + return 1 + return len(self.content) + + + + +class Group(object): + ''' + Class that models a group of data. Every value (int, float, long, tuple + or list) passed is converted to a list of Data. + It can receive: + - A single number (int, float, long); + - A list of numbers; + - A tuple of numbers; + - An instance of Data; + - A list of Data; + + Obs: If a tuple with 2 or 3 items is passed it is converted to a point. + If a tuple with only 1 item is passed it's converted to a number; + If a tuple with more than 2 items is passed it's converted to a + list of numbers + ''' + def __init__(self, group=None, name=None, parent=None): + ''' + Starts main atributes in Group instance. + @data_list - a list of data which forms the group; + @range - a range that represent the x axis of possible functions; + @name - name of the data group; + @parent - the Serie parent of this group. + + Usage: + >>> g = Group(13, 'simple number'); print g + simple number ['13'] + >>> g = Group((1,2), 'simple point'); print g + simple point ['(1, 2)'] + >>> g = Group([1,2,3,4], 'list of numbers'); print g + list of numbers ['1', '2', '3', '4'] + >>> g = Group((1,2,3,4),'int in tuple'); print g + int in tuple ['1', '2', '3', '4'] + >>> g = Group([(1,2),(2,3),(3,4)], 'list of points'); print g + list of points ['(1, 2)', '(2, 3)', '(3, 4)'] + >>> g = Group([[1,2,3],[1,2,3]], '2D coordinate lists'); print g + 2D coordinated lists ['(1, 1)', '(2, 2)', '(3, 3)'] + >>> g = Group([[1,2],[1,2],[1,2]], '3D coordinate lists'); print g + 3D coordinated lists ['(1, 1, 1)', '(2, 2, 2)'] + ''' + # Initial values + self.__data_list = [] + self.__range = [] + self.__name = None + + + self.parent = parent + self.name = name + self.data_list = group + + # Name property + @apply + def name(): + doc = ''' + Name is a read/write property that controls the input of name. + - If passed an invalid value it cleans the name with None + + Usage: + >>> g = Group(13); g.name = 'name_test'; print g + name_test ['13'] + >>> g.name = 11; print g + ['13'] + >>> g.name = 'other_name'; print g + other_name ['13'] + >>> g.name = None; print g + ['13'] + >>> g.name = 'last_name'; print g + last_name ['13'] + >>> g.name = ''; print g + ['13'] + ''' + def fget(self): + ''' + Returns the name as a string + ''' + return self.__name + + def fset(self, name): + ''' + Sets the name of the Group + ''' + if type(name) in STRTYPES and len(name) > 0: + self.__name = name + else: + self.__name = None + + return property(**locals()) + + # data_list property + @apply + def data_list(): + doc = ''' + The data_list is a read/write property that can be a list of + numbers, a list of points or a list of 2 or 3 coordinate lists. This + property uses mainly the self.add_data method. + + Usage: + >>> g = Group(); g.data_list = 13; print g + ['13'] + >>> g.data_list = (1,2); print g + ['(1, 2)'] + >>> g.data_list = Data((1,2),'point a'); print g + ['point a: (1, 2)'] + >>> g.data_list = [1,2,3]; print g + ['1', '2', '3'] + >>> g.data_list = (1,2,3,4); print g + ['1', '2', '3', '4'] + >>> g.data_list = [(1,2),(2,3),(3,4)]; print g + ['(1, 2)', '(2, 3)', '(3, 4)'] + >>> g.data_list = [[1,2],[1,2]]; print g + ['(1, 1)', '(2, 2)'] + >>> g.data_list = [[1,2],[1,2],[1,2]]; print g + ['(1, 1, 1)', '(2, 2, 2)'] + >>> g.range = (10); g.data_list = lambda x:x**2; print g + ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)'] + ''' + def fget(self): + ''' + Returns the value of data_list + ''' + return self.__data_list + + def fset(self, group): + ''' + Ensures that group is valid. + ''' + # None + if group is None: + self.__data_list = [] + + # Int/float/long or Instance of Data + elif type(group) in NUMTYPES or isinstance(group, Data): + # Clean data_list + self.__data_list = [] + self.add_data(group) + + # One point + elif type(group) is tuple and len(group) in (2,3): + self.__data_list = [] + self.add_data(group) + + # list of items + elif type(group) in LISTTYPES and type(group[0]) is not list: + # Clean data_list + self.__data_list = [] + for item in group: + # try to append and catch an exception + self.add_data(item) + + # function lambda + elif callable(group): + # Explicit is better than implicit + function = group + # Has range + if len(self.range) is not 0: + # Clean data_list + self.__data_list = [] + # Generate values for the lambda function + for x in self.range: + #self.add_data((x,round(group(x),2))) + self.add_data((x,function(x))) + + # Only have range in parent + elif self.parent is not None and len(self.parent.range) is not 0: + # Copy parent range + self.__range = self.parent.range[:] + # Clean data_list + self.__data_list = [] + # Generate values for the lambda function + for x in self.range: + #self.add_data((x,round(group(x),2))) + self.add_data((x,function(x))) + + # Don't have range anywhere + else: + # x_data don't exist + raise Exception, "Data argument is valid but to use function type please set x_range first" + + # Coordinate Lists + elif type(group) in LISTTYPES and type(group[0]) is list: + # Clean data_list + self.__data_list = [] + data = [] + if len(group) == 3: + data = zip(group[0], group[1], group[2]) + elif len(group) == 2: + data = zip(group[0], group[1]) + else: + raise TypeError, "Only one list of coordinates was received." + + for item in data: + self.add_data(item) + + else: + raise TypeError, "Group type not supported" + + return property(**locals()) + + @apply + def range(): + doc = ''' + The range is a read/write property that generates a range of values + for the x axis of the functions. When passed a tuple it almost works + like the built-in range funtion: + - 1 item, represent the end of the range started from 0; + - 2 items, represents the start and the end, respectively; + - 3 items, the last one represents the step; + + When passed a list the range function understands as a valid range. + + Usage: + >>> g = Group(); g.range = 10; print g.range + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0] + >>> g = Group(); g.range = (5); print g.range + [0.0, 1.0, 2.0, 3.0, 4.0] + >>> g = Group(); g.range = (1,7); print g.range + [1.0, 2.0, 3.0, 4.0, 5.0, 6.0] + >>> g = Group(); g.range = (0,10,2); print g.range + [0.0, 2.0, 4.0, 6.0, 8.0] + >>> + >>> g = Group(); g.range = [0]; print g.range + [0.0] + >>> g = Group(); g.range = [0,10,20]; print g.range + [0.0, 10.0, 20.0] + ''' + def fget(self): + ''' + Returns the range + ''' + return self.__range + + def fset(self, x_range): + ''' + Controls the input of a valid type and generate the range + ''' + # if passed a simple number convert to tuple + if type(x_range) in NUMTYPES: + x_range = (x_range,) + + # A list, just convert to float + if type(x_range) is list and len(x_range) > 0: + # Convert all to float + x_range = map(float, x_range) + # Prevents repeated values and convert back to list + self.__range = list(set(x_range[:])) + # Sort the list to ascending order + self.__range.sort() + + # A tuple, must check the lengths and generate the values + elif type(x_range) is tuple and len(x_range) in (1,2,3): + # Convert all to float + x_range = map(float, x_range) + + # Inital values + start = 0.0 + step = 1.0 + end = 0.0 + + # Only the end and it can't be less or iqual to 0 + if len(x_range) is 1 and x_range > 0: + end = x_range[0] + + # The start and the end but the start must be less then the end + elif len(x_range) is 2 and x_range[0] < x_range[1]: + start = x_range[0] + end = x_range[1] + + # All 3, but the start must be less then the end + elif x_range[0] <= x_range[1]: + start = x_range[0] + end = x_range[1] + step = x_range[2] + + # Starts the range + self.__range = [] + # Generate the range + # Can't use the range function because it doesn't support float values + while start < end: + self.__range.append(start) + start += step + + # Incorrect type + else: + raise Exception, "x_range must be a list with one or more items or a tuple with 2 or 3 items" + + return property(**locals()) + + def add_data(self, data, name=None): + ''' + Append a new data to the data_list. + - If data is an instance of Data, append it + - If it's an int, float, tuple or list create an instance of Data and append it + + Usage: + >>> g = Group() + >>> g.add_data(12); print g + ['12'] + >>> g.add_data(7,'other'); print g + ['12', 'other: 7'] + >>> + >>> g = Group() + >>> g.add_data((1,1),'a'); print g + ['a: (1, 1)'] + >>> g.add_data((2,2),'b'); print g + ['a: (1, 1)', 'b: (2, 2)'] + >>> + >>> g.add_data(Data((1,2),'c')); print g + ['a: (1, 1)', 'b: (2, 2)', 'c: (1, 2)'] + ''' + if not isinstance(data, Data): + # Try to convert + data = Data(data,name,self) + + if data.content is not None: + self.__data_list.append(data.copy()) + self.__data_list[-1].parent = self + + + def to_list(self): + ''' + Returns the group as a list of numbers (int, float or long) or a + list of tuples (points 2D or 3D). + + Usage: + >>> g = Group([1,2,3,4],'g1'); g.to_list() + [1, 2, 3, 4] + >>> g = Group([(1,2),(2,3),(3,4)],'g2'); g.to_list() + [(1, 2), (2, 3), (3, 4)] + >>> g = Group([(1,2,3),(3,4,5)],'g2'); g.to_list() + [(1, 2, 3), (3, 4, 5)] + ''' + return [data.content for data in self] + + def copy(self): + ''' + Returns a copy of this group + ''' + new_group = Group() + new_group.__name = self.__name + if self.__range is not None: + new_group.__range = self.__range[:] + for data in self: + new_group.add_data(data.copy()) + return new_group + + def get_names(self): + ''' + Return a list with the names of all data in this group + ''' + names = [] + for data in self: + if data.name is None: + names.append('Data '+str(data.index()+1)) + else: + names.append(data.name) + return names + + + def __str__ (self): + ''' + Returns a string representing the Group + ''' + ret = "" + if self.name is not None: + ret += self.name + " " + if len(self) > 0: + list_str = [str(item) for item in self] + ret += str(list_str) + else: + ret += "[]" + return ret + + def __getitem__(self, key): + ''' + Makes a Group iterable, based in the data_list property + ''' + return self.data_list[key] + + def __len__(self): + ''' + Returns the length of the Group, based in the data_list property + ''' + return len(self.data_list) + + +class Colors(object): + ''' + Class that models the colors its labels (names) and its properties, RGB + and filling type. + + It can receive: + - A list where each item is a list with 3 or 4 items. The + first 3 items represent the RGB values and the last argument + defines the filling type. The list will be converted to a dict + and each color will receve a name based in its position in the + list. + - A dictionary where each key will be the color name and its item + can be a list with 3 or 4 items. The first 3 items represent + the RGB colors and the last argument defines the filling type. + ''' + def __init__(self, color_list=None): + ''' + Start the color_list property + @ color_list - the list or dict contaning the colors properties. + ''' + self.__color_list = None + + self.color_list = color_list + + @apply + def color_list(): + doc = ''' + >>> c = Colors([[1,1,1],[2,2,2,'linear'],[3,3,3,'gradient']]) + >>> print c.color_list + {'Color 2': [2, 2, 2, 'linear'], 'Color 3': [3, 3, 3, 'gradient'], 'Color 1': [1, 1, 1, 'solid']} + >>> c.color_list = [[1,1,1],(2,2,2,'solid'),(3,3,3,'linear')] + >>> print c.color_list + {'Color 2': [2, 2, 2, 'solid'], 'Color 3': [3, 3, 3, 'linear'], 'Color 1': [1, 1, 1, 'solid']} + >>> c.color_list = {'a':[1,1,1],'b':(2,2,2,'solid'),'c':(3,3,3,'linear'), 'd':(4,4,4)} + >>> print c.color_list + {'a': [1, 1, 1, 'solid'], 'c': [3, 3, 3, 'linear'], 'b': [2, 2, 2, 'solid'], 'd': [4, 4, 4, 'solid']} + ''' + def fget(self): + ''' + Return the color list + ''' + return self.__color_list + + def fset(self, color_list): + ''' + Format the color list to a dictionary + ''' + if color_list is None: + self.__color_list = None + return + + if type(color_list) in LISTTYPES and type(color_list[0]) in LISTTYPES: + old_color_list = color_list[:] + color_list = {} + for index, color in enumerate(old_color_list): + if len(color) is 3 and max(map(type, color)) in NUMTYPES: + color_list['Color '+str(index+1)] = list(color)+[DEFAULT_COLOR_FILLING] + elif len(color) is 4 and max(map(type, color[:-1])) in NUMTYPES and color[-1] in FILLING_TYPES: + color_list['Color '+str(index+1)] = list(color) + else: + raise TypeError, "Unsuported color format" + elif type(color_list) is not dict: + raise TypeError, "Unsuported color format" + + for name, color in color_list.items(): + if len(color) is 3: + if max(map(type, color)) in NUMTYPES: + color_list[name] = list(color)+[DEFAULT_COLOR_FILLING] + else: + raise TypeError, "Unsuported color format" + elif len(color) is 4: + if max(map(type, color[:-1])) in NUMTYPES and color[-1] in FILLING_TYPES: + color_list[name] = list(color) + else: + raise TypeError, "Unsuported color format" + self.__color_list = color_list.copy() + + return property(**locals()) + + +class Series(object): + ''' + Class that models a Series (group of groups). Every value (int, float, + long, tuple or list) passed is converted to a list of Group or Data. + It can receive: + - a single number or point, will be converted to a Group of one Data; + - a list of numbers, will be converted to a group of numbers; + - a list of tuples, will converted to a single Group of points; + - a list of lists of numbers, each 'sublist' will be converted to a + group of numbers; + - a list of lists of tuples, each 'sublist' will be converted to a + group of points; + - a list of lists of lists, the content of the 'sublist' will be + processed as coordinated lists and the result will be converted to + a group of points; + - a Dictionary where each item can be the same of the list: number, + point, list of numbers, list of points or list of lists (coordinated + lists); + - an instance of Data; + - an instance of group. + ''' + def __init__(self, series=None, name=None, property=[], colors=None): + ''' + Starts main atributes in Group instance. + @series - a list, dict of data of which the series is composed; + @name - name of the series; + @property - a list/dict of properties to be used in the plots of + this Series + + Usage: + >>> print Series([1,2,3,4]) + ["Group 1 ['1', '2', '3', '4']"] + >>> print Series([[1,2,3],[4,5,6]]) + ["Group 1 ['1', '2', '3']", "Group 2 ['4', '5', '6']"] + >>> print Series((1,2)) + ["Group 1 ['(1, 2)']"] + >>> print Series([(1,2),(2,3)]) + ["Group 1 ['(1, 2)', '(2, 3)']"] + >>> print Series([[(1,2),(2,3)],[(4,5),(5,6)]]) + ["Group 1 ['(1, 2)', '(2, 3)']", "Group 2 ['(4, 5)', '(5, 6)']"] + >>> print Series([[[1,2,3],[1,2,3],[1,2,3]]]) + ["Group 1 ['(1, 1, 1)', '(2, 2, 2)', '(3, 3, 3)']"] + >>> print Series({'g1':[1,2,3], 'g2':[4,5,6]}) + ["g1 ['1', '2', '3']", "g2 ['4', '5', '6']"] + >>> print Series({'g1':[(1,2),(2,3)], 'g2':[(4,5),(5,6)]}) + ["g1 ['(1, 2)', '(2, 3)']", "g2 ['(4, 5)', '(5, 6)']"] + >>> print Series({'g1':[[1,2],[1,2]], 'g2':[[4,5],[4,5]]}) + ["g1 ['(1, 1)', '(2, 2)']", "g2 ['(4, 4)', '(5, 5)']"] + >>> print Series(Data(1,'d1')) + ["Group 1 ['d1: 1']"] + >>> print Series(Group([(1,2),(2,3)],'g1')) + ["g1 ['(1, 2)', '(2, 3)']"] + ''' + # Intial values + self.__group_list = [] + self.__name = None + self.__range = None + + # TODO: Implement colors with filling + self.__colors = None + + self.name = name + self.group_list = series + self.colors = colors + + # Name property + @apply + def name(): + doc = ''' + Name is a read/write property that controls the input of name. + - If passed an invalid value it cleans the name with None + + Usage: + >>> s = Series(13); s.name = 'name_test'; print s + name_test ["Group 1 ['13']"] + >>> s.name = 11; print s + ["Group 1 ['13']"] + >>> s.name = 'other_name'; print s + other_name ["Group 1 ['13']"] + >>> s.name = None; print s + ["Group 1 ['13']"] + >>> s.name = 'last_name'; print s + last_name ["Group 1 ['13']"] + >>> s.name = ''; print s + ["Group 1 ['13']"] + ''' + def fget(self): + ''' + Returns the name as a string + ''' + return self.__name + + def fset(self, name): + ''' + Sets the name of the Group + ''' + if type(name) in STRTYPES and len(name) > 0: + self.__name = name + else: + self.__name = None + + return property(**locals()) + + + + # Colors property + @apply + def colors(): + doc = ''' + >>> s = Series() + >>> s.colors = [[1,1,1],[2,2,2,'linear'],[3,3,3,'gradient']] + >>> print s.colors + {'Color 2': [2, 2, 2, 'linear'], 'Color 3': [3, 3, 3, 'gradient'], 'Color 1': [1, 1, 1, 'solid']} + >>> s.colors = [[1,1,1],(2,2,2,'solid'),(3,3,3,'linear')] + >>> print s.colors + {'Color 2': [2, 2, 2, 'solid'], 'Color 3': [3, 3, 3, 'linear'], 'Color 1': [1, 1, 1, 'solid']} + >>> s.colors = {'a':[1,1,1],'b':(2,2,2,'solid'),'c':(3,3,3,'linear'), 'd':(4,4,4)} + >>> print s.colors + {'a': [1, 1, 1, 'solid'], 'c': [3, 3, 3, 'linear'], 'b': [2, 2, 2, 'solid'], 'd': [4, 4, 4, 'solid']} + ''' + def fget(self): + ''' + Return the color list + ''' + return self.__colors.color_list + + def fset(self, colors): + ''' + Format the color list to a dictionary + ''' + self.__colors = Colors(colors) + + return property(**locals()) + + @apply + def range(): + doc = ''' + The range is a read/write property that generates a range of values + for the x axis of the functions. When passed a tuple it almost works + like the built-in range funtion: + - 1 item, represent the end of the range started from 0; + - 2 items, represents the start and the end, respectively; + - 3 items, the last one represents the step; + + When passed a list the range function understands as a valid range. + + Usage: + >>> s = Series(); s.range = 10; print s.range + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0] + >>> s = Series(); s.range = (5); print s.range + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0] + >>> s = Series(); s.range = (1,7); print s.range + [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0] + >>> s = Series(); s.range = (0,10,2); print s.range + [0.0, 2.0, 4.0, 6.0, 8.0, 10.0] + >>> + >>> s = Series(); s.range = [0]; print s.range + [0.0] + >>> s = Series(); s.range = [0,10,20]; print s.range + [0.0, 10.0, 20.0] + ''' + def fget(self): + ''' + Returns the range + ''' + return self.__range + + def fset(self, x_range): + ''' + Controls the input of a valid type and generate the range + ''' + # if passed a simple number convert to tuple + if type(x_range) in NUMTYPES: + x_range = (x_range,) + + # A list, just convert to float + if type(x_range) is list and len(x_range) > 0: + # Convert all to float + x_range = map(float, x_range) + # Prevents repeated values and convert back to list + self.__range = list(set(x_range[:])) + # Sort the list to ascending order + self.__range.sort() + + # A tuple, must check the lengths and generate the values + elif type(x_range) is tuple and len(x_range) in (1,2,3): + # Convert all to float + x_range = map(float, x_range) + + # Inital values + start = 0.0 + step = 1.0 + end = 0.0 + + # Only the end and it can't be less or iqual to 0 + if len(x_range) is 1 and x_range > 0: + end = x_range[0] + + # The start and the end but the start must be lesser then the end + elif len(x_range) is 2 and x_range[0] < x_range[1]: + start = x_range[0] + end = x_range[1] + + # All 3, but the start must be lesser then the end + elif x_range[0] < x_range[1]: + start = x_range[0] + end = x_range[1] + step = x_range[2] + + # Starts the range + self.__range = [] + # Generate the range + # Cnat use the range function becouse it don't suport float values + while start <= end: + self.__range.append(start) + start += step + + # Incorrect type + else: + raise Exception, "x_range must be a list with one or more item or a tuple with 2 or 3 items" + + return property(**locals()) + + @apply + def group_list(): + doc = ''' + The group_list is a read/write property used to pre-process the list + of Groups. + It can be: + - a single number, point or lambda, will be converted to a single + Group of one Data; + - a list of numbers, will be converted to a group of numbers; + - a list of tuples, will converted to a single Group of points; + - a list of lists of numbers, each 'sublist' will be converted to + a group of numbers; + - a list of lists of tuples, each 'sublist' will be converted to a + group of points; + - a list of lists of lists, the content of the 'sublist' will be + processed as coordinated lists and the result will be converted + to a group of points; + - a list of lambdas, each lambda represents a Group; + - a Dictionary where each item can be the same of the list: number, + point, list of numbers, list of points, list of lists + (coordinated lists) or lambdas + - an instance of Data; + - an instance of group. + + Usage: + >>> s = Series() + >>> s.group_list = [1,2,3,4]; print s + ["Group 1 ['1', '2', '3', '4']"] + >>> s.group_list = [[1,2,3],[4,5,6]]; print s + ["Group 1 ['1', '2', '3']", "Group 2 ['4', '5', '6']"] + >>> s.group_list = (1,2); print s + ["Group 1 ['(1, 2)']"] + >>> s.group_list = [(1,2),(2,3)]; print s + ["Group 1 ['(1, 2)', '(2, 3)']"] + >>> s.group_list = [[(1,2),(2,3)],[(4,5),(5,6)]]; print s + ["Group 1 ['(1, 2)', '(2, 3)']", "Group 2 ['(4, 5)', '(5, 6)']"] + >>> s.group_list = [[[1,2,3],[1,2,3],[1,2,3]]]; print s + ["Group 1 ['(1, 1, 1)', '(2, 2, 2)', '(3, 3, 3)']"] + >>> s.group_list = [(0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,9)]; print s + ["Group 1 ['(0.5, 5.5)']", "Group 2 ['(0, 4)', '(6, 8)']", "Group 3 ['(5.5, 7)']", "Group 4 ['(7, 9)']"] + >>> s.group_list = {'g1':[1,2,3], 'g2':[4,5,6]}; print s + ["g1 ['1', '2', '3']", "g2 ['4', '5', '6']"] + >>> s.group_list = {'g1':[(1,2),(2,3)], 'g2':[(4,5),(5,6)]}; print s + ["g1 ['(1, 2)', '(2, 3)']", "g2 ['(4, 5)', '(5, 6)']"] + >>> s.group_list = {'g1':[[1,2],[1,2]], 'g2':[[4,5],[4,5]]}; print s + ["g1 ['(1, 1)', '(2, 2)']", "g2 ['(4, 4)', '(5, 5)']"] + >>> s.range = 10 + >>> s.group_list = lambda x:x*2 + >>> s.group_list = [lambda x:x*2, lambda x:x**2, lambda x:x**3]; print s + ["Group 1 ['(0.0, 0.0)', '(1.0, 2.0)', '(2.0, 4.0)', '(3.0, 6.0)', '(4.0, 8.0)', '(5.0, 10.0)', '(6.0, 12.0)', '(7.0, 14.0)', '(8.0, 16.0)', '(9.0, 18.0)', '(10.0, 20.0)']", "Group 2 ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)', '(10.0, 100.0)']", "Group 3 ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 8.0)', '(3.0, 27.0)', '(4.0, 64.0)', '(5.0, 125.0)', '(6.0, 216.0)', '(7.0, 343.0)', '(8.0, 512.0)', '(9.0, 729.0)', '(10.0, 1000.0)']"] + >>> s.group_list = {'linear':lambda x:x*2, 'square':lambda x:x**2, 'cubic':lambda x:x**3}; print s + ["cubic ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 8.0)', '(3.0, 27.0)', '(4.0, 64.0)', '(5.0, 125.0)', '(6.0, 216.0)', '(7.0, 343.0)', '(8.0, 512.0)', '(9.0, 729.0)', '(10.0, 1000.0)']", "linear ['(0.0, 0.0)', '(1.0, 2.0)', '(2.0, 4.0)', '(3.0, 6.0)', '(4.0, 8.0)', '(5.0, 10.0)', '(6.0, 12.0)', '(7.0, 14.0)', '(8.0, 16.0)', '(9.0, 18.0)', '(10.0, 20.0)']", "square ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)', '(10.0, 100.0)']"] + >>> s.group_list = Data(1,'d1'); print s + ["Group 1 ['d1: 1']"] + >>> s.group_list = Group([(1,2),(2,3)],'g1'); print s + ["g1 ['(1, 2)', '(2, 3)']"] + ''' + def fget(self): + ''' + Return the group list. + ''' + return self.__group_list + + def fset(self, series): + ''' + Controls the input of a valid group list. + ''' + #TODO: Add support to the following strem of data: [ (0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,9)] + + # Type: None + if series is None: + self.__group_list = [] + + # List or Tuple + elif type(series) in LISTTYPES: + self.__group_list = [] + + is_function = lambda x: callable(x) + # Groups + if list in map(type, series) or max(map(is_function, series)): + for group in series: + self.add_group(group) + + # single group + else: + self.add_group(series) + + #old code + ## List of numbers + #if type(series[0]) in NUMTYPES or type(series[0]) is tuple: + # print series + # self.add_group(series) + # + ## List of anything else + #else: + # for group in series: + # self.add_group(group) + + # Dict representing series of groups + elif type(series) is dict: + self.__group_list = [] + names = series.keys() + names.sort() + for name in names: + self.add_group(Group(series[name],name,self)) + + # A single lambda + elif callable(series): + self.__group_list = [] + self.add_group(series) + + # Int/float, instance of Group or Data + elif type(series) in NUMTYPES or isinstance(series, Group) or isinstance(series, Data): + self.__group_list = [] + self.add_group(series) + + # Default + else: + raise TypeError, "Serie type not supported" + + return property(**locals()) + + def add_group(self, group, name=None): + ''' + Append a new group in group_list + ''' + if not isinstance(group, Group): + #Try to convert + group = Group(group, name, self) + + if len(group.data_list) is not 0: + # Auto naming groups + if group.name is None: + group.name = "Group "+str(len(self.__group_list)+1) + + self.__group_list.append(group) + self.__group_list[-1].parent = self + + def copy(self): + ''' + Returns a copy of the Series + ''' + new_series = Series() + new_series.__name = self.__name + if self.__range is not None: + new_series.__range = self.__range[:] + #Add color property in the copy method + #self.__colors = None + + for group in self: + new_series.add_group(group.copy()) + + return new_series + + def get_names(self): + ''' + Returns a list of the names of all groups in the Serie + ''' + names = [] + for group in self: + if group.name is None: + names.append('Group '+str(group.index()+1)) + else: + names.append(group.name) + + return names + + def to_list(self): + ''' + Returns a list with the content of all groups and data + ''' + big_list = [] + for group in self: + for data in group: + if type(data.content) in NUMTYPES: + big_list.append(data.content) + else: + big_list = big_list + list(data.content) + return big_list + + def __getitem__(self, key): + ''' + Makes the Series iterable, based in the group_list property + ''' + return self.__group_list[key] + + def __str__(self): + ''' + Returns a string that represents the Series + ''' + ret = "" + if self.name is not None: + ret += self.name + " " + if len(self) > 0: + list_str = [str(item) for item in self] + ret += str(list_str) + else: + ret += "[]" + return ret + + def __len__(self): + ''' + Returns the length of the Series, based in the group_lsit property + ''' + return len(self.group_list) + + +if __name__ == '__main__': + doctest.testmod() diff --git a/python/examples/sched_switch.py b/python/examples/sched_switch.py new file mode 100644 index 0000000..7f73248 --- /dev/null +++ b/python/examples/sched_switch.py @@ -0,0 +1,111 @@ +# The script takes one optional argument (pid) +# The script will read events based on pid and +# print the scheduler switches happening with the process. +# If no arguments are passed, it displays all the scheduler switches. +# This can be used to understand which tasks schedule out the current +# process being traced, and when it gets scheduled in again. +# The trace needs PID context (lttng add-context -k -t pid) + +import sys +from babeltrace import * + +if len(sys.argv) < 2 or len(sys.argv) > 3: + raise TypeError("Usage: python sched_switch.py [pid] path/to/trace") +elif len(sys.argv) == 3: + usePID = True +else: + usePID = False + + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +while event is not None: + while True: + if event.get_name() == "sched_switch": + # Getting scope definition + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) + if sco is None: + print("ERROR: Cannot get definition scope for {}".format( + event.get_name())) + break # Next event + + # Getting PID + pid_field = event.get_field(sco, "_pid") + pid = pid_field.get_int64() + + if ctf.field_error(): + print("ERROR: Missing PID info for sched_switch") + break # Next event + + if usePID and (pid != long(sys.argv[1])): + break # Next event + + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + + # prev_comm + field = event.get_field(sco, "_prev_comm") + prev_comm = field.get_char_array() + if ctf.field_error(): + print("ERROR: Missing prev_comm context info") + + # prev_tid + field = event.get_field(sco, "_prev_tid") + prev_tid = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing prev_tid context info") + + # prev_prio + field = event.get_field(sco, "_prev_prio") + prev_prio = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing prev_prio context info") + + # prev_state + field = event.get_field(sco, "_prev_state") + prev_state = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing prev_state context info") + + # next_comm + field = event.get_field(sco, "_next_comm") + next_comm = field.get_char_array() + if ctf.field_error(): + print("ERROR: Missing next_comm context info") + + # next_tid + field = event.get_field(sco, "_next_tid") + next_tid = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing next_tid context info") + + # next_prio + field = event.get_field(sco, "_next_prio") + next_prio = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing next_prio context info") + + # Output + print("sched_switch, pid = {}, TS = {}, prev_comm = {},\n\t" + "prev_tid = {}, prev_prio = {}, prev_state = {},\n\t" + "next_comm = {}, next_tid = {}, next_prio = {}".format( + pid, event.get_timestamp(), prev_comm, prev_tid, + prev_prio, prev_state, next_comm, next_tid, next_prio)) + + break # Next event + + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it diff --git a/python/examples/schedtimes.py b/python/examples/schedtimes.py new file mode 100644 index 0000000..4680be5 --- /dev/null +++ b/python/examples/schedtimes.py @@ -0,0 +1,92 @@ +# The script checks the trace for the amount of time that +# each process spends in running, sleeping, queuing, and +# waiting for io. Then, it prints out the accumulated time +# for each state of processes observed. +# Optionally, this script can be used with specified PIDs. +# The trace needs PID context (lttng add-context -k -t pid) + +import sys +from babeltrace import * +from output_format_modules.pprint_table import pprint_table as pprint + +if len(sys.argv) < 2 : + raise TypeError("Usage: python sched_switch.py [pid1 [pid2 [...]]] path/to/trace") +elif len(sys.argv) > 2: + usePID = True +else: + usePID = False + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +# Dict used to store data and +# function to add data to the dict +pid_data = {} +def pid_data_append(pid, event_name, time): + global pid_data + if pid in pid_data: + if event_name in pid_data[pid]: + pid_data[pid][event_name] += time + else: + pid_data[pid][event_name] = time + else: + event_timer = {event_name: time} + pid_data[pid] = event_timer + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +while event is not None: + + ts = event.get_timestamp() + event_name = event.get_name() + error = True + + if event_name.find("sched_stat") != -1: + error = False + # Getting scope definition + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) + if sco is None: + print("ERROR: Cannot get definition scope for {}".format( + event.get_name())) + error = True + else: + # Getting PID + pid_field = event.get_field(sco, "_pid") + pid = pid_field.get_int64() + + if ctf.field_error(): + print("ERROR: Missing PID info for {}".format( + event.get_name())) + error = True + + # Read next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + + if not error and event is not None: + # Appending data to dict + spent_time = event.get_timestamp() - ts + if usePID: + for pid_to_use in sys.argv[1:len(sys.argv)-1]: + if pid == long(pid_to_use): + pid_data_append(pid, event_name, spent_time) + else: + pid_data_append(pid, event_name, spent_time) + +del ctf_it + +# Formatting output data +table = [] +for one_pid in sorted(pid_data): + for one_event in sorted(pid_data[one_pid]): + table.append([one_pid, one_event, pid_data[one_pid][one_event]]) +table.insert(0, ["PID", "EVENT", "TOTAL TIME (ns)"]) +pprint(table, 2) diff --git a/python/examples/softirqtimes.py b/python/examples/softirqtimes.py new file mode 100644 index 0000000..4e20bab --- /dev/null +++ b/python/examples/softirqtimes.py @@ -0,0 +1,120 @@ +# The script checks the trace for the amount of time +# spent from each softirq_raise to after softirq_exit. +# It prints out the min, max (with timestamp), +# average times, the standard deviation and the total count. +# Using the cairoplot module, a .svg graph is also outputted +# showing the taken time in function of the time since the +# beginning of the trace. + +import sys, math +from output_format_modules import cairoplot +from babeltrace import * + +if len(sys.argv) < 2: + raise TypeError("Usage: python softirqtimes.py path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +time_taken = [] +graph_data = [] +max_time = (0.0, 0.0) # (val, ts) + +# tmp template: {(cpu_id, vec):TS raise} +tmp = {} + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +start_time = event.get_timestamp() +while(event is not None): + + event_name = event.get_name() + error = True + appendNext = False + + if event_name == 'softirq_raise' or event_name == 'softirq_exit': + # Recover cpu_id and vec values to make a key to tmp + error = False + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) + field = event.get_field(scope, "cpu_id") + cpu_id = field.get_uint64() + if ctf.field_error(): + print("ERROR: Missing cpu_id info for {}".format( + event.get_name())) + error = True + + scope = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + field = event.get_field(scope, "_vec") + vec = field.get_uint64() + if ctf.field_error(): + print("ERROR: Missing vec info for {}".format( + event.get_name())) + error = True + key = (cpu_id, vec) + + if event_name == 'softirq_raise' and not error: + # Add timestamp to tmp + if key in tmp: + # If key already exists + key = (cpu_id, vec, 0) + if key in tmp: + # If the same key appears more than twice + # This part would need improvement, but seems + # to work so far + raise KeyError("key already exists") + tmp[key] = event.get_timestamp() + + if event_name == 'softirq_exit' and not error: + # Wait for reading of next timestamp + appendNext = True + + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + + if appendNext and event: + # Saving data for output + if not (key in tmp) and ((key[0], key[1], 0) in tmp): + key = (key[0], key[1], 0) + raise_timestamp = tmp[key] + time_data = event.get_timestamp() - tmp.pop(key) + if time_data > max_time[0]: + # max_time = (val, ts) + max_time = (time_data, raise_timestamp) + time_taken.append(time_data) + graph_data.append((raise_timestamp - start_time, time_data)) + +del ctf_it + +# Standard dev. calc. +try: + mean = sum(time_taken)/float(len(time_taken)) +except ZeroDivisionError: + raise TypeError("empty data") +deviations_squared = [] +for x in time_taken: + deviations_squared.append(math.pow((x - mean), 2)) +try: + stddev = math.sqrt(sum(deviations_squared) / (len(deviations_squared) - 1)) +except ZeroDivisionError: + stddev = '-' + +# Terminal output +print("AVG TIME: {} ns".format(mean)) +print("MIN TIME: {} ns".format(min(time_taken))) +print("MAX TIME: {} ns, TS: {}".format(max_time[0], max_time[1])) +print("STD DEV: {}".format(stddev)) +print("TOTAL COUNT: {}".format(len(time_taken))) + +# Graph output +cairoplot.scatter_plot ( 'softirqtimes.svg', data = graph_data, + width = 5000, height = 4000, border = 20, axis = True, + grid = True, series_colors = ["red"] ) diff --git a/python/examples/syscalls_by_pid.py b/python/examples/syscalls_by_pid.py new file mode 100644 index 0000000..f6127ed --- /dev/null +++ b/python/examples/syscalls_by_pid.py @@ -0,0 +1,61 @@ +# The script checks all syscall in the trace and prints a list +# showing the number of systemcalls executed by each PID +# ordered from greatest to least number of syscalls. +# The trace needs PID context (lttng add-context -k -t pid) + +import sys +from babeltrace import * +from output_format_modules.pprint_table import pprint_table as pprint + +if len(sys.argv) < 2 : + raise TypeError("Usage: python syscalls_by_pid.py path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +data = {} + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +while event is not None: + if event.get_name().find("sys") >= 0: + # Getting scope definition + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) + if sco is None: + print("ERROR: Cannot get definition scope for {}".format( + event.get_name())) + else: + # Getting PID + pid_field = event.get_field(sco, "_pid") + pid = pid_field.get_int64() + + if ctf.field_error(): + print("ERROR: Missing PID info for sched_switch".format( + event.get_name())) + elif pid in data: + data[pid] += 1 + else: + data[pid] = 1 + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it + +# Setting table for output +table = [] +for item in data: + table.append([data[item], item]) # [count, pid] +table.sort(reverse = True) # [big count first, pid] +for i in range(len(table)): + table[i].reverse() # [pid, big count first] +table.insert(0, ["PID", "SYSCALL COUNT"]) +pprint(table) diff --git a/python/examples/timestats.py b/python/examples/timestats.py new file mode 100644 index 0000000..51e6334 --- /dev/null +++ b/python/examples/timestats.py @@ -0,0 +1,116 @@ +# The script checks the time taken for each event listed as +# the first command line argument or for all events if none +# specified. It prints out the average, minimum, and maximum +# times in microseconds followed by the standard deviation +# and a count of times that the function was called. +# If events are specified, a .svg graph of latency vs +# tracing time is outputted using the cairoplot module. + +import sys, math +from random import randrange +from babeltrace import * +from output_format_modules.pprint_table import pprint_table as pprint +from output_format_modules import cairoplot + +if len(sys.argv) < 2: + raise TypeError("Usage: python timestats.py [event1 [event2 [...]]] path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +event_data = {} +graph_data = {} + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +start_time = event.get_timestamp() +old_ts = start_time + +while(event is not None): + event_name = event.get_name() + + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + + if event is None: + break + + if len(sys.argv) > 2: # Only check specified events + # Event check + for event_type in sys.argv[1:len(sys.argv)-1]: + if event_type == event_name: + # Microsec. + time = float(event.get_timestamp() - old_ts)/1000 + + if event_name in event_data: + event_data[event_name].append(time) + else: + event_data[event_name] = [time] + + # Setting graph data + if event_name in graph_data: + # x coord. (time of event (us)) + graph_data[event_name][0].append( + (event.get_timestamp() - start_time)/1000) + # y coord. (taken time (us)) + graph_data[event_name][1].append(time) + else: + graph_data[event_name] = [ + [(event.get_timestamp() - start_time)/1000], + [time] ] + + else: #If no event specified, keep track of all + + # Microsec. + time = float(event.get_timestamp() - old_ts)/1000 + + if event_name in event_data: + event_data[event_name].append(time) + else: + event_data[event_name] = [time] + + old_ts = event.get_timestamp() + +del ctf_it + +# Appending data to table for output +table = [] +for item in event_data: + + # Standard deviation calc. + mean = sum(event_data[item])/len(event_data[item]) + deviations_squared = [] + for x in event_data[item]: + deviations_squared.append(math.pow((x - mean), 2)) + try: + stddev = math.sqrt(sum(deviations_squared) / (len(deviations_squared) - 1)) + except ZeroDivisionError: + stddev = '-' + + table.append([item, mean, min(event_data[item]), + max(event_data[item]), stddev, len(event_data[item])]) +table = sorted(table) +table.insert(0, ["EVENT", "AVG", "MIN", "MAX", "STD DEV", "COUNT"]) +pprint(table) + +# Graph output +if len(sys.argv) > 2: + colors = [(0.5,0,0)] + for i in range(1, len(graph_data.keys())): + new_color = list(colors[i-1]) + new_color[randrange(3)] += 0.5 + colors.append(tuple(new_color)) + cairoplot.scatter_plot ( 'timestats.svg', data = graph_data, + width = 5500, height = 4000, border = 20, + axis = True, discrete = False, dots = 3, grid = True, + x_title = "time of event (us)", y_title = "time taken (us)", + series_legend = True, series_colors = colors ) diff --git a/python/python-complements.c b/python/python-complements.c new file mode 100644 index 0000000..8c4d811 --- /dev/null +++ b/python/python-complements.c @@ -0,0 +1,105 @@ +/* python-complements.c + Needed functions for python binding +*/ + +#include "python-complements.h" + +/* FILE functions + ---------------------------------------------------- +*/ + +FILE *_bt_file_open(char *file_path, char *mode) +{ + FILE *fp = stdout; + if (file_path != NULL) + fp = fopen(file_path, mode); + return fp; +} + +void _bt_file_close(FILE *fp) +{ + if (fp != NULL) + fclose(fp); +} + + +/* List-related functions + ---------------------------------------------------- +*/ + +/* ctf-field-list */ +struct definition **_bt_python_field_listcaller( + const struct bt_ctf_event *ctf_event, + const struct definition *scope) +{ + struct definition **list; + unsigned int count; + int ret; + + ret = bt_ctf_get_field_list(ctf_event, scope, + (const struct definition * const **)&list, &count); + + if (ret < 0) /* For python to know an error occured */ + list = NULL; + else /* For python to know the end is reached */ + list[count] = NULL; + + return list; +} + +struct definition *_bt_python_field_one_from_list( + struct definition **list, int index) +{ + return list[index]; +} + +/* event_decl_list */ +struct bt_ctf_event_decl **_bt_python_event_decl_listcaller( + int handle_id, struct bt_context *ctx) +{ + struct bt_ctf_event_decl **list; + unsigned int count; + int ret; + + ret = bt_ctf_get_event_decl_list(handle_id, ctx, + (struct bt_ctf_event_decl * const **)&list, &count); + + if (ret < 0) /* For python to know an error occured */ + list = NULL; + else /* For python to know the end is reached */ + list[count] = NULL; + + return list; +} + +struct bt_ctf_event_decl *_bt_python_decl_one_from_list( + struct bt_ctf_event_decl **list, int index) +{ + return list[index]; +} + +/* decl_fields */ +struct bt_ctf_field_decl **_by_python_field_decl_listcaller( + struct bt_ctf_event_decl *event_decl, + enum bt_ctf_scope scope) +{ + struct bt_ctf_field_decl **list; + unsigned int count; + int ret; + + ret = bt_ctf_get_decl_fields(event_decl, scope, + (const struct bt_ctf_field_decl * const **)&list, &count); + + if (ret < 0) /* For python to know an error occured */ + list = NULL; + else /* For python to know the end is reached */ + list[count] = NULL; + + return list; +} + +struct bt_ctf_field_decl *_bt_python_field_decl_one_from_list( + struct bt_ctf_field_decl **list, int index) +{ + return list[index]; +} diff --git a/python/python-complements.h b/python/python-complements.h new file mode 100644 index 0000000..cdd5528 --- /dev/null +++ b/python/python-complements.h @@ -0,0 +1,36 @@ +/* python-complements.h + Needed functions for python binding +*/ + +#include +#include +#include +#include +#include +#include +#include +#include + +/* File */ +FILE *_bt_file_open(char *file_path, char *mode); +void _bt_file_close(FILE *fp); + +/* ctf-field-list */ +struct definition **_bt_python_field_listcaller( + const struct bt_ctf_event *ctf_event, + const struct definition *scope); +struct definition *_bt_python_field_one_from_list( + struct definition **list, int index); + +/* event_decl_list */ +struct bt_ctf_event_decl **_bt_python_event_decl_listcaller( + int handle_id, struct bt_context *ctx); +struct bt_ctf_event_decl *_bt_python_decl_one_from_list( + struct bt_ctf_event_decl **list, int index); + +/* decl_fields */ +struct bt_ctf_field_decl **_by_python_field_decl_listcaller( + struct bt_ctf_event_decl *event_decl, + enum bt_ctf_scope scope); +struct bt_ctf_field_decl *_bt_python_field_decl_one_from_list( + struct bt_ctf_field_decl **list, int index); diff --git a/tests/tests-python.py b/tests/tests-python.py new file mode 100644 index 0000000..0bd71c2 --- /dev/null +++ b/tests/tests-python.py @@ -0,0 +1,115 @@ +import unittest +import sys +from babeltrace import * + +class TestBabeltracePythonModule(unittest.TestCase): + + def test_handle_decl(self): + #Context creation, adding trace + ctx = Context() + trace_handle = ctx.add_trace( + "ctf-traces/succeed/lttng-modules-2.0-pre5", + "ctf") + self.assertIsNotNone(trace_handle, "Error adding trace") + + #TraceHandle test + ts_begin = trace_handle.get_timestamp_begin(ctx, CLOCK_REAL) + ts_end = trace_handle.get_timestamp_end(ctx, CLOCK_REAL) + self.assertGreater(ts_end, ts_begin, "Error get_timestamp from trace_handle") + + lst = ctf.get_event_decl_list(trace_handle, ctx) + self.assertIsNotNone(lst, "Error get_event_decl_list") + + name = lst[0].get_name() + self.assertIsNotNone(name) + + fields = lst[0].get_decl_fields(ctf.scope.EVENT_FIELDS) + self.assertIsNotNone(fields, "Error getting FieldDecl list") + + if len(fields) > 0: + self.assertIsNotNone(fields[0].get_name(), + "Error getting name from FieldDecl") + + #Remove trace + ctx.remove_trace(trace_handle) + del ctx + del trace_handle + + + def test_iterator_event(self): + #Context creation, adding trace + ctx = Context() + trace_handle = ctx.add_trace( + "ctf-traces/succeed/lttng-modules-2.0-pre5", + "ctf") + self.assertIsNotNone(trace_handle, "Error adding trace") + + begin_pos = IterPos(SEEK_BEGIN) + it = ctf.Iterator(ctx, begin_pos) + self.assertIsNotNone(it, "Error creating iterator") + + event = it.read_event() + self.assertIsNotNone(event, "Error reading event") + + handle = event.get_handle() + self.assertIsNotNone(handle, "Error getting handle") + + context = event.get_context() + self.assertIsNotNone(context, "Error getting context") + + name = "" + while event is not None and name != "sched_switch": + name = event.get_name() + self.assertIsNotNone(name, "Error getting event name") + + ts = event.get_timestamp() + self.assertGreaterEqual(ts, 0, "Error getting timestamp") + + cycles = event.get_cycles() + self.assertGreaterEqual(cycles, 0, "Error getting cycles") + + if name == "sched_switch": + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) + self.assertIsNotNone(scope, "Error getting scope definition") + + field = event.get_field(scope, "cpu_id") + prev_comm_str = field.get_uint64() + self.assertEqual(ctf.field_error(), 0, "Error getting vec info") + + field_lst = event.get_field_list(scope) + self.assertIsNotNone(field_lst, "Error getting field list") + + fname = field.field_name() + self.assertIsNotNone(fname, "Error getting field name") + + ftype = field.field_type() + self.assertIsNot(ftype, -1, "Error getting field type (unknown)") + + ret = it.next() + self.assertGreaterEqual(ret, 0, "Error moving iterator") + + event = it.read_event() + + pos = it.get_pos() + self.assertIsNotNone(pos._pos, "Error getting iterator position") + + ret = it.set_pos(pos) + self.assertEqual(ret, 0, "Error setting iterator position") + + pos = it.create_time_pos(ts) + self.assertIsNotNone(pos._pos, "Error creating time-based iterator position") + + del it, pos + set_pos = IterPos(SEEK_TIME, ts) + it = ctf.Iterator(ctx, begin_pos, set_pos) + self.assertIsNotNone(it, "Error creating iterator with end_pos") + + + def test_file_class(self): + f = File("test-bitfield.c") + self.assertIsNotNone(f, "Error opening file") + f.close() + + +if __name__ == "__main__": + unittest.main() -- 1.7.9.5 From dgoulet at efficios.com Tue Aug 7 15:09:19 2012 From: dgoulet at efficios.com (David Goulet) Date: Tue, 07 Aug 2012 15:09:19 -0400 Subject: [lttng-dev] [RFC] LTTng address API proposal v3 Message-ID: <502167DF.5000806@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 After a discussion with Mathieu and Yannick, here is some big changes. I hope I did not forget anything important. Cheers! David -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQIWfcAAoJEELoaioR9I02SAkH/iic+EYwOs5dw0X+yS37sBrp rbP8VHm+v7OP4Lc+x84pfHBnz0pk5qM7MN5/lmMpRfRwCiYNW8CbxC13V5Au6UgC PsNQdZGduVb56xmB5zhV1cD1laiU2W+0vdQzahIN0YDDZowYa6uJda9yjav/b6z8 +duI2Zuw9NPOUnIDCSIpb3hKA4NZIZFsQGn+/IToikqWyRt/4wwiufaZNtlC85/w C0pGJL/9TBTY4rFmyZ/e19lJRUoNEygrFRRoJb5qxKWH5nHvfRXdFGw78xjDgim7 g9AvrWZ7g73KZaS7LEpmAcofgRicWafIMd0+jUXv0MMrj3NeT/xD66UbtcUusl8= =jRvr -----END PGP SIGNATURE----- -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0004-lttng-address-api.txt URL: From mathieu.desnoyers at efficios.com Tue Aug 7 15:19:15 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 7 Aug 2012 15:19:15 -0400 Subject: [lttng-dev] [RFC] LTTng address API proposal v3 In-Reply-To: <502167DF.5000806@efficios.com> References: <502167DF.5000806@efficios.com> Message-ID: <20120807191915.GB454@Krystal> * David Goulet (dgoulet at efficios.com) wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > After a discussion with Mathieu and Yannick, here is some big changes. > > I hope I did not forget anything important. > > Cheers! > David > -----BEGIN PGP SIGNATURE----- > > iQEcBAEBCgAGBQJQIWfcAAoJEELoaioR9I02SAkH/iic+EYwOs5dw0X+yS37sBrp > rbP8VHm+v7OP4Lc+x84pfHBnz0pk5qM7MN5/lmMpRfRwCiYNW8CbxC13V5Au6UgC > PsNQdZGduVb56xmB5zhV1cD1laiU2W+0vdQzahIN0YDDZowYa6uJda9yjav/b6z8 > +duI2Zuw9NPOUnIDCSIpb3hKA4NZIZFsQGn+/IToikqWyRt/4wwiufaZNtlC85/w > C0pGJL/9TBTY4rFmyZ/e19lJRUoNEygrFRRoJb5qxKWH5nHvfRXdFGw78xjDgim7 > g9AvrWZ7g73KZaS7LEpmAcofgRicWafIMd0+jUXv0MMrj3NeT/xD66UbtcUusl8= > =jRvr > -----END PGP SIGNATURE----- > RFC - LTTng address API proposal > > Author: David Goulet > > Contributors: > * Mathieu Desnoyers > * Yannick Brosseau > > Version: > - v0.1: 31/07/2012 > * Initial proposal > - v0.2: 07/08/0212 > * Remove lttng_create_session_addr > * Describe URL string format > * Add set_consumer_url examples > > Introduction > ----------------- > > This document proposes the use of string URLs to the command line interface and > API which will deprecate a function and propose new ones. > > The purpose of this proposal is to support network streaming using URL string > format that you can find in proposal doc/proposals/0003-network.consumer.txt, > remove the lttng_uri structure from the API and integrate the URL string to the > API. > > API > ----------------- > > In order NOT to expose the new lttng_uri structure used to identify trace > location for lttng consumer, the public API will only use string address where > it will be converted in a lttng_uri and sent to the session daemon. > > [*] Create session: > > With the introduction of the enable-consumer command used for network > streaming, the create session command has been modified so the user could > define a consumer location either on the network or local with the command. > > This does NOT change the current API call but rather simply rename _path_ to > _url_ and how it is used in the lttng-ctl library. > > Call changed from: > --> lttng_create_session(const char *name, const char *path); > > To: > --> lttng_create_session(const char *name, const char *url); > > The _name_ argument is the session name and _url_ is a string representing the > URL specified by the user which is define like so: > > PROTO://[HOSTNAME|IP][:PORT][/PATH] > > The proto supported at this stage are (6 means IPv6): > > * net, net6, tcp, tcp6, file > > If the proto is NOT recognized, the string is considered to be a simple path on > the local filesystem. you could specify that it is a path relative to the process CWD (unless it starts with a /). > > The PATH section is the destination path on the _remote_ host where the trace > data will be written with the append after. The last sentence is not clear (after reading it 2-3 times, it's still not obvious what it explains). Simply saying that the PATH is relative to a subdirectory "hostname", under the remote relayd "virtual" root directory, might be enough. > The relayd > defines a default location ... for its virtual root directory... > if none is given using the -o, --output option. This > default is at: > > * $USER/lttng-traces > > For example, this URL results in writing the trace data in > "$USER/lttng-traces//foo/bar". > > * net://hostname/foo/bar > > The PATH part can not be bigger than PATH_MAX (define in limits.h) which is > 4096 bytes at the time of this proposal. Moreover, "../" is ignored and > removed. For instance, using "net://localhost/../../" will set the path to the > default one. > > The protocol has a special case where the user can input two ports > respectively being the control and data port. > > * net://[HOSTNAME|IP][:CTRL_PORT][:DATA_PORT][/PATH] > > If _url_ is not NULL, the lttng_create_session will use two API calls. If _url_ is not NULL, in addition to creating the session, the lttng_create_session will use the two following API calls: > > 1) lttng_set_consumer_url(handle, url); > 2) lttng_enable_consumer(handle); > > If _url_ is NULL, then NO consumer is created for this tracing session and > subsequent calls are needed to set up a consumer (i.e lttng_enable_consumer and > lttng_set_consumer_url). > > [*] Consumer: > > This call is simply renamed. > > From: > --> lttng_set_consumer_uri(...) > > To: > --> lttng_set_consumer_url(struct lttng_handle *handle, > const char *url); > > For both functions (consumer and create), the _url_ will be parsed into a > lttng_uri in the liblttng-ctl and sent to the session daemon. > > Example: > > lttng_set_consumer_url(handle, "net://42.42.42.2"); > > > With all this, the lttng_uri data structure will NOT be exposed to the public > API and the user command line interface. The rest looks good! Thanks, Mathieu > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From dgoulet at efficios.com Tue Aug 7 15:21:48 2012 From: dgoulet at efficios.com (David Goulet) Date: Tue, 07 Aug 2012 15:21:48 -0400 Subject: [lttng-dev] [RFC] LTTng address API proposal v3 In-Reply-To: <20120807191915.GB454@Krystal> References: <502167DF.5000806@efficios.com> <20120807191915.GB454@Krystal> Message-ID: <50216ACC.4030208@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Mathieu Desnoyers: > * David Goulet (dgoulet at efficios.com) wrote: After a discussion > with Mathieu and Yannick, here is some big changes. > > I hope I did not forget anything important. > > Cheers! David > >> RFC - LTTng address API proposal >> >> Author: David Goulet >> >> Contributors: * Mathieu Desnoyers >> * Yannick Brosseau >> >> >> Version: - v0.1: 31/07/2012 * Initial proposal - v0.2: >> 07/08/0212 * Remove lttng_create_session_addr * Describe URL >> string format * Add set_consumer_url examples >> >> Introduction ----------------- >> >> This document proposes the use of string URLs to the command line >> interface and API which will deprecate a function and propose new >> ones. >> >> The purpose of this proposal is to support network streaming >> using URL string format that you can find in proposal >> doc/proposals/0003-network.consumer.txt, remove the lttng_uri >> structure from the API and integrate the URL string to the API. >> >> API ----------------- >> >> In order NOT to expose the new lttng_uri structure used to >> identify trace location for lttng consumer, the public API will >> only use string address where it will be converted in a lttng_uri >> and sent to the session daemon. >> >> [*] Create session: >> >> With the introduction of the enable-consumer command used for >> network streaming, the create session command has been modified >> so the user could define a consumer location either on the >> network or local with the command. >> >> This does NOT change the current API call but rather simply >> rename _path_ to _url_ and how it is used in the lttng-ctl >> library. >> >> Call changed from: --> lttng_create_session(const char *name, >> const char *path); >> >> To: --> lttng_create_session(const char *name, const char *url); >> >> The _name_ argument is the session name and _url_ is a string >> representing the URL specified by the user which is define like >> so: >> >> PROTO://[HOSTNAME|IP][:PORT][/PATH] >> >> The proto supported at this stage are (6 means IPv6): >> >> * net, net6, tcp, tcp6, file >> >> If the proto is NOT recognized, the string is considered to be a >> simple path on the local filesystem. > > you could specify that it is a path relative to the process CWD > (unless it starts with a /). Done > >> >> The PATH section is the destination path on the _remote_ host >> where the trace data will be written with the >> append after. > > The last sentence is not clear (after reading it 2-3 times, it's > still not obvious what it explains). > > Simply saying that the PATH is relative to a subdirectory > "hostname", under the remote relayd "virtual" root directory, might > be enough. > Done > >> The relayd defines a default location > > ... for its virtual root directory... Done > >> if none is given using the -o, --output option. This default is >> at: >> >> * $USER/lttng-traces >> >> For example, this URL results in writing the trace data in >> "$USER/lttng-traces//foo/bar". >> >> * net://hostname/foo/bar >> >> The PATH part can not be bigger than PATH_MAX (define in >> limits.h) which is 4096 bytes at the time of this proposal. >> Moreover, "../" is ignored and removed. For instance, using >> "net://localhost/../../" will set the path to the default one. >> >> The protocol has a special case where the user can input >> two ports respectively being the control and data port. >> >> * net://[HOSTNAME|IP][:CTRL_PORT][:DATA_PORT][/PATH] >> >> If _url_ is not NULL, the lttng_create_session will use two API >> calls. > > If _url_ is not NULL, in addition to creating the session, the > lttng_create_session will use the two following API calls: Done Eazy pizy. David > >> >> 1) lttng_set_consumer_url(handle, url); 2) >> lttng_enable_consumer(handle); >> >> If _url_ is NULL, then NO consumer is created for this tracing >> session and subsequent calls are needed to set up a consumer (i.e >> lttng_enable_consumer and lttng_set_consumer_url). >> >> [*] Consumer: >> >> This call is simply renamed. >> >> From: --> lttng_set_consumer_uri(...) >> >> To: --> lttng_set_consumer_url(struct lttng_handle *handle, const >> char *url); >> >> For both functions (consumer and create), the _url_ will be >> parsed into a lttng_uri in the liblttng-ctl and sent to the >> session daemon. >> >> Example: >> >> lttng_set_consumer_url(handle, "net://42.42.42.2"); >> >> >> With all this, the lttng_uri data structure will NOT be exposed >> to the public API and the user command line interface. > > The rest looks good! > > Thanks, > > Mathieu > > >> _______________________________________________ lttng-dev mailing >> list lttng-dev at lists.lttng.org >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQIWrMAAoJEELoaioR9I020qwH/RcBwYZUvR3u5u7BcrG1wp7K xgJ8HmYG9vTYSCMZv1dnMBu1aKCUwzQrwMTI/wi9qmM0oXHi7yk+XMQAbUlDmXoL 5vU4KtdejDyH3jPVMKsAfZRpKu+AghszsvqhUtPON6bM5jYXTWL4E8Wd29QhGS8O xnkl6WUVN0Cb/+I1m3sIJEJFkOjEuKjaoRgZd/fO9f/ZLV90/jId8/nr3Yiar6Sa mgIyl3mfEza8C/oeqBdEmGHSF2emAYEvHBZ9p2ewGfk0T57yVhMEoWGRG1o8pAQj 5BO6fJa+JJT62UAPnh3EFkveHLReoj0eVXVDJ2o2UNU20cTKiJR/TQWDD+CBKL0= =pOdd -----END PGP SIGNATURE----- From mathieu.desnoyers at efficios.com Tue Aug 7 15:42:49 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 7 Aug 2012 15:42:49 -0400 Subject: [lttng-dev] [BABELTRACE PATCH] Fix: wrong type in bt_ctf_get_uint64/int64 In-Reply-To: <1344368148-468-1-git-send-email-jdesfossez@efficios.com> References: <1344368148-468-1-git-send-email-jdesfossez@efficios.com> Message-ID: <20120807194249.GA1485@Krystal> * Julien Desfossez (jdesfossez at efficios.com) wrote: > Signed-off-by: Julien Desfossez merged, thanks to you both! Mathieu > --- > formats/ctf/events.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/formats/ctf/events.c b/formats/ctf/events.c > index 9693413..a8b08f6 100644 > --- a/formats/ctf/events.c > +++ b/formats/ctf/events.c > @@ -407,7 +407,7 @@ int bt_ctf_get_array_len(const struct definition *field) > > uint64_t bt_ctf_get_uint64(const struct definition *field) > { > - unsigned int ret = 0; > + uint64_t ret = 0; > > if (field && bt_ctf_field_type(field) == CTF_TYPE_INTEGER) > ret = get_unsigned_int(field); > @@ -419,7 +419,7 @@ uint64_t bt_ctf_get_uint64(const struct definition *field) > > int64_t bt_ctf_get_int64(const struct definition *field) > { > - int ret = 0; > + int64_t ret = 0; > > if (field && bt_ctf_field_type(field) == CTF_TYPE_INTEGER) > ret = get_signed_int(field); > -- > 1.7.10.4 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From laijs at cn.fujitsu.com Tue Aug 7 21:03:49 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Wed, 08 Aug 2012 09:03:49 +0800 Subject: [lttng-dev] urcu rfc: privatize urcu/futex.h In-Reply-To: <20120807140616.GA29280@Krystal> References: <5020DBFF.3020702@cn.fujitsu.com> <20120807140616.GA29280@Krystal> Message-ID: <5021BAF5.4080507@cn.fujitsu.com> On 08/07/2012 10:06 PM, Mathieu Desnoyers wrote: > Hi Lai, > > * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: >> futex.h is in urcu/, it means it can be used by users out of the library. >> >> But >> >> If the user's system has futex, he should use #include . >> If the user's system don't have futex, it is not good that if the user >> use this compat_futex. >> >> Because the compat_futex_async()'s behavior is different from the >> futex in linux. If the caller don't change the value of @uaddr, the >> futex(FUTEX_WAKE) in linux can also wake a thread, but >> compat_futex_async() don't. (I guess no one use this behavior, but if >> there are someone, we give them a wrong thing) > > Hrm, passing a NULL uaddr parameter is really not the targeted use-case. > We could probably just document that uaddr should not be NULL when using > compat_futex ? Sorry, I wanted to show you such un-normal behavior: /* keep the value of *uaddr unchanged, wake up a thread to do something */ futex(FUTEX_WAKE, uaddr, ...). This can't not wake up any thread for current compat_futex_async() implement. But the sys_futex() can. From laijs at cn.fujitsu.com Tue Aug 7 21:20:36 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Wed, 08 Aug 2012 09:20:36 +0800 Subject: [lttng-dev] [PATCH] urcu: fix compat_futex_noasync() Message-ID: <5021BEE4.6090907@cn.fujitsu.com> This patch fix two critical problems: 1) compat_futex_cond is not bound to any @uaddr, it services all @uaddr, if you wakeup only one thread(pthread_cond_signal), the @uaddr of this waking thread and the @uaddr of the woken-up thread may be different. the the woken-up thread will very probably go to sleep again because his own condition is not true. *And* this waking thread(FUTEX_WAKE) wake up NOTHING. 2) If the caller want to wake up all waiting threads, he will use INT_MAX for @val. and for (i = 0; i < INT_MAX; i++) pthread_cond_signal(&compat_futex_cond); becomes almost infinity loop. Signed-off-by: Lai Jiangshan --- diff --git a/compat_futex.c b/compat_futex.c index 04de596..bb928e6 100644 --- a/compat_futex.c +++ b/compat_futex.c @@ -43,7 +43,7 @@ static pthread_cond_t compat_futex_cond = PTHREAD_COND_INITIALIZER; int compat_futex_noasync(int32_t *uaddr, int op, int32_t val, const struct timespec *timeout, int32_t *uaddr2, int32_t val3) { - int ret, i, gret = 0; + int ret, gret = 0; /* * Check if NULL. Don't let users expect that they are taken into @@ -67,8 +67,7 @@ int compat_futex_noasync(int32_t *uaddr, int op, int32_t val, pthread_cond_wait(&compat_futex_cond, &compat_futex_lock); break; case FUTEX_WAKE: - for (i = 0; i < val; i++) - pthread_cond_signal(&compat_futex_cond); + pthread_cond_broadcast(&compat_futex_cond); break; default: gret = -EINVAL; From laijs at cn.fujitsu.com Tue Aug 7 21:32:21 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Wed, 08 Aug 2012 09:32:21 +0800 Subject: [lttng-dev] urcu rfc: privatize urcu/futex.h In-Reply-To: <5021BAF5.4080507@cn.fujitsu.com> References: <5020DBFF.3020702@cn.fujitsu.com> <20120807140616.GA29280@Krystal> <5021BAF5.4080507@cn.fujitsu.com> Message-ID: <5021C1A5.5090404@cn.fujitsu.com> (Added a CC email address) On 08/08/2012 09:03 AM, Lai Jiangshan wrote: > On 08/07/2012 10:06 PM, Mathieu Desnoyers wrote: >> Hi Lai, >> >> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: >>> futex.h is in urcu/, it means it can be used by users out of the library. >>> >>> But >>> >>> If the user's system has futex, he should use #include . >>> If the user's system don't have futex, it is not good that if the user >>> use this compat_futex. >>> >>> Because the compat_futex_async()'s behavior is different from the >>> futex in linux. If the caller don't change the value of @uaddr, the >>> futex(FUTEX_WAKE) in linux can also wake a thread, but >>> compat_futex_async() don't. (I guess no one use this behavior, but if >>> there are someone, we give them a wrong thing) >> >> Hrm, passing a NULL uaddr parameter is really not the targeted use-case. >> We could probably just document that uaddr should not be NULL when using >> compat_futex ? > > Sorry, I wanted to show you such un-normal behavior: > > /* keep the value of *uaddr unchanged, wake up a thread to do something */ > futex(FUTEX_WAKE, uaddr, ...). > > This can't not wake up any thread for current compat_futex_async() implement. > But the sys_futex() can. > > It is hard to implement compat_futex_async() to make its behavior compatible with sys_futex().(I don't like the two alternatives too, I just wrote what I have thought) so I sent rfc email to suggest to privatize urcu/futex.h. Thanks, Lai From Zheng.Chang at emc.com Tue Aug 7 22:31:04 2012 From: Zheng.Chang at emc.com (Chang, Zheng) Date: Tue, 7 Aug 2012 22:31:04 -0400 Subject: [lttng-dev] Uprobe and extractor for coredump Message-ID: <6539770C71C3814BB0BFC2DBEBD10508CC6967@CORPUSMX30B.corp.emc.com> Hi, Recently Linus merged uprobe into kernel mainline and perf has supported user-space dynamic tracing now. Is there any plan or motion on this with LTTng? I did some research on buffer management of LTTng. And I'm trying to figure out a way to extract trace info from coredump. Here is an immature idea about the extractor: - Resolve the addresses of necessary symbols from coredump with binary format library/tool - Restructure the data structures - Following the data structures, dump the buffer data page by page Any good ideas or suggestions or concerns? BR Zheng -------------- next part -------------- An HTML attachment was scrubbed... URL: From laijs at cn.fujitsu.com Wed Aug 8 04:31:12 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Wed, 8 Aug 2012 16:31:12 +0800 Subject: [lttng-dev] [PATCH 1/2] urcu: add hint to DEFINE_URCU_TLS() for compound types Message-ID: <1344414673-14714-1-git-send-email-laijs@cn.fujitsu.com> Just a hint. Signed-off-by: Lai Jiangshan --- urcu/tls-compat.h | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-) diff --git a/urcu/tls-compat.h b/urcu/tls-compat.h index 9686eca..192a536 100644 --- a/urcu/tls-compat.h +++ b/urcu/tls-compat.h @@ -34,6 +34,21 @@ extern "C" { #ifdef CONFIG_RCU_TLS /* Based on ax_tls.m4 */ +/* + * Hint: How to define/declare TLS variables of compound types + * such as array or function pointers? + * + * Answer: Use typedef to assign a type_name to the compound type. + * Example: Define a TLS variable which is an int array with len=4: + * + * typedef int my_int_array_type[4]; + * DEFINE_URCU_TLS(my_int_array_type, var_name); + * + * Another exmaple: + * typedef void (*call_rcu_flavor)(struct rcu_head *, XXXX); + * DECLARE_URCU_TLS(call_rcu_flavor, p_call_rcu); + */ + # define DECLARE_URCU_TLS(type, name) \ CONFIG_RCU_TLS type name -- 1.7.4.4 From laijs at cn.fujitsu.com Wed Aug 8 04:31:13 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Wed, 8 Aug 2012 16:31:13 +0800 Subject: [lttng-dev] [PATCH 2/2] urcu: add notice to URCU_TLS() for it is not async-signal-safe In-Reply-To: <1344414673-14714-1-git-send-email-laijs@cn.fujitsu.com> References: <1344414673-14714-1-git-send-email-laijs@cn.fujitsu.com> Message-ID: <1344414673-14714-2-git-send-email-laijs@cn.fujitsu.com> Signed-off-by: Lai Jiangshan --- urcu/tls-compat.h | 14 ++++++++++++++ 1 files changed, 14 insertions(+), 0 deletions(-) diff --git a/urcu/tls-compat.h b/urcu/tls-compat.h index 192a536..b7bf363 100644 --- a/urcu/tls-compat.h +++ b/urcu/tls-compat.h @@ -59,6 +59,20 @@ extern "C" { #else /* #ifndef CONFIG_RCU_TLS */ +/* + * NOTE: URCU_TLS() is NOT async-signal-safe, you can't use it + * inside any function which can be called from signal handler. + * + * But if pthread_getspecific() is async-signal-safe in your + * platform, you can make URCU_TLS() async-signal-safe via: + * ensuring the first call to URCU_TLS() of a given TLS variable of + * all threads is called earliest from a non-signal handler function. + * + * Exmaple: In any thread, the first call of URCU_TLS(rcu_reader) + * is called from rcu_register_thread(), so we can ensure all later + * URCU_TLS(rcu_reader) in any thread is async-signal-safe. + */ + # include struct urcu_tls { -- 1.7.4.4 From dgoulet at efficios.com Wed Aug 8 13:17:21 2012 From: dgoulet at efficios.com (David Goulet) Date: Wed, 08 Aug 2012 13:17:21 -0400 Subject: [lttng-dev] [RFC] LTTng address API proposal v3 In-Reply-To: <502167DF.5000806@efficios.com> References: <502167DF.5000806@efficios.com> Message-ID: <50229F21.4080400@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 ADDENDUM: The set_consumer_url() call needs an extra arguments since we can both set the CONTROL or STREAM path using this call (the stream type). So, I would like to propose a change: From: lttng_set_consumer_url(struct lttng_handle *handle, const char *url); To: lttng_set_consumer_url(struct lttng_handle *handle, const char *url, enum lttng_url_stream_type stype); With: enum lttng_url_stream_type { LTTNG_STREAM_CONTROL, LTTNG_STREAM_DATA, LTTNG_STREAM_ALL, }; An _url_ argument with "net://localhost" for instance implies LTTNG_STREAM_ALL as a type however with "tcp://", we have to specify it. The second possible idea is to add specialize calls like: set_consumer_control_url() set_consumer_stream_url() Though? Cheers! David David Goulet: > After a discussion with Mathieu and Yannick, here is some big > changes. > > I hope I did not forget anything important. > > Cheers! David > > > > This body part will be downloaded on demand. -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQIp8aAAoJEELoaioR9I02cBYIAJYI9cFpWwTUq7kAImWB/ZAZ 20NLdDhjvYkHkLdHs76++P5d5DowHYZBY4dFrUKDlRTA5INQ/Q3OeZMCiCs9HkDM i4JZCJdIrgI6Y1Wu5aVKU3kjN11z+169qAx4YYarKoXsordWcPzsBfPazRsJNfVy zAhcY0LQFOgxRDnxAg7zxVIQFimYS/OZBM976fRCRxwV6YBNscEQj9OYgBq8nCOm 76vR1p5WPb93jX/wqe2/ilHAhRslVWedn0p+OT1kNWW0puPyWf2NHxY/KqqhkhAy yFkMeVXJtvnGig1UPgm7H7eouCRAAM04aS6qCVfZz9vBJALEk42UVbcLPO6GzQg= =VobI -----END PGP SIGNATURE----- From danny.serres at efficios.com Wed Aug 8 14:56:28 2012 From: danny.serres at efficios.com (Danny Serres) Date: Wed, 8 Aug 2012 14:56:28 -0400 Subject: [lttng-dev] [lttng-tools PATCH] lttng-tools python module v2 Message-ID: <1344452188-7579-1-git-send-email-danny.serres@efficios.com> The lttng-tools Python module can be used to directly control the lttng-tools API inside Python, using 'import lttng'. Therefore, it becomes possible to create a trace, add events, start/stop tracing, destroy a session and so on from within Python. The module does not include URI-related functions. SWIG >= 2.0 is used to create the wrapper and its 'warning md variable unused' bug is patched in Makefile.am. In the interface file, struct and enum are directly copied from lttng.h (all changes must be made in both files). Signed-off-by: Danny Serres Signed-off-by: Yannick Brosseau --- .gitignore | 4 + Makefile.am | 1 + README | 18 + config/ax_pkg_swig.m4 | 135 ++++ configure.ac | 46 ++ doc/python-howto.txt | 70 ++ extras/Makefile.am | 1 + extras/bindings/Makefile.am | 1 + extras/bindings/swig/Makefile.am | 3 + extras/bindings/swig/python/Makefile.am | 25 + extras/bindings/swig/python/lttng.i.in | 1029 ++++++++++++++++++++++++++ extras/bindings/swig/python/tests/example.py | 109 +++ extras/bindings/swig/python/tests/run.sh | 1 + extras/bindings/swig/python/tests/tests.py | 310 ++++++++ 14 files changed, 1753 insertions(+) create mode 100644 config/ax_pkg_swig.m4 create mode 100644 doc/python-howto.txt create mode 100644 extras/Makefile.am create mode 100644 extras/bindings/Makefile.am create mode 100644 extras/bindings/swig/Makefile.am create mode 100644 extras/bindings/swig/python/Makefile.am create mode 100644 extras/bindings/swig/python/lttng.i.in create mode 100644 extras/bindings/swig/python/tests/example.py create mode 100644 extras/bindings/swig/python/tests/run.sh create mode 100644 extras/bindings/swig/python/tests/tests.py diff --git a/.gitignore b/.gitignore index 6d04272..c94f759 100644 --- a/.gitignore +++ b/.gitignore @@ -42,6 +42,10 @@ src/lib/lttng-ctl/filter-parser.c src/lib/lttng-ctl/filter-parser.h src/lib/lttng-ctl/filter-parser.output +extras/bindings/swig/python/lttng.i +extras/bindings/swig/python/lttng.py +extras/bindings/swig/python/lttng_wrap.c + # Tests test_sessions test_kernel_data_trace diff --git a/Makefile.am b/Makefile.am index 13d3f93..b0537ce 100644 --- a/Makefile.am +++ b/Makefile.am @@ -2,6 +2,7 @@ ACLOCAL_AMFLAGS = -I config SUBDIRS = src \ tests \ + extras \ include \ doc diff --git a/README b/README index fe9e4c2..9b91288 100644 --- a/README +++ b/README @@ -28,6 +28,16 @@ REQUIREMENTS: * Debian/Ubuntu package: libpopt-dev + - SWIG >= 2.0 + Needed for Python bindings + + * Debian/Ubuntu package: swig2.0 + + - python-dev + Python headers + + * Debian/Ubuntu package: python-dev + - For kernel tracing: modprobe For developers using the git tree: @@ -62,6 +72,8 @@ INSTALLATION INSTRUCTIONS: If compiling from the git repository, run ./bootstrap before running the configure script, to generate it. + If you do not want Python bindings, run ./configure --disable-python. + USAGE: Please see doc/quickstart.txt to help you start tracing. You can also use the @@ -71,6 +83,9 @@ lttng enable-event -h). A network streaming HOWTO can be found in doc/streaming-howto.txt which quickly helps you understand how to stream a LTTng 2.0 trace. +A Python HOWTO can be found in doc/python-howto.txt which quickly +helps you understand how to use the Python module to control the LTTng API. + PACKAGE CONTENTS: This package contains the following elements: @@ -121,6 +136,9 @@ PACKAGE CONTENTS: - include (lttng.h --> installed in $(includedir)/lttng/lttng.h) The liblttngctl API header file. + - python + The Python bindings + - tests Various test programs. diff --git a/config/ax_pkg_swig.m4 b/config/ax_pkg_swig.m4 new file mode 100644 index 0000000..e112f3d --- /dev/null +++ b/config/ax_pkg_swig.m4 @@ -0,0 +1,135 @@ +# =========================================================================== +# http://www.gnu.org/software/autoconf-archive/ax_pkg_swig.html +# =========================================================================== +# +# SYNOPSIS +# +# AX_PKG_SWIG([major.minor.micro], [action-if-found], [action-if-not-found]) +# +# DESCRIPTION +# +# This macro searches for a SWIG installation on your system. If found, +# then SWIG is AC_SUBST'd; if not found, then $SWIG is empty. If SWIG is +# found, then SWIG_LIB is set to the SWIG library path, and AC_SUBST'd. +# +# You can use the optional first argument to check if the version of the +# available SWIG is greater than or equal to the value of the argument. It +# should have the format: N[.N[.N]] (N is a number between 0 and 999. Only +# the first N is mandatory.) If the version argument is given (e.g. +# 1.3.17), AX_PKG_SWIG checks that the swig package is this version number +# or higher. +# +# As usual, action-if-found is executed if SWIG is found, otherwise +# action-if-not-found is executed. +# +# In configure.in, use as: +# +# AX_PKG_SWIG(1.3.17, [], [ AC_MSG_ERROR([SWIG is required to build..]) ]) +# AX_SWIG_ENABLE_CXX +# AX_SWIG_MULTI_MODULE_SUPPORT +# AX_SWIG_PYTHON +# +# LICENSE +# +# Copyright (c) 2008 Sebastian Huber +# Copyright (c) 2008 Alan W. Irwin +# Copyright (c) 2008 Rafael Laboissiere +# Copyright (c) 2008 Andrew Collier +# Copyright (c) 2011 Murray Cumming +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation; either version 2 of the License, or (at your +# option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program. If not, see . +# +# As a special exception, the respective Autoconf Macro's copyright owner +# gives unlimited permission to copy, distribute and modify the configure +# scripts that are the output of Autoconf when processing the Macro. You +# need not follow the terms of the GNU General Public License when using +# or distributing such scripts, even though portions of the text of the +# Macro appear in them. The GNU General Public License (GPL) does govern +# all other use of the material that constitutes the Autoconf Macro. +# +# This special exception to the GPL applies to versions of the Autoconf +# Macro released by the Autoconf Archive. When you make and distribute a +# modified version of the Autoconf Macro, you may extend this special +# exception to the GPL to apply to your modified version as well. + +#serial 8 + +AC_DEFUN([AX_PKG_SWIG],[ + # Ubuntu has swig 2.0 as /usr/bin/swig2.0 + AC_PATH_PROGS([SWIG],[swig swig2.0]) + if test -z "$SWIG" ; then + m4_ifval([$3],[$3],[:]) + elif test -n "$1" ; then + AC_MSG_CHECKING([SWIG version]) + [swig_version=`$SWIG -version 2>&1 | grep 'SWIG Version' | sed 's/.*\([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\).*/\1/g'`] + AC_MSG_RESULT([$swig_version]) + if test -n "$swig_version" ; then + # Calculate the required version number components + [required=$1] + [required_major=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_major" ; then + [required_major=0] + fi + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] + [required_minor=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_minor" ; then + [required_minor=0] + fi + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] + [required_patch=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_patch" ; then + [required_patch=0] + fi + # Calculate the available version number components + [available=$swig_version] + [available_major=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_major" ; then + [available_major=0] + fi + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] + [available_minor=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_minor" ; then + [available_minor=0] + fi + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] + [available_patch=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_patch" ; then + [available_patch=0] + fi + # Convert the version tuple into a single number for easier comparison. + # Using base 100 should be safe since SWIG internally uses BCD values + # to encode its version number. + required_swig_vernum=`expr $required_major \* 10000 \ + \+ $required_minor \* 100 \+ $required_patch` + available_swig_vernum=`expr $available_major \* 10000 \ + \+ $available_minor \* 100 \+ $available_patch` + + if test $available_swig_vernum -lt $required_swig_vernum; then + AC_MSG_WARN([SWIG version >= $1 is required. You have $swig_version.]) + SWIG='' + m4_ifval([$3],[$3],[]) + else + AC_MSG_CHECKING([for SWIG library]) + SWIG_LIB=`$SWIG -swiglib` + AC_MSG_RESULT([$SWIG_LIB]) + m4_ifval([$2],[$2],[]) + fi + else + AC_MSG_WARN([cannot determine SWIG version]) + SWIG='' + m4_ifval([$3],[$3],[]) + fi + fi + AC_SUBST([SWIG_LIB]) +]) diff --git a/configure.ac b/configure.ac index 17e6b67..267dad1 100644 --- a/configure.ac +++ b/configure.ac @@ -154,6 +154,40 @@ AC_CHECK_LIB([c], [open_memstream], ] ) +# For Python +# SWIG version needed or newer: +swig_version=2.0.0 + +AC_ARG_ENABLE([python], + [AC_HELP_STRING([--disable-python], + [do not compile Python bindings])], + [], [enable_python=yes]) + +AM_CONDITIONAL([USE_PYTHON], [test "x${enable_python:-yes}" = xyes]) + +if test "x${enable_python:-yes}" = xyes; then + AC_MSG_NOTICE([You may configure with --disable-python ]dnl +[if you do not want Python bindings.]) + + AX_PKG_SWIG($swig_version, [], [ AC_MSG_ERROR([SWIG $swig_version or newer is needed]) ]) + AM_PATH_PYTHON + + AC_ARG_VAR([PYTHON_INCLUDE], [Include flags for python, bypassing python-config]) + AC_ARG_VAR([PYTHON_CONFIG], [Path to python-config]) + AS_IF([test -z "$PYTHON_INCLUDE"], [ + AS_IF([test -z "$PYTHON_CONFIG"], [ + AC_PATH_PROGS([PYTHON_CONFIG], + [python$PYTHON_VERSION-config python-config], + [no], + [`dirname $PYTHON`]) + AS_IF([test "$PYTHON_CONFIG" = no], [AC_MSG_ERROR([cannot find python-config for $PYTHON.])]) + ]) + AC_MSG_CHECKING([python include flags]) + PYTHON_INCLUDE=`$PYTHON_CONFIG --includes` + AC_MSG_RESULT([$PYTHON_INCLUDE]) + ]) +fi + # Option to only build the consumer daemon and its libraries AC_ARG_WITH([consumerd-only], AS_HELP_STRING([--with-consumerd-only],[Only build the consumer daemon [default=no]]), @@ -198,6 +232,10 @@ AC_CONFIG_FILES([ doc/Makefile doc/man/Makefile include/Makefile + extras/Makefile + extras/bindings/Makefile + extras/bindings/swig/Makefile + extras/bindings/swig/python/Makefile src/Makefile src/common/Makefile src/common/kernel-ctl/Makefile @@ -260,6 +298,14 @@ AS_IF([test "x$lttng_ust_support" = "xyes"],[ AS_ECHO("Disabled") ]) +#Python binding enabled/disabled +AS_ECHO_N("Python binding: ") +AS_IF([test "x${enable_python:-yes}" = xyes], [ + AS_ECHO("Enabled") +],[ + AS_ECHO("Disabled") +]) + # Do we build only the consumerd, or everything AS_IF([test "x$consumerd_only" = "xyes"],[ AS_ECHO("Only the consumerd daemon will be built.") diff --git a/doc/python-howto.txt b/doc/python-howto.txt new file mode 100644 index 0000000..796c5b9 --- /dev/null +++ b/doc/python-howto.txt @@ -0,0 +1,70 @@ +PYTHON BINDINGS +---------------- + +This is a brief howto for using the lttng-tools Python module. + +By default, the Python bindings are installed. +If you do not wish the Python bindings, you can configure with the +--disable-python option during the installation procedure: + + $ ./configure --disable-python + +The Python module is automatically generated using SWIG, therefore the +swig2.0 package on Debian/Ubuntu is requied. + +Once installed, the Python module can be used by importing it in Python. +In the Python interpreter: + + >>> import lttng + +Example: +---------------- + +Quick example using Python to trace with LTTng. + +1) Run python + + $ python + +2) Import the lttng module + + >>> import lttng + +3) Create a session + + >>> lttng.create("session-name", "path/to/trace") + +4) Create a handle for the tracing session and domain + + >>> domain = lttng.Domain() + >>> domain.type = lttng.DOMAIN_KERNEL * + >>> handle = lttng.Handle("session-name", domain) + +* This line is somewhat useless since domain.type is set to 0 + by default, the corresponding value of lttng.DOMAIN_KERNEL + +5) Enable all Kernel events + + >>> event = lttng.Event() + >>> event.type = lttng.EVENT_TRACEPOINT * + >>> event.loglevel_type = lttng.EVENT_LOGLEVEL_ALL * + >>> lttng.enable_event(handle, event, None) + +* These two lines are somewhat useless since event.type + and event.loglevel_type are by default set to 0, the + corresponding value of lttng.EVENT_TRACEPOINT and + lttng.EVENT_LOGLEVEL_ALL + +5) Start tracing + + >>> lttng.start("session-name") + +6) Stop tracing + + >>> lttng.stop("session-name") + +7) Destroy the tracing session + + >>> lttng.destroy("session-name") + +For an example script with more details, see extras/bindings/swig/python/tests/example.py diff --git a/extras/Makefile.am b/extras/Makefile.am new file mode 100644 index 0000000..925dc2e --- /dev/null +++ b/extras/Makefile.am @@ -0,0 +1 @@ +SUBDIRS = bindings diff --git a/extras/bindings/Makefile.am b/extras/bindings/Makefile.am new file mode 100644 index 0000000..1585422 --- /dev/null +++ b/extras/bindings/Makefile.am @@ -0,0 +1 @@ +SUBDIRS = swig diff --git a/extras/bindings/swig/Makefile.am b/extras/bindings/swig/Makefile.am new file mode 100644 index 0000000..dcd868d --- /dev/null +++ b/extras/bindings/swig/Makefile.am @@ -0,0 +1,3 @@ +if USE_PYTHON +SUBDIRS = python +endif diff --git a/extras/bindings/swig/python/Makefile.am b/extras/bindings/swig/python/Makefile.am new file mode 100644 index 0000000..85237a4 --- /dev/null +++ b/extras/bindings/swig/python/Makefile.am @@ -0,0 +1,25 @@ +lttng.i: lttng.i.in + sed "s/LTTNG_VERSION_STR/LTTng $(PACKAGE_VERSION)/g" lttng.i + +AM_CFLAGS = -I$(PYTHON_INCLUDE) -I$(top_srcdir)/lib/lttng-ctl -I../common \ + $(BUDDY_CFLAGS) + +EXTRA_DIST = lttng.i +python_PYTHON = lttng.py +pyexec_LTLIBRARIES = _lttng.la + +MAINTAINERCLEANFILES = lttng_wrap.c lttng.py + +_lttng_la_SOURCES = lttng_wrap.c + +_lttng_la_LDFLAGS = -module + +_lttng_la_LIBADD = $(top_srcdir)/src/lib/lttng-ctl/liblttng-ctl.la \ + $(top_srcdir)/src/common/sessiond-comm/libsessiond-comm.la + +# SWIG 'warning md variable unused' fixed after SWIG build: +lttng_wrap.c: lttng.i + $(SWIG) -python -I. -I$(top_srcdir)/src/common/sessiond-comm lttng.i + sed -i "s/PyObject \*m, \*d, \*md;/PyObject \*m, \*d;\n#if defined(SWIGPYTHON_BUILTIN)\nPyObject *md;\n#endif/g" lttng_wrap.c + sed -i "s/md = d/d/g" lttng_wrap.c + sed -i "s/(void)public_symbol;/(void)public_symbol;\n md = d;/g" lttng_wrap.c diff --git a/extras/bindings/swig/python/lttng.i.in b/extras/bindings/swig/python/lttng.i.in new file mode 100644 index 0000000..0d6d1e9 --- /dev/null +++ b/extras/bindings/swig/python/lttng.i.in @@ -0,0 +1,1029 @@ +%define DOCSTRING +"LTTNG_VERSION_STR + +The LTTng project aims at providing highly efficient tracing tools for Linux. +It's tracers help tracking down performance issues and debugging problems involving +multiple concurrent processes and threads. Tracing across multiple systems is also possible." +%enddef + +%module(docstring=DOCSTRING) lttng + +%include "typemaps.i" +%include "pyabc.i" +%{ +#define SWIG_FILE_WITH_INIT +#include +%} + +typedef unsigned int uint32_t; +typedef int int32_t; +typedef unsigned long long uint64_t; +typedef long pid_t; + + +// ============================================= +// ENUMS +// These are directly taken from lttng.h. +// Any change to these enums must also be +// made here. +// ============================================= + +%rename("DOMAIN_KERNEL") LTTNG_DOMAIN_KERNEL; +%rename("DOMAIN_UST") LTTNG_DOMAIN_UST; +enum lttng_domain_type { + LTTNG_DOMAIN_KERNEL = 1, + LTTNG_DOMAIN_UST = 2, +}; + +%rename("EVENT_ALL") LTTNG_EVENT_ALL; +%rename("EVENT_TRACEPOINT") LTTNG_EVENT_TRACEPOINT; +%rename("EVENT_PROBE") LTTNG_EVENT_PROBE; +%rename("EVENT_FUNCTION")LTTNG_EVENT_FUNCTION; +%rename("EVENT_FUNCTION_ENTRY") LTTNG_EVENT_FUNCTION_ENTRY; +%rename("EVENT_NOOP") LTTNG_EVENT_NOOP; +%rename("EVENT_SYSCALL") LTTNG_EVENT_SYSCALL; +enum lttng_event_type { + LTTNG_EVENT_ALL = -1, + LTTNG_EVENT_TRACEPOINT = 0, + LTTNG_EVENT_PROBE = 1, + LTTNG_EVENT_FUNCTION = 2, + LTTNG_EVENT_FUNCTION_ENTRY = 3, + LTTNG_EVENT_NOOP = 4, + LTTNG_EVENT_SYSCALL = 5, +}; + +%rename("EVENT_LOGLEVEL_ALL") LTTNG_EVENT_LOGLEVEL_ALL; +%rename("EVENT_LOGLEVEL_RANGE") LTTNG_EVENT_LOGLEVEL_RANGE; +%rename("EVENT_LOGLEVEL_SINGLE") LTTNG_EVENT_LOGLEVEL_SINGLE; +enum lttng_loglevel_type { + LTTNG_EVENT_LOGLEVEL_ALL = 0, + LTTNG_EVENT_LOGLEVEL_RANGE = 1, + LTTNG_EVENT_LOGLEVEL_SINGLE = 2, +}; + +%rename("LOGLEVEL_EMERG") LTTNG_LOGLEVEL_EMERG; +%rename("LOGLEVEL_ALERT") LTTNG_LOGLEVEL_ALERT; +%rename("LOGLEVEL_CRIT") LTTNG_LOGLEVEL_CRIT; +%rename("LOGLEVEL_ERR") LTTNG_LOGLEVEL_ERR; +%rename("LOGLEVEL_WARNING") LTTNG_LOGLEVEL_WARNING; +%rename("LOGLEVEL_NOTICE") LTTNG_LOGLEVEL_NOTICE; +%rename("LOGLEVEL_INFO") LTTNG_LOGLEVEL_INFO; +%rename("LOGLEVEL_DEBUG_SYSTEM") LTTNG_LOGLEVEL_DEBUG_SYSTEM; +%rename("LOGLEVEL_DEBUG_PROGRAM") LTTNG_LOGLEVEL_DEBUG_PROGRAM; +%rename("LOGLEVEL_DEBUG_PROCESS") LTTNG_LOGLEVEL_DEBUG_PROCESS; +%rename("LOGLEVEL_DEBUG_MODULE") LTTNG_LOGLEVEL_DEBUG_MODULE; +%rename("LOGLEVEL_DEBUG_UNIT") LTTNG_LOGLEVEL_DEBUG_UNIT; +%rename("LOGLEVEL_DEBUG_FUNCTION") LTTNG_LOGLEVEL_DEBUG_FUNCTION; +%rename("LOGLEVEL_DEBUG_LINE") LTTNG_LOGLEVEL_DEBUG_LINE; +%rename("LOGLEVEL_DEBUG") LTTNG_LOGLEVEL_DEBUG; +enum lttng_loglevel { + LTTNG_LOGLEVEL_EMERG = 0, + LTTNG_LOGLEVEL_ALERT = 1, + LTTNG_LOGLEVEL_CRIT = 2, + LTTNG_LOGLEVEL_ERR = 3, + LTTNG_LOGLEVEL_WARNING = 4, + LTTNG_LOGLEVEL_NOTICE = 5, + LTTNG_LOGLEVEL_INFO = 6, + LTTNG_LOGLEVEL_DEBUG_SYSTEM = 7, + LTTNG_LOGLEVEL_DEBUG_PROGRAM = 8, + LTTNG_LOGLEVEL_DEBUG_PROCESS = 9, + LTTNG_LOGLEVEL_DEBUG_MODULE = 10, + LTTNG_LOGLEVEL_DEBUG_UNIT = 11, + LTTNG_LOGLEVEL_DEBUG_FUNCTION = 12, + LTTNG_LOGLEVEL_DEBUG_LINE = 13, + LTTNG_LOGLEVEL_DEBUG = 14, +}; + +%rename("EVENT_SPLICE") LTTNG_EVENT_SPLICE; +%rename("EVENT_MMAP") LTTNG_EVENT_MMAP; +enum lttng_event_output { + LTTNG_EVENT_SPLICE = 0, + LTTNG_EVENT_MMAP = 1, +}; + +%rename("EVENT_CONTEXT_PID") LTTNG_EVENT_CONTEXT_PID; +%rename("EVENT_CONTEXT_PERF_COUNTER") LTTNG_EVENT_CONTEXT_PERF_COUNTER; +%rename("EVENT_CONTEXT_PROCNAME") LTTNG_EVENT_CONTEXT_PROCNAME; +%rename("EVENT_CONTEXT_PRIO") LTTNG_EVENT_CONTEXT_PRIO; +%rename("EVENT_CONTEXT_NICE") LTTNG_EVENT_CONTEXT_NICE; +%rename("EVENT_CONTEXT_VPID") LTTNG_EVENT_CONTEXT_VPID; +%rename("EVENT_CONTEXT_TID") LTTNG_EVENT_CONTEXT_TID; +%rename("EVENT_CONTEXT_VTID") LTTNG_EVENT_CONTEXT_VTID; +%rename("EVENT_CONTEXT_PPID") LTTNG_EVENT_CONTEXT_PPID; +%rename("EVENT_CONTEXT_VPPID") LTTNG_EVENT_CONTEXT_VPPID; +%rename("EVENT_CONTEXT_PTHREAD_ID") LTTNG_EVENT_CONTEXT_PTHREAD_ID; +enum lttng_event_context_type { + LTTNG_EVENT_CONTEXT_PID = 0, + LTTNG_EVENT_CONTEXT_PERF_COUNTER = 1, + LTTNG_EVENT_CONTEXT_PROCNAME = 2, + LTTNG_EVENT_CONTEXT_PRIO = 3, + LTTNG_EVENT_CONTEXT_NICE = 4, + LTTNG_EVENT_CONTEXT_VPID = 5, + LTTNG_EVENT_CONTEXT_TID = 6, + LTTNG_EVENT_CONTEXT_VTID = 7, + LTTNG_EVENT_CONTEXT_PPID = 8, + LTTNG_EVENT_CONTEXT_VPPID = 9, + LTTNG_EVENT_CONTEXT_PTHREAD_ID = 10, +}; + +%rename("CALIBRATE_FUNCTION") LTTNG_CALIBRATE_FUNCTION; +enum lttng_calibrate_type { + LTTNG_CALIBRATE_FUNCTION = 0, +}; + + + +// ============================================= +// TYPEMAPS +// ============================================= + +//list_sessions +%typemap(argout) struct lttng_session **sessions{ + + int l = PyInt_AsSsize_t($result); + if (l >= 0) + { + PyObject *sessions = PyList_New(0); + int i; + for(i=0; i= 0) + { + PyObject *dom = PyList_New(0); + int i; + for(i=0; i= 0) + { + PyObject *chan = PyList_New(0); + int i; + for(i=0; i= 0) + { + PyObject *events = PyList_New(0); + int i; + for(i=0; i int + +Create a new tracing session using name and path. +Returns size of returned session payload data or a negative error code." +int lttng_create_session(const char *name, const char *path); + + +%feature("docstring")"destroy(str name) -> int + +Tear down tracing session using name. +Returns size of returned session payload data or a negative error code." +int lttng_destroy_session(const char *name); + + +%feature("docstring")"session_daemon_alive() -> int + +Check if session daemon is alive. +Return 1 if alive or 0 if not. +On error returns a negative value." +int lttng_session_daemon_alive(void); + + +%feature("docstring")"set_tracing_group(str name) -> int + +Sets the tracing_group variable with name. +This function allocates memory pointed to by tracing_group. +On success, returns 0, on error, returns -1 (null name) or -ENOMEM." +int lttng_set_tracing_group(const char *name); + + +%feature("docstring")"strerror(int code) -> char + +Returns a human readable string describing +the error code (a negative value)." +const char *lttng_strerror(int code); + + +%feature("docstring")"start(str session_name) -> int + +Start tracing for all traces of the session. +Returns size of returned session payload data or a negative error code." +int lttng_start_tracing(const char *session_name); + + +%feature("docstring")"stop(str session_name) -> int + +Stop tracing for all traces of the session. +Returns size of returned session payload data or a negative error code." +int lttng_stop_tracing(const char *session_name); + + +%feature("docstring")"channel_set_default_attr(Domain domain, ChannelAttr attr) + +Set default channel attributes. +If either or both of the arguments are null, attr content is zeroe'd." +void lttng_channel_set_default_attr(struct lttng_domain *domain, struct lttng_channel_attr *attr); + + +// ============================================= +// Python redefinition of some functions +// (List and Handle-related) +// ============================================= + +%feature("docstring")"" +%pythoncode %{ + +def list_sessions(): + """ + list_sessions() -> dict + + Ask the session daemon for all available sessions. + Returns a dict of Session instances, the key is the name; + on error, returns a negative value. + """ + + ses_list = _lttng_list_sessions() + if type(ses_list) is int: + return ses_list + + sessions = {} + + for ses_elements in ses_list: + ses = Session() + ses.name = ses_elements[0] + ses.path = ses_elements[1] + ses.enabled = ses_elements[2] + ses.padding = ses_elements[3] + + sessions[ses.name] = ses + + return sessions + + +def list_domains(session_name): + """ + list_domains(str session_name) -> list + + Ask the session daemon for all available domains of a session. + Returns a list of Domain instances; + on error, returns a negative value. + """ + + dom_list = _lttng_list_domains(session_name) + if type(dom_list) is int: + return dom_list + + domains = [] + + for dom_elements in dom_list: + dom = Domain() + dom.type = dom_elements[0] + dom.paddinf = dom_elements[1] + dom.attr.pid = dom_elements[2] + dom.attr.exec_name = dom_elements[3] + dom.attr.padding = dom_elements[4] + + domains.append(dom) + + return domains + + +def list_channels(handle): + """ + list_channels(Handle handle) -> dict + + Ask the session daemon for all available channels of a session. + Returns a dict of Channel instances, the key is the name; + on error, returns a negative value. + """ + + try: + chan_list = _lttng_list_channels(handle._h) + except AttributeError: + raise TypeError("in method 'list_channels', argument 1 must be a Handle instance") + + if type(chan_list) is int: + return chan_list + + channels = {} + + for channel_elements in chan_list: + chan = Channel() + chan.name = channel_elements[0] + chan.enabled = channel_elements[1] + chan.padding = channel_elements[2] + chan.attr.overwrite = channel_elements[3][0] + chan.attr.subbuf_size = channel_elements[3][1] + chan.attr.num_subbuf = channel_elements[3][2] + chan.attr.switch_timer_interval = channel_elements[3][3] + chan.attr.read_timer_interval = channel_elements[3][4] + chan.attr.output = channel_elements[3][5] + chan.attr.padding = channel_elements[3][6] + + channels[chan.name] = chan + + return channels + + +def list_events(handle, channel_name): + """ + list_events(Handle handle, str channel_name) -> dict + + Ask the session daemon for all available events of a session channel. + Returns a dict of Event instances, the key is the name; + on error, returns a negative value. + """ + + try: + ev_list = _lttng_list_events(handle._h, channel_name) + except AttributeError: + raise TypeError("in method 'list_events', argument 1 must be a Handle instance") + + if type(ev_list) is int: + return ev_list + + events = {} + + for ev_elements in ev_list: + ev = Event() + ev.name = ev_elements[0] + ev.type = ev_elements[1] + ev.loglevel_type = ev_elements[2] + ev.loglevel = ev_elements[3] + ev.enabled = ev_elements[4] + ev.pid = ev_elements[5] + ev.attr.padding = ev_elements[6] + ev.attr.probe.addr = ev_elements[7][0] + ev.attr.probe.offset = ev_elements[7][1] + ev.attr.probe.symbol_name = ev_elements[7][2] + ev.attr.probe.padding = ev_elements[7][3] + ev.attr.ftrace.symbol_name = ev_elements[8][0] + ev.attr.ftrace.padding = ev_elements[8][1] + ev.attr.padding = ev_elements[9] + + events[ev.name] = ev + + return events + + +def list_tracepoints(handle): + """ + list_tracepoints(Handle handle) -> dict + + Returns a dict of Event instances, the key is the name; + on error, returns a negative value. + """ + + try: + ev_list = _lttng_list_tracepoints(handle._h) + except AttributeError: + raise TypeError("in method 'list_tracepoints', argument 1 must be a Handle instance") + + if type(ev_list) is int: + return ev_list + + events = {} + + for ev_elements in ev_list: + ev = Event() + ev.name = ev_elements[0] + ev.type = ev_elements[1] + ev.loglevel_type = ev_elements[2] + ev.loglevel = ev_elements[3] + ev.enabled = ev_elements[4] + ev.pid = ev_elements[5] + ev.attr.padding = ev_elements[6] + ev.attr.probe.addr = ev_elements[7][0] + ev.attr.probe.offset = ev_elements[7][1] + ev.attr.probe.symbol_name = ev_elements[7][2] + ev.attr.probe.padding = ev_elements[7][3] + ev.attr.ftrace.symbol_name = ev_elements[8][0] + ev.attr.ftrace.padding = ev_elements[8][1] + ev.attr.padding = ev_elements[9] + + events[ev.name] = ev + + return events + + +def register_consumer(handle, socket_path): + """ + register_consumer(Handle handle, str socket_path) -> int + + Register an outside consumer. + Returns size of returned session payload data or a negative error code. + """ + + try: + return _lttng_register_consumer(handle._h, socket_path) + except AttributeError: + raise TypeError("in method 'register_consumer', argument 1 must be a Handle instance") + + +def add_context(handle, event_context, event_name, channel_name): + """ + add_context(Handle handle, EventContext ctx, + str event_name, str channel_name) -> int + + Add context to event and/or channel. + If event_name is None, the context is applied to all events of the channel. + If channel_name is None, a lookup of the event's channel is done. + If both are None, the context is applied to all events of all channels. + Returns the size of the returned payload data or a negative error code. + """ + + try: + return _lttng_add_context(handle._h, event_context, event_name, channel_name) + except AttributeError: + raise TypeError("in method 'add_context', argument 1 must be a Handle instance") + + +def enable_event(handle, event, channel_name): + """ + enable_event(Handle handle, Event event, + str channel_name) -> int + + Enable event(s) for a channel. + If no event name is specified, all events are enabled. + If no channel name is specified, the default 'channel0' is used. + Returns size of returned session payload data or a negative error code. + """ + + try: + return _lttng_enable_event(handle._h, event, channel_name) + except AttributeError: + raise TypeError("in method 'enable_event', argument 1 must be a Handle instance") + + +def enable_channel(handle, channel): + """ + enable_channel(Handle handle, Channel channel -> int + + Enable channel per domain + Returns size of returned session payload data or a negative error code. + """ + + try: + return _lttng_enable_channel(handle._h, channel) + except AttributeError: + raise TypeError("in method 'enable_channel', argument 1 must be a Handle instance") + + +def disable_event(handle, name, channel_name): + """ + disable_event(Handle handle, str name, str channel_name) -> int + + Disable event(s) of a channel and domain. + If no event name is specified, all events are disabled. + If no channel name is specified, the default 'channel0' is used. + Returns size of returned session payload data or a negative error code + """ + + try: + return _lttng_disable_event(handle._h, name, channel_name) + except AttributeError: + raise TypeError("in method 'disable_event', argument 1 must be a Handle instance") + + +def disable_channel(handle, name): + """ + disable_channel(Handle handle, str name) -> int + + All tracing will be stopped for registered events of the channel. + Returns size of returned session payload data or a negative error code. + """ + + try: + return _lttng_disable_channel(handle._h, name) + except AttributeError: + raise TypeError("in method 'disable_channel', argument 1 must be a Handle instance") + + +def calibrate(handle, calibrate): + """ + calibrate(Handle handle, Calibrate calibrate) -> int + + Quantify LTTng overhead. + Returns size of returned session payload data or a negative error code. + """ + + try: + return _lttng_calibrate(handle._h, calibrate) + except AttributeError: + raise TypeError("in method 'calibrate', argument 1 must be a Handle instance") +%} + + +// ============================================= +// Handle class +// Used to prevent freeing unallocated memory +// ============================================= + +%feature("docstring")"" +%feature("autodoc", "1"); + +%pythoncode %{ +class Handle: + """ + Manages a handle. + Takes two arguments: (str session_name, Domain domain) + """ + + __frozen = False + + def __init__(self, session_name, domain): + if type(session_name) is not str: + raise TypeError("in method '__init__', argument 2 of type 'str'") + if type(domain) is not Domain and domain is not None: + raise TypeError("in method '__init__', argument 3 of type 'lttng.Domain'") + + self._sname = session_name + if domain is None: + self._domtype = None + else: + self._domtype = domain.type + self._h = _lttng_create_handle(session_name, domain) + self.__frozen = True + + def __del__(self): + _lttng_destroy_handle(self._h) + + def __repr__(self): + if self._domtype == 1: + domstr = "DOMAIN_KERNEL" + elif self._domtype == 2: + domstr = "DOMAIN_UST" + else: + domstr = self._domtype + + return "lttng.Handle; session('{}'), domain.type({})".format( + self._sname, domstr) + + def __setattr__(self, attr, val): + if self.__frozen: + raise NotImplementedError("cannot modify attributes") + else: + self.__dict__[attr] = val +%} + + +// ============================================= +// STRUCTURES +// These are directly taken from lttng.h. +// Any change to these structures must also be +// made here. +// ============================================= + +%rename("Domain") lttng_domain; +%rename("EventContext") lttng_event_context; +%rename("Event") lttng_event; +%rename("Calibrate") lttng_calibrate; +%rename("ChannelAttr") lttng_channel_attr; +%rename("Channel") lttng_channel; +%rename("Session") lttng_session; + +struct lttng_domain{ + enum lttng_domain_type type; + char padding[LTTNG_DOMAIN_PADDING1]; + + union { + pid_t pid; + char exec_name[NAME_MAX]; + char padding[LTTNG_DOMAIN_PADDING2]; + } attr; + + %extend { + char *__repr__() { + static char temp[256]; + switch ( $self->type ) { + case 1: + sprintf(temp, "lttng.Domain; type(DOMAIN_KERNEL)"); + break; + case 2: + sprintf(temp, "lttng.Domain; type(DOMAIN_UST)"); + break; + default: + sprintf(temp, "lttng.Domain; type(%i)", $self->type); + break; + } + return &temp[0]; + } + } +}; + +struct lttng_event_context { + enum lttng_event_context_type ctx; + char padding[LTTNG_EVENT_CONTEXT_PADDING1]; + + union { + struct lttng_event_perf_counter_ctx perf_counter; + char padding[LTTNG_EVENT_CONTEXT_PADDING2]; + } u; + + %extend { + char *__repr__() { + static char temp[256]; + switch ( $self->ctx ) { + case 0: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_PID)"); + break; + case 1: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_PERF_COUNTER)"); + break; + case 2: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_PROCNAME)"); + break; + case 3: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_PRIO)"); + break; + case 4: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_NICE)"); + break; + case 5: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_VPID)"); + break; + case 6: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_TID)"); + break; + case 7: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_VTID)"); + break; + case 8: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_PPID)"); + break; + case 9: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_VPPID)"); + break; + case 10: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_PTHREAD_ID)"); + break; + default: + sprintf(temp, "lttng.EventContext; type(%i)", $self->ctx); + break; + } + return &temp[0]; + } + } +}; + +struct lttng_event_probe_attr { + uint64_t addr; + uint64_t offset; + char symbol_name[LTTNG_SYMBOL_NAME_LEN]; + char padding[LTTNG_EVENT_PROBE_PADDING1]; +}; + +struct lttng_event_function_attr { + char symbol_name[LTTNG_SYMBOL_NAME_LEN]; + char padding[LTTNG_EVENT_FUNCTION_PADDING1]; +}; + +struct lttng_event { + enum lttng_event_type type; + char name[LTTNG_SYMBOL_NAME_LEN]; + + enum lttng_loglevel_type loglevel_type; + int loglevel; + + int32_t enabled; + pid_t pid; + + char padding[LTTNG_EVENT_PADDING1]; + + union { + struct lttng_event_probe_attr probe; + struct lttng_event_function_attr ftrace; + + char padding[LTTNG_EVENT_PADDING2]; + } attr; + + %extend { + char *__repr__() { + static char temp[512]; + char evtype[50]; + char logtype[50]; + + switch ( $self->type ) { + case -1: + sprintf(evtype, "EVENT_ALL"); + break; + case 0: + sprintf(evtype, "EVENT_TRACEPOINT"); + break; + case 1: + sprintf(evtype, "EVENT_PROBE"); + break; + case 2: + sprintf(evtype, "EVENT_FUNCTION"); + break; + case 3: + sprintf(evtype, "EVENT_FUNCTION_ENTRY"); + break; + case 4: + sprintf(evtype, "EVENT_NOOP"); + break; + case 5: + sprintf(evtype, "EVENT_SYSCALL"); + break; + default: + sprintf(evtype, "%i", $self->type); + break; + } + + switch ( $self->loglevel_type ) { + case 0: + sprintf(logtype, "EVENT_LOGLEVEL_ALL"); + break; + case 1: + sprintf(logtype, "EVENT_LOGLEVEL_RANGE"); + break; + case 2: + sprintf(logtype, "EVENT_LOGLEVEL_SINGLE"); + break; + default: + sprintf(logtype, "%i", $self->loglevel_type); + break; + } + + sprintf(temp, "lttng.Event; name('%s'), type(%s), " + "loglevel_type(%s), loglevel(%i), " + "enabled(%s), pid(%i)", + $self->name, evtype, logtype, $self->loglevel, + $self->enabled ? "True" : "False", $self->pid); + return &temp[0]; + } + } +}; + +struct lttng_calibrate { + enum lttng_calibrate_type type; + char padding[LTTNG_CALIBRATE_PADDING1]; + + %extend { + char *__repr__() { + static char temp[256]; + switch ( $self->type ) { + case 0: + sprintf(temp, "lttng.Calibrate; type(CALIBRATE_FUNCTION)"); + break; + default: + sprintf(temp, "lttng.Calibrate; type(%i)", $self->type); + break; + } + return &temp[0]; + } + } +}; + +struct lttng_channel_attr { + int overwrite; + uint64_t subbuf_size; + uint64_t num_subbuf; + unsigned int switch_timer_interval; + unsigned int read_timer_interval; + enum lttng_event_output output; + + char padding[LTTNG_CHANNEL_ATTR_PADDING1]; + + %extend { + char *__repr__() { + static char temp[256]; + char evout[25]; + + switch ( $self->output ) { + case 0: + sprintf(evout, "EVENT_SPLICE"); + break; + case 1: + sprintf(evout, "EVENT_MMAP"); + break; + default: + sprintf(evout, "%i", $self->output); + break; + } + sprintf(temp, "lttng.ChannelAttr; overwrite(%i), subbuf_size(%lu), " + "num_subbuf(%lu), switch_timer_interval(%u), " + "read_timer_interval(%u), output(%s)", + $self->overwrite, $self->subbuf_size, $self->num_subbuf, + $self->switch_timer_interval, $self->read_timer_interval, + evout); + return &temp[0]; + } + } +}; + +struct lttng_channel { + char name[LTTNG_SYMBOL_NAME_LEN]; + uint32_t enabled; + struct lttng_channel_attr attr; + char padding[LTTNG_CHANNEL_PADDING1]; + + %extend { + char *__repr__() { + static char temp[512]; + sprintf(temp, "lttng.Channel; name('%s'), enabled(%s)", + $self->name, $self->enabled ? "True" : "False"); + return &temp[0]; + } + } +}; + +struct lttng_session { + char name[NAME_MAX]; + char path[PATH_MAX]; + uint32_t enabled; + char padding[LTTNG_SESSION_PADDING1]; + + %extend { + char *__repr__() { + static char temp[512]; + sprintf(temp, "lttng.Session; name('%s'), path('%s'), enabled(%s)", + $self->name, $self->path, + $self->enabled ? "True" : "False"); + return &temp[0]; + } + } +}; diff --git a/extras/bindings/swig/python/tests/example.py b/extras/bindings/swig/python/tests/example.py new file mode 100644 index 0000000..9703170 --- /dev/null +++ b/extras/bindings/swig/python/tests/example.py @@ -0,0 +1,109 @@ +#This example shows basically how to use the lttng-tools python module + +from lttng import * + +# This error will be raised is something goes wrong +class LTTngError(Exception): + def __init__(self, value): + self.value = value + def __str__(self): + return repr(self.value) + +#Setting up the domain to use +dom = Domain() +dom.type = DOMAIN_KERNEL + +#Setting up a channel to use +channel = Channel() +channel.name = "mychan" +channel.attr.overwrite = 0 +channel.attr.subbuf_size = 4096 +channel.attr.num_subbuf = 8 +channel.attr.switch_timer_interval = 0 +channel.attr.read_timer_interval = 200 +channel.attr.output = EVENT_SPLICE + +#Setting up some events that will be used +event = Event() +event.type = EVENT_TRACEPOINT +event.loglevel_type = EVENT_LOGLEVEL_ALL + +sched_switch = Event() +sched_switch.name = "sched_switch" +sched_switch.type = EVENT_TRACEPOINT +sched_switch.loglevel_type = EVENT_LOGLEVEL_ALL + +sched_process_exit = Event() +sched_process_exit.name = "sched_process_exit" +sched_process_exit.type = EVENT_TRACEPOINT +sched_process_exit.loglevel_type = EVENT_LOGLEVEL_ALL + +sched_process_free = Event() +sched_process_free.name = "sched_process_free" +sched_process_free.type = EVENT_TRACEPOINT +sched_process_free.loglevel_type = EVENT_LOGLEVEL_ALL + + +#Creating a new session +res = create("test","/lttng-traces/test") +if res<0: + raise LTTngError(strerror(res)) + +#Creating handle +han = None +han = Handle("test", dom) +if han is None: + raise LTTngError("Handle not created") + +#Enabling the kernel channel +res = enable_channel(han, channel) +if res<0: + raise LTTngError(strerror(res)) + +#Enabling some events in given channel +#To enable all events in default channel, use +#enable_event(han, event, None) +res = enable_event(han, sched_switch, channel.name) +if res<0: + raise LTTngError(strerror(res)) + +res = enable_event(han, sched_process_exit, channel.name) +if res<0: + raise LTTngError(strerror(res)) + +res = enable_event(han, sched_process_free, channel.name) +if res<0: + raise LTTngError(strerror(res)) + +#Disabling an event +res = disable_event(han, sched_switch.name, channel.name) +if res<0: + raise LTTngError(strerror(res)) + +#Getting a list of the channels +l = list_channels(han) +if type(l) is int: + raise LTTngError(strerror(l)) + +#Starting the trace +res = start("test") +if res<0: + raise LTTngError(strerror(res)) + +#Stopping the trace +res = stop("test") +if res<0: + raise LTTngError(strerror(res)) + +#Disabling a channel +res = disable_channel(han, channel.name) +if res<0: + raise LTTngError(strerror(res)) + +#Destroying the handle +del han + +#Destroying the session +res = destroy("test") +if res<0: + raise LTTngError(strerror(res)) diff --git a/extras/bindings/swig/python/tests/run.sh b/extras/bindings/swig/python/tests/run.sh new file mode 100644 index 0000000..7de819b --- /dev/null +++ b/extras/bindings/swig/python/tests/run.sh @@ -0,0 +1 @@ +python tests.py diff --git a/extras/bindings/swig/python/tests/tests.py b/extras/bindings/swig/python/tests/tests.py new file mode 100644 index 0000000..a4be981 --- /dev/null +++ b/extras/bindings/swig/python/tests/tests.py @@ -0,0 +1,310 @@ +import unittest +import os +import time +from lttng import * + +class TestLttngPythonModule (unittest.TestCase): + + def test_kernel_all_events(self): + dom = Domain() + dom.type = DOMAIN_KERNEL + + event = Event() + event.type = EVENT_TRACEPOINT + event.loglevel_type = EVENT_LOGLEVEL_ALL + + han = Handle("test_kernel_all_ev", dom) + + r = create("test_kernel_all_ev","/lttng-traces/test") + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, event, None) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = start("test_kernel_all_ev") + self.assertGreaterEqual(r, 0, strerror(r)) + time.sleep(2) + + r = stop("test_kernel_all_ev") + self.assertGreaterEqual(r, 0, strerror(r)) + + r = destroy("test_kernel_all_ev") + self.assertGreaterEqual(r, 0, strerror(r)) + + + def test_kernel_event(self): + + dom = Domain() + dom.type = DOMAIN_KERNEL + + channel = Channel() + channel.name="mychan" + channel.attr.overwrite = 0 + channel.attr.subbuf_size = 4096 + channel.attr.num_subbuf = 8 + channel.attr.switch_timer_interval = 0 + channel.attr.read_timer_interval = 200 + channel.attr.output = EVENT_SPLICE + + sched_switch = Event() + sched_switch.name = "sched_switch" + sched_switch.type = EVENT_TRACEPOINT + sched_switch.loglevel_type = EVENT_LOGLEVEL_ALL + + sched_process_exit = Event() + sched_process_exit.name = "sched_process_exit" + sched_process_exit.type = EVENT_TRACEPOINT + sched_process_exit.loglevel_type = EVENT_LOGLEVEL_ALL + + sched_process_free = Event() + sched_process_free.name = "sched_process_free" + sched_process_free.type = EVENT_TRACEPOINT + sched_process_free.loglevel_type = EVENT_LOGLEVEL_ALL + + han = Handle("test_kernel_event", dom) + + #Create session test + r = create("test_kernel_event","/lttng-traces/test") + self.assertGreaterEqual(r, 0, strerror(r)) + + #Enabling channel tests + r = enable_channel(han, channel) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Enabling events tests + r = enable_event(han, sched_switch, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, sched_process_exit, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, sched_process_free, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Disabling events tests + r = disable_event(han, sched_switch.name, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = disable_event(han, sched_process_free.name, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Renabling events tests + r = enable_event(han, sched_switch, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, sched_process_free, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Start, stop, destroy + r = start("test_kernel_event") + self.assertGreaterEqual(r, 0, strerror(r)) + time.sleep(2) + + r = stop("test_kernel_event") + self.assertGreaterEqual(r, 0, strerror(r)) + + r=disable_channel(han, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r=destroy("test_kernel_event") + self.assertGreaterEqual(r, 0, strerror(r)) + + + + def test_ust_all_events(self): + dom = Domain() + dom.type = DOMAIN_UST + + event = Event() + event.type = EVENT_TRACEPOINT + event.loglevel_type = EVENT_LOGLEVEL_ALL + + han = Handle("test_ust_all_ev", dom) + + r = create("test_ust_all_ev","/lttng-traces/test") + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, event, None) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = start("test_ust_all_ev") + self.assertGreaterEqual(r, 0, strerror(r)) + time.sleep(2) + + r = stop("test_ust_all_ev") + self.assertGreaterEqual(r, 0, strerror(r)) + + r = destroy("test_ust_all_ev") + self.assertGreaterEqual(r, 0, strerror(r)) + + + def test_ust_event(self): + + dom = Domain() + dom.type = DOMAIN_UST + + channel = Channel() + channel.name="mychan" + channel.attr.overwrite = 0 + channel.attr.subbuf_size = 4096 + channel.attr.num_subbuf = 8 + channel.attr.switch_timer_interval = 0 + channel.attr.read_timer_interval = 200 + channel.attr.output = EVENT_MMAP + + ev1 = Event() + ev1.name = "tp1" + ev1.type = EVENT_TRACEPOINT + ev1.loglevel_type = EVENT_LOGLEVEL_ALL + + ev2 = Event() + ev2.name = "ev2" + ev2.type = EVENT_TRACEPOINT + ev2.loglevel_type = EVENT_LOGLEVEL_ALL + + ev3 = Event() + ev3.name = "ev3" + ev3.type = EVENT_TRACEPOINT + ev3.loglevel_type = EVENT_LOGLEVEL_ALL + + han = Handle("test_ust_event", dom) + + #Create session test + r = create("test_ust_event","/lttng-traces/test") + self.assertGreaterEqual(r, 0, strerror(r)) + + #Enabling channel tests + r = enable_channel(han, channel) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Enabling events tests + r = enable_event(han, ev1, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, ev2, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, ev3, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Disabling events tests + r = disable_event(han, ev1.name, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = disable_event(han, ev3.name, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Renabling events tests + r = enable_event(han, ev1, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, ev3, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Start, stop + r = start("test_ust_event") + self.assertGreaterEqual(r, 0, strerror(r)) + time.sleep(2) + + r = stop("test_ust_event") + self.assertGreaterEqual(r, 0, strerror(r)) + + #Restart/restop + r = start("test_ust_event") + self.assertGreaterEqual(r, 0, strerror(r)) + time.sleep(2) + + r = stop("test_ust_event") + self.assertGreaterEqual(r, 0, strerror(r)) + + #Disabling channel and destroy + r=disable_channel(han, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r=destroy("test_ust_event") + self.assertGreaterEqual(r, 0, strerror(r)) + + + def test_other_functions(self): + dom = Domain() + dom.type=DOMAIN_KERNEL + + event=Event() + event.type=EVENT_TRACEPOINT + event.loglevel_type=EVENT_LOGLEVEL_ALL + + calib = Calibrate() + calib.type = CALIBRATE_FUNCTION + + ctx = EventContext() + ctx.type=EVENT_CONTEXT_PID + + chattr = ChannelAttr() + chattr.overwrite = 0 + chattr.subbuf_size = 4096 + chattr.num_subbuf = 8 + chattr.switch_timer_interval = 0 + chattr.read_timer_interval = 200 + chattr.output = EVENT_SPLICE + + han = Handle("test_otherf" , dom) + + r = create("test_otherf","/lttng-traces/test") + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, event, None) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Calibrate test + r = calibrate(han , calib) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Context test + r = add_context(han, ctx, "sched_switch", "channel0") + self.assertGreaterEqual(r, 0, strerror(r)) + #Any channel + r = add_context(han, ctx, "sched_wakeup", None) + self.assertGreaterEqual(r, 0, strerror(r)) + #All events + r = add_context(han, ctx, None, None) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Def. channel attr + channel_set_default_attr(dom, chattr) + channel_set_default_attr(None, None) + + #Ses Daemon alive + r = session_daemon_alive() + self.assertTrue(r == 1 or r == 0, strerror(r)) + + #Setting trace group + r = set_tracing_group("testing") + self.assertGreaterEqual(r, 0, strerror(r)) + + + r = start("test_otherf") + self.assertGreaterEqual(r, 0, strerror(r)) + time.sleep(2) + + r = stop("test_otherf") + self.assertGreaterEqual(r, 0, strerror(r)) + + del han + + r = destroy("test_otherf") + self.assertGreaterEqual(r, 0, strerror(r)) + + +if __name__ == "__main__": + # CHECK IF ROOT + if os.geteuid() == 0: + #Make sure session names don't already exist: + destroy("test_kernel_event") + destroy("test_kernel_all_events") + destroy("test_ust_all_events") + destroy("test_ust_event") + destroy("test_otherf") + + unittest.main() + else: + print('Script must be run as root') -- 1.7.9.5 From yannick.brosseau at gmail.com Wed Aug 8 16:42:41 2012 From: yannick.brosseau at gmail.com (Yannick Brosseau) Date: Wed, 08 Aug 2012 16:42:41 -0400 Subject: [lttng-dev] [RFC] LTTng address API proposal v3 In-Reply-To: <50229F21.4080400@efficios.com> References: <502167DF.5000806@efficios.com> <50229F21.4080400@efficios.com> Message-ID: <5022CF41.9030605@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-08-08 13:17, David Goulet wrote: > ADDENDUM: > > The set_consumer_url() call needs an extra arguments since we can both > set the CONTROL or STREAM path using this call (the stream type). > > So, I would like to propose a change: > > From: > lttng_set_consumer_url(struct lttng_handle *handle, const char *url); > > To: > lttng_set_consumer_url(struct lttng_handle *handle, const char *url, > enum lttng_url_stream_type stype); > > With: > enum lttng_url_stream_type { > LTTNG_STREAM_CONTROL, > LTTNG_STREAM_DATA, > LTTNG_STREAM_ALL, > }; > > An _url_ argument with "net://localhost" for instance implies > LTTNG_STREAM_ALL as a type however with "tcp://", we have to specify it. > > The second possible idea is to add specialize calls like: > > set_consumer_control_url() > set_consumer_stream_url() Just brainstorming: lttng_set_consumer_url(struct lttng_handle *handle, const char *control_url, const char *data_url) Yannick -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlAiz0AACgkQFQrZ7GzHX2qd/ACg4uX+tLdSIuBNIz9LvT+6Gy7R O+oAoMevaOacHiR2VI8LA18eeAB0jnCD =iRYC -----END PGP SIGNATURE----- From yannick.brosseau at gmail.com Wed Aug 8 16:44:15 2012 From: yannick.brosseau at gmail.com (Yannick Brosseau) Date: Wed, 08 Aug 2012 16:44:15 -0400 Subject: [lttng-dev] [lttng-tools PATCH] lttng-tools python module v2 In-Reply-To: <1344452188-7579-1-git-send-email-danny.serres@efficios.com> References: <1344452188-7579-1-git-send-email-danny.serres@efficios.com> Message-ID: <5022CF9F.9090500@gmail.com> On 2012-08-08 14:56, Danny Serres wrote: > The lttng-tools Python module can be used to directly control > the lttng-tools API inside Python, using 'import lttng'. > > Therefore, it becomes possible to create a trace, add events, > start/stop tracing, destroy a session and so on from within Python. > > The module does not include URI-related functions. > > SWIG >= 2.0 is used to create the wrapper and its > 'warning md variable unused' bug is patched in Makefile.am. > But I don't think that stuff in "extra" should be in the build system. From david.goulet at polymtl.ca Wed Aug 8 16:48:01 2012 From: david.goulet at polymtl.ca (David Goulet) Date: Wed, 08 Aug 2012 16:48:01 -0400 Subject: [lttng-dev] [RFC] LTTng address API proposal v3 In-Reply-To: <5022CF41.9030605@gmail.com> References: <502167DF.5000806@efficios.com> <50229F21.4080400@efficios.com> <5022CF41.9030605@gmail.com> Message-ID: <5022D081.3080805@polymtl.ca> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Yannick Brosseau: > > On 2012-08-08 13:17, David Goulet wrote: >> ADDENDUM: > >> The set_consumer_url() call needs an extra arguments since we can >> both set the CONTROL or STREAM path using this call (the stream >> type). > >> So, I would like to propose a change: > >> From: lttng_set_consumer_url(struct lttng_handle *handle, const >> char *url); > >> To: lttng_set_consumer_url(struct lttng_handle *handle, const >> char *url, enum lttng_url_stream_type stype); > >> With: enum lttng_url_stream_type { LTTNG_STREAM_CONTROL, >> LTTNG_STREAM_DATA, LTTNG_STREAM_ALL, }; > >> An _url_ argument with "net://localhost" for instance implies >> LTTNG_STREAM_ALL as a type however with "tcp://", we have to >> specify it. > >> The second possible idea is to add specialize calls like: > >> set_consumer_control_url() set_consumer_stream_url() > > Just brainstorming: lttng_set_consumer_url(struct lttng_handle > *handle, const char *control_url, const char *data_url) Just a bit difficult when you have --control-url and --data-url option possible on the command line... pass NULL to one to set only one.. ? so this call *can* accept NULL value of one of the arg ? David > > Yannick > > > _______________________________________________ lttng-dev mailing > list lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQItCBAAoJEELoaioR9I02fJ4H/R213HDtvvu4hbcciu/4mxLR XlDaz3nIWmxPVcpJSiHnEChMjCD1aRh40aUdVbnG5gd/L3MGXV7dbmu/XCcH5OSo gLzk/T86ijvobjgFE6bwXFdGWxCoLTuTk2buzb1v1ppVLC9NeBo8W8wkPURYCR5+ ACFOAMY12X2p543hkp1/B6VSCjdrED59s9QPHpWE4i4FIBSMQrXrXOLLHUwbIrhq x4/14HFYDLkPJeLDOQezbvW1r6B0yovU+NDWimiyr2uG5CPO1Rp5fGl8wjQTh8p+ gcnigccnrONW2ZjQnKlDFeGQmMq8Werznm5QBc38E3SA0+N8BRNRRqoKqqNpsUE= =DUoy -----END PGP SIGNATURE----- From laijs at cn.fujitsu.com Thu Aug 9 04:46:25 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Thu, 9 Aug 2012 16:46:25 +0800 Subject: [lttng-dev] [PATCH 1/2] urcu: move exsit code and name it ___cds_wfq_node_sync_next() Message-ID: <1344501986-23151-1-git-send-email-laijs@cn.fujitsu.com> This code which waits for a node's next pointer until it appears, will be used many times, move it to a help function and name it ___cds_wfq_node_sync_next(). Signed-off-by: Lai Jiangshan --- urcu/static/wfqueue.h | 36 +++++++++++++++++++++++++----------- 1 files changed, 25 insertions(+), 11 deletions(-) diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h index 19314f5..636e1af 100644 --- a/urcu/static/wfqueue.h +++ b/urcu/static/wfqueue.h @@ -85,6 +85,29 @@ static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, } /* + * Waiting for enqueuer to complete enqueue and return the next node + */ +static inline struct cds_wfq_node * +___cds_wfq_node_sync_next(struct cds_wfq_node *node) +{ + struct cds_wfq_node *next; + int attempt = 0; + + /* + * Adaptative busy-looping waiting for enqueuer to complete enqueue. + */ + while ((next = CMM_LOAD_SHARED(node->next)) == NULL) { + if (++attempt >= WFQ_ADAPT_ATTEMPTS) { + poll(NULL, 0, WFQ_WAIT); /* Wait for 10ms */ + attempt = 0; + } else + caa_cpu_relax(); + } + + return next; +} + +/* * It is valid to reuse and free a dequeued node immediately. * * No need to go on a waitqueue here, as there is no possible state in which the @@ -96,7 +119,6 @@ static inline struct cds_wfq_node * ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) { struct cds_wfq_node *node, *next; - int attempt = 0; /* * Queue is empty if it only contains the dummy node. @@ -105,16 +127,8 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) return NULL; node = q->head; - /* - * Adaptative busy-looping waiting for enqueuer to complete enqueue. - */ - while ((next = CMM_LOAD_SHARED(node->next)) == NULL) { - if (++attempt >= WFQ_ADAPT_ATTEMPTS) { - poll(NULL, 0, WFQ_WAIT); /* Wait for 10ms */ - attempt = 0; - } else - caa_cpu_relax(); - } + next = ___cds_wfq_node_sync_next(node); + /* * Move queue head forward. */ -- 1.7.7 From laijs at cn.fujitsu.com Thu Aug 9 04:46:26 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Thu, 9 Aug 2012 16:46:26 +0800 Subject: [lttng-dev] [PATCH 2/2] urcu: new wfqueue implementation In-Reply-To: <1344501986-23151-1-git-send-email-laijs@cn.fujitsu.com> References: <1344501986-23151-1-git-send-email-laijs@cn.fujitsu.com> Message-ID: <1344501986-23151-2-git-send-email-laijs@cn.fujitsu.com> Some guys would be surprised by this fact: There are already TWO implementations of wfqueue in urcu. The first one is in urcu/static/wfqueue.h: 1) enqueue: exchange the tail and then update previous->next 2) dequeue: wait for first node's next pointer and them shift, a dummy node is introduced to avoid the queue->tail become NULL when shift. The second one shares some code with the first one, and the left code are spreading in urcu-call-rcu-impl.h: 1) enqueue: share with the first one 2) no dequeue operation: and no shift, so it don't need dummy node, Although the dummy node is queued when initialization, but it is removed after the first dequeue_all operation in call_rcu_thread(). call_rcu_data_free() forgets to handle the dummy node if it is not removed. 3)dequeue_all: record the old head and tail, and queue->head become the special tail node.(atomic record the tail and change the tail). The second implementation's code are spreading, bad for review, and it is not tested by tests/test_urcu_wfq. So we need a better implementation avoid the dummy node dancing and can service both generic wfqueue APIs and dequeue_all API for call rcu. The new implementation: 1) enqueue: share with the first one/original implementation. 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. no dummy node, save memory. 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail and return the old head.next. More implementation details are in the code. tests/test_urcu_wfq will be update in future for testing new APIs. Signed-off-by: Lai Jiangshan --- urcu-call-rcu-impl.h | 50 ++++++++++-------------- urcu/static/wfqueue.h | 104 ++++++++++++++++++++++++++++++++++++------------ urcu/wfqueue.h | 25 ++++++++++-- wfqueue.c | 29 ++++++++++++++ 4 files changed, 149 insertions(+), 59 deletions(-) diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h index 13b24ff..dbfb410 100644 --- a/urcu-call-rcu-impl.h +++ b/urcu-call-rcu-impl.h @@ -221,7 +221,7 @@ static void *call_rcu_thread(void *arg) { unsigned long cbcount; struct cds_wfq_node *cbs; - struct cds_wfq_node **cbs_tail; + struct cds_wfq_node *cbs_tail; struct call_rcu_data *crdp = (struct call_rcu_data *)arg; struct rcu_head *rhp; int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); @@ -243,24 +243,18 @@ static void *call_rcu_thread(void *arg) cmm_smp_mb(); } for (;;) { - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) - poll(NULL, 0, 1); - _CMM_STORE_SHARED(crdp->cbs.head, NULL); - cbs_tail = (struct cds_wfq_node **) - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); + cbs = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &cbs_tail); + if (cbs) { synchronize_rcu(); cbcount = 0; do { - while (cbs->next == NULL && - &cbs->next != cbs_tail) - poll(NULL, 0, 1); - if (cbs == &crdp->cbs.dummy) { - cbs = cbs->next; - continue; - } rhp = (struct rcu_head *)cbs; - cbs = cbs->next; + + if (cbs != cbs_tail) + cbs = __cds_wfq_node_sync_next(cbs); + else + cbs = NULL; + rhp->func(rhp); cbcount++; } while (cbs != NULL); @@ -270,8 +264,7 @@ static void *call_rcu_thread(void *arg) break; rcu_thread_offline(); if (!rt) { - if (&crdp->cbs.head - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { + if (cds_wfq_empty(&crdp->cbs)) { call_rcu_wait(crdp); poll(NULL, 0, 10); uatomic_dec(&crdp->futex); @@ -625,32 +618,31 @@ void call_rcu(struct rcu_head *head, */ void call_rcu_data_free(struct call_rcu_data *crdp) { - struct cds_wfq_node *cbs; - struct cds_wfq_node **cbs_tail; - struct cds_wfq_node **cbs_endprev; + struct cds_wfq_node *head, *tail; if (crdp == NULL || crdp == default_call_rcu_data) { return; } + if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); wake_call_rcu_thread(crdp); while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) poll(NULL, 0, 1); } - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) - poll(NULL, 0, 1); - _CMM_STORE_SHARED(crdp->cbs.head, NULL); - cbs_tail = (struct cds_wfq_node **) - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); + + if (!cds_wfq_empty(&crdp->cbs)) { + head = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &tail); + assert(head); + /* Create default call rcu data if need be */ (void) get_default_call_rcu_data(); - cbs_endprev = (struct cds_wfq_node **) - uatomic_xchg(&default_call_rcu_data, cbs_tail); - *cbs_endprev = cbs; + + __cds_wfq_append_list(&default_call_rcu_data->cbs, head, tail); + uatomic_add(&default_call_rcu_data->qlen, uatomic_read(&crdp->qlen)); + wake_call_rcu_thread(default_call_rcu_data); } diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h index 636e1af..15ea9fc 100644 --- a/urcu/static/wfqueue.h +++ b/urcu/static/wfqueue.h @@ -10,6 +10,7 @@ * dynamically with the userspace rcu library. * * Copyright 2010 - Mathieu Desnoyers + * Copyright 2011-2012 - Lai Jiangshan * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -29,6 +30,7 @@ #include #include #include +#include #include #include @@ -38,8 +40,6 @@ extern "C" { /* * Queue with wait-free enqueue/blocking dequeue. - * This implementation adds a dummy head node when the queue is empty to ensure - * we can always update the queue locklessly. * * Inspired from half-wait-free/half-blocking queue implementation done by * Paul E. McKenney. @@ -57,31 +57,43 @@ static inline void _cds_wfq_init(struct cds_wfq_queue *q) { int ret; - _cds_wfq_node_init(&q->dummy); /* Set queue head and tail */ - q->head = &q->dummy; - q->tail = &q->dummy.next; + _cds_wfq_node_init(&q->head); + q->tail = &q->head; ret = pthread_mutex_init(&q->lock, NULL); assert(!ret); } -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, - struct cds_wfq_node *node) +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) { - struct cds_wfq_node **old_tail; + /* + * Queue is empty if no node is pointed by q->head.next nor q->tail. + */ + return q->head.next == NULL && CMM_LOAD_SHARED(q->tail) == &q->head; +} +static inline void ___cds_wfq_append_list(struct cds_wfq_queue *q, + struct cds_wfq_node *head, struct cds_wfq_node *tail) +{ /* * uatomic_xchg() implicit memory barrier orders earlier stores to data * structure containing node and setting node->next to NULL before * publication. */ - old_tail = uatomic_xchg(&q->tail, &node->next); + tail = uatomic_xchg(&q->tail, tail); + /* - * At this point, dequeuers see a NULL old_tail->next, which indicates + * At this point, dequeuers see a NULL tail->next, which indicates * that the queue is being appended to. The following store will append * "node" to the queue from a dequeuer perspective. */ - CMM_STORE_SHARED(*old_tail, node); + CMM_STORE_SHARED(tail->next, head); +} + +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, + struct cds_wfq_node *node) +{ + ___cds_wfq_append_list(q, node, node); } /* @@ -120,27 +132,46 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) { struct cds_wfq_node *node, *next; - /* - * Queue is empty if it only contains the dummy node. - */ - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) + if (_cds_wfq_empty(q)) return NULL; - node = q->head; - next = ___cds_wfq_node_sync_next(node); + node = ___cds_wfq_node_sync_next(&q->head); + + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { + if (CMM_LOAD_SHARED(q->tail) == node) { + /* + * @node is the only node in the queue. + * Try to move the tail to &q->head + */ + _cds_wfq_node_init(&q->head); + if (uatomic_cmpxchg(&q->tail, node, &q->head) == node) + return node; + } + next = ___cds_wfq_node_sync_next(node); + } /* * Move queue head forward. */ - q->head = next; - /* - * Requeue dummy node if we just dequeued it. - */ - if (node == &q->dummy) { - _cds_wfq_node_init(node); - _cds_wfq_enqueue(q, node); - return ___cds_wfq_dequeue_blocking(q); - } + q->head.next = next; + + return node; +} + +/* dequeue all nodes, the nodes are not synchronized for the next pointer */ +static inline struct cds_wfq_node * +___cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, + struct cds_wfq_node **tail) +{ + struct cds_wfq_node *node; + + if (_cds_wfq_empty(q)) + return NULL; + + node = ___cds_wfq_node_sync_next(&q->head); + _cds_wfq_node_init(&q->head); + *tail = uatomic_xchg(&q->tail, &q->head); + return node; } @@ -158,6 +189,27 @@ _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) return retnode; } +static inline struct cds_wfq_node * +_cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, + struct cds_wfq_node **tail) +{ + struct cds_wfq_node *node, *next; + int ret; + + ret = pthread_mutex_lock(&q->lock); + assert(!ret); + node = ___cds_wfq_dequeue_all_blocking(q, tail); + ret = pthread_mutex_unlock(&q->lock); + assert(!ret); + + /* synchronize all nodes' next pointer */ + next = node; + while (next != *tail) + next = ___cds_wfq_node_sync_next(next); + + return node; +} + #ifdef __cplusplus } #endif diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h index 03a73f1..985f540 100644 --- a/urcu/wfqueue.h +++ b/urcu/wfqueue.h @@ -7,6 +7,7 @@ * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue * * Copyright 2010 - Mathieu Desnoyers + * Copyright 2011-2012 - Lai Jiangshan * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -25,6 +26,7 @@ #include #include +#include #include #ifdef __cplusplus @@ -33,8 +35,6 @@ extern "C" { /* * Queue with wait-free enqueue/blocking dequeue. - * This implementation adds a dummy head node when the queue is empty to ensure - * we can always update the queue locklessly. * * Inspired from half-wait-free/half-blocking queue implementation done by * Paul E. McKenney. @@ -45,8 +45,7 @@ struct cds_wfq_node { }; struct cds_wfq_queue { - struct cds_wfq_node *head, **tail; - struct cds_wfq_node dummy; /* Dummy node */ + struct cds_wfq_node head, *tail; pthread_mutex_t lock; }; @@ -56,18 +55,36 @@ struct cds_wfq_queue { #define cds_wfq_node_init _cds_wfq_node_init #define cds_wfq_init _cds_wfq_init +#define cds_wfq_empty _cds_wfq_empty +#define __cds_wfq_append_list ___cds_wfq_append_list #define cds_wfq_enqueue _cds_wfq_enqueue #define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking #define cds_wfq_dequeue_blocking _cds_wfq_dequeue_blocking +#define __cds_wfq_node_sync_next ___cds_wfq_node_sync_next +#define __cds_wfq_dequeue_all_blocking ___cds_wfq_dequeue_all_blocking +#define cds_wfq_dequeue_all_blocking _cds_wfq_dequeue_all_blocking #else /* !_LGPL_SOURCE */ extern void cds_wfq_node_init(struct cds_wfq_node *node); extern void cds_wfq_init(struct cds_wfq_queue *q); +extern bool cds_wfq_empty(struct cds_wfq_queue *q); +/* __cds_wfq_append_list: caller ensures mutual exclusion between dequeues */ +extern void __cds_wfq_append_list(struct cds_wfq_queue *q, + struct cds_wfq_node *head, struct cds_wfq_node *tail); extern void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node); /* __cds_wfq_dequeue_blocking: caller ensures mutual exclusion between dequeues */ extern struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); extern struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); +extern struct cds_wfq_node *__cds_wfq_node_sync_next(struct cds_wfq_node *node); +/* + * __cds_wfq_dequeue_all_blocking: caller ensures mutual exclusion between + * dequeues, and need synchronize next pointer berfore use it. + */ +extern struct cds_wfq_node *__cds_wfq_dequeue_all_blocking( + struct cds_wfq_queue *q, struct cds_wfq_node **tail); +extern struct cds_wfq_node *cds_wfq_dequeue_all_blocking( + struct cds_wfq_queue *q, struct cds_wfq_node **tail); #endif /* !_LGPL_SOURCE */ diff --git a/wfqueue.c b/wfqueue.c index 3337171..28a7b58 100644 --- a/wfqueue.c +++ b/wfqueue.c @@ -4,6 +4,7 @@ * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue * * Copyright 2010 - Mathieu Desnoyers + * Copyright 2011-2012 - Lai Jiangshan * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -38,6 +39,17 @@ void cds_wfq_init(struct cds_wfq_queue *q) _cds_wfq_init(q); } +bool cds_wfq_empty(struct cds_wfq_queue *q) +{ + return _cds_wfq_empty(q); +} + +void __cds_wfq_append_list(struct cds_wfq_queue *q, + struct cds_wfq_node *head, struct cds_wfq_node *tail) +{ + return ___cds_wfq_append_list(q, head, tail); +} + void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node) { _cds_wfq_enqueue(q, node); @@ -52,3 +64,20 @@ struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) { return _cds_wfq_dequeue_blocking(q); } + +struct cds_wfq_node *__cds_wfq_node_sync_next(struct cds_wfq_node *node) +{ + return ___cds_wfq_node_sync_next(node); +} + +struct cds_wfq_node *__cds_wfq_dequeue_all_blocking( + struct cds_wfq_queue *q, struct cds_wfq_node **tail) +{ + return ___cds_wfq_dequeue_all_blocking(q, tail); +} + +struct cds_wfq_node *cds_wfq_dequeue_all_blocking( + struct cds_wfq_queue *q, struct cds_wfq_node **tail) +{ + return _cds_wfq_dequeue_all_blocking(q, tail); +} -- 1.7.7 From mathieu.desnoyers at efficios.com Thu Aug 9 09:13:14 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 09:13:14 -0400 Subject: [lttng-dev] Question about Transformation to CTF file In-Reply-To: References: Message-ID: <20120809131314.GA8081@Krystal> CCing lttng-dev and eclipse lttng plugin developers, * Runhua Xu (xurunhua23 at hotmail.com) wrote: > > Hi Mathieu, > I am a student and I am now writing my graduation thesis. The purpose > of my thesis is to develop a kernel module and trace some syscalls > generated by playing games on Android, and analyze the results. > Currently, my tracer module works fine and I can save a txt file with > the information I need, like PID, time, scheduler information, syscall > numbers and so on, in the sdcard. Now I am working on the final phase, > which is to show the logs vividly. I found that Eclipse Linux Tools > are really helpful, therefore, I want to transfer my .txt trace file > into the Common Trace Format, and use the Eclipse Linux Tools to show > the statistics and graphics. > I installed babeltrace, and use babeltrace-log to transfer my file > into the corresponding CTF file, by using "cat myfile.txt | > babeltrace-log -t CTF_File" (I am not sure whether it is the correct babeltrace-log expects a dmesg linux kernel output, so I'm not sure this is what you provide as input. Especially the "-t" option expects the dmesg formatting that starts with the timestamp between brackets at the beginning of each line. You might want to drop the -t ? > way, but I couldn't find more information about that). However, when I > use the Eclipse Linux Tools to open the CTF file, it reports error > "Problems occurred when invoking code from plug-in: > "org.eclipse.jface". I tried the tools with the sample CTF file, which > is provided by LTTng, it succeeded. Then I looked into the metadata of > the transferred CTF file (see below codes in blue), it seems that it > remains the original format, doesn't change based on my input file. This looks like a bug in the eclipse tool. I'll let Alexandre handle this part. Please test with "babeltrace" too to see if it is parsed correctly by the C parser. > Which then results into the error message in eclipse. > I got stuck here for several days. I read the documents regarding > tracebabel and CTF on efficios, but didn't get too much insight how > should I generate my own CTF file for the current .txt log. And there > are not too much information online. Would you please kindly give me > some tips, how should I continue? Did I use the correct way for > babeltrace-log? Or, I should write my own metadata and write some C > codes to generate the corresponding datastream? I am a newbee and > don't know too much about what to do with CTF. Please help me. > Thanks a lot in advance. > Regards,Ryan The output has been wrapped by your mail client into a single line please allow me to reformat it: > /* CTF 1.8 */ > typealias integer { size = 8; align = 8; signed = false; } := uint8_t; > typealias integer { size = 32; align = 32; signed = false; } := uint32_t; > trace { > major = 1; > minor = 0; > uuid = "c1aa954c-7771-464e-9045-d67343a31c1d"; > byte_order = le; > packet.header := struct { > uint32_t magic; > uint8_t uuid[16]; > }; > }; > stream { > packet.context := struct { > uint32_t content_size; > uint32_t packet_size; > }; > typealias integer { size = 64; align = 64; signed = false; } := uint64_t; > event.header := struct { > uint64_t timestamp; > }; > }; What I see here is that I think the eclipse CTF parser might have issues with the typealias declaration nested within the stream declaration. Thanks, Mathieu -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 9 09:26:29 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 09:26:29 -0400 Subject: [lttng-dev] syscall options In-Reply-To: References: Message-ID: <20120809132629.GB8382@Krystal> * tchak adim (montarcyber at gmail.com) wrote: > Hi , > > i wanna display channel name (kernel,fs,status) and File name (for example > :in the case of sys-open or sys-close) for differents syscall traces > generated by Lttng2.0 . > i didn't find any indication of what should i do to add these information . The channel name is not relevant in lttng 2.0, because it is not associated with the traced data semantic as is was in 0.x. It is rather dynamically selected by the user on a per event basis. The filenames for sys open are present by default when you enable syscall tracing. With babeltrace, the filenames for close are not present, because close takes a file descriptor. This info can be regenerated by eclise lttng plugin et lttv though. Thanks, Mathieu > > > thanks in advance ! > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 9 09:36:19 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 09:36:19 -0400 Subject: [lttng-dev] [PATCH lttng-tools 1/2] Fix: Libtool fails to find dependent libraries when cross-compiling lttng-tools In-Reply-To: References: Message-ID: <20120809133619.GF8382@Krystal> * Christian Babeux (christian.babeux at efficios.com) wrote: > This problem arise when cross compiling and linking libraries with > indirect libraries dependencies (such as liblttng-ust). This "bug" is > caused by an upstream modification in the libtool package on Debian > system. The libtool "link_all_deplibs" flag is set to "no" by default > on linux targets (AFAIK, other distros set it to "unknown"). > > The chosen solution is to detect such cases via the configure script > and automagically patch the libtool.m4 by forcing the "link_all_deplibs" > to "unknown". > > This fixup can be disabled with the appropriate configure flag: > > ./configure --disable-libtool-linkdep-fixup > > Sample configure output on affected systems: > > checking for occurence(s) of link_all_deplibs = no in > ./config/libtool.m4... 3 > configure: WARNING: the detected libtool will not link all > dependencies, forcing link_all_deplibs = unknown > > Signed-off-by: Christian Babeux Acked-by: Mathieu Desnoyers > --- > configure.ac | 25 +++++++++++++++++++++++++ > 1 file changed, 25 insertions(+) > > diff --git a/configure.ac b/configure.ac > index 17e6b67..3a023cd 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -14,6 +14,31 @@ AC_DEFINE_UNQUOTED([VERSION_DESCRIPTION], ["$version_description"], "") > > AC_CONFIG_HEADERS([include/config.h]) > > +AC_PROG_GREP > +# libtool link_all_deplibs fixup. See http://bugs.lttng.org/issues/321. > +AC_ARG_ENABLE(libtool-linkdep-fixup, > + AS_HELP_STRING([--disable-libtool-linkdep-fixup], > + [disable the libtool fixup for linking all dependent libraries (link_all_deplibs)]), > + libtool_fixup=$enableval, > + libtool_fixup=yes) > + > +AS_IF([test "x$libtool_fixup" = "xyes"], > + [ > + libtool_m4="$srcdir/config/libtool.m4" > + libtool_flag_pattern=".*link_all_deplibs\s*,\s*\$1\s*)" > + AC_MSG_CHECKING([for occurence(s) of link_all_deplibs = no in $libtool_m4]) > + libtool_flag_pattern_count=$(grep -c "$libtool_flag_pattern\s*=\s*no" $libtool_m4) > + AS_IF([test $libtool_flag_pattern_count -ne 0], > + [ > + AC_MSG_RESULT([$libtool_flag_pattern_count]) > + AC_MSG_WARN([the detected libtool will not link all dependencies, forcing link_all_deplibs = unknown]) > + sed -i "s/\($libtool_flag_pattern\)\s*=\s*no/\1=unknown/g" $libtool_m4 > + ], > + [ > + AC_MSG_RESULT([none]) > + ]) > + ]) > + > AC_CHECK_HEADERS([ \ > sys/types.h unistd.h fcntl.h string.h pthread.h limits.h \ > signal.h stdlib.h sys/un.h sys/socket.h stdlib.h stdio.h \ > -- > 1.7.11.3 > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 9 09:46:21 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 09:46:21 -0400 Subject: [lttng-dev] [PATCH lttng-tools 2/2] Fix: Missing libs dependencies in configure check for lttng-ust-ctl In-Reply-To: <884d75d90550f85492dfcaac2b0088f78ced542b.1344025572.git.christian.babeux@efficios.com> References: <884d75d90550f85492dfcaac2b0088f78ced542b.1344025572.git.christian.babeux@efficios.com> Message-ID: <20120809134621.GG8382@Krystal> * Christian Babeux (christian.babeux at efficios.com) wrote: > The lttng-ust-ctl library depends on liburcu-{common,bp,cds}. > The AC_CHECK_LIBRARY macro can't automatically resolve dependents > libraries (ala libtool), so any additionnals dependencies must > be manually specified. > > Also, the AC_CHECK_LIB action-if-found case for the lttng-ust-ctl > check is modified to have a similar behavior as the default action, > which is to define HAVE_LIBxxx and append -lxxx to $LIBS, *except* > for the later step. This is to ensure that any future addition of > AC_CHECK_LIB after the one for lttng-ust-ctl will not need to append > the liburcu dependencies or fail unexpectedly. not sure I understand why all the changes you are doing are needed. What happens if you just do: - [AC_MSG_ERROR([Cannot find LTTng-UST. Use [LDFLAGS]=-Ldir to specify its location, or specify --disable-lttng-ust to build lttng-tools without LTTng-UST support.])] + [AC_MSG_ERROR([Cannot find LTTng-UST. Use [LDFLAGS]=-Ldir to specify its location, or specify --disable-lttng-ust to build lttng-tools without LTTng-UST support.])], + [-lurcu-common -lurcu-bp -lurcu-cds] ? > > Signed-off-by: Christian Babeux > --- > configure.ac | 13 ++++++++----- > 1 file changed, 8 insertions(+), 5 deletions(-) > > diff --git a/configure.ac b/configure.ac > index 3a023cd..e4b9eb1 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -138,13 +138,16 @@ AC_ARG_ENABLE(lttng-ust, > lttng_ust_support=$enableval, lttng_ust_support=yes) > > AS_IF([test "x$lttng_ust_support" = "xyes"], [ > - AC_CHECK_LIB([lttng-ust-ctl], [ustctl_create_session], [], > - [AC_MSG_ERROR([Cannot find LTTng-UST. Use [LDFLAGS]=-Ldir to specify its location, or specify --disable-lttng-ust to build lttng-tools without LTTng-UST support.])] > + AC_CHECK_LIB([lttng-ust-ctl], [ustctl_create_session], > + [ > + AC_DEFINE([HAVE_LIBLTTNG_UST_CTL], [1], [has LTTng-UST control support]) wasn't HAVE_LIBLTTNG_UST_CTL already defined by the AM_CONDITIONAL below ? > + lttng_ust_ctl_found=yes why are you adding this ? > + ], > + [AC_MSG_ERROR([Cannot find LTTng-UST. Use [LDFLAGS]=-Ldir to specify its location, or specify --disable-lttng-ust to build lttng-tools without LTTng-UST support.])], > + [-lurcu-common -lurcu-bp -lurcu-cds] > ) > ]) > - > -AM_CONDITIONAL([HAVE_LIBLTTNG_UST_CTL], [ test "x$ac_cv_lib_lttng_ust_ctl_ustctl_create_session" = "xyes" ]) > - > +AM_CONDITIONAL([HAVE_LIBLTTNG_UST_CTL], [test "x$lttng_ust_ctl_found" = xyes]) why are you changing this line ? Thanks, Mathieu > AC_CHECK_FUNCS([sched_getcpu sysconf]) > > # check for dlopen > -- > 1.7.11.3 > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 9 09:50:38 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 09:50:38 -0400 Subject: [lttng-dev] ERROR: No buffer space available In-Reply-To: References: <9ac1de00b4084e6a84dbdd60eced4477@UMEX001.cgcent.miami.edu> Message-ID: <20120809135038.GI8382@Krystal> CCing lttng-dev, as other people might want to help. Please be aware that the TRACE_EVENT LTTng implementation is slightly different from the mainline kernel implementation, so the documentation you refer to is not a perfect fit. Thanks, Mathieu * Rui Han (r.han at umiami.edu) wrote: > Hi Mathieu, > > I did reboot the system. I am wondering if it is because the header file > need to be modified. I am not quite understand how the headers worked. I > read the article at http://lwn.net/Articles/383362/, but I am not > completely understand the essence of the discussion about the header. Could > you give me some hint that I can understand lttng and debug the code to get > what I wanted? > > Thank you very much. > > Regards, > Rui > > On Sun, Aug 5, 2012 at 1:03 PM, Mathieu Desnoyers < > mathieu.desnoyers at efficios.com> wrote: > > > * Rui Han (r.han at umiami.edu) wrote: > > > Hi, > > > > > > I try to modified the lttng2.0 source code for my research. For example, > > I > > > try to log the "comm" for the write system call, I add the following > > filed > > > in the file x86-64-syscalls-3.0.4_pointers.h located at the path > > > lttng-modules-2.0.3/instrumentation/syscalls/headers. I modify the > > > TRACE_EVENTS() macro from the original one: > > > > > > #ifndef OVERRIDE_64_sys_write > > > SC_TRACE_EVENT(sys_write, > > > TP_PROTO(unsigned int fd, const char * buf, size_t count), > > > TP_ARGS(fd, buf, count), > > > TP_STRUCT__entry(__field(unsigned int, fd) __field_hex(const char *, buf) > > > __field(size_t, count)), > > > TP_fast_assign(tp_assign(fd, fd) tp_assign(buf, buf) tp_assign(count, > > > count)), > > > TP_printk() > > > ) > > > #endif > > > > > > to the following one: > > > > > > #ifndef OVERRIDE_64_sys_write > > > SC_TRACE_EVENT(sys_write, > > > TP_PROTO(unsigned int fd, const char * buf, size_t count, struct > > > task_struct *p), > > > TP_ARGS(fd, buf, count, p), > > > TP_STRUCT__entry( __field(unsigned int, fd) __field_hex(const char *, > > > buf) __field(size_t, > > > count) __array_text(char, comm, TASK_COMM_LEN)), > > > TP_fast_assign( tp_assign(fd, fd) tp_assign(buf, buf) tp_assign(count, > > > 2064) tp_memcpy(comm, > > > p->comm, TASK_COMM_LEN)), > > > TP_printk() > > > ) > > > #endif > > > > > > I add one argument *p and one filed "comm" for the macro. It was able to > > > be compiled, however, after reload the lttng modules, It will show the > > > following error messages when I start lttng. > > > > > > PERROR: ioctl start session: No buffer space available [in > > > kernel_start_session() at kernel.c:397] > > > Error: Starting kernel trace failed > > > > I don't see how modifying the system call tracing macro would trigger > > this error at start. Try restarting lttng-sessiond. Try unloading the > > lttng modules, and, maybe, as last resort, rebooting your system. > > > > Thanks, > > > > Mathieu > > > > > > > > It won't work. Please help me out with this. My next step is try to add > > > extra fields in the sys_connect macros to enable the logging of the > > > connected IP/port from the socket. Is this method right? what are other > > > possible solutions in order to logging the details of the socket > > > information? > > > > > > Thank you very much. > > > > > > Regards, > > > Rui > > > > -- > > Mathieu Desnoyers > > Operating System Efficiency R&D Consultant > > EfficiOS Inc. > > http://www.efficios.com > > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From david.goulet at polymtl.ca Thu Aug 9 10:01:36 2012 From: david.goulet at polymtl.ca (David Goulet) Date: Thu, 09 Aug 2012 10:01:36 -0400 Subject: [lttng-dev] [PATCH lttng-tools 0/2] Configure fixes for cross-compilation In-Reply-To: References: Message-ID: <5023C2C0.5050000@polymtl.ca> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Merged! Christian Babeux: > Hi all, > > Here are a few fixes to the configure.ac script to accomodate > cross-compilation of lttng-tools. > > Patch 1: > > Same fix that was merged in lttng-ust to fix libtool.m4 on target > where link_all_deplibs = no. > > Patch 2: > > Fix an issue encountered when cross-compiling. The configure > script simply fails saying it cannot find the lttng-ust-ctl library > even thought the proper "-L/path/to/ust" is specified in $LDFLAGS. > The AC_CHECK_LIB test fails because it can't find the libraries > that liblttng-ust-ctl depends on. The fix is to specify the > dependents libraries (liburcu-{common,bp,cds}) in the > other-lib-case of AC_CHECK_LIB. > > Thanks, > > Christian > > Christian Babeux (2): Fix: Libtool fails to find dependent > libraries when cross-compiling lttng-tools Fix: Missing libs > dependencies in configure check for lttng-ust-ctl > > configure.ac | 38 +++++++++++++++++++++++++++++++++----- 1 file > changed, 33 insertions(+), 5 deletions(-) > -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQI8K9AAoJEELoaioR9I02GYoIALy1oupw5BbMgfNV5i/gyPrf 3ywUW+s0/Ltl+H4L6hOUq4daUTyUpt+Tla9bBNiP+HXkRX7rxjMnePjN2lk8QJR6 4lW1fWD7IlUNwQkVzKNZh+fv3xu4QyIiWd9zaEmMWIFwibaAcV8yzaI4ZkdrkHXX r+b2nVeGScx2kamwj5B/CWXr+FLovsstGEdKUy2KmJZ7A9eLg4G4Eyn8R3U7Sn5m X3t04liPawiZ6g8NTVQiw+O2vBs6ngdklj5DTQ1v4vLb/a2IwPPEnHjrznV1Np6A T1ONQex+063MYdxp6a0948pXx2X1jBEgeESphoew/Go15LQqgkUBFZHTye3kKdI= =GgYi -----END PGP SIGNATURE----- From mathieu.desnoyers at efficios.com Thu Aug 9 10:02:40 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 10:02:40 -0400 Subject: [lttng-dev] Uprobe and extractor for coredump In-Reply-To: <6539770C71C3814BB0BFC2DBEBD10508CC6967@CORPUSMX30B.corp.emc.com> References: <6539770C71C3814BB0BFC2DBEBD10508CC6967@CORPUSMX30B.corp.emc.com> Message-ID: <20120809140240.GB8937@Krystal> * Chang, Zheng (Zheng.Chang at emc.com) wrote: > Hi, > > > > Recently Linus merged uprobe into kernel mainline and perf has supported > user-space dynamic tracing now. > > Is there any plan or motion on this with LTTng? this would certainly be a nice feature! If you want to work on adding support within lttng-modules and lttng-tools to enable tracing of userspace through uprobes, it would certainly be welcome! > > I did some research on buffer management of LTTng. And I'm trying to > figure out a way to extract trace info from coredump. > > Here is an immature idea about the extractor: > > - Resolve the addresses of necessary symbols from coredump with > binary format library/tool > > - Restructure the data structures > > - Following the data structures, dump the buffer data page by > page > > > > Any good ideas or suggestions or concerns? One main thing I want us to be careful about: if we do that, we need some kind of versioning information in the tracer that can be checked by the core dumper, so we can do changes internally without breaking the code dump. Thoughts ? Thanks, Mathieu > > > > BR > > Zheng > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 9 10:11:21 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 10:11:21 -0400 Subject: [lttng-dev] [PATCH 1/2] urcu: add hint to DEFINE_URCU_TLS() for compound types In-Reply-To: <1344414673-14714-1-git-send-email-laijs@cn.fujitsu.com> References: <1344414673-14714-1-git-send-email-laijs@cn.fujitsu.com> Message-ID: <20120809141121.GB9064@Krystal> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > Just a hint. merged, thanks! Mathieu > > Signed-off-by: Lai Jiangshan > --- > urcu/tls-compat.h | 15 +++++++++++++++ > 1 files changed, 15 insertions(+), 0 deletions(-) > > diff --git a/urcu/tls-compat.h b/urcu/tls-compat.h > index 9686eca..192a536 100644 > --- a/urcu/tls-compat.h > +++ b/urcu/tls-compat.h > @@ -34,6 +34,21 @@ extern "C" { > > #ifdef CONFIG_RCU_TLS /* Based on ax_tls.m4 */ > > +/* > + * Hint: How to define/declare TLS variables of compound types > + * such as array or function pointers? > + * > + * Answer: Use typedef to assign a type_name to the compound type. > + * Example: Define a TLS variable which is an int array with len=4: > + * > + * typedef int my_int_array_type[4]; > + * DEFINE_URCU_TLS(my_int_array_type, var_name); > + * > + * Another exmaple: > + * typedef void (*call_rcu_flavor)(struct rcu_head *, XXXX); > + * DECLARE_URCU_TLS(call_rcu_flavor, p_call_rcu); > + */ > + > # define DECLARE_URCU_TLS(type, name) \ > CONFIG_RCU_TLS type name > > -- > 1.7.4.4 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 9 10:14:17 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 10:14:17 -0400 Subject: [lttng-dev] [PATCH 2/2] urcu: add notice to URCU_TLS() for it is not async-signal-safe In-Reply-To: <1344414673-14714-2-git-send-email-laijs@cn.fujitsu.com> References: <1344414673-14714-1-git-send-email-laijs@cn.fujitsu.com> <1344414673-14714-2-git-send-email-laijs@cn.fujitsu.com> Message-ID: <20120809141417.GC9064@Krystal> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > Signed-off-by: Lai Jiangshan > --- > urcu/tls-compat.h | 14 ++++++++++++++ > 1 files changed, 14 insertions(+), 0 deletions(-) > > diff --git a/urcu/tls-compat.h b/urcu/tls-compat.h > index 192a536..b7bf363 100644 > --- a/urcu/tls-compat.h > +++ b/urcu/tls-compat.h > @@ -59,6 +59,20 @@ extern "C" { > > #else /* #ifndef CONFIG_RCU_TLS */ > > +/* > + * NOTE: URCU_TLS() is NOT async-signal-safe, you can't use it > + * inside any function which can be called from signal handler. > + * > + * But if pthread_getspecific() is async-signal-safe in your > + * platform, you can make URCU_TLS() async-signal-safe via: > + * ensuring the first call to URCU_TLS() of a given TLS variable of > + * all threads is called earliest from a non-signal handler function. > + * > + * Exmaple: In any thread, the first call of URCU_TLS(rcu_reader) > + * is called from rcu_register_thread(), so we can ensure all later > + * URCU_TLS(rcu_reader) in any thread is async-signal-safe. Hrm. We could also just block all signals within type *__tls_access_ ## name(void) (in tls-compat.h) and make sure it is async-signal-safe I guess ? I would prefer that solution: it would make the code more robust for a rarely taken performance hit. Thoughts ? Thanks, Mathieu > + */ > + > # include > > struct urcu_tls { > -- > 1.7.4.4 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 9 10:20:48 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 10:20:48 -0400 Subject: [lttng-dev] [PATCH] urcu: fix compat_futex_noasync() In-Reply-To: <5021BEE4.6090907@cn.fujitsu.com> References: <5021BEE4.6090907@cn.fujitsu.com> Message-ID: <20120809142048.GA9211@Krystal> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > This patch fix two critical problems: > > 1) compat_futex_cond is not bound to any @uaddr, it services all @uaddr, > if you wakeup only one thread(pthread_cond_signal), the @uaddr of > this waking thread and the @uaddr of the woken-up thread may be different. > the the woken-up thread will very probably go to sleep again > because his own condition is not true. > > *And* this waking thread(FUTEX_WAKE) wake up NOTHING. > > 2) If the caller want to wake up all waiting threads, he will use INT_MAX for @val. > and > for (i = 0; i < INT_MAX; i++) > pthread_cond_signal(&compat_futex_cond); > becomes almost infinity loop. > > Signed-off-by: Lai Jiangshan Good catch ! Merged, thanks! Mathieu > --- > diff --git a/compat_futex.c b/compat_futex.c > index 04de596..bb928e6 100644 > --- a/compat_futex.c > +++ b/compat_futex.c > @@ -43,7 +43,7 @@ static pthread_cond_t compat_futex_cond = PTHREAD_COND_INITIALIZER; > int compat_futex_noasync(int32_t *uaddr, int op, int32_t val, > const struct timespec *timeout, int32_t *uaddr2, int32_t val3) > { > - int ret, i, gret = 0; > + int ret, gret = 0; > > /* > * Check if NULL. Don't let users expect that they are taken into > @@ -67,8 +67,7 @@ int compat_futex_noasync(int32_t *uaddr, int op, int32_t val, > pthread_cond_wait(&compat_futex_cond, &compat_futex_lock); > break; > case FUTEX_WAKE: > - for (i = 0; i < val; i++) > - pthread_cond_signal(&compat_futex_cond); > + pthread_cond_broadcast(&compat_futex_cond); > break; > default: > gret = -EINVAL; -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From dgoulet at efficios.com Thu Aug 9 10:22:14 2012 From: dgoulet at efficios.com (David Goulet) Date: Thu, 09 Aug 2012 10:22:14 -0400 Subject: [lttng-dev] [lttng-tools PATCH] lttng-tools python module v2 In-Reply-To: <5022CF9F.9090500@gmail.com> References: <1344452188-7579-1-git-send-email-danny.serres@efficios.com> <5022CF9F.9090500@gmail.com> Message-ID: <5023C796.2030205@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Yannick Brosseau: > On 2012-08-08 14:56, Danny Serres wrote: >> The lttng-tools Python module can be used to directly control the >> lttng-tools API inside Python, using 'import lttng'. >> >> Therefore, it becomes possible to create a trace, add events, >> start/stop tracing, destroy a session and so on from within >> Python. >> >> The module does not include URI-related functions. >> >> SWIG >= 2.0 is used to create the wrapper and its 'warning md >> variable unused' bug is patched in Makefile.am. >> > > But I don't think that stuff in "extra" should be in the build > system. Indeed. It should not. Is it possible to create the Makefile on "configure" but not building the bindings with "make" ? David > > _______________________________________________ lttng-dev mailing > list lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQI8eTAAoJEELoaioR9I02H24H/iepIFDK0qRvL1pU4S0H2EQx iMhDNGF/CLzG/FnOWEyZgoNR6lhc1lSUjaFs6FL56VcVueBuCW07rsTJDu2NgCYz TsHO29KAR3qrk+ymwSR53TeK2qXt8gr5AXYbc8mK0/DBWWQAxrmJXKSOy8qQ2tmV O0OBWVayDRWttusOknZgabAZF7JBitsoIQ2ZRNRlNtqIUxlbdsQlliujEpzjBu8b iCuZ/ZXxZ/oeClF1Pnoh64+yziHTfsBVFEbee4CpnBiRBn/6CsxEOF/ZscgWWkOl DOUWxj739xHcbtEtWBDo8iu6eqtOCuqVi4RKPqtmk4mqme8Tj7fTQg6YIJ+SZbU= =wKL0 -----END PGP SIGNATURE----- From mathieu.desnoyers at efficios.com Thu Aug 9 10:25:36 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 10:25:36 -0400 Subject: [lttng-dev] [PATCH 1/2] urcu: move exsit code and name it ___cds_wfq_node_sync_next() In-Reply-To: <1344501986-23151-1-git-send-email-laijs@cn.fujitsu.com> References: <1344501986-23151-1-git-send-email-laijs@cn.fujitsu.com> Message-ID: <20120809142536.GB9211@Krystal> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > This code which waits for a node's next pointer until it appears, will be used > many times, move it to a help function and name it ___cds_wfq_node_sync_next(). Merged as: commit b9103f30e18bf81ffbebb6a23215f19f621cd76b Author: Lai Jiangshan Date: Thu Aug 9 10:24:38 2012 -0400 urcu: move busy-wait code and name it ___cds_wfq_node_sync_next() This code which waits for a node's next pointer until it appears, will be used many times, move it to a help function and name it ___cds_wfq_node_sync_next(). Signed-off-by: Lai Jiangshan Signed-off-by: Mathieu Desnoyers Thanks! Mathieu > > Signed-off-by: Lai Jiangshan > --- > urcu/static/wfqueue.h | 36 +++++++++++++++++++++++++----------- > 1 files changed, 25 insertions(+), 11 deletions(-) > > diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h > index 19314f5..636e1af 100644 > --- a/urcu/static/wfqueue.h > +++ b/urcu/static/wfqueue.h > @@ -85,6 +85,29 @@ static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > } > > /* > + * Waiting for enqueuer to complete enqueue and return the next node > + */ > +static inline struct cds_wfq_node * > +___cds_wfq_node_sync_next(struct cds_wfq_node *node) > +{ > + struct cds_wfq_node *next; > + int attempt = 0; > + > + /* > + * Adaptative busy-looping waiting for enqueuer to complete enqueue. > + */ > + while ((next = CMM_LOAD_SHARED(node->next)) == NULL) { > + if (++attempt >= WFQ_ADAPT_ATTEMPTS) { > + poll(NULL, 0, WFQ_WAIT); /* Wait for 10ms */ > + attempt = 0; > + } else > + caa_cpu_relax(); > + } > + > + return next; > +} > + > +/* > * It is valid to reuse and free a dequeued node immediately. > * > * No need to go on a waitqueue here, as there is no possible state in which the > @@ -96,7 +119,6 @@ static inline struct cds_wfq_node * > ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > { > struct cds_wfq_node *node, *next; > - int attempt = 0; > > /* > * Queue is empty if it only contains the dummy node. > @@ -105,16 +127,8 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > return NULL; > node = q->head; > > - /* > - * Adaptative busy-looping waiting for enqueuer to complete enqueue. > - */ > - while ((next = CMM_LOAD_SHARED(node->next)) == NULL) { > - if (++attempt >= WFQ_ADAPT_ATTEMPTS) { > - poll(NULL, 0, WFQ_WAIT); /* Wait for 10ms */ > - attempt = 0; > - } else > - caa_cpu_relax(); > - } > + next = ___cds_wfq_node_sync_next(node); > + > /* > * Move queue head forward. > */ > -- > 1.7.7 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From serres at live.ca Thu Aug 9 10:43:39 2012 From: serres at live.ca (Danny Serres) Date: Thu, 9 Aug 2012 10:43:39 -0400 Subject: [lttng-dev] [lttng-tools PATCH] lttng-tools python module v2 In-Reply-To: <5023C796.2030205@efficios.com> References: <1344452188-7579-1-git-send-email-danny.serres@efficios.com>, <5022CF9F.9090500@gmail.com>, <5023C796.2030205@efficios.com> Message-ID: > Date: Thu, 9 Aug 2012 10:22:14 -0400 > From: dgoulet at efficios.com > To: yannick.brosseau at gmail.com > CC: danny.serres at efficios.com; lttng-dev at lists.lttng.org > Subject: Re: [lttng-dev] [lttng-tools PATCH] lttng-tools python module v2 > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > > Yannick Brosseau: > > On 2012-08-08 14:56, Danny Serres wrote: > >> The lttng-tools Python module can be used to directly control the > >> lttng-tools API inside Python, using 'import lttng'. > >> > >> Therefore, it becomes possible to create a trace, add events, > >> start/stop tracing, destroy a session and so on from within > >> Python. > >> > >> The module does not include URI-related functions. > >> > >> SWIG >= 2.0 is used to create the wrapper and its 'warning md > >> variable unused' bug is patched in Makefile.am. > >> > > > > But I don't think that stuff in "extra" should be in the build > > system. > > Indeed. It should not. > > Is it possible to create the Makefile on "configure" but not building > the bindings with "make" ? > Sure I'll just switch the --disable-python option to --enable-python in the configure.ac Danny > David > > > > > _______________________________________________ lttng-dev mailing > > list lttng-dev at lists.lttng.org > > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > -----BEGIN PGP SIGNATURE----- > > iQEcBAEBCgAGBQJQI8eTAAoJEELoaioR9I02H24H/iepIFDK0qRvL1pU4S0H2EQx > iMhDNGF/CLzG/FnOWEyZgoNR6lhc1lSUjaFs6FL56VcVueBuCW07rsTJDu2NgCYz > TsHO29KAR3qrk+ymwSR53TeK2qXt8gr5AXYbc8mK0/DBWWQAxrmJXKSOy8qQ2tmV > O0OBWVayDRWttusOknZgabAZF7JBitsoIQ2ZRNRlNtqIUxlbdsQlliujEpzjBu8b > iCuZ/ZXxZ/oeClF1Pnoh64+yziHTfsBVFEbee4CpnBiRBn/6CsxEOF/ZscgWWkOl > DOUWxj739xHcbtEtWBDo8iu6eqtOCuqVi4RKPqtmk4mqme8Tj7fTQg6YIJ+SZbU= > =wKL0 > -----END PGP SIGNATURE----- > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexandre.montplaisir at polymtl.ca Thu Aug 9 10:43:15 2012 From: alexandre.montplaisir at polymtl.ca (Alexandre Montplaisir) Date: Thu, 09 Aug 2012 10:43:15 -0400 Subject: [lttng-dev] Question about Transformation to CTF file In-Reply-To: <20120809131314.GA8081@Krystal> References: <20120809131314.GA8081@Krystal> Message-ID: <5023CC83.9070107@polymtl.ca> Hi, On 12-08-09 09:13 AM, Mathieu Desnoyers wrote: > CCing lttng-dev and eclipse lttng plugin developers, > > * Runhua Xu (xurunhua23 at hotmail.com) wrote: > [...] >> way, but I couldn't find more information about that). However, when I >> use the Eclipse Linux Tools to open the CTF file, it reports error >> "Problems occurred when invoking code from plug-in: >> "org.eclipse.jface". I tried the tools with the sample CTF file, which >> is provided by LTTng, it succeeded. Then I looked into the metadata of >> the transferred CTF file (see below codes in blue), it seems that it >> remains the original format, doesn't change based on my input file. > This looks like a bug in the eclipse tool. I'll let Alexandre handle > this part. Please test with "babeltrace" too to see if it is parsed > correctly by the C parser. Could we get a copy of that trace please? If the CTF is invalid, then of course it might have problems parsing, but it could be interesting to see if the error handling is done correctly. To convert traces to and from CTF format, you might want to check out the not-yet-officially-released Javeltrace tool: http://www.dorsal.polymtl.ca/~pproulx/javeltrace/ It won't do what you want to do out of the box, but it really helps understanding the CTF format. Cheers, -- Alexandre Montplaisir DORSAL lab, ?cole Polytechnique de Montr?al From dgoulet at efficios.com Thu Aug 9 10:52:10 2012 From: dgoulet at efficios.com (David Goulet) Date: Thu, 09 Aug 2012 10:52:10 -0400 Subject: [lttng-dev] [lttng-tools PATCH] lttng-tools python module v2 In-Reply-To: References: <1344452188-7579-1-git-send-email-danny.serres@efficios.com>, <5022CF9F.9090500@gmail.com>, <5023C796.2030205@efficios.com> Message-ID: <5023CE9A.9080505@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Danny Serres: > > >> Date: Thu, 9 Aug 2012 10:22:14 -0400 From: dgoulet at efficios.com >> To: yannick.brosseau at gmail.com CC: danny.serres at efficios.com; >> lttng-dev at lists.lttng.org Subject: Re: [lttng-dev] [lttng-tools >> PATCH] lttng-tools python module v2 >> > > Yannick Brosseau: >> On 2012-08-08 14:56, Danny Serres wrote: >>> The lttng-tools Python module can be used to directly control >>> the lttng-tools API inside Python, using 'import lttng'. > >>> Therefore, it becomes possible to create a trace, add events, >>> start/stop tracing, destroy a session and so on from within >>> Python. > >>> The module does not include URI-related functions. > >>> SWIG >= 2.0 is used to create the wrapper and its 'warning md >>> variable unused' bug is patched in Makefile.am. > > >> But I don't think that stuff in "extra" should be in the build >> system. > > Indeed. It should not. > > Is it possible to create the Makefile on "configure" but not > building the bindings with "make" ? > > >> Sure I'll just switch the --disable-python option to >> --enable-python in the configure.ac This seems like a good idea. Does "--enable-python" is right. Should we have something like "--enable-python-bindings" or "--enable-lttngctl-python"... or ... (you get the idea) ? David > >> Danny > > > David > > >> _______________________________________________ lttng-dev >> mailing list lttng-dev at lists.lttng.org >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >> >> _______________________________________________ lttng-dev mailing >> list lttng-dev at lists.lttng.org >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQI86XAAoJEELoaioR9I02XGcIAILmRSD/4ghVjJjggfTjxSuA E0aKMWnXpxBO70I6CgP3Gvkab5gShjxS0juv1L2BqoqvfWfolhfRJ19kbg6+fHbl oCYR+n1XL+7pjjQwjrKN006hEHzQXCInImX1tY0cIFZ3nfBCG6jRnyg57MM36hdC nDs1KYHyhFiUcSKOoZ06Vis3H+YvUYo9zB7mHEf5y7pvdQgaIP01Y5d94MP33GEK CZQpOMDfrlFD/qGe2BsFRw+zMc+O1irkXRszawK7bAekFpHuN+od1nEED2ZhxO8e Et7mU5TQn0U9usrbCHw7lkEVDCaHri/9rjoDqWWpqbHPIdHG7O8uwqQQEtg0Qd8= =Gsvm -----END PGP SIGNATURE----- From serres at live.ca Thu Aug 9 11:10:17 2012 From: serres at live.ca (Danny Serres) Date: Thu, 9 Aug 2012 11:10:17 -0400 Subject: [lttng-dev] [lttng-tools PATCH] lttng-tools python module v2 In-Reply-To: <5023CE9A.9080505@efficios.com> References: <1344452188-7579-1-git-send-email-danny.serres@efficios.com>, <5022CF9F.9090500@gmail.com>, <5023C796.2030205@efficios.com> , <5023CE9A.9080505@efficios.com> Message-ID: > Date: Thu, 9 Aug 2012 10:52:10 -0400 > From: dgoulet at efficios.com > To: serres at live.ca > CC: yannick.brosseau at gmail.com; lttng-dev at lists.lttng.org > Subject: Re: [lttng-dev] [lttng-tools PATCH] lttng-tools python module v2 > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Danny Serres: > > > > > >> Date: Thu, 9 Aug 2012 10:22:14 -0400 From: dgoulet at efficios.com > >> To: yannick.brosseau at gmail.com CC: danny.serres at efficios.com; > >> lttng-dev at lists.lttng.org Subject: Re: [lttng-dev] [lttng-tools > >> PATCH] lttng-tools python module v2 > >> > > > > Yannick Brosseau: > >> On 2012-08-08 14:56, Danny Serres wrote: > >>> The lttng-tools Python module can be used to directly control > >>> the lttng-tools API inside Python, using 'import lttng'. > > > >>> Therefore, it becomes possible to create a trace, add events, > >>> start/stop tracing, destroy a session and so on from within > >>> Python. > > > >>> The module does not include URI-related functions. > > > >>> SWIG >= 2.0 is used to create the wrapper and its 'warning md > >>> variable unused' bug is patched in Makefile.am. > > > > > >> But I don't think that stuff in "extra" should be in the build > >> system. > > > > Indeed. It should not. > > > > Is it possible to create the Makefile on "configure" but not > > building the bindings with "make" ? > > > > > >> Sure I'll just switch the --disable-python option to > >> --enable-python in the configure.ac > > This seems like a good idea. Does "--enable-python" is right. Should > we have something like "--enable-python-bindings" or > "--enable-lttngctl-python"... or ... (you get the idea) ? > --enable-python-bindings seems more elucidatory, I'll go with that Danny > David > > > > >> Danny > > > > > > David > > > > > >> _______________________________________________ lttng-dev > >> mailing list lttng-dev at lists.lttng.org > >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > >> > >> _______________________________________________ lttng-dev mailing > >> list lttng-dev at lists.lttng.org > >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > -----BEGIN PGP SIGNATURE----- > > iQEcBAEBCgAGBQJQI86XAAoJEELoaioR9I02XGcIAILmRSD/4ghVjJjggfTjxSuA > E0aKMWnXpxBO70I6CgP3Gvkab5gShjxS0juv1L2BqoqvfWfolhfRJ19kbg6+fHbl > oCYR+n1XL+7pjjQwjrKN006hEHzQXCInImX1tY0cIFZ3nfBCG6jRnyg57MM36hdC > nDs1KYHyhFiUcSKOoZ06Vis3H+YvUYo9zB7mHEf5y7pvdQgaIP01Y5d94MP33GEK > CZQpOMDfrlFD/qGe2BsFRw+zMc+O1irkXRszawK7bAekFpHuN+od1nEED2ZhxO8e > Et7mU5TQn0U9usrbCHw7lkEVDCaHri/9rjoDqWWpqbHPIdHG7O8uwqQQEtg0Qd8= > =Gsvm > -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From danny.serres at efficios.com Thu Aug 9 11:24:02 2012 From: danny.serres at efficios.com (Danny Serres) Date: Thu, 9 Aug 2012 11:24:02 -0400 Subject: [lttng-dev] [lttng-tools PATCH] lttng-tools python module v3 Message-ID: <1344525842-10176-1-git-send-email-danny.serres@efficios.com> The lttng-tools Python module can be used to directly control the lttng-tools API inside Python, using 'import lttng'. Therefore, it becomes possible to create a trace, add events, start/stop tracing, destroy a session and so on from within Python. The module does not include URI-related functions. SWIG >= 2.0 is used to create the wrapper and its 'warning md variable unused' bug is patched in Makefile.am. In the interface file, struct and enum are directly copied from lttng.h (all changes must be made in both files). To install with python bingings, configure using --enable-python-bindings Signed-off-by: Danny Serres Signed-off-by: Yannick Brosseau --- .gitignore | 4 + Makefile.am | 1 + README | 15 + config/ax_pkg_swig.m4 | 135 ++++ configure.ac | 48 ++ doc/python-howto.txt | 70 ++ extras/Makefile.am | 1 + extras/bindings/Makefile.am | 1 + extras/bindings/swig/Makefile.am | 3 + extras/bindings/swig/python/Makefile.am | 25 + extras/bindings/swig/python/lttng.i.in | 1029 ++++++++++++++++++++++++++ extras/bindings/swig/python/tests/example.py | 109 +++ extras/bindings/swig/python/tests/run.sh | 1 + extras/bindings/swig/python/tests/tests.py | 310 ++++++++ 14 files changed, 1752 insertions(+) create mode 100644 config/ax_pkg_swig.m4 create mode 100644 doc/python-howto.txt create mode 100644 extras/Makefile.am create mode 100644 extras/bindings/Makefile.am create mode 100644 extras/bindings/swig/Makefile.am create mode 100644 extras/bindings/swig/python/Makefile.am create mode 100644 extras/bindings/swig/python/lttng.i.in create mode 100644 extras/bindings/swig/python/tests/example.py create mode 100644 extras/bindings/swig/python/tests/run.sh create mode 100644 extras/bindings/swig/python/tests/tests.py diff --git a/.gitignore b/.gitignore index 6d04272..c94f759 100644 --- a/.gitignore +++ b/.gitignore @@ -42,6 +42,10 @@ src/lib/lttng-ctl/filter-parser.c src/lib/lttng-ctl/filter-parser.h src/lib/lttng-ctl/filter-parser.output +extras/bindings/swig/python/lttng.i +extras/bindings/swig/python/lttng.py +extras/bindings/swig/python/lttng_wrap.c + # Tests test_sessions test_kernel_data_trace diff --git a/Makefile.am b/Makefile.am index 13d3f93..b0537ce 100644 --- a/Makefile.am +++ b/Makefile.am @@ -2,6 +2,7 @@ ACLOCAL_AMFLAGS = -I config SUBDIRS = src \ tests \ + extras \ include \ doc diff --git a/README b/README index fe9e4c2..1898af9 100644 --- a/README +++ b/README @@ -28,6 +28,16 @@ REQUIREMENTS: * Debian/Ubuntu package: libpopt-dev + - SWIG >= 2.0 (optional) + Needed for Python bindings + + * Debian/Ubuntu package: swig2.0 + + - python-dev (optional) + Python headers + + * Debian/Ubuntu package: python-dev + - For kernel tracing: modprobe For developers using the git tree: @@ -62,6 +72,8 @@ INSTALLATION INSTRUCTIONS: If compiling from the git repository, run ./bootstrap before running the configure script, to generate it. + If you want Python bindings, run ./configure --enable-python-bindings. + USAGE: Please see doc/quickstart.txt to help you start tracing. You can also use the @@ -71,6 +83,9 @@ lttng enable-event -h). A network streaming HOWTO can be found in doc/streaming-howto.txt which quickly helps you understand how to stream a LTTng 2.0 trace. +A Python HOWTO can be found in doc/python-howto.txt which quickly +helps you understand how to use the Python module to control the LTTng API. + PACKAGE CONTENTS: This package contains the following elements: diff --git a/config/ax_pkg_swig.m4 b/config/ax_pkg_swig.m4 new file mode 100644 index 0000000..e112f3d --- /dev/null +++ b/config/ax_pkg_swig.m4 @@ -0,0 +1,135 @@ +# =========================================================================== +# http://www.gnu.org/software/autoconf-archive/ax_pkg_swig.html +# =========================================================================== +# +# SYNOPSIS +# +# AX_PKG_SWIG([major.minor.micro], [action-if-found], [action-if-not-found]) +# +# DESCRIPTION +# +# This macro searches for a SWIG installation on your system. If found, +# then SWIG is AC_SUBST'd; if not found, then $SWIG is empty. If SWIG is +# found, then SWIG_LIB is set to the SWIG library path, and AC_SUBST'd. +# +# You can use the optional first argument to check if the version of the +# available SWIG is greater than or equal to the value of the argument. It +# should have the format: N[.N[.N]] (N is a number between 0 and 999. Only +# the first N is mandatory.) If the version argument is given (e.g. +# 1.3.17), AX_PKG_SWIG checks that the swig package is this version number +# or higher. +# +# As usual, action-if-found is executed if SWIG is found, otherwise +# action-if-not-found is executed. +# +# In configure.in, use as: +# +# AX_PKG_SWIG(1.3.17, [], [ AC_MSG_ERROR([SWIG is required to build..]) ]) +# AX_SWIG_ENABLE_CXX +# AX_SWIG_MULTI_MODULE_SUPPORT +# AX_SWIG_PYTHON +# +# LICENSE +# +# Copyright (c) 2008 Sebastian Huber +# Copyright (c) 2008 Alan W. Irwin +# Copyright (c) 2008 Rafael Laboissiere +# Copyright (c) 2008 Andrew Collier +# Copyright (c) 2011 Murray Cumming +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation; either version 2 of the License, or (at your +# option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program. If not, see . +# +# As a special exception, the respective Autoconf Macro's copyright owner +# gives unlimited permission to copy, distribute and modify the configure +# scripts that are the output of Autoconf when processing the Macro. You +# need not follow the terms of the GNU General Public License when using +# or distributing such scripts, even though portions of the text of the +# Macro appear in them. The GNU General Public License (GPL) does govern +# all other use of the material that constitutes the Autoconf Macro. +# +# This special exception to the GPL applies to versions of the Autoconf +# Macro released by the Autoconf Archive. When you make and distribute a +# modified version of the Autoconf Macro, you may extend this special +# exception to the GPL to apply to your modified version as well. + +#serial 8 + +AC_DEFUN([AX_PKG_SWIG],[ + # Ubuntu has swig 2.0 as /usr/bin/swig2.0 + AC_PATH_PROGS([SWIG],[swig swig2.0]) + if test -z "$SWIG" ; then + m4_ifval([$3],[$3],[:]) + elif test -n "$1" ; then + AC_MSG_CHECKING([SWIG version]) + [swig_version=`$SWIG -version 2>&1 | grep 'SWIG Version' | sed 's/.*\([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\).*/\1/g'`] + AC_MSG_RESULT([$swig_version]) + if test -n "$swig_version" ; then + # Calculate the required version number components + [required=$1] + [required_major=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_major" ; then + [required_major=0] + fi + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] + [required_minor=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_minor" ; then + [required_minor=0] + fi + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] + [required_patch=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_patch" ; then + [required_patch=0] + fi + # Calculate the available version number components + [available=$swig_version] + [available_major=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_major" ; then + [available_major=0] + fi + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] + [available_minor=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_minor" ; then + [available_minor=0] + fi + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] + [available_patch=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_patch" ; then + [available_patch=0] + fi + # Convert the version tuple into a single number for easier comparison. + # Using base 100 should be safe since SWIG internally uses BCD values + # to encode its version number. + required_swig_vernum=`expr $required_major \* 10000 \ + \+ $required_minor \* 100 \+ $required_patch` + available_swig_vernum=`expr $available_major \* 10000 \ + \+ $available_minor \* 100 \+ $available_patch` + + if test $available_swig_vernum -lt $required_swig_vernum; then + AC_MSG_WARN([SWIG version >= $1 is required. You have $swig_version.]) + SWIG='' + m4_ifval([$3],[$3],[]) + else + AC_MSG_CHECKING([for SWIG library]) + SWIG_LIB=`$SWIG -swiglib` + AC_MSG_RESULT([$SWIG_LIB]) + m4_ifval([$2],[$2],[]) + fi + else + AC_MSG_WARN([cannot determine SWIG version]) + SWIG='' + m4_ifval([$3],[$3],[]) + fi + fi + AC_SUBST([SWIG_LIB]) +]) diff --git a/configure.ac b/configure.ac index 17e6b67..5e8edb4 100644 --- a/configure.ac +++ b/configure.ac @@ -154,6 +154,42 @@ AC_CHECK_LIB([c], [open_memstream], ] ) +# For Python +# SWIG version needed or newer: +swig_version=2.0.0 + +AC_ARG_ENABLE([python-bindings], + [AC_HELP_STRING([--enable-python-bindings], + [compile Python bindings])], + [enable_python=yes], [enable_python=no]) + +AM_CONDITIONAL([USE_PYTHON], [test "x${enable_python:-yes}" = xyes]) + +if test "x${enable_python:-yes}" = xyes; then + AX_PKG_SWIG($swig_version, [], [ AC_MSG_ERROR([SWIG $swig_version or newer is needed]) ]) + AM_PATH_PYTHON + + AC_ARG_VAR([PYTHON_INCLUDE], [Include flags for python, bypassing python-config]) + AC_ARG_VAR([PYTHON_CONFIG], [Path to python-config]) + AS_IF([test -z "$PYTHON_INCLUDE"], [ + AS_IF([test -z "$PYTHON_CONFIG"], [ + AC_PATH_PROGS([PYTHON_CONFIG], + [python$PYTHON_VERSION-config python-config], + [no], + [`dirname $PYTHON`]) + AS_IF([test "$PYTHON_CONFIG" = no], [AC_MSG_ERROR([cannot find python-config for $PYTHON.])]) + ]) + AC_MSG_CHECKING([python include flags]) + PYTHON_INCLUDE=`$PYTHON_CONFIG --includes` + AC_MSG_RESULT([$PYTHON_INCLUDE]) + ]) + +else + AC_MSG_NOTICE([You may configure with --enable-python-bindings ]dnl +[if you want Python bindings.]) + +fi + # Option to only build the consumer daemon and its libraries AC_ARG_WITH([consumerd-only], AS_HELP_STRING([--with-consumerd-only],[Only build the consumer daemon [default=no]]), @@ -198,6 +234,10 @@ AC_CONFIG_FILES([ doc/Makefile doc/man/Makefile include/Makefile + extras/Makefile + extras/bindings/Makefile + extras/bindings/swig/Makefile + extras/bindings/swig/python/Makefile src/Makefile src/common/Makefile src/common/kernel-ctl/Makefile @@ -260,6 +300,14 @@ AS_IF([test "x$lttng_ust_support" = "xyes"],[ AS_ECHO("Disabled") ]) +#Python binding enabled/disabled +AS_ECHO_N("Python binding: ") +AS_IF([test "x${enable_python:-yes}" = xyes], [ + AS_ECHO("Enabled") +],[ + AS_ECHO("Disabled") +]) + # Do we build only the consumerd, or everything AS_IF([test "x$consumerd_only" = "xyes"],[ AS_ECHO("Only the consumerd daemon will be built.") diff --git a/doc/python-howto.txt b/doc/python-howto.txt new file mode 100644 index 0000000..5f7fcfb --- /dev/null +++ b/doc/python-howto.txt @@ -0,0 +1,70 @@ +PYTHON BINDINGS +---------------- + +This is a brief howto for using the lttng-tools Python module. + +By default, the Python bindings are not installed. +If you wish the Python bindings, you can configure with the +--enable-python-bindings option during the installation procedure: + + $ ./configure --enable-python-bindings + +The Python module is automatically generated using SWIG, therefore the +swig2.0 package on Debian/Ubuntu is requied. + +Once installed, the Python module can be used by importing it in Python. +In the Python interpreter: + + >>> import lttng + +Example: +---------------- + +Quick example using Python to trace with LTTng. + +1) Run python + + $ python + +2) Import the lttng module + + >>> import lttng + +3) Create a session + + >>> lttng.create("session-name", "path/to/trace") + +4) Create a handle for the tracing session and domain + + >>> domain = lttng.Domain() + >>> domain.type = lttng.DOMAIN_KERNEL * + >>> handle = lttng.Handle("session-name", domain) + +* This line is somewhat useless since domain.type is set to 0 + by default, the corresponding value of lttng.DOMAIN_KERNEL + +5) Enable all Kernel events + + >>> event = lttng.Event() + >>> event.type = lttng.EVENT_TRACEPOINT * + >>> event.loglevel_type = lttng.EVENT_LOGLEVEL_ALL * + >>> lttng.enable_event(handle, event, None) + +* These two lines are somewhat useless since event.type + and event.loglevel_type are by default set to 0, the + corresponding value of lttng.EVENT_TRACEPOINT and + lttng.EVENT_LOGLEVEL_ALL + +5) Start tracing + + >>> lttng.start("session-name") + +6) Stop tracing + + >>> lttng.stop("session-name") + +7) Destroy the tracing session + + >>> lttng.destroy("session-name") + +For an example script with more details, see extras/bindings/swig/python/tests/example.py diff --git a/extras/Makefile.am b/extras/Makefile.am new file mode 100644 index 0000000..925dc2e --- /dev/null +++ b/extras/Makefile.am @@ -0,0 +1 @@ +SUBDIRS = bindings diff --git a/extras/bindings/Makefile.am b/extras/bindings/Makefile.am new file mode 100644 index 0000000..1585422 --- /dev/null +++ b/extras/bindings/Makefile.am @@ -0,0 +1 @@ +SUBDIRS = swig diff --git a/extras/bindings/swig/Makefile.am b/extras/bindings/swig/Makefile.am new file mode 100644 index 0000000..dcd868d --- /dev/null +++ b/extras/bindings/swig/Makefile.am @@ -0,0 +1,3 @@ +if USE_PYTHON +SUBDIRS = python +endif diff --git a/extras/bindings/swig/python/Makefile.am b/extras/bindings/swig/python/Makefile.am new file mode 100644 index 0000000..85237a4 --- /dev/null +++ b/extras/bindings/swig/python/Makefile.am @@ -0,0 +1,25 @@ +lttng.i: lttng.i.in + sed "s/LTTNG_VERSION_STR/LTTng $(PACKAGE_VERSION)/g" lttng.i + +AM_CFLAGS = -I$(PYTHON_INCLUDE) -I$(top_srcdir)/lib/lttng-ctl -I../common \ + $(BUDDY_CFLAGS) + +EXTRA_DIST = lttng.i +python_PYTHON = lttng.py +pyexec_LTLIBRARIES = _lttng.la + +MAINTAINERCLEANFILES = lttng_wrap.c lttng.py + +_lttng_la_SOURCES = lttng_wrap.c + +_lttng_la_LDFLAGS = -module + +_lttng_la_LIBADD = $(top_srcdir)/src/lib/lttng-ctl/liblttng-ctl.la \ + $(top_srcdir)/src/common/sessiond-comm/libsessiond-comm.la + +# SWIG 'warning md variable unused' fixed after SWIG build: +lttng_wrap.c: lttng.i + $(SWIG) -python -I. -I$(top_srcdir)/src/common/sessiond-comm lttng.i + sed -i "s/PyObject \*m, \*d, \*md;/PyObject \*m, \*d;\n#if defined(SWIGPYTHON_BUILTIN)\nPyObject *md;\n#endif/g" lttng_wrap.c + sed -i "s/md = d/d/g" lttng_wrap.c + sed -i "s/(void)public_symbol;/(void)public_symbol;\n md = d;/g" lttng_wrap.c diff --git a/extras/bindings/swig/python/lttng.i.in b/extras/bindings/swig/python/lttng.i.in new file mode 100644 index 0000000..0d6d1e9 --- /dev/null +++ b/extras/bindings/swig/python/lttng.i.in @@ -0,0 +1,1029 @@ +%define DOCSTRING +"LTTNG_VERSION_STR + +The LTTng project aims at providing highly efficient tracing tools for Linux. +It's tracers help tracking down performance issues and debugging problems involving +multiple concurrent processes and threads. Tracing across multiple systems is also possible." +%enddef + +%module(docstring=DOCSTRING) lttng + +%include "typemaps.i" +%include "pyabc.i" +%{ +#define SWIG_FILE_WITH_INIT +#include +%} + +typedef unsigned int uint32_t; +typedef int int32_t; +typedef unsigned long long uint64_t; +typedef long pid_t; + + +// ============================================= +// ENUMS +// These are directly taken from lttng.h. +// Any change to these enums must also be +// made here. +// ============================================= + +%rename("DOMAIN_KERNEL") LTTNG_DOMAIN_KERNEL; +%rename("DOMAIN_UST") LTTNG_DOMAIN_UST; +enum lttng_domain_type { + LTTNG_DOMAIN_KERNEL = 1, + LTTNG_DOMAIN_UST = 2, +}; + +%rename("EVENT_ALL") LTTNG_EVENT_ALL; +%rename("EVENT_TRACEPOINT") LTTNG_EVENT_TRACEPOINT; +%rename("EVENT_PROBE") LTTNG_EVENT_PROBE; +%rename("EVENT_FUNCTION")LTTNG_EVENT_FUNCTION; +%rename("EVENT_FUNCTION_ENTRY") LTTNG_EVENT_FUNCTION_ENTRY; +%rename("EVENT_NOOP") LTTNG_EVENT_NOOP; +%rename("EVENT_SYSCALL") LTTNG_EVENT_SYSCALL; +enum lttng_event_type { + LTTNG_EVENT_ALL = -1, + LTTNG_EVENT_TRACEPOINT = 0, + LTTNG_EVENT_PROBE = 1, + LTTNG_EVENT_FUNCTION = 2, + LTTNG_EVENT_FUNCTION_ENTRY = 3, + LTTNG_EVENT_NOOP = 4, + LTTNG_EVENT_SYSCALL = 5, +}; + +%rename("EVENT_LOGLEVEL_ALL") LTTNG_EVENT_LOGLEVEL_ALL; +%rename("EVENT_LOGLEVEL_RANGE") LTTNG_EVENT_LOGLEVEL_RANGE; +%rename("EVENT_LOGLEVEL_SINGLE") LTTNG_EVENT_LOGLEVEL_SINGLE; +enum lttng_loglevel_type { + LTTNG_EVENT_LOGLEVEL_ALL = 0, + LTTNG_EVENT_LOGLEVEL_RANGE = 1, + LTTNG_EVENT_LOGLEVEL_SINGLE = 2, +}; + +%rename("LOGLEVEL_EMERG") LTTNG_LOGLEVEL_EMERG; +%rename("LOGLEVEL_ALERT") LTTNG_LOGLEVEL_ALERT; +%rename("LOGLEVEL_CRIT") LTTNG_LOGLEVEL_CRIT; +%rename("LOGLEVEL_ERR") LTTNG_LOGLEVEL_ERR; +%rename("LOGLEVEL_WARNING") LTTNG_LOGLEVEL_WARNING; +%rename("LOGLEVEL_NOTICE") LTTNG_LOGLEVEL_NOTICE; +%rename("LOGLEVEL_INFO") LTTNG_LOGLEVEL_INFO; +%rename("LOGLEVEL_DEBUG_SYSTEM") LTTNG_LOGLEVEL_DEBUG_SYSTEM; +%rename("LOGLEVEL_DEBUG_PROGRAM") LTTNG_LOGLEVEL_DEBUG_PROGRAM; +%rename("LOGLEVEL_DEBUG_PROCESS") LTTNG_LOGLEVEL_DEBUG_PROCESS; +%rename("LOGLEVEL_DEBUG_MODULE") LTTNG_LOGLEVEL_DEBUG_MODULE; +%rename("LOGLEVEL_DEBUG_UNIT") LTTNG_LOGLEVEL_DEBUG_UNIT; +%rename("LOGLEVEL_DEBUG_FUNCTION") LTTNG_LOGLEVEL_DEBUG_FUNCTION; +%rename("LOGLEVEL_DEBUG_LINE") LTTNG_LOGLEVEL_DEBUG_LINE; +%rename("LOGLEVEL_DEBUG") LTTNG_LOGLEVEL_DEBUG; +enum lttng_loglevel { + LTTNG_LOGLEVEL_EMERG = 0, + LTTNG_LOGLEVEL_ALERT = 1, + LTTNG_LOGLEVEL_CRIT = 2, + LTTNG_LOGLEVEL_ERR = 3, + LTTNG_LOGLEVEL_WARNING = 4, + LTTNG_LOGLEVEL_NOTICE = 5, + LTTNG_LOGLEVEL_INFO = 6, + LTTNG_LOGLEVEL_DEBUG_SYSTEM = 7, + LTTNG_LOGLEVEL_DEBUG_PROGRAM = 8, + LTTNG_LOGLEVEL_DEBUG_PROCESS = 9, + LTTNG_LOGLEVEL_DEBUG_MODULE = 10, + LTTNG_LOGLEVEL_DEBUG_UNIT = 11, + LTTNG_LOGLEVEL_DEBUG_FUNCTION = 12, + LTTNG_LOGLEVEL_DEBUG_LINE = 13, + LTTNG_LOGLEVEL_DEBUG = 14, +}; + +%rename("EVENT_SPLICE") LTTNG_EVENT_SPLICE; +%rename("EVENT_MMAP") LTTNG_EVENT_MMAP; +enum lttng_event_output { + LTTNG_EVENT_SPLICE = 0, + LTTNG_EVENT_MMAP = 1, +}; + +%rename("EVENT_CONTEXT_PID") LTTNG_EVENT_CONTEXT_PID; +%rename("EVENT_CONTEXT_PERF_COUNTER") LTTNG_EVENT_CONTEXT_PERF_COUNTER; +%rename("EVENT_CONTEXT_PROCNAME") LTTNG_EVENT_CONTEXT_PROCNAME; +%rename("EVENT_CONTEXT_PRIO") LTTNG_EVENT_CONTEXT_PRIO; +%rename("EVENT_CONTEXT_NICE") LTTNG_EVENT_CONTEXT_NICE; +%rename("EVENT_CONTEXT_VPID") LTTNG_EVENT_CONTEXT_VPID; +%rename("EVENT_CONTEXT_TID") LTTNG_EVENT_CONTEXT_TID; +%rename("EVENT_CONTEXT_VTID") LTTNG_EVENT_CONTEXT_VTID; +%rename("EVENT_CONTEXT_PPID") LTTNG_EVENT_CONTEXT_PPID; +%rename("EVENT_CONTEXT_VPPID") LTTNG_EVENT_CONTEXT_VPPID; +%rename("EVENT_CONTEXT_PTHREAD_ID") LTTNG_EVENT_CONTEXT_PTHREAD_ID; +enum lttng_event_context_type { + LTTNG_EVENT_CONTEXT_PID = 0, + LTTNG_EVENT_CONTEXT_PERF_COUNTER = 1, + LTTNG_EVENT_CONTEXT_PROCNAME = 2, + LTTNG_EVENT_CONTEXT_PRIO = 3, + LTTNG_EVENT_CONTEXT_NICE = 4, + LTTNG_EVENT_CONTEXT_VPID = 5, + LTTNG_EVENT_CONTEXT_TID = 6, + LTTNG_EVENT_CONTEXT_VTID = 7, + LTTNG_EVENT_CONTEXT_PPID = 8, + LTTNG_EVENT_CONTEXT_VPPID = 9, + LTTNG_EVENT_CONTEXT_PTHREAD_ID = 10, +}; + +%rename("CALIBRATE_FUNCTION") LTTNG_CALIBRATE_FUNCTION; +enum lttng_calibrate_type { + LTTNG_CALIBRATE_FUNCTION = 0, +}; + + + +// ============================================= +// TYPEMAPS +// ============================================= + +//list_sessions +%typemap(argout) struct lttng_session **sessions{ + + int l = PyInt_AsSsize_t($result); + if (l >= 0) + { + PyObject *sessions = PyList_New(0); + int i; + for(i=0; i= 0) + { + PyObject *dom = PyList_New(0); + int i; + for(i=0; i= 0) + { + PyObject *chan = PyList_New(0); + int i; + for(i=0; i= 0) + { + PyObject *events = PyList_New(0); + int i; + for(i=0; i int + +Create a new tracing session using name and path. +Returns size of returned session payload data or a negative error code." +int lttng_create_session(const char *name, const char *path); + + +%feature("docstring")"destroy(str name) -> int + +Tear down tracing session using name. +Returns size of returned session payload data or a negative error code." +int lttng_destroy_session(const char *name); + + +%feature("docstring")"session_daemon_alive() -> int + +Check if session daemon is alive. +Return 1 if alive or 0 if not. +On error returns a negative value." +int lttng_session_daemon_alive(void); + + +%feature("docstring")"set_tracing_group(str name) -> int + +Sets the tracing_group variable with name. +This function allocates memory pointed to by tracing_group. +On success, returns 0, on error, returns -1 (null name) or -ENOMEM." +int lttng_set_tracing_group(const char *name); + + +%feature("docstring")"strerror(int code) -> char + +Returns a human readable string describing +the error code (a negative value)." +const char *lttng_strerror(int code); + + +%feature("docstring")"start(str session_name) -> int + +Start tracing for all traces of the session. +Returns size of returned session payload data or a negative error code." +int lttng_start_tracing(const char *session_name); + + +%feature("docstring")"stop(str session_name) -> int + +Stop tracing for all traces of the session. +Returns size of returned session payload data or a negative error code." +int lttng_stop_tracing(const char *session_name); + + +%feature("docstring")"channel_set_default_attr(Domain domain, ChannelAttr attr) + +Set default channel attributes. +If either or both of the arguments are null, attr content is zeroe'd." +void lttng_channel_set_default_attr(struct lttng_domain *domain, struct lttng_channel_attr *attr); + + +// ============================================= +// Python redefinition of some functions +// (List and Handle-related) +// ============================================= + +%feature("docstring")"" +%pythoncode %{ + +def list_sessions(): + """ + list_sessions() -> dict + + Ask the session daemon for all available sessions. + Returns a dict of Session instances, the key is the name; + on error, returns a negative value. + """ + + ses_list = _lttng_list_sessions() + if type(ses_list) is int: + return ses_list + + sessions = {} + + for ses_elements in ses_list: + ses = Session() + ses.name = ses_elements[0] + ses.path = ses_elements[1] + ses.enabled = ses_elements[2] + ses.padding = ses_elements[3] + + sessions[ses.name] = ses + + return sessions + + +def list_domains(session_name): + """ + list_domains(str session_name) -> list + + Ask the session daemon for all available domains of a session. + Returns a list of Domain instances; + on error, returns a negative value. + """ + + dom_list = _lttng_list_domains(session_name) + if type(dom_list) is int: + return dom_list + + domains = [] + + for dom_elements in dom_list: + dom = Domain() + dom.type = dom_elements[0] + dom.paddinf = dom_elements[1] + dom.attr.pid = dom_elements[2] + dom.attr.exec_name = dom_elements[3] + dom.attr.padding = dom_elements[4] + + domains.append(dom) + + return domains + + +def list_channels(handle): + """ + list_channels(Handle handle) -> dict + + Ask the session daemon for all available channels of a session. + Returns a dict of Channel instances, the key is the name; + on error, returns a negative value. + """ + + try: + chan_list = _lttng_list_channels(handle._h) + except AttributeError: + raise TypeError("in method 'list_channels', argument 1 must be a Handle instance") + + if type(chan_list) is int: + return chan_list + + channels = {} + + for channel_elements in chan_list: + chan = Channel() + chan.name = channel_elements[0] + chan.enabled = channel_elements[1] + chan.padding = channel_elements[2] + chan.attr.overwrite = channel_elements[3][0] + chan.attr.subbuf_size = channel_elements[3][1] + chan.attr.num_subbuf = channel_elements[3][2] + chan.attr.switch_timer_interval = channel_elements[3][3] + chan.attr.read_timer_interval = channel_elements[3][4] + chan.attr.output = channel_elements[3][5] + chan.attr.padding = channel_elements[3][6] + + channels[chan.name] = chan + + return channels + + +def list_events(handle, channel_name): + """ + list_events(Handle handle, str channel_name) -> dict + + Ask the session daemon for all available events of a session channel. + Returns a dict of Event instances, the key is the name; + on error, returns a negative value. + """ + + try: + ev_list = _lttng_list_events(handle._h, channel_name) + except AttributeError: + raise TypeError("in method 'list_events', argument 1 must be a Handle instance") + + if type(ev_list) is int: + return ev_list + + events = {} + + for ev_elements in ev_list: + ev = Event() + ev.name = ev_elements[0] + ev.type = ev_elements[1] + ev.loglevel_type = ev_elements[2] + ev.loglevel = ev_elements[3] + ev.enabled = ev_elements[4] + ev.pid = ev_elements[5] + ev.attr.padding = ev_elements[6] + ev.attr.probe.addr = ev_elements[7][0] + ev.attr.probe.offset = ev_elements[7][1] + ev.attr.probe.symbol_name = ev_elements[7][2] + ev.attr.probe.padding = ev_elements[7][3] + ev.attr.ftrace.symbol_name = ev_elements[8][0] + ev.attr.ftrace.padding = ev_elements[8][1] + ev.attr.padding = ev_elements[9] + + events[ev.name] = ev + + return events + + +def list_tracepoints(handle): + """ + list_tracepoints(Handle handle) -> dict + + Returns a dict of Event instances, the key is the name; + on error, returns a negative value. + """ + + try: + ev_list = _lttng_list_tracepoints(handle._h) + except AttributeError: + raise TypeError("in method 'list_tracepoints', argument 1 must be a Handle instance") + + if type(ev_list) is int: + return ev_list + + events = {} + + for ev_elements in ev_list: + ev = Event() + ev.name = ev_elements[0] + ev.type = ev_elements[1] + ev.loglevel_type = ev_elements[2] + ev.loglevel = ev_elements[3] + ev.enabled = ev_elements[4] + ev.pid = ev_elements[5] + ev.attr.padding = ev_elements[6] + ev.attr.probe.addr = ev_elements[7][0] + ev.attr.probe.offset = ev_elements[7][1] + ev.attr.probe.symbol_name = ev_elements[7][2] + ev.attr.probe.padding = ev_elements[7][3] + ev.attr.ftrace.symbol_name = ev_elements[8][0] + ev.attr.ftrace.padding = ev_elements[8][1] + ev.attr.padding = ev_elements[9] + + events[ev.name] = ev + + return events + + +def register_consumer(handle, socket_path): + """ + register_consumer(Handle handle, str socket_path) -> int + + Register an outside consumer. + Returns size of returned session payload data or a negative error code. + """ + + try: + return _lttng_register_consumer(handle._h, socket_path) + except AttributeError: + raise TypeError("in method 'register_consumer', argument 1 must be a Handle instance") + + +def add_context(handle, event_context, event_name, channel_name): + """ + add_context(Handle handle, EventContext ctx, + str event_name, str channel_name) -> int + + Add context to event and/or channel. + If event_name is None, the context is applied to all events of the channel. + If channel_name is None, a lookup of the event's channel is done. + If both are None, the context is applied to all events of all channels. + Returns the size of the returned payload data or a negative error code. + """ + + try: + return _lttng_add_context(handle._h, event_context, event_name, channel_name) + except AttributeError: + raise TypeError("in method 'add_context', argument 1 must be a Handle instance") + + +def enable_event(handle, event, channel_name): + """ + enable_event(Handle handle, Event event, + str channel_name) -> int + + Enable event(s) for a channel. + If no event name is specified, all events are enabled. + If no channel name is specified, the default 'channel0' is used. + Returns size of returned session payload data or a negative error code. + """ + + try: + return _lttng_enable_event(handle._h, event, channel_name) + except AttributeError: + raise TypeError("in method 'enable_event', argument 1 must be a Handle instance") + + +def enable_channel(handle, channel): + """ + enable_channel(Handle handle, Channel channel -> int + + Enable channel per domain + Returns size of returned session payload data or a negative error code. + """ + + try: + return _lttng_enable_channel(handle._h, channel) + except AttributeError: + raise TypeError("in method 'enable_channel', argument 1 must be a Handle instance") + + +def disable_event(handle, name, channel_name): + """ + disable_event(Handle handle, str name, str channel_name) -> int + + Disable event(s) of a channel and domain. + If no event name is specified, all events are disabled. + If no channel name is specified, the default 'channel0' is used. + Returns size of returned session payload data or a negative error code + """ + + try: + return _lttng_disable_event(handle._h, name, channel_name) + except AttributeError: + raise TypeError("in method 'disable_event', argument 1 must be a Handle instance") + + +def disable_channel(handle, name): + """ + disable_channel(Handle handle, str name) -> int + + All tracing will be stopped for registered events of the channel. + Returns size of returned session payload data or a negative error code. + """ + + try: + return _lttng_disable_channel(handle._h, name) + except AttributeError: + raise TypeError("in method 'disable_channel', argument 1 must be a Handle instance") + + +def calibrate(handle, calibrate): + """ + calibrate(Handle handle, Calibrate calibrate) -> int + + Quantify LTTng overhead. + Returns size of returned session payload data or a negative error code. + """ + + try: + return _lttng_calibrate(handle._h, calibrate) + except AttributeError: + raise TypeError("in method 'calibrate', argument 1 must be a Handle instance") +%} + + +// ============================================= +// Handle class +// Used to prevent freeing unallocated memory +// ============================================= + +%feature("docstring")"" +%feature("autodoc", "1"); + +%pythoncode %{ +class Handle: + """ + Manages a handle. + Takes two arguments: (str session_name, Domain domain) + """ + + __frozen = False + + def __init__(self, session_name, domain): + if type(session_name) is not str: + raise TypeError("in method '__init__', argument 2 of type 'str'") + if type(domain) is not Domain and domain is not None: + raise TypeError("in method '__init__', argument 3 of type 'lttng.Domain'") + + self._sname = session_name + if domain is None: + self._domtype = None + else: + self._domtype = domain.type + self._h = _lttng_create_handle(session_name, domain) + self.__frozen = True + + def __del__(self): + _lttng_destroy_handle(self._h) + + def __repr__(self): + if self._domtype == 1: + domstr = "DOMAIN_KERNEL" + elif self._domtype == 2: + domstr = "DOMAIN_UST" + else: + domstr = self._domtype + + return "lttng.Handle; session('{}'), domain.type({})".format( + self._sname, domstr) + + def __setattr__(self, attr, val): + if self.__frozen: + raise NotImplementedError("cannot modify attributes") + else: + self.__dict__[attr] = val +%} + + +// ============================================= +// STRUCTURES +// These are directly taken from lttng.h. +// Any change to these structures must also be +// made here. +// ============================================= + +%rename("Domain") lttng_domain; +%rename("EventContext") lttng_event_context; +%rename("Event") lttng_event; +%rename("Calibrate") lttng_calibrate; +%rename("ChannelAttr") lttng_channel_attr; +%rename("Channel") lttng_channel; +%rename("Session") lttng_session; + +struct lttng_domain{ + enum lttng_domain_type type; + char padding[LTTNG_DOMAIN_PADDING1]; + + union { + pid_t pid; + char exec_name[NAME_MAX]; + char padding[LTTNG_DOMAIN_PADDING2]; + } attr; + + %extend { + char *__repr__() { + static char temp[256]; + switch ( $self->type ) { + case 1: + sprintf(temp, "lttng.Domain; type(DOMAIN_KERNEL)"); + break; + case 2: + sprintf(temp, "lttng.Domain; type(DOMAIN_UST)"); + break; + default: + sprintf(temp, "lttng.Domain; type(%i)", $self->type); + break; + } + return &temp[0]; + } + } +}; + +struct lttng_event_context { + enum lttng_event_context_type ctx; + char padding[LTTNG_EVENT_CONTEXT_PADDING1]; + + union { + struct lttng_event_perf_counter_ctx perf_counter; + char padding[LTTNG_EVENT_CONTEXT_PADDING2]; + } u; + + %extend { + char *__repr__() { + static char temp[256]; + switch ( $self->ctx ) { + case 0: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_PID)"); + break; + case 1: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_PERF_COUNTER)"); + break; + case 2: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_PROCNAME)"); + break; + case 3: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_PRIO)"); + break; + case 4: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_NICE)"); + break; + case 5: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_VPID)"); + break; + case 6: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_TID)"); + break; + case 7: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_VTID)"); + break; + case 8: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_PPID)"); + break; + case 9: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_VPPID)"); + break; + case 10: + sprintf(temp, "lttng.EventContext; ctx(EVENT_CONTEXT_PTHREAD_ID)"); + break; + default: + sprintf(temp, "lttng.EventContext; type(%i)", $self->ctx); + break; + } + return &temp[0]; + } + } +}; + +struct lttng_event_probe_attr { + uint64_t addr; + uint64_t offset; + char symbol_name[LTTNG_SYMBOL_NAME_LEN]; + char padding[LTTNG_EVENT_PROBE_PADDING1]; +}; + +struct lttng_event_function_attr { + char symbol_name[LTTNG_SYMBOL_NAME_LEN]; + char padding[LTTNG_EVENT_FUNCTION_PADDING1]; +}; + +struct lttng_event { + enum lttng_event_type type; + char name[LTTNG_SYMBOL_NAME_LEN]; + + enum lttng_loglevel_type loglevel_type; + int loglevel; + + int32_t enabled; + pid_t pid; + + char padding[LTTNG_EVENT_PADDING1]; + + union { + struct lttng_event_probe_attr probe; + struct lttng_event_function_attr ftrace; + + char padding[LTTNG_EVENT_PADDING2]; + } attr; + + %extend { + char *__repr__() { + static char temp[512]; + char evtype[50]; + char logtype[50]; + + switch ( $self->type ) { + case -1: + sprintf(evtype, "EVENT_ALL"); + break; + case 0: + sprintf(evtype, "EVENT_TRACEPOINT"); + break; + case 1: + sprintf(evtype, "EVENT_PROBE"); + break; + case 2: + sprintf(evtype, "EVENT_FUNCTION"); + break; + case 3: + sprintf(evtype, "EVENT_FUNCTION_ENTRY"); + break; + case 4: + sprintf(evtype, "EVENT_NOOP"); + break; + case 5: + sprintf(evtype, "EVENT_SYSCALL"); + break; + default: + sprintf(evtype, "%i", $self->type); + break; + } + + switch ( $self->loglevel_type ) { + case 0: + sprintf(logtype, "EVENT_LOGLEVEL_ALL"); + break; + case 1: + sprintf(logtype, "EVENT_LOGLEVEL_RANGE"); + break; + case 2: + sprintf(logtype, "EVENT_LOGLEVEL_SINGLE"); + break; + default: + sprintf(logtype, "%i", $self->loglevel_type); + break; + } + + sprintf(temp, "lttng.Event; name('%s'), type(%s), " + "loglevel_type(%s), loglevel(%i), " + "enabled(%s), pid(%i)", + $self->name, evtype, logtype, $self->loglevel, + $self->enabled ? "True" : "False", $self->pid); + return &temp[0]; + } + } +}; + +struct lttng_calibrate { + enum lttng_calibrate_type type; + char padding[LTTNG_CALIBRATE_PADDING1]; + + %extend { + char *__repr__() { + static char temp[256]; + switch ( $self->type ) { + case 0: + sprintf(temp, "lttng.Calibrate; type(CALIBRATE_FUNCTION)"); + break; + default: + sprintf(temp, "lttng.Calibrate; type(%i)", $self->type); + break; + } + return &temp[0]; + } + } +}; + +struct lttng_channel_attr { + int overwrite; + uint64_t subbuf_size; + uint64_t num_subbuf; + unsigned int switch_timer_interval; + unsigned int read_timer_interval; + enum lttng_event_output output; + + char padding[LTTNG_CHANNEL_ATTR_PADDING1]; + + %extend { + char *__repr__() { + static char temp[256]; + char evout[25]; + + switch ( $self->output ) { + case 0: + sprintf(evout, "EVENT_SPLICE"); + break; + case 1: + sprintf(evout, "EVENT_MMAP"); + break; + default: + sprintf(evout, "%i", $self->output); + break; + } + sprintf(temp, "lttng.ChannelAttr; overwrite(%i), subbuf_size(%lu), " + "num_subbuf(%lu), switch_timer_interval(%u), " + "read_timer_interval(%u), output(%s)", + $self->overwrite, $self->subbuf_size, $self->num_subbuf, + $self->switch_timer_interval, $self->read_timer_interval, + evout); + return &temp[0]; + } + } +}; + +struct lttng_channel { + char name[LTTNG_SYMBOL_NAME_LEN]; + uint32_t enabled; + struct lttng_channel_attr attr; + char padding[LTTNG_CHANNEL_PADDING1]; + + %extend { + char *__repr__() { + static char temp[512]; + sprintf(temp, "lttng.Channel; name('%s'), enabled(%s)", + $self->name, $self->enabled ? "True" : "False"); + return &temp[0]; + } + } +}; + +struct lttng_session { + char name[NAME_MAX]; + char path[PATH_MAX]; + uint32_t enabled; + char padding[LTTNG_SESSION_PADDING1]; + + %extend { + char *__repr__() { + static char temp[512]; + sprintf(temp, "lttng.Session; name('%s'), path('%s'), enabled(%s)", + $self->name, $self->path, + $self->enabled ? "True" : "False"); + return &temp[0]; + } + } +}; diff --git a/extras/bindings/swig/python/tests/example.py b/extras/bindings/swig/python/tests/example.py new file mode 100644 index 0000000..9703170 --- /dev/null +++ b/extras/bindings/swig/python/tests/example.py @@ -0,0 +1,109 @@ +#This example shows basically how to use the lttng-tools python module + +from lttng import * + +# This error will be raised is something goes wrong +class LTTngError(Exception): + def __init__(self, value): + self.value = value + def __str__(self): + return repr(self.value) + +#Setting up the domain to use +dom = Domain() +dom.type = DOMAIN_KERNEL + +#Setting up a channel to use +channel = Channel() +channel.name = "mychan" +channel.attr.overwrite = 0 +channel.attr.subbuf_size = 4096 +channel.attr.num_subbuf = 8 +channel.attr.switch_timer_interval = 0 +channel.attr.read_timer_interval = 200 +channel.attr.output = EVENT_SPLICE + +#Setting up some events that will be used +event = Event() +event.type = EVENT_TRACEPOINT +event.loglevel_type = EVENT_LOGLEVEL_ALL + +sched_switch = Event() +sched_switch.name = "sched_switch" +sched_switch.type = EVENT_TRACEPOINT +sched_switch.loglevel_type = EVENT_LOGLEVEL_ALL + +sched_process_exit = Event() +sched_process_exit.name = "sched_process_exit" +sched_process_exit.type = EVENT_TRACEPOINT +sched_process_exit.loglevel_type = EVENT_LOGLEVEL_ALL + +sched_process_free = Event() +sched_process_free.name = "sched_process_free" +sched_process_free.type = EVENT_TRACEPOINT +sched_process_free.loglevel_type = EVENT_LOGLEVEL_ALL + + +#Creating a new session +res = create("test","/lttng-traces/test") +if res<0: + raise LTTngError(strerror(res)) + +#Creating handle +han = None +han = Handle("test", dom) +if han is None: + raise LTTngError("Handle not created") + +#Enabling the kernel channel +res = enable_channel(han, channel) +if res<0: + raise LTTngError(strerror(res)) + +#Enabling some events in given channel +#To enable all events in default channel, use +#enable_event(han, event, None) +res = enable_event(han, sched_switch, channel.name) +if res<0: + raise LTTngError(strerror(res)) + +res = enable_event(han, sched_process_exit, channel.name) +if res<0: + raise LTTngError(strerror(res)) + +res = enable_event(han, sched_process_free, channel.name) +if res<0: + raise LTTngError(strerror(res)) + +#Disabling an event +res = disable_event(han, sched_switch.name, channel.name) +if res<0: + raise LTTngError(strerror(res)) + +#Getting a list of the channels +l = list_channels(han) +if type(l) is int: + raise LTTngError(strerror(l)) + +#Starting the trace +res = start("test") +if res<0: + raise LTTngError(strerror(res)) + +#Stopping the trace +res = stop("test") +if res<0: + raise LTTngError(strerror(res)) + +#Disabling a channel +res = disable_channel(han, channel.name) +if res<0: + raise LTTngError(strerror(res)) + +#Destroying the handle +del han + +#Destroying the session +res = destroy("test") +if res<0: + raise LTTngError(strerror(res)) diff --git a/extras/bindings/swig/python/tests/run.sh b/extras/bindings/swig/python/tests/run.sh new file mode 100644 index 0000000..7de819b --- /dev/null +++ b/extras/bindings/swig/python/tests/run.sh @@ -0,0 +1 @@ +python tests.py diff --git a/extras/bindings/swig/python/tests/tests.py b/extras/bindings/swig/python/tests/tests.py new file mode 100644 index 0000000..a4be981 --- /dev/null +++ b/extras/bindings/swig/python/tests/tests.py @@ -0,0 +1,310 @@ +import unittest +import os +import time +from lttng import * + +class TestLttngPythonModule (unittest.TestCase): + + def test_kernel_all_events(self): + dom = Domain() + dom.type = DOMAIN_KERNEL + + event = Event() + event.type = EVENT_TRACEPOINT + event.loglevel_type = EVENT_LOGLEVEL_ALL + + han = Handle("test_kernel_all_ev", dom) + + r = create("test_kernel_all_ev","/lttng-traces/test") + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, event, None) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = start("test_kernel_all_ev") + self.assertGreaterEqual(r, 0, strerror(r)) + time.sleep(2) + + r = stop("test_kernel_all_ev") + self.assertGreaterEqual(r, 0, strerror(r)) + + r = destroy("test_kernel_all_ev") + self.assertGreaterEqual(r, 0, strerror(r)) + + + def test_kernel_event(self): + + dom = Domain() + dom.type = DOMAIN_KERNEL + + channel = Channel() + channel.name="mychan" + channel.attr.overwrite = 0 + channel.attr.subbuf_size = 4096 + channel.attr.num_subbuf = 8 + channel.attr.switch_timer_interval = 0 + channel.attr.read_timer_interval = 200 + channel.attr.output = EVENT_SPLICE + + sched_switch = Event() + sched_switch.name = "sched_switch" + sched_switch.type = EVENT_TRACEPOINT + sched_switch.loglevel_type = EVENT_LOGLEVEL_ALL + + sched_process_exit = Event() + sched_process_exit.name = "sched_process_exit" + sched_process_exit.type = EVENT_TRACEPOINT + sched_process_exit.loglevel_type = EVENT_LOGLEVEL_ALL + + sched_process_free = Event() + sched_process_free.name = "sched_process_free" + sched_process_free.type = EVENT_TRACEPOINT + sched_process_free.loglevel_type = EVENT_LOGLEVEL_ALL + + han = Handle("test_kernel_event", dom) + + #Create session test + r = create("test_kernel_event","/lttng-traces/test") + self.assertGreaterEqual(r, 0, strerror(r)) + + #Enabling channel tests + r = enable_channel(han, channel) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Enabling events tests + r = enable_event(han, sched_switch, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, sched_process_exit, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, sched_process_free, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Disabling events tests + r = disable_event(han, sched_switch.name, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = disable_event(han, sched_process_free.name, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Renabling events tests + r = enable_event(han, sched_switch, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, sched_process_free, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Start, stop, destroy + r = start("test_kernel_event") + self.assertGreaterEqual(r, 0, strerror(r)) + time.sleep(2) + + r = stop("test_kernel_event") + self.assertGreaterEqual(r, 0, strerror(r)) + + r=disable_channel(han, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r=destroy("test_kernel_event") + self.assertGreaterEqual(r, 0, strerror(r)) + + + + def test_ust_all_events(self): + dom = Domain() + dom.type = DOMAIN_UST + + event = Event() + event.type = EVENT_TRACEPOINT + event.loglevel_type = EVENT_LOGLEVEL_ALL + + han = Handle("test_ust_all_ev", dom) + + r = create("test_ust_all_ev","/lttng-traces/test") + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, event, None) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = start("test_ust_all_ev") + self.assertGreaterEqual(r, 0, strerror(r)) + time.sleep(2) + + r = stop("test_ust_all_ev") + self.assertGreaterEqual(r, 0, strerror(r)) + + r = destroy("test_ust_all_ev") + self.assertGreaterEqual(r, 0, strerror(r)) + + + def test_ust_event(self): + + dom = Domain() + dom.type = DOMAIN_UST + + channel = Channel() + channel.name="mychan" + channel.attr.overwrite = 0 + channel.attr.subbuf_size = 4096 + channel.attr.num_subbuf = 8 + channel.attr.switch_timer_interval = 0 + channel.attr.read_timer_interval = 200 + channel.attr.output = EVENT_MMAP + + ev1 = Event() + ev1.name = "tp1" + ev1.type = EVENT_TRACEPOINT + ev1.loglevel_type = EVENT_LOGLEVEL_ALL + + ev2 = Event() + ev2.name = "ev2" + ev2.type = EVENT_TRACEPOINT + ev2.loglevel_type = EVENT_LOGLEVEL_ALL + + ev3 = Event() + ev3.name = "ev3" + ev3.type = EVENT_TRACEPOINT + ev3.loglevel_type = EVENT_LOGLEVEL_ALL + + han = Handle("test_ust_event", dom) + + #Create session test + r = create("test_ust_event","/lttng-traces/test") + self.assertGreaterEqual(r, 0, strerror(r)) + + #Enabling channel tests + r = enable_channel(han, channel) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Enabling events tests + r = enable_event(han, ev1, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, ev2, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, ev3, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Disabling events tests + r = disable_event(han, ev1.name, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = disable_event(han, ev3.name, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Renabling events tests + r = enable_event(han, ev1, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, ev3, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Start, stop + r = start("test_ust_event") + self.assertGreaterEqual(r, 0, strerror(r)) + time.sleep(2) + + r = stop("test_ust_event") + self.assertGreaterEqual(r, 0, strerror(r)) + + #Restart/restop + r = start("test_ust_event") + self.assertGreaterEqual(r, 0, strerror(r)) + time.sleep(2) + + r = stop("test_ust_event") + self.assertGreaterEqual(r, 0, strerror(r)) + + #Disabling channel and destroy + r=disable_channel(han, channel.name) + self.assertGreaterEqual(r, 0, strerror(r)) + + r=destroy("test_ust_event") + self.assertGreaterEqual(r, 0, strerror(r)) + + + def test_other_functions(self): + dom = Domain() + dom.type=DOMAIN_KERNEL + + event=Event() + event.type=EVENT_TRACEPOINT + event.loglevel_type=EVENT_LOGLEVEL_ALL + + calib = Calibrate() + calib.type = CALIBRATE_FUNCTION + + ctx = EventContext() + ctx.type=EVENT_CONTEXT_PID + + chattr = ChannelAttr() + chattr.overwrite = 0 + chattr.subbuf_size = 4096 + chattr.num_subbuf = 8 + chattr.switch_timer_interval = 0 + chattr.read_timer_interval = 200 + chattr.output = EVENT_SPLICE + + han = Handle("test_otherf" , dom) + + r = create("test_otherf","/lttng-traces/test") + self.assertGreaterEqual(r, 0, strerror(r)) + + r = enable_event(han, event, None) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Calibrate test + r = calibrate(han , calib) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Context test + r = add_context(han, ctx, "sched_switch", "channel0") + self.assertGreaterEqual(r, 0, strerror(r)) + #Any channel + r = add_context(han, ctx, "sched_wakeup", None) + self.assertGreaterEqual(r, 0, strerror(r)) + #All events + r = add_context(han, ctx, None, None) + self.assertGreaterEqual(r, 0, strerror(r)) + + #Def. channel attr + channel_set_default_attr(dom, chattr) + channel_set_default_attr(None, None) + + #Ses Daemon alive + r = session_daemon_alive() + self.assertTrue(r == 1 or r == 0, strerror(r)) + + #Setting trace group + r = set_tracing_group("testing") + self.assertGreaterEqual(r, 0, strerror(r)) + + + r = start("test_otherf") + self.assertGreaterEqual(r, 0, strerror(r)) + time.sleep(2) + + r = stop("test_otherf") + self.assertGreaterEqual(r, 0, strerror(r)) + + del han + + r = destroy("test_otherf") + self.assertGreaterEqual(r, 0, strerror(r)) + + +if __name__ == "__main__": + # CHECK IF ROOT + if os.geteuid() == 0: + #Make sure session names don't already exist: + destroy("test_kernel_event") + destroy("test_kernel_all_events") + destroy("test_ust_all_events") + destroy("test_ust_event") + destroy("test_otherf") + + unittest.main() + else: + print('Script must be run as root') -- 1.7.9.5 From christian.babeux at efficios.com Thu Aug 9 11:24:54 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Thu, 9 Aug 2012 11:24:54 -0400 Subject: [lttng-dev] [PATCH lttng-tools 2/2] Fix: Missing libs dependencies in configure check for lttng-ust-ctl In-Reply-To: <20120809134621.GG8382@Krystal> References: <884d75d90550f85492dfcaac2b0088f78ced542b.1344025572.git.christian.babeux@efficios.com> <20120809134621.GG8382@Krystal> Message-ID: Hi Mathieu, > not sure I understand why all the changes you are doing are needed. What > happens if you just do: > > - [AC_MSG_ERROR([Cannot find LTTng-UST. Use [LDFLAGS]=-Ldir to specify its location, or specify --disable-lttng-ust to build lttng-tools without LTTng-UST support.])] > + [AC_MSG_ERROR([Cannot find LTTng-UST. Use [LDFLAGS]=-Ldir to specify its location, or specify --disable-lttng-ust to build lttng-tools without LTTng-UST support.])], > + [-lurcu-common -lurcu-bp -lurcu-cds] > > ? If we only change the other-lib case of the AC_CHECK_LIB, subsequent calls to this macro will fail unexpectedly. For example, the next call to this macro is the check for dlopen in the dl lib. Since the previous call (the check for lttng-ust-ctl) succeeded, -llttng-ust-ctl has been prepended to the $LIBS variable. The problem arise when trying to link a test program (the actual test of AC_CHECK_LIB) to check for the dl lib. The linking will fail for the same reason, because the dependencies to urcu-{common,bp,cds} have not been manually specified and lttng-ust-ctl has been added to the $LIBS flags passed to the linker. Hence, the test for the dl lib will fail, not because it can't find the dl lib, but because the linking fail as a side effect of adding the lttng-ust-ctl lib. There is possibly three other ways to fix this issue that I know of: - The check for lttng-ust-ctl lib should be the last AC_CHECK_LIB. This is exactly the situation I was trying to avoid. Someone could inadvertently add one after and silently break cross-compilation builds. - Add AC_CHECK_LIB checks for the dependents URCU libs. This will add a -lurcu-{common,bp,cds} when linking. I'm not fully aware of the repercussions this could have. - Use PKG_CONFIG support in configure.ac. This might be a more elegant and long-term solution so we don't have to rely on these kind of trickery to check for libraries and correctly support cross-compilation builds. >> + AC_CHECK_LIB([lttng-ust-ctl], [ustctl_create_session], >> + [ >> + AC_DEFINE([HAVE_LIBLTTNG_UST_CTL], [1], [has LTTng-UST control support]) > > wasn't HAVE_LIBLTTNG_UST_CTL already defined by the AM_CONDITIONAL below ? > >> + lttng_ust_ctl_found=yes > > why are you adding this ? > The default behavior of the action-if-found in AC_CHECK_LIB is to prepend '-llibrary' and defining 'HAVE_LIBlibrary' [1]. Since we don't want to prepend -llttng-ust-ctl, we must manually define HAVE_LIBlibrary. The AM_CONDITIONAL does not define the HAVE_LIBLTTNG_UST_CTL (as in the preprocessor sense) it is the AC_CHECK_LIB check that normally does (via an underlying call to AC_DEFINE). AM_CONDITIONAL only make the HAVE_LIBLTTNG_UST_CTL variable available to use in Makefile.am [2]. >> - >> -AM_CONDITIONAL([HAVE_LIBLTTNG_UST_CTL], [ test "x$ac_cv_lib_lttng_ust_ctl_ustctl_create_session" = "xyes" ]) >> - >> +AM_CONDITIONAL([HAVE_LIBLTTNG_UST_CTL], [test "x$lttng_ust_ctl_found" = xyes]) > > why are you changing this line ? > I only changed the test flag to lttng_ust_ctl_found for clarity sake. It might not be obvious to everyone that the $ac_cv_lib_lttng_ust_ctl_ustctl_create_session variable is a cached test result by autoconf. I think this patch is not the most obvious. I would like to know if anyone has any other ideas to resolve this issue. Thoughts? Thanks, Christian 1 - https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Libraries.html#Libraries 2 - https://www.gnu.org/software/automake/manual/html_node/Usage-of-Conditionals.html From danny.serres at efficios.com Thu Aug 9 11:45:00 2012 From: danny.serres at efficios.com (Danny Serres) Date: Thu, 9 Aug 2012 11:45:00 -0400 Subject: [lttng-dev] [babeltrace PATCH] Fix: correct name of bt_ctf_field_get_error in comments and typo in man page Message-ID: <1344527100-10782-1-git-send-email-danny.serres@efficios.com> Signed-off-by: Danny Serres --- doc/babeltrace.1 | 2 +- formats/ctf/events.c | 2 +- include/babeltrace/ctf/events.h | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/babeltrace.1 b/doc/babeltrace.1 index db07caf..db462f4 100644 --- a/doc/babeltrace.1 +++ b/doc/babeltrace.1 @@ -97,7 +97,7 @@ babeltrace-log(1), lttng(1), lttng-ust(3), lttng-sessiond(8) .SH "BUGS" .PP -No knows bugs at this point. +No known bugs at this point. If you encounter any issues or usability problem, please report it on our mailing list to help improve this diff --git a/formats/ctf/events.c b/formats/ctf/events.c index a8b08f6..5d92b08 100644 --- a/formats/ctf/events.c +++ b/formats/ctf/events.c @@ -34,7 +34,7 @@ /* * thread local storage to store the last error that occured * while reading a field, this variable must be accessed by - * bt_ctf_field_error only + * bt_ctf_field_get_error only */ __thread int bt_ctf_last_field_error = 0; diff --git a/include/babeltrace/ctf/events.h b/include/babeltrace/ctf/events.h index 4232feb..2bf9c62 100644 --- a/include/babeltrace/ctf/events.h +++ b/include/babeltrace/ctf/events.h @@ -182,7 +182,7 @@ int bt_ctf_get_array_len(const struct definition *field); * * If the field does not exist or is not of the type requested, the value * returned is undefined. To check if an error occured, use the - * bt_ctf_field_error() function after accessing a field. + * bt_ctf_field_get_error() function after accessing a field. */ uint64_t bt_ctf_get_uint64(const struct definition *field); int64_t bt_ctf_get_int64(const struct definition *field); @@ -190,7 +190,7 @@ char *bt_ctf_get_char_array(const struct definition *field); char *bt_ctf_get_string(const struct definition *field); /* - * bt_ctf_field_error: returns the last error code encountered while + * bt_ctf_field_get_error: returns the last error code encountered while * accessing a field and reset the error flag. * Return 0 if no error, a negative value otherwise. */ -- 1.7.9.5 From mathieu.desnoyers at efficios.com Thu Aug 9 12:32:45 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 12:32:45 -0400 Subject: [lttng-dev] [PATCH lttng-tools 2/2] Fix: Missing libs dependencies in configure check for lttng-ust-ctl In-Reply-To: References: <884d75d90550f85492dfcaac2b0088f78ced542b.1344025572.git.christian.babeux@efficios.com> <20120809134621.GG8382@Krystal> Message-ID: <20120809163245.GA10886@Krystal> * Christian Babeux (christian.babeux at efficios.com) wrote: > Hi Mathieu, > > > not sure I understand why all the changes you are doing are needed. What > > happens if you just do: > > > > - [AC_MSG_ERROR([Cannot find LTTng-UST. Use [LDFLAGS]=-Ldir to specify its location, or specify --disable-lttng-ust to build lttng-tools without LTTng-UST support.])] > > + [AC_MSG_ERROR([Cannot find LTTng-UST. Use [LDFLAGS]=-Ldir to specify its location, or specify --disable-lttng-ust to build lttng-tools without LTTng-UST support.])], > > + [-lurcu-common -lurcu-bp -lurcu-cds] > > > > ? > > If we only change the other-lib case of the AC_CHECK_LIB, subsequent > calls to this macro will fail unexpectedly. For example, the next call > to this macro is the check for dlopen in the dl lib. Since the previous > call (the check for lttng-ust-ctl) succeeded, -llttng-ust-ctl has been > prepended to the $LIBS variable. > > The problem arise when trying to link a test program > (the actual test of AC_CHECK_LIB) to check for the dl lib. The linking > will fail for the same reason, because the dependencies to > urcu-{common,bp,cds} have not been manually specified and lttng-ust-ctl > has been added to the $LIBS flags passed to the linker. Hence, > the test for the dl lib will fail, not because it can't find the dl lib, > but because the linking fail as a side effect of adding the lttng-ust-ctl lib. > > There is possibly three other ways to fix this issue that I know of: > > - The check for lttng-ust-ctl lib should be the last AC_CHECK_LIB. > This is exactly the situation I was trying to avoid. Someone could > inadvertently add one after and silently break cross-compilation > builds. > > - Add AC_CHECK_LIB checks for the dependents URCU libs. > This will add a -lurcu-{common,bp,cds} when linking. > I'm not fully aware of the repercussions this could have. > > - Use PKG_CONFIG support in configure.ac. > This might be a more elegant and long-term solution so we don't have > to rely on these kind of trickery to check for libraries > and correctly support cross-compilation builds. > > > >> + AC_CHECK_LIB([lttng-ust-ctl], [ustctl_create_session], > >> + [ > >> + AC_DEFINE([HAVE_LIBLTTNG_UST_CTL], [1], [has LTTng-UST control support]) > > > > wasn't HAVE_LIBLTTNG_UST_CTL already defined by the AM_CONDITIONAL below ? > > > >> + lttng_ust_ctl_found=yes > > > > why are you adding this ? > > > > The default behavior of the action-if-found in AC_CHECK_LIB is to > prepend '-llibrary' and defining 'HAVE_LIBlibrary' [1]. > Since we don't want to prepend -llttng-ust-ctl, we must manually define > HAVE_LIBlibrary. > > The AM_CONDITIONAL does not define the HAVE_LIBLTTNG_UST_CTL > (as in the preprocessor sense) it is the AC_CHECK_LIB check > that normally does (via an underlying call to AC_DEFINE). > AM_CONDITIONAL only make the HAVE_LIBLTTNG_UST_CTL variable > available to use in Makefile.am [2]. > > >> - > >> -AM_CONDITIONAL([HAVE_LIBLTTNG_UST_CTL], [ test "x$ac_cv_lib_lttng_ust_ctl_ustctl_create_session" = "xyes" ]) > >> - > >> +AM_CONDITIONAL([HAVE_LIBLTTNG_UST_CTL], [test "x$lttng_ust_ctl_found" = xyes]) > > > > why are you changing this line ? > > > > I only changed the test flag to lttng_ust_ctl_found for clarity sake. > It might not be obvious to everyone that the > $ac_cv_lib_lttng_ust_ctl_ustctl_create_session variable is a cached > test result by autoconf. > > I think this patch is not the most obvious. I would like to know > if anyone has any other ideas to resolve this issue. ok, this explanation is fine. Now I understand that by specifying anything other than [] in the AC_CHECK_LIB, you override the default behavior, and hence you don't have the LIBS+=. Good! Acked-by: Mathieu Desnoyers > > Thoughts? > > Thanks, > > Christian > > 1 - https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Libraries.html#Libraries > 2 - https://www.gnu.org/software/automake/manual/html_node/Usage-of-Conditionals.html -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 9 12:35:36 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 12:35:36 -0400 Subject: [lttng-dev] [babeltrace PATCH] Fix: correct name of bt_ctf_field_get_error in comments and typo in man page In-Reply-To: <1344527100-10782-1-git-send-email-danny.serres@efficios.com> References: <1344527100-10782-1-git-send-email-danny.serres@efficios.com> Message-ID: <20120809163536.GB10886@Krystal> * Danny Serres (danny.serres at efficios.com) wrote: > Signed-off-by: Danny Serres merged, thanks ! Mathieu > --- > doc/babeltrace.1 | 2 +- > formats/ctf/events.c | 2 +- > include/babeltrace/ctf/events.h | 4 ++-- > 3 files changed, 4 insertions(+), 4 deletions(-) > > diff --git a/doc/babeltrace.1 b/doc/babeltrace.1 > index db07caf..db462f4 100644 > --- a/doc/babeltrace.1 > +++ b/doc/babeltrace.1 > @@ -97,7 +97,7 @@ babeltrace-log(1), lttng(1), lttng-ust(3), lttng-sessiond(8) > .SH "BUGS" > > .PP > -No knows bugs at this point. > +No known bugs at this point. > > If you encounter any issues or usability problem, please report it on > our mailing list to help improve this > diff --git a/formats/ctf/events.c b/formats/ctf/events.c > index a8b08f6..5d92b08 100644 > --- a/formats/ctf/events.c > +++ b/formats/ctf/events.c > @@ -34,7 +34,7 @@ > /* > * thread local storage to store the last error that occured > * while reading a field, this variable must be accessed by > - * bt_ctf_field_error only > + * bt_ctf_field_get_error only > */ > __thread int bt_ctf_last_field_error = 0; > > diff --git a/include/babeltrace/ctf/events.h b/include/babeltrace/ctf/events.h > index 4232feb..2bf9c62 100644 > --- a/include/babeltrace/ctf/events.h > +++ b/include/babeltrace/ctf/events.h > @@ -182,7 +182,7 @@ int bt_ctf_get_array_len(const struct definition *field); > * > * If the field does not exist or is not of the type requested, the value > * returned is undefined. To check if an error occured, use the > - * bt_ctf_field_error() function after accessing a field. > + * bt_ctf_field_get_error() function after accessing a field. > */ > uint64_t bt_ctf_get_uint64(const struct definition *field); > int64_t bt_ctf_get_int64(const struct definition *field); > @@ -190,7 +190,7 @@ char *bt_ctf_get_char_array(const struct definition *field); > char *bt_ctf_get_string(const struct definition *field); > > /* > - * bt_ctf_field_error: returns the last error code encountered while > + * bt_ctf_field_get_error: returns the last error code encountered while > * accessing a field and reset the error flag. > * Return 0 if no error, a negative value otherwise. > */ > -- > 1.7.9.5 > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From eag0628 at gmail.com Thu Aug 9 12:16:05 2012 From: eag0628 at gmail.com (Lai Jiangshan) Date: Fri, 10 Aug 2012 00:16:05 +0800 Subject: [lttng-dev] [PATCH 2/2] urcu: add notice to URCU_TLS() for it is not async-signal-safe In-Reply-To: <20120809141417.GC9064@Krystal> References: <1344414673-14714-1-git-send-email-laijs@cn.fujitsu.com> <1344414673-14714-2-git-send-email-laijs@cn.fujitsu.com> <20120809141417.GC9064@Krystal> Message-ID: OK for me. Please do it. On Thu, Aug 9, 2012 at 10:14 PM, Mathieu Desnoyers wrote: > * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: >> Signed-off-by: Lai Jiangshan >> --- >> urcu/tls-compat.h | 14 ++++++++++++++ >> 1 files changed, 14 insertions(+), 0 deletions(-) >> >> diff --git a/urcu/tls-compat.h b/urcu/tls-compat.h >> index 192a536..b7bf363 100644 >> --- a/urcu/tls-compat.h >> +++ b/urcu/tls-compat.h >> @@ -59,6 +59,20 @@ extern "C" { >> >> #else /* #ifndef CONFIG_RCU_TLS */ >> >> +/* >> + * NOTE: URCU_TLS() is NOT async-signal-safe, you can't use it >> + * inside any function which can be called from signal handler. >> + * >> + * But if pthread_getspecific() is async-signal-safe in your >> + * platform, you can make URCU_TLS() async-signal-safe via: >> + * ensuring the first call to URCU_TLS() of a given TLS variable of >> + * all threads is called earliest from a non-signal handler function. >> + * >> + * Exmaple: In any thread, the first call of URCU_TLS(rcu_reader) >> + * is called from rcu_register_thread(), so we can ensure all later >> + * URCU_TLS(rcu_reader) in any thread is async-signal-safe. > > Hrm. We could also just block all signals within type *__tls_access_ ## > name(void) (in tls-compat.h) and make sure it is async-signal-safe I > guess ? I would prefer that solution: it would make the code more robust > for a rarely taken performance hit. > > Thoughts ? > > Thanks, > > Mathieu > >> + */ >> + >> # include >> >> struct urcu_tls { >> -- >> 1.7.4.4 >> > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com From danny.serres at efficios.com Thu Aug 9 14:20:59 2012 From: danny.serres at efficios.com (Danny Serres) Date: Thu, 9 Aug 2012 14:20:59 -0400 Subject: [lttng-dev] =?utf-8?q?=5Bbabeltrace_PATCH=5D_Babeltrace_python_mo?= =?utf-8?q?dule_v2?= Message-ID: <1344536459-12742-1-git-send-email-danny.serres@efficios.com> The Babeltrace Python module can be used to directly control the Babeltrace API inside Python, using 'import babeltrace'. Therefore, it becomes possible to create a Context, add a trace to it, iterate on it, read events and so on from within Python. SWIG >= 2.0 is used to create the wrapper and its 'warning md variable unused' bug is patched in Makefile.am In the interface file, struct and enum are directly copied from the include files. All changes to struct bt_iter_pos and to enums in ctf/events.h and clock-types.h must also be made in the interface file. Signed-off-by: Danny Serres Signed-off-by: Yannick Brosseau --- .gitignore | 4 + Makefile.am | 2 +- README | 6 + bindings/Makefile.am | 3 + bindings/python/Makefile.am | 28 + bindings/python/babeltrace.i.in | 1079 +++++++++ bindings/python/examples/babeltrace_and_lttng.py | 107 + bindings/python/examples/eventcount.py | 66 + bindings/python/examples/eventcountlist.py | 65 + bindings/python/examples/events_per_cpu.py | 96 + bindings/python/examples/example-api-test.py | 59 + bindings/python/examples/histogram.py | 121 + .../examples/output_format_modules/cairoplot.py | 2336 ++++++++++++++++++++ .../examples/output_format_modules/pprint_table.py | 35 + .../examples/output_format_modules/series.py | 1140 ++++++++++ bindings/python/examples/sched_switch.py | 110 + bindings/python/examples/softirqtimes.py | 130 ++ bindings/python/examples/syscalls_by_pid.py | 61 + bindings/python/python-complements.c | 105 + bindings/python/python-complements.h | 36 + bootstrap | 2 +- configure.ac | 37 + doc/python-howto.txt | 70 + m4/ax_pkg_swig.m4 | 135 ++ tests/tests-python.py | 115 + 25 files changed, 5946 insertions(+), 2 deletions(-) create mode 100644 bindings/Makefile.am create mode 100644 bindings/python/Makefile.am create mode 100644 bindings/python/babeltrace.i.in create mode 100644 bindings/python/examples/babeltrace_and_lttng.py create mode 100644 bindings/python/examples/eventcount.py create mode 100644 bindings/python/examples/eventcountlist.py create mode 100644 bindings/python/examples/events_per_cpu.py create mode 100644 bindings/python/examples/example-api-test.py create mode 100644 bindings/python/examples/histogram.py create mode 100644 bindings/python/examples/output_format_modules/__init__.py create mode 100755 bindings/python/examples/output_format_modules/cairoplot.py create mode 100644 bindings/python/examples/output_format_modules/pprint_table.py create mode 100755 bindings/python/examples/output_format_modules/series.py create mode 100644 bindings/python/examples/sched_switch.py create mode 100644 bindings/python/examples/softirqtimes.py create mode 100644 bindings/python/examples/syscalls_by_pid.py create mode 100644 bindings/python/python-complements.c create mode 100644 bindings/python/python-complements.h create mode 100644 doc/python-howto.txt create mode 100644 m4/ax_pkg_swig.m4 create mode 100644 tests/tests-python.py diff --git a/.gitignore b/.gitignore index d6098ac..3fe60e7 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ /tests/test-bitfield +*~ *.o *.a *.la @@ -26,3 +27,6 @@ converter/babeltrace-log core formats/ctf/metadata/ctf-parser.output stamp-h1 +bindings/python/babeltrace.i +bindings/python/babeltrace.py +bindings/python/babeltrace_wrap.c diff --git a/Makefile.am b/Makefile.am index 308ee16..6584c5d 100644 --- a/Makefile.am +++ b/Makefile.am @@ -2,7 +2,7 @@ AM_CFLAGS = $(PACKAGE_CFLAGS) -I$(top_srcdir)/include ACLOCAL_AMFLAGS = -I m4 -SUBDIRS = include types lib formats converter tests doc +SUBDIRS = include types lib formats converter bindings tests doc dist_doc_DATA = ChangeLog LICENSE mit-license.txt gpl-2.0.txt \ std-ext-lib.txt diff --git a/README b/README index 75bf0cf..1687075 100644 --- a/README +++ b/README @@ -25,6 +25,7 @@ BUILDING make install ldconfig + If you do not want Python bindings, run ./configure --disable-python. DEPENDENCIES ------------ @@ -44,6 +45,11 @@ To compile Babeltrace, you will need: libpopt >= 1.13 development libraries (Debian : libpopt-dev) (Fedora : popt) + python headers (optional) + (Debian/Ubuntu : python-dev) + swig >= 2.0 (optional) + (Debian/Ubuntu : swig2.0) + For developers using the git tree: diff --git a/bindings/Makefile.am b/bindings/Makefile.am new file mode 100644 index 0000000..dcd868d --- /dev/null +++ b/bindings/Makefile.am @@ -0,0 +1,3 @@ +if USE_PYTHON +SUBDIRS = python +endif diff --git a/bindings/python/Makefile.am b/bindings/python/Makefile.am new file mode 100644 index 0000000..579759f --- /dev/null +++ b/bindings/python/Makefile.am @@ -0,0 +1,28 @@ +babeltrace.i: babeltrace.i.in + sed "s/BABELTRACE_VERSION_STR/Babeltrace $(PACKAGE_VERSION)/g" babeltrace.i + +AM_CFLAGS = -I$(PYTHON_INCLUDE) -I$(top_srcdir)/include/ + +EXTRA_DIST = babeltrace.i +python_PYTHON = babeltrace.py +pyexec_LTLIBRARIES = _babeltrace.la + +MAINTAINERCLEANFILES = babeltrace_wrap.c babeltrace.py + +_babeltrace_la_SOURCES = babeltrace_wrap.c python-complements.c + +_babeltrace_la_LDFLAGS = -module + +_babeltrace_la_CFLAGS = $(GLIB_CFLAGS) $(AM_CFLAGS) + +_babeltrace_la_LIBS = $(GLIB_LIBS) + +_babeltrace_la_LIBADD = $(top_srcdir)/formats/ctf/libbabeltrace-ctf.la \ + $(top_srcdir)/formats/ctf-text/libbabeltrace-ctf-text.la + +# SWIG 'warning md variable unused' fixed after SWIG build: +babeltrace_wrap.c: babeltrace.i + $(SWIG) -python -Wall -I. -I$(top_srcdir)/include babeltrace.i + sed -i "s/PyObject \*m, \*d, \*md;/PyObject \*m, \*d;\n#if defined(SWIGPYTHON_BUILTIN)\nPyObject *md;\n#endif/g" babeltrace_wrap.c + sed -i "s/md = d/d/g" babeltrace_wrap.c + sed -i "s/(void)public_symbol;/(void)public_symbol;\n md = d;/g" babeltrace_wrap.c diff --git a/bindings/python/babeltrace.i.in b/bindings/python/babeltrace.i.in new file mode 100644 index 0000000..49636d1 --- /dev/null +++ b/bindings/python/babeltrace.i.in @@ -0,0 +1,1079 @@ +/* BABELTRACE PYTHON MODULE interface file */ + +%define DOCSTRING +"BABELTRACE_VERSION_STR + +Babeltrace is a trace viewer and converter reading and writing the +Common Trace Format (CTF). Its main use is to pretty-print CTF +traces into a human-readable text output. + +To use this module, the first step is to create a Context and add a +trace to it." +%enddef + +%module(docstring=DOCSTRING) babeltrace + +%include "typemaps.i" +%{ +#define SWIG_FILE_WITH_INIT +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "python-complements.h" +%} + +typedef unsigned long long uint64_t; +typedef long long int64_t; +typedef int bt_intern_str; + +/* ================================================================= + CONTEXT.H, CONTEXT-INTERNAL.H + ????????????????????????????? +*/ + +%rename("_bt_context_create") bt_context_create(void); +%rename("_bt_context_add_trace") bt_context_add_trace( + struct bt_context *ctx, const char *path, const char *format, + void (*packet_seek)(struct stream_pos *pos, size_t index, int whence), + struct mmap_stream_list *stream_list, FILE *metadata); +%rename("_bt_context_remove_trace") bt_context_remove_trace( + struct bt_context *ctx, int trace_id); +%rename("_bt_context_get") bt_context_get(struct bt_context *ctx); +%rename("_bt_context_put") bt_context_put(struct bt_context *ctx); +%rename("_bt_ctf_event_get_context") bt_ctf_event_get_context( + const struct bt_ctf_event *event); + +struct bt_context *bt_context_create(void); +int bt_context_add_trace(struct bt_context *ctx, const char *path, const char *format, + void (*packet_seek)(struct stream_pos *pos, size_t index, int whence), + struct mmap_stream_list *stream_list, FILE *metadata); +void bt_context_remove_trace(struct bt_context *ctx, int trace_id); +void bt_context_get(struct bt_context *ctx); +void bt_context_put(struct bt_context *ctx); +struct bt_context *bt_ctf_event_get_context(const struct bt_ctf_event *event); + +// class Context to prevent direct access to struct bt_context +%pythoncode%{ +class Context: + """ + The context represents the object in which a trace_collection is + open. As long as this structure is allocated, the trace_collection + is open and the traces it contains can be read and seeked by the + iterators and callbacks. + """ + + def __init__(self): + self._c = _bt_context_create() + + def __del__(self): + _bt_context_put(self._c) + + def add_trace(self, path, format_str, + packet_seek=None, stream_list=None, metadata=None): + """ + Add a trace by path to the context. + + Open a trace. + + path is the path to the trace, it is not recursive. + If "path" is None, stream_list is used instead as a list + of mmap streams to open for the trace. + + format is a string containing the format name in which the trace was + produced. + + packet_seek is not implemented for Python. Should be left None to + use the default packet_seek handler provided by the trace format. + + stream_list is a linked list of streams, it is used to open a trace + where the trace data is located in memory mapped areas instead of + trace files, this argument should be None when path is not None. + + The metadata parameter acts as a metadata override when not None, + otherwise the format handles the metadata opening. + + Return: the corresponding TraceHandle on success or None on error. + """ + if metadata is not None: + metadata = metadata._file + + ret = _bt_context_add_trace(self._c, path, format_str, packet_seek, + stream_list, metadata) + if ret < 0: + return None + + th = TraceHandle.__new__(TraceHandle) + th._id = ret + return th + + def add_traces_recursive(self, path, format_str): + """ + Open a trace recursively. + + Find each trace present in the subdirectory starting from the given + path, and add them to the context. + + Return a dict of TraceHandle instances (the full path is the key). + Return None on error. + """ + + import os + + trace_handles = {} + + noTrace = True + error = False + + for fullpath, dirs, files in os.walk(path): + if "metadata" in files: + trace_handle = self.add_trace(fullpath, format_str) + if trace_handle is None: + error = True + continue + + trace_handles[fullpath] = trace_handle + noTrace = False + + if noTrace and error: + return None + return trace_handles + + def remove_trace(self, trace_handle): + """ + Remove a trace from the context. + Effectively closing the trace. + """ + try: + _bt_context_remove_trace(self._c, trace_handle._id) + except AttributeError: + raise TypeError("in remove_trace, " + "argument 2 must be a TraceHandle instance") +%} + + + +/* ================================================================= + FORMAT.H, REGISTRY + ?????????????????? +*/ + +%rename("lookup_format") bt_lookup_format(bt_intern_str qname); +%rename("_bt_print_format_list") bt_fprintf_format_list(FILE *fp); +%rename("register_format") bt_register_format(struct format *format); + +extern struct format *bt_lookup_format(bt_intern_str qname); +extern void bt_fprintf_format_list(FILE *fp); +extern int bt_register_format(struct format *format); + +void format_init(void); +void format_finalize(void); + +%pythoncode %{ + +def print_format_list(babeltrace_file): + """ + Print a list of available formats to file. + + babeltrace_file must be a File instance opened in write mode. + """ + try: + if babeltrace_file._file is not None: + _bt_print_format_list(babeltrace_file._file) + except AttributeError: + raise TypeError("in print_format_list, " + "argument 1 must be a File instance") + +%} + + +/* ================================================================= + ITERATOR.H, ITERATOR-INTERNAL.H + ??????????????????????????????? +*/ + +%rename("_bt_iter_create") bt_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, const struct bt_iter_pos *end_pos); +%rename("_bt_iter_destroy") bt_iter_destroy(struct bt_iter *iter); +%rename("_bt_iter_next") bt_iter_next(struct bt_iter *iter); +%rename("_bt_iter_get_pos") bt_iter_get_pos(struct bt_iter *iter); +%rename("_bt_iter_free_pos") bt_iter_free_pos(struct bt_iter_pos *pos); +%rename("_bt_iter_set_pos") bt_iter_set_pos(struct bt_iter *iter, + const struct bt_iter_pos *pos); +%rename("_bt_iter_create_time_pos") bt_iter_create_time_pos(struct bt_iter *iter, + uint64_t timestamp); + +struct bt_iter *bt_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, const struct bt_iter_pos *end_pos); +void bt_iter_destroy(struct bt_iter *iter); +int bt_iter_next(struct bt_iter *iter); +struct bt_iter_pos *bt_iter_get_pos(struct bt_iter *iter); +void bt_iter_free_pos(struct bt_iter_pos *pos); +int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *pos); +struct bt_iter_pos *bt_iter_create_time_pos(struct bt_iter *iter, uint64_t timestamp); + +%rename("_bt_iter_pos") bt_iter_pos; +%rename("SEEK_TIME") BT_SEEK_TIME; +%rename("SEEK_RESTORE") BT_SEEK_RESTORE; +%rename("SEEK_CUR") BT_SEEK_CUR; +%rename("SEEK_BEGIN") BT_SEEK_BEGIN; +%rename("SEEK_END") BT_SEEK_END; + + +// This struct is taken from iterator.h +// All changes to the struct must also be made here +struct bt_iter_pos { + enum { + BT_SEEK_TIME, /* uses u.seek_time */ + BT_SEEK_RESTORE, /* uses u.restore */ + BT_SEEK_CUR, + BT_SEEK_BEGIN, + BT_SEEK_END, + } type; + union { + uint64_t seek_time; + struct bt_saved_pos *restore; + } u; +}; + + +%pythoncode%{ + +class IterPos: + """This class represents the position where to set an iterator.""" + + __can_access = False + + def __init__(self, seek_type, seek_time = None): + """ + seek_type represents the type of seek to use. + seek_time is the timestamp to seek to when using SEEK_TIME, it + is expressed in nanoseconds + Only use SEEK_RESTORE on IterPos obtained from the get_pos function + in Iter class. + """ + + self._pos = _bt_iter_pos() + self._pos.type = seek_type + if seek_time and seek_type == SEEK_TIME: + self._pos.u.seek_time = seek_time + self.__can_access = True + + def __del__(self): + if not self.__can_access: + _bt_iter_free_pos(self._pos) + + def _get_type(self): + if not __can_access: + raise AttributeError("seek_type is not available") + return self._pos.type + + def _set_type(self, seek_type): + if not __can_access: + raise AttributeError("seek_type is not available") + self._pos.type = seek_type + + def _get_time(self): + if not __can_access: + raise AttributeError("seek_time is not available") + + elif self._pos.type is not SEEK_TIME: + raise TypeError("seek_type is not SEEK_TIME") + + return self._pos.u.seek_time + + def _set_time(self, time): + if not __can_access: + raise AttributeError("seek_time is not available") + + elif self._pos.type is not SEEK_TIME: + raise TypeError("seek_type is not SEEK_TIME") + + self._pos.u.seek_time = time + + def _get_pos(self): + return self._pos + + + seek_type = property(_get_type, _set_type) + seek_time = property(_get_time, _set_time) + + +class Iterator: + + __with_init = False + + def __init__(self, context, begin_pos = None, end_pos = None, _no_init = None): + """ + Allocate a trace collection iterator. + + begin_pos and end_pos are optional parameters to specify the + position at which the trace collection should be seeked upon + iterator creation, and the position at which iteration will + start returning "EOF". + + By default, if begin_pos is None, a BT_SEEK_CUR is performed at + creation. By default, if end_pos is None, a BT_SEEK_END (end of + trace) is the EOF criterion. + """ + if _no_init is None: + if begin_pos is None: + bp = None + else: + try: + bp = begin_pos._pos + except AttributeError: + raise TypeError("in __init__, " + "argument 3 must be a IterPos instance") + + if end_pos is None: + ep = None + else: + try: + ep = end_pos._pos + except AttributeError: + raise TypeError("in __init__, " + "argument 4 must be a IterPos instance") + + try: + self._bi = _bt_iter_create(context._c, bp, ep) + except AttributeError: + raise TypeError("in __init__, " + "argument 2 must be a Context instance") + + self.__with_init = True + + else: + self._bi = _no_init + + def __del__(self): + if self.__with_init: + _bt_iter_destroy(self._bi) + + def next(self): + """ + Move trace collection position to the next event. + Returns 0 on success, a negative value on error. + """ + return _bt_iter_next(self._bi) + + def get_pos(self): + """Return a IterPos class of the current iterator position.""" + ret = IterPos(0) + ret.__can_access = False + ret._pos = _bt_iter_get_pos(self._bi) + return ret + + def set_pos(self, pos): + """ + Move the iterator to a given position. + + On error, the stream_heap is reinitialized and returned empty. + Return 0 for success. + Return EOF if the position requested is after the last event of the + trace collection. + Return -EINVAL when called with invalid parameter. + Return -ENOMEM if the stream_heap could not be properly initialized. + """ + try: + return _bt_iter_set_pos(self._bi, pos._pos) + except AttributeError: + raise TypeError("in set_pos, " + "argument 2 must be a IterPos instance") + + def create_time_pos(self, timestamp): + """ + Create a position based on time + This function allocates and returns a new IterPos to be able to + restore an iterator position based on a timestamp. + """ + + if timestamp < 0: + raise TypeError("timestamp must be an unsigned int") + + ret = IterPos(0) + ret.__can_access = False + ret._pos = _bt_iter_create_time_pos(self._bi, timestamp) + return ret +%} + + +/* ================================================================= + CLOCK-TYPE.H + ???????????? + *** Enum copied from clock-type.h? + All changes must also be made here +*/ +%rename("CLOCK_CYCLES") BT_CLOCK_CYCLES; +%rename("CLOCK_REAL") BT_CLOCK_REAL; + +enum bt_clock_type { + BT_CLOCK_CYCLES = 0, + BT_CLOCK_REAL +}; + +/* ================================================================= + TRACE-HANDLE.H, TRACE-HANDLE-INTERNAL.H + ??????????????????????????????????????? +*/ + +%rename("_bt_trace_handle_create") bt_trace_handle_create(struct bt_context *ctx); +%rename("_bt_trace_handle_destroy") bt_trace_handle_destroy(struct bt_trace_handle *bt); +struct bt_trace_handle *bt_trace_handle_create(struct bt_context *ctx); +void bt_trace_handle_destroy(struct bt_trace_handle *bt); + +%rename("_bt_trace_handle_get_path") bt_trace_handle_get_path(struct bt_context *ctx, + int handle_id); +%rename("_bt_trace_handle_get_timestamp_begin") bt_trace_handle_get_timestamp_begin( + struct bt_context *ctx, int handle_id, enum bt_clock_type type); +%rename("_bt_trace_handle_get_timestamp_end") bt_trace_handle_get_timestamp_end( + struct bt_context *ctx, int handle_id, enum bt_clock_type type); +%rename("_bt_trace_handle_get_id") bt_trace_handle_get_id(struct bt_trace_handle *th); +int bt_trace_handle_get_id(struct bt_trace_handle *th); +const char *bt_trace_handle_get_path(struct bt_context *ctx, int handle_id); +uint64_t bt_trace_handle_get_timestamp_begin(struct bt_context *ctx, int handle_id, + enum bt_clock_type type); +uint64_t bt_trace_handle_get_timestamp_end(struct bt_context *ctx, int handle_id, + enum bt_clock_type type); + +%rename("_bt_ctf_event_get_handle_id") bt_ctf_event_get_handle_id( + const struct bt_ctf_event *event); +int bt_ctf_event_get_handle_id(const struct bt_ctf_event *event); + + +%pythoncode%{ + +class TraceHandle(object): + """ + The TraceHandle allows the user to manipulate a trace file directly. + It is a unique identifier representing a trace file. + Do not instantiate. + """ + + def __init__(self): + raise NotImplementedError("TraceHandle cannot be instantiated") + + def __repr__(self): + return "Babeltrace TraceHandle: trace_id('{}')".format(self._id) + + def get_id(self): + """Return the TraceHandle id.""" + return self._id + + def get_path(self, context): + """Return the path of a TraceHandle.""" + try: + return _bt_trace_handle_get_path(context._c, self._id) + except AttributeError: + raise TypeError("in get_path, " + "argument 2 must be a Context instance") + + def get_timestamp_begin(self, context, clock_type): + """Return the creation time of the buffers of a trace.""" + try: + return _bt_trace_handle_get_timestamp_begin( + context._c, self._id,clock_type) + except AttributeError: + raise TypeError("in get_timestamp_begin, " + "argument 2 must be a Context instance") + + def get_timestamp_end(self, context, clock_type): + """Return the destruction timestamp of the buffers of a trace.""" + try: + return _bt_trace_handle_get_timestamp_end( + context._c, self._id, clock_type) + except AttributeError: + raise TypeError("in get_timestamp_end, " + "argument 2 must be a Context instance") + +%} + + + +// ================================================================= +// CTF +// ================================================================= + +/* ================================================================= + ITERATOR.H, EVENTS.H + ???????????????????? +*/ + +//Iterator +%rename("_bt_ctf_iter_create") bt_ctf_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, + const struct bt_iter_pos *end_pos); +%rename("_bt_ctf_get_iter") bt_ctf_get_iter(struct bt_ctf_iter *iter); +%rename("_bt_ctf_iter_destroy") bt_ctf_iter_destroy(struct bt_ctf_iter *iter); +%rename("_bt_ctf_iter_read_event") bt_ctf_iter_read_event(struct bt_ctf_iter *iter); + +struct bt_ctf_iter *bt_ctf_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, + const struct bt_iter_pos *end_pos); +struct bt_iter *bt_ctf_get_iter(struct bt_ctf_iter *iter); +void bt_ctf_iter_destroy(struct bt_ctf_iter *iter); +struct bt_ctf_event *bt_ctf_iter_read_event(struct bt_ctf_iter *iter); + + +//Events + +%rename("_bt_ctf_get_top_level_scope") bt_ctf_get_top_level_scope(const struct + bt_ctf_event *event, enum bt_ctf_scope scope); +%rename("_bt_ctf_event_name") bt_ctf_event_name(const struct bt_ctf_event *ctf_event); +%rename("_bt_ctf_get_timestamp") bt_ctf_get_timestamp( + const struct bt_ctf_event *ctf_event); +%rename("_bt_ctf_get_cycles") bt_ctf_get_cycles( + const struct bt_ctf_event *ctf_event); + +%rename("_bt_ctf_get_field") bt_ctf_get_field(const struct bt_ctf_event *ctf_event, + const struct definition *scope, const char *field); +%rename("_bt_ctf_get_index") bt_ctf_get_index(const struct bt_ctf_event *ctf_event, + const struct definition *field, unsigned int index); +%rename("_bt_ctf_field_name") bt_ctf_field_name(const struct definition *def); +%rename("_bt_ctf_field_type") bt_ctf_field_type(const struct definition *def); +%rename("_bt_ctf_get_int_signedness") bt_ctf_get_int_signedness( + const struct definition *field); +%rename("_bt_ctf_get_int_base") bt_ctf_get_int_base(const struct definition *field); +%rename("_bt_ctf_get_int_byte_order") bt_ctf_get_int_byte_order( + const struct definition *field); +%rename("_bt_ctf_get_int_len") bt_ctf_get_int_len(const struct definition *field); +%rename("_bt_ctf_get_encoding") bt_ctf_get_encoding(const struct definition *field); +%rename("_bt_ctf_get_array_len") bt_ctf_get_array_len(const struct definition *field); +%rename("_bt_ctf_get_uint64") bt_ctf_get_uint64(const struct definition *field); +%rename("_bt_ctf_get_int64") bt_ctf_get_int64(const struct definition *field); +%rename("_bt_ctf_get_char_array") bt_ctf_get_char_array(const struct definition *field); +%rename("_bt_ctf_get_string") bt_ctf_get_string(const struct definition *field); +%rename("_bt_ctf_field_get_error") bt_ctf_field_get_error(void); +%rename("_bt_ctf_get_decl_event_name") bt_ctf_get_decl_event_name(const struct + bt_ctf_event_decl *event); +%rename("_bt_ctf_get_decl_field_name") bt_ctf_get_decl_field_name( + const struct bt_ctf_field_decl *field); + +const struct definition *bt_ctf_get_top_level_scope(const struct bt_ctf_event *ctf_event, + enum bt_ctf_scope scope); +const char *bt_ctf_event_name(const struct bt_ctf_event *ctf_event); +uint64_t bt_ctf_get_timestamp(const struct bt_ctf_event *ctf_event); +uint64_t bt_ctf_get_cycles(const struct bt_ctf_event *ctf_event); +const struct definition *bt_ctf_get_field(const struct bt_ctf_event *ctf_event, + const struct definition *scope, + const char *field); +const struct definition *bt_ctf_get_index(const struct bt_ctf_event *ctf_event, + const struct definition *field, + unsigned int index); +const char *bt_ctf_field_name(const struct definition *def); +enum ctf_type_id bt_ctf_field_type(const struct definition *def); +int bt_ctf_get_int_signedness(const struct definition *field); +int bt_ctf_get_int_base(const struct definition *field); +int bt_ctf_get_int_byte_order(const struct definition *field); +ssize_t bt_ctf_get_int_len(const struct definition *field); +enum ctf_string_encoding bt_ctf_get_encoding(const struct definition *field); +int bt_ctf_get_array_len(const struct definition *field); +uint64_t bt_ctf_get_uint64(const struct definition *field); +int64_t bt_ctf_get_int64(const struct definition *field); +char *bt_ctf_get_char_array(const struct definition *field); +char *bt_ctf_get_string(const struct definition *field); +int bt_ctf_field_get_error(void); +const char *bt_ctf_get_decl_event_name(const struct bt_ctf_event_decl *event); +const char *bt_ctf_get_decl_field_name(const struct bt_ctf_field_decl *field); + +%pythoncode%{ + +class ctf: + + #enum equivalent, accessible constants + #These are taken directly from ctf/events.h + #All changes to enums must also be made here + class type_id: + UNKNOWN = 0 + INTEGER = 1 + FLOAT = 2 + ENUM = 3 + STRING = 4 + STRUCT = 5 + UNTAGGED_VARIANT = 6 + VARIANT = 7 + ARRAY = 8 + SEQUENCE = 9 + NR_CTF_TYPES = 10 + + class scope: + TRACE_PACKET_HEADER = 0 + STREAM_PACKET_CONTEXT = 1 + STREAM_EVENT_HEADER = 2 + STREAM_EVENT_CONTEXT = 3 + EVENT_CONTEXT = 4 + EVENT_FIELDS = 5 + + class string_encoding: + NONE = 0 + UTF8 = 1 + ASCII = 2 + UNKNOWN = 3 + + class Iterator(Iterator, object): + """ + Allocate a CTF trace collection iterator. + + begin_pos and end_pos are optional parameters to specify the + position at which the trace collection should be seeked upon + iterator creation, and the position at which iteration will + start returning "EOF". + + By default, if begin_pos is None, a SEEK_CUR is performed at + creation. By default, if end_pos is None, a SEEK_END (end of + trace) is the EOF criterion. + + Only one iterator can be created against a context. If more than one + iterator is being created for the same context, the second creation + will return None. The previous iterator must be destroyed before + creation of the new iterator for this function to succeed. + """ + + def __new__(cls, context, begin_pos = None, end_pos = None): + # __new__ is used to control the return value + # as the ctf.Iterator class should return None + # if bt_ctf_iter_create returns NULL + + if begin_pos is None: + bp = None + else: + bp = begin_pos._pos + if end_pos is None: + ep = None + else: + ep = end_pos._pos + try: + it = _bt_ctf_iter_create(context._c, bp, ep) + except AttributeError: + raise TypeError("in __init__, " + "argument 2 must be a Context instance") + if it is None: + return None + + ret_class = super(ctf.Iterator, cls).__new__(cls) + ret_class._i = it + return ret_class + + def __init__(self, context, begin_pos = None, end_pos = None): + Iterator.__init__(self, None, None, None, + _bt_ctf_get_iter(self._i)) + + def __del__(self): + _bt_ctf_iter_destroy(self._i) + + def read_event(self): + """ + Read the iterator's current event data. + Return current event on success, None on end of trace. + """ + ret = _bt_ctf_iter_read_event(self._i) + if ret is None: + return ret + ev = ctf.Event.__new__(ctf.Event) + ev._e = ret + return ev + + + class Event(object): + """ + This class represents an event from the trace. + It is obtained with read_event() from ctf.Iterator. + Do not instantiate. + """ + + def __init__(self): + raise NotImplementedError("ctf.Event cannot be instantiated") + + def get_top_level_scope(self, scope): + """ + Return a definition of the top-level scope + Top-level scopes are defined in ctf.scope. + In order to get a field or a field list, the user needs to pass a + scope as argument, this scope can be a top-level scope or a scope + relative to an arbitrary field. This function provides the mapping + between the scope and the actual definition of top-level scopes. + On error return None. + """ + evDef = ctf.Definition.__new__(ctf.Definition) + evDef._d = _bt_ctf_get_top_level_scope(self._e, scope) + if evDef._d is None: + return None + return evDef + + def get_name(self): + """Return the name of the event or None on error.""" + return _bt_ctf_event_name(self._e) + + def get_cycles(self): + """ + Return the timestamp of the event as written in + the packet (in cycles) or -1ULL on error. + """ + return _bt_ctf_get_cycles(self._e) + + def get_timestamp(self): + """ + Return the timestamp of the event offsetted with the + system clock source or -1ULL on error. + """ + return _bt_ctf_get_timestamp(self._e) + + def get_field(self, scope, field): + """Return the definition of a specific field.""" + evDef = ctf.Definition.__new__(ctf.Definition) + try: + evDef._d = _bt_ctf_get_field(self._e, scope._d, field) + except AttributeError: + raise TypeError("in get_field, argument 2 must be a " + "Definition (scope) instance") + return evDef + + def get_field_list(self, scope): + """ + Return a list of Definitions + Return None on error. + """ + try: + field_lc = _bt_python_field_listcaller(self._e, scope._d) + except AttributeError: + raise TypeError("in get_field_list, argument 2 must be a " + "Definition (scope) instance") + + if field_lc is None: + return None + + def_list = [] + i = 0 + while True: + tmp = ctf.Definition.__new__(ctf.Definition) + tmp._d = _bt_python_field_one_from_list(field_lc, i) + + if tmp._d is None: + #Last item of list is None, assured in + #_bt_python_field_listcaller + break + + def_list.append(tmp) + i += 1 + return def_list + + def get_index(self, field, index): + """ + If the field is an array or a sequence, return the element + at position index, otherwise return None + """ + evDef = ctf.Definition.__new__(ctf.Definition) + try: + evDef._d = _bt_ctf_get_index(self._e, field._d, index) + except AttributeError: + raise TypeError("in get_index, argument 2 must be a " + "Definition (field) instance") + + if evDef._d is None: + return None + return evDef + + def get_handle(self): + """ + Get the TraceHandle associated with an event + Return None on error + """ + ret = _bt_ctf_event_get_handle_id(self._e) + if ret < 0: + return None + + th = TraceHandle.__new__(TraceHandle) + th._id = ret + return th + + def get_context(self): + """ + Get the context associated with an event. + Return None on error. + """ + ctx = Context() + ctx._c = _bt_ctf_event_get_context(self._e); + if ctx._c is None: + return None + else: + return ctx + + + class Definition(object): + """Definition class. Do not instantiate.""" + + def __init__(self): + raise NotImplementedError("ctf.Definition cannot be instantiated") + + def __repr__(self): + return "Babeltrace Definition: name('{}'), type({})".format( + self.field_name(), self.field_type()) + + def field_name(self): + """Return the name of a field or None on error.""" + return _bt_ctf_field_name(self._d) + + def field_type(self): + """Return the type of a field or -1 if unknown.""" + return _bt_ctf_field_type(self._d) + + def get_int_signedness(self): + """ + Return the signedness of an integer: + 0 if unsigned; 1 if signed; -1 on error. + """ + return _bt_ctf_get_int_signedness(self._d) + + def get_int_base(self): + """Return the base of an int or a negative value on error.""" + return _bt_ctf_get_int_base(self._d) + + def get_int_byte_order(self): + """ + Return the byte order of an int or a negative + value on error. + """ + return _bt_ctf_get_int_byte_order(self._d) + + def get_int_len(self): + """ + Return the size, in bits, of an int or a negative + value on error. + """ + return _bt_ctf_get_int_len(self._d) + + def get_encoding(self): + """ + Return the encoding of an int or a string. + Return a negative value on error. + """ + return _bt_ctf_get_encoding(self._d) + + def get_array_len(self): + """ + Return the len of an array or a negative + value on error. + """ + return _bt_ctf_get_array_len(self._d) + + def get_uint64(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_uint64(self._d) + + def get_int64(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_int64(self._d) + + def get_char_array(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_char_array(self._d) + + def get_str(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_string(self._d) + + + class EventDecl(object): + """Event declaration class. Do not instantiate.""" + + def __init__(self): + raise NotImplementedError("ctf.EventDecl cannot be instantiated") + + def __repr__(self): + return "Babeltrace EventDecl: name {}".format(self.get_name()) + + def get_name(self): + """Return the name of the event or None on error""" + return _bt_ctf_get_decl_event_name(self._d) + + def get_decl_fields(self, scope): + """ + Return a list of ctf.FieldDecl + Return None on error. + """ + ptr_list = _by_python_field_decl_listcaller(self._d, scope) + + if ptr_list is None: + return None + + decl_list = [] + i = 0 + while True: + tmp = ctf.FieldDecl.__new__(ctf.FieldDecl) + tmp._d = _bt_python_field_decl_one_from_list( + ptr_list, i) + + if tmp._d is None: + #Last item of list is None + break + + decl_list.append(tmp) + i += 1 + return decl_list + + + class FieldDecl(object): + """Field declaration class. Do not instantiate.""" + + def __init__(self): + raise NotImplementedError("ctf.FieldDecl cannot be instantiated") + + def __repr__(self): + return "Babeltrace FieldDecl: name {}".format(self.get_name()) + + def get_name(self): + """Return the name of a FieldDecl or None on error""" + return _bt_ctf_get_decl_field_name(self._d) + + + @staticmethod + def field_error(): + """ + Return the last error code encountered while + accessing a field and reset the error flag. + Return 0 if no error, a negative value otherwise. + """ + return _bt_ctf_field_get_error() + + @staticmethod + def get_event_decl_list(trace_handle, context): + """ + Return a list of ctf.EventDecl + Return None on error. + """ + try: + handle_id = trace_handle._id + except AttributeError: + raise TypeError("in get_event_decl_list, " + "argument 1 must be a TraceHandle instance") + try: + ptr_list = _bt_python_event_decl_listcaller(handle_id, context._c) + except AttributeError: + raise TypeError("in get_event_decl_list, " + "argument 2 must be a Context instance") + + if ptr_list is None: + return None + + decl_list = [] + i = 0 + while True: + tmp = ctf.EventDecl.__new__(ctf.EventDecl) + tmp._d = _bt_python_decl_one_from_list(ptr_list, i) + + if tmp._d is None: + #Last item of list is None + break + + decl_list.append(tmp) + i += 1 + return decl_list + +%} + + + +// ================================================================= +// NEW FUNCTIONS +// File and list-related +// python-complements.h +// ================================================================= + +%include python-complements.c + +%pythoncode %{ + +class File(object): + """ + Open a file for babeltrace. + + file_path is a string containing the path or None to use the + standard output in writing mode. + + The mode can be 'r', 'w' or 'a' for reading (default), writing or + appending. The file will be created if it doesn't exist when + opened for writing or appending; it will be truncated when opened + for writing. Add a 'b' to the mode for binary files. Add a '+' + to the mode to allow simultaneous reading and writing. + """ + + def __new__(cls, file_path, mode='r'): + # __new__ is used to control the return value + # as the File class should return None + # if _bt_file_open returns NULL + + # Type check + if file_path is not None and type(file_path) is not str: + raise TypeError("in method __init__, argument 2 of type 'str'") + if type(mode) is not str: + raise TypeError("in method __init__, argument 3 of type 'str'") + + # Opening file + file_ptr = _bt_file_open(file_path, mode) + if file_ptr is None: + return None + + # Class instantiation + file_inst = super(File, cls).__new__(cls) + file_inst._file = file_ptr + return file_inst + + def __init__(self, file_path, mode='r'): + self._opened = True + self._use_stdout = False + + if file_path is None: + # use stdout + file_path = "stdout" + mode = 'w' + self._use_stdout = True + + self._file_path = file_path + self._mode = mode + + def __del__(self): + self.close() + + def __repr__(self): + if self._opened: + stat = 'opened' + else: + stat = 'closed' + return "{} babeltrace File; file_path('{}'), mode('{}')".format( + stat, self._file_path, self._mode) + + def close(self): + """Close the file. Is also called using del.""" + if self._opened and not self._use_stdout: + _bt_file_close(self._file) + self._opened = False +%} diff --git a/bindings/python/examples/babeltrace_and_lttng.py b/bindings/python/examples/babeltrace_and_lttng.py new file mode 100644 index 0000000..ef0e35c --- /dev/null +++ b/bindings/python/examples/babeltrace_and_lttng.py @@ -0,0 +1,107 @@ +# This script uses both lttng-tools and babeltrace +# python modules. It creates a session, enables +# events, starts tracing for 2 seconds, stops tracing, +# destroys the session and outputs the trace in the +# specified output file. +# +# WARNING: will destroy any existing trace having +# the same name as ses_name + + +# ------------------------------------------------------ +ses_name = "babeltrace-lttng-test" +trace_path = "/lttng-traces/babeltrace-lttng-trace/" +out_file = "babeltrace-lttng-trace-text-output.txt" +# ------------------------------------------------------ + + +import time +try: + import babeltrace, lttng +except ImportError: + raise ImportError( "both babeltrace and lttng-tools " + "python modules must be installed" ) + + +# Errors to raise if something goes wrong +class LTTngError(Exception): + pass +class BabeltraceError(Exception): + pass + + +# LTTNG-TOOLS + +# Making sure session does not already exist +lttng.destroy(ses_name) + +# Creating a new session and handle +ret = lttng.create(ses_name,trace_path) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + +han = None +han = lttng.Handle(ses_name, lttng.Domain()) +if han is None: + raise LTTngError("Handle not created") + + +# Enabling all events +ret = lttng.enable_event(han, lttng.Event(), None) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + + +# Start, wait, stop +ret = lttng.start(ses_name) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) +print("Tracing...") +time.sleep(2) +print("Stopped.") +ret = lttng.stop(ses_name) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + + +# Destroying tracing session +ret = lttng.destroy(ses_name) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + + +# BABELTRACE + +# Create context and add trace: +ctx = babeltrace.Context() +ret = ctx.add_trace(trace_path + "/kernel", "ctf") +if ret is None: + raise BabeltraceError("Error adding trace") + +# Iterator setup +bp = babeltrace.IterPos(babeltrace.SEEK_BEGIN) +ctf_it = babeltrace.ctf.Iterator(ctx,bp) + +# Reading events from trace +# and outputting timestamps and event names +# in out_file +print("Writing trace file...") +output = open(out_file, "wt") + +event = ctf_it.read_event() +while(event is not None): + output.write("TS: {}, {} : {}\n".format(event.get_timestamp(), + event.get_cycles(), event.get_name())) + + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +# Closing file +output.close() + +# Destroying dynamic elements +del ctf_it, han +print("Done.") diff --git a/bindings/python/examples/eventcount.py b/bindings/python/examples/eventcount.py new file mode 100644 index 0000000..2a63b74 --- /dev/null +++ b/bindings/python/examples/eventcount.py @@ -0,0 +1,66 @@ +# The script prints a count of specified events and +# their related tid's in a given trace. +# The trace needs TID context (lttng add-context -k -t tid) + +import sys +from babeltrace import * +from output_format_modules.pprint_table import pprint_table as pprint + +if len(sys.argv) < 3: + raise TypeError("Usage: python eventcount.py event1 [event2 ...] path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +counts = {} + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +while(event is not None): + for event_type in sys.argv[1:len(sys.argv)-1]: + if event_type == event.get_name(): + + # Getting scope definition + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) + if sco is None: + print("ERROR: Cannot get definition scope for {}".format( + event.get_name())) + continue + + # Getting TID + tid_field = event.get_field(sco, "_tid") + tid = tid_field.get_int64() + + if ctf.field_error(): + print("ERROR: Missing TID info for {}".format( + event.get_name())) + continue + + tmp = (tid, event.get_name()) + + if tmp in counts: + counts[tmp] += 1 + else: + counts[tmp] = 1 + + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it + +# Appending data to table for output +table = [] +for item in counts: + table.append([item[0], item[1], counts[item]]) +table = sorted(table) +table.insert(0,["TID", "EVENT", "COUNT"]) +pprint(table, 2) diff --git a/bindings/python/examples/eventcountlist.py b/bindings/python/examples/eventcountlist.py new file mode 100644 index 0000000..800996c --- /dev/null +++ b/bindings/python/examples/eventcountlist.py @@ -0,0 +1,65 @@ +# The script prints a count and rate of events. +# It also outputs a bar graph of count per event, using the cairoplot module. + +import sys +from babeltrace import * +from output_format_modules import cairoplot +from output_format_modules.pprint_table import pprint_table as pprint + +# Check for path arg: +if len(sys.argv) < 2: + raise TypeError("Usage: python eventcountlist.py path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +# Events and their assossiated count +# will be stored as a dict: +events_count = {} + +# Setting iterator: +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx,bp) + +prev_event = None +event = ctf_it.read_event() + +start_time = event.get_timestamp() + +# Reading events: +while(event is not None): + if event.get_name() in events_count: + events_count[event.get_name()] += 1 + else: + events_count[event.get_name()] = 1 + + ret = ctf_it.next() + if ret < 0: + break + else: + prev_event = event + event = ctf_it.read_event() + +if event: + total_time = event.get_timestamp() - start_time +else: + total_time = prev_event.get_timestamp() - start_time + +del ctf_it + +# Printing encountered events with respective count and rate: +print("Total time: {} ns".format(total_time)) +table = [["EVENT", "COUNT", "RATE (Hz)"]] +for item in sorted(events_count.iterkeys()): + tmp = [item, events_count[item], + events_count[item]/(total_time/1000000000.0)] + table.append(tmp) +pprint(table) + +# Exporting data as bar graph +cairoplot.vertical_bar_plot ( 'eventcountlist.svg', events_count, 50+85*len(events_count), + 800, border = 20, display_values = True, grid = True, + rounded_corners = True, + x_labels = sorted(events_count.keys()) ) diff --git a/bindings/python/examples/events_per_cpu.py b/bindings/python/examples/events_per_cpu.py new file mode 100644 index 0000000..b464735 --- /dev/null +++ b/bindings/python/examples/events_per_cpu.py @@ -0,0 +1,96 @@ +# The script opens a trace and prints out CPU statistics +# for the given trace (event count per CPU, total active +# time and % of time processing events). +# It also outputs a .txt file showing each time interval +# (since the beginning of the trace) in which each CPU +# was active and the corresponding event. + +import sys, multiprocessing +from output_format_modules.pprint_table import pprint_table as pprint +from babeltrace import * + +if len(sys.argv) < 2: + raise TypeError("Usage: python events_per_cpu.py path/to/trace") + +# Adding trace +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +cpu_usage = [] +usage_time = [] +nbEvents = 0 +i = 0 +while i < multiprocessing.cpu_count(): + cpu_usage.append([]) + usage_time.append(0) + i += 1 + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +start_time = event.get_timestamp() +old_ts = start_time + +while(event is not None): + + skip = False + event_name = event.get_name() + + # Getting cpu_id + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) + field = event.get_field(scope, "cpu_id") + cpu_id = field.get_uint64() + if ctf.field_error(): + print("ERROR: Missing cpu_id info for {}".format(event.get_name())) + skip = True + + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + if event is None: + break + + ts = event.get_timestamp() + + if not skip: + cpu_usage[cpu_id].append( (old_ts - start_time, ts - start_time, event_name) ) + usage_time[cpu_id] += ts - old_ts + nbEvents += 1 + + old_ts = ts + +trace_time = old_ts - start_time + +# Outputting +table = [] +output = open("events_per_cpu.txt", "wt") +output.write("(start time (ns since beginning of trace), end time (ns), event)\n") + +for cpu in range(len(usage_time)): + # Setting table + time_str = str(100.0 * usage_time[cpu] / trace_time) + '000' + event_str = str(100.0 * len(cpu_usage[cpu]) / nbEvents) + '000' + # % is printed with 2 decimals + table.append([cpu, len(cpu_usage[cpu]), event_str[0:event_str.find('.') + 3] + ' %', + usage_time[cpu], time_str[0:time_str.find('.') + 3] + ' %']) + + # Writing to file + output.write("\n\n\n----------------------\n") + output.write("CPU {}\n\n".format(cpu)) + for event in cpu_usage[cpu]: + output.write(str(event) + '\n') + +# Printing table +table.insert(0, ["CPU ID", "EVENT COUNT", "TRACE EVENT %", "TOTAL ACTIVE TIME (ns)", "TRACE TIME %"]) +pprint(table) +print("Total event count: {}".format(nbEvents)) +print("Total trace time: {} ns".format(trace_time)) + +output.close() diff --git a/bindings/python/examples/example-api-test.py b/bindings/python/examples/example-api-test.py new file mode 100644 index 0000000..0cfc3ed --- /dev/null +++ b/bindings/python/examples/example-api-test.py @@ -0,0 +1,59 @@ +# This example uses the babeltrace python module +# to partially test the api. + +import sys +from babeltrace import * + +# Check for path arg: +if len(sys.argv) < 2: + raise TypeError("Usage: python example-api-test.py path/to/file") + +# Create context and add trace: +ctx = Context() +trace_handle = ctx.add_trace(sys.argv[1], "ctf") +if trace_handle is None: + raise IOError("Error adding trace") + +# Listing events +lst = ctf.get_event_decl_list(trace_handle, ctx) +print("--- Event list ---") +for item in lst: + print("event : {}".format(item.get_name())) +print("--- Done ---") + +# Iter trace +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx,bp) +event = ctf_it.read_event() + +while(event is not None): + print("TS: {}, {} : {}".format(event.get_timestamp(), + event.get_cycles(), event.get_name())) + + if event.get_name() == "sched_switch": + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + prev_field = event.get_field(sco, "_prev_comm") + prev_comm = prev_field.get_char_array() + + if ctf.field_error(): + print("ERROR: Missing prev_comm context info") + else: + print("sched_switch prev_comm: {}".format(prev_comm)) + + if event.get_name() == "exit_syscall": + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + ret_field = event.get_field(sco, "_ret") + ret_code = ret_field.get_int64() + + if ctf.field_error(): + print("ERROR: Unable to extract ret") + else: + print("exit_syscall ret: {}".format(ret_code)) + + ret = ctf_it.next() + if ret < 0: + break + else: + event = ctf_it.read_event() + +del ctf_it diff --git a/bindings/python/examples/histogram.py b/bindings/python/examples/histogram.py new file mode 100644 index 0000000..a24c00d --- /dev/null +++ b/bindings/python/examples/histogram.py @@ -0,0 +1,121 @@ +# The script checks the number of events in the trace +# and outputs a table and a .svg histogram for the specified +# range (microseconds) or the total trace if no range specified. +# The graph is generated using the cairoplot module. + +import sys +from babeltrace import * +from output_format_modules import cairoplot +from output_format_modules.pprint_table import pprint_table as pprint + +# ------------------------------------------------ +# Output settings + +# number of intervals: +nbDiv = 25 # Should not be over 150 + # for usable graph output + +# table output stream (file-like object): +out = sys.stdout +# ------------------------------------------------- + +if len(sys.argv) < 2 or len(sys.argv) > 4: + raise TypeError("Usage: python histogram.py [ start_time [end_time] ] path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +# Check when to start/stop graphing +sinceBegin = True +beginTime = 0.0 +if len(sys.argv) > 2: + sinceBegin = False + beginTime = float(sys.argv[1]) +untilEnd = True +if len(sys.argv) == 4: + untilEnd = False + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +start_time = event.get_timestamp() +time = 0 +count = {} + +while(event is not None): + # Microsec. + time = (event.get_timestamp() - start_time)/1000.0 + + # Check if in range + if not sinceBegin: + if time < beginTime: + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + continue + if not untilEnd: + if time > float(sys.argv[2]): + break + + # Counting events per timestamp: + if time in count: + count[time] += 1 + else: + count[time] = 1 + + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it + +# Setting data for output +interval = (time - beginTime)/nbDiv +div_begin_time = beginTime +div_end_time = beginTime + interval +data = {} + +# Prefix for string sorting, considering +# there should not be over 150 intervals. +# This would work up to 9999 intervals. +# If needed, add zeros. +prefix = 0.0001 + +while div_end_time <= time: + key = str(prefix) + '[' + str(div_begin_time) + ';' + str(div_end_time) + '[' + for tmp in count: + if tmp >= div_begin_time and tmp < div_end_time: + if key in data: + data[key] += count[tmp] + else: + data[key] = count[tmp] + if not key in data: + data[key] = 0 + div_begin_time = div_end_time + div_end_time += interval + # Prefix increment + prefix += 0.001 + +table = [] +x_labels = [] +for key in sorted(data): + table.append([key[key.find('['):], data[key]]) + x_labels.append(key[key.find('['):]) + +# Table output +table.insert(0, ["INTERVAL (us)", "COUNT"]) +pprint(table, 1, out) + +# Graph output +cairoplot.vertical_bar_plot ( 'histogram.svg', data, 50 + 150*nbDiv, 50*nbDiv, + border = 20, display_values = True, grid = True, + x_labels = x_labels, rounded_corners = True ) diff --git a/bindings/python/examples/output_format_modules/__init__.py b/bindings/python/examples/output_format_modules/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/bindings/python/examples/output_format_modules/cairoplot.py b/bindings/python/examples/output_format_modules/cairoplot.py new file mode 100755 index 0000000..a27113f --- /dev/null +++ b/bindings/python/examples/output_format_modules/cairoplot.py @@ -0,0 +1,2336 @@ +?#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# CairoPlot.py +# +# Copyright (c) 2008 Rodrigo Moreira Ara?jo +# +# Author: Rodrigo Moreiro Araujo +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public License +# as published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 +# USA + +#Contributor: Jo?o S. O. Bueno + +#TODO: review BarPlot Code +#TODO: x_label colision problem on Horizontal Bar Plot +#TODO: y_label's eat too much space on HBP + + +__version__ = 1.2 + +import cairo +import math +import random +from series import Series, Group, Data + +HORZ = 0 +VERT = 1 +NORM = 2 + +COLORS = {"red" : (1.0,0.0,0.0,1.0), "lime" : (0.0,1.0,0.0,1.0), "blue" : (0.0,0.0,1.0,1.0), + "maroon" : (0.5,0.0,0.0,1.0), "green" : (0.0,0.5,0.0,1.0), "navy" : (0.0,0.0,0.5,1.0), + "yellow" : (1.0,1.0,0.0,1.0), "magenta" : (1.0,0.0,1.0,1.0), "cyan" : (0.0,1.0,1.0,1.0), + "orange" : (1.0,0.5,0.0,1.0), "white" : (1.0,1.0,1.0,1.0), "black" : (0.0,0.0,0.0,1.0), + "gray" : (0.5,0.5,0.5,1.0), "light_gray" : (0.9,0.9,0.9,1.0), + "transparent" : (0.0,0.0,0.0,0.0)} + +THEMES = {"black_red" : [(0.0,0.0,0.0,1.0), (1.0,0.0,0.0,1.0)], + "red_green_blue" : [(1.0,0.0,0.0,1.0), (0.0,1.0,0.0,1.0), (0.0,0.0,1.0,1.0)], + "red_orange_yellow" : [(1.0,0.2,0.0,1.0), (1.0,0.7,0.0,1.0), (1.0,1.0,0.0,1.0)], + "yellow_orange_red" : [(1.0,1.0,0.0,1.0), (1.0,0.7,0.0,1.0), (1.0,0.2,0.0,1.0)], + "rainbow" : [(1.0,0.0,0.0,1.0), (1.0,0.5,0.0,1.0), (1.0,1.0,0.0,1.0), (0.0,1.0,0.0,1.0), (0.0,0.0,1.0,1.0), (0.3, 0.0, 0.5,1.0), (0.5, 0.0, 1.0, 1.0)]} + +def colors_from_theme( theme, series_length, mode = 'solid' ): + colors = [] + if theme not in THEMES.keys() : + raise Exception, "Theme not defined" + color_steps = THEMES[theme] + n_colors = len(color_steps) + if series_length <= n_colors: + colors = [color + tuple([mode]) for color in color_steps[0:n_colors]] + else: + iterations = [(series_length - n_colors)/(n_colors - 1) for i in color_steps[:-1]] + over_iterations = (series_length - n_colors) % (n_colors - 1) + for i in range(n_colors - 1): + if over_iterations <= 0: + break + iterations[i] += 1 + over_iterations -= 1 + for index,color in enumerate(color_steps[:-1]): + colors.append(color + tuple([mode])) + if iterations[index] == 0: + continue + next_color = color_steps[index+1] + color_step = ((next_color[0] - color[0])/(iterations[index] + 1), + (next_color[1] - color[1])/(iterations[index] + 1), + (next_color[2] - color[2])/(iterations[index] + 1), + (next_color[3] - color[3])/(iterations[index] + 1)) + for i in range( iterations[index] ): + colors.append((color[0] + color_step[0]*(i+1), + color[1] + color_step[1]*(i+1), + color[2] + color_step[2]*(i+1), + color[3] + color_step[3]*(i+1), + mode)) + colors.append(color_steps[-1] + tuple([mode])) + return colors + + +def other_direction(direction): + "explicit is better than implicit" + if direction == HORZ: + return VERT + else: + return HORZ + +#Class definition + +class Plot(object): + def __init__(self, + surface=None, + data=None, + width=640, + height=480, + background=None, + border = 0, + x_labels = None, + y_labels = None, + series_colors = None): + random.seed(2) + self.create_surface(surface, width, height) + self.dimensions = {} + self.dimensions[HORZ] = width + self.dimensions[VERT] = height + self.context = cairo.Context(self.surface) + self.labels={} + self.labels[HORZ] = x_labels + self.labels[VERT] = y_labels + self.load_series(data, x_labels, y_labels, series_colors) + self.font_size = 10 + self.set_background (background) + self.border = border + self.borders = {} + self.line_color = (0.5, 0.5, 0.5) + self.line_width = 0.5 + self.label_color = (0.0, 0.0, 0.0) + self.grid_color = (0.8, 0.8, 0.8) + + def create_surface(self, surface, width=None, height=None): + self.filename = None + if isinstance(surface, cairo.Surface): + self.surface = surface + return + if not type(surface) in (str, unicode): + raise TypeError("Surface should be either a Cairo surface or a filename, not %s" % surface) + sufix = surface.rsplit(".")[-1].lower() + self.filename = surface + if sufix == "png": + self.surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, width, height) + elif sufix == "ps": + self.surface = cairo.PSSurface(surface, width, height) + elif sufix == "pdf": + self.surface = cairo.PSSurface(surface, width, height) + else: + if sufix != "svg": + self.filename += ".svg" + self.surface = cairo.SVGSurface(self.filename, width, height) + + def commit(self): + try: + self.context.show_page() + if self.filename and self.filename.endswith(".png"): + self.surface.write_to_png(self.filename) + else: + self.surface.finish() + except cairo.Error: + pass + + def load_series (self, data, x_labels=None, y_labels=None, series_colors=None): + self.series_labels = [] + self.series = None + + #The pretty way + #if not isinstance(data, Series): + # # Not an instance of Series + # self.series = Series(data) + #else: + # self.series = data + # + #self.series_labels = self.series.get_names() + + #TODO: Remove on next version + # The ugly way, keeping retrocompatibility... + if callable(data) or type(data) is list and callable(data[0]): # Lambda or List of lambdas + self.series = data + self.series_labels = None + elif isinstance(data, Series): # Instance of Series + self.series = data + self.series_labels = data.get_names() + else: # Anything else + self.series = Series(data) + self.series_labels = self.series.get_names() + + #TODO: allow user passed series_widths + self.series_widths = [1.0 for group in self.series] + + #TODO: Remove on next version + self.process_colors( series_colors ) + + def process_colors( self, series_colors, length = None, mode = 'solid' ): + #series_colors might be None, a theme, a string of colors names or a list of color tuples + if length is None : + length = len( self.series.to_list() ) + + #no colors passed + if not series_colors: + #Randomize colors + self.series_colors = [ [random.random() for i in range(3)] + [1.0, mode] for series in range( length ) ] + else: + #Just theme pattern + if not hasattr( series_colors, "__iter__" ): + theme = series_colors + self.series_colors = colors_from_theme( theme.lower(), length ) + + #Theme pattern and mode + elif not hasattr(series_colors, '__delitem__') and not hasattr( series_colors[0], "__iter__" ): + theme = series_colors[0] + mode = series_colors[1] + self.series_colors = colors_from_theme( theme.lower(), length, mode ) + + #List + else: + self.series_colors = series_colors + for index, color in enumerate( self.series_colors ): + #element is a color name + if not hasattr(color, "__iter__"): + self.series_colors[index] = COLORS[color.lower()] + tuple([mode]) + #element is rgb tuple instead of rgba + elif len( color ) == 3 : + self.series_colors[index] += (1.0,mode) + #element has 4 elements, might be rgba tuple or rgb tuple with mode + elif len( color ) == 4 : + #last element is mode + if not hasattr(color[3], "__iter__"): + self.series_colors[index] += tuple([color[3]]) + self.series_colors[index][3] = 1.0 + #last element is alpha + else: + self.series_colors[index] += tuple([mode]) + + def get_width(self): + return self.surface.get_width() + + def get_height(self): + return self.surface.get_height() + + def set_background(self, background): + if background is None: + self.background = (0.0,0.0,0.0,0.0) + elif type(background) in (cairo.LinearGradient, tuple): + self.background = background + elif not hasattr(background,"__iter__"): + colors = background.split(" ") + if len(colors) == 1 and colors[0] in COLORS: + self.background = COLORS[background] + elif len(colors) > 1: + self.background = cairo.LinearGradient(self.dimensions[HORZ] / 2, 0, self.dimensions[HORZ] / 2, self.dimensions[VERT]) + for index,color in enumerate(colors): + self.background.add_color_stop_rgba(float(index)/(len(colors)-1),*COLORS[color]) + else: + raise TypeError ("Background should be either cairo.LinearGradient or a 3/4-tuple, not %s" % type(background)) + + def render_background(self): + if isinstance(self.background, cairo.LinearGradient): + self.context.set_source(self.background) + else: + self.context.set_source_rgba(*self.background) + self.context.rectangle(0,0, self.dimensions[HORZ], self.dimensions[VERT]) + self.context.fill() + + def render_bounding_box(self): + self.context.set_source_rgba(*self.line_color) + self.context.set_line_width(self.line_width) + self.context.rectangle(self.border, self.border, + self.dimensions[HORZ] - 2 * self.border, + self.dimensions[VERT] - 2 * self.border) + self.context.stroke() + + def render(self): + pass + +class ScatterPlot( Plot ): + def __init__(self, + surface=None, + data=None, + errorx=None, + errory=None, + width=640, + height=480, + background=None, + border=0, + axis = False, + dash = False, + discrete = False, + dots = 0, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + z_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + circle_colors = None ): + + self.bounds = {} + self.bounds[HORZ] = x_bounds + self.bounds[VERT] = y_bounds + self.bounds[NORM] = z_bounds + self.titles = {} + self.titles[HORZ] = x_title + self.titles[VERT] = y_title + self.max_value = {} + self.axis = axis + self.discrete = discrete + self.dots = dots + self.grid = grid + self.series_legend = series_legend + self.variable_radius = False + self.x_label_angle = math.pi / 2.5 + self.circle_colors = circle_colors + + Plot.__init__(self, surface, data, width, height, background, border, x_labels, y_labels, series_colors) + + self.dash = None + if dash: + if hasattr(dash, "keys"): + self.dash = [dash[key] for key in self.series_labels] + elif max([hasattr(item,'__delitem__') for item in data]) : + self.dash = dash + else: + self.dash = [dash] + + self.load_errors(errorx, errory) + + def convert_list_to_tuple(self, data): + #Data must be converted from lists of coordinates to a single + # list of tuples + out_data = zip(*data) + if len(data) == 3: + self.variable_radius = True + return out_data + + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): + #TODO: In cairoplot 2.0 keep only the Series instances + + # Convert Data and Group to Series + if isinstance(data, Data) or isinstance(data, Group): + data = Series(data) + + # Series + if isinstance(data, Series): + for group in data: + for item in group: + if len(item) is 3: + self.variable_radius = True + + #Dictionary with lists + if hasattr(data, "keys") : + if hasattr( data.values()[0][0], "__delitem__" ) : + for key in data.keys() : + data[key] = self.convert_list_to_tuple(data[key]) + elif len(data.values()[0][0]) == 3: + self.variable_radius = True + #List + elif hasattr(data[0], "__delitem__") : + #List of lists + if hasattr(data[0][0], "__delitem__") : + for index,value in enumerate(data) : + data[index] = self.convert_list_to_tuple(value) + #List + elif type(data[0][0]) != type((0,0)): + data = self.convert_list_to_tuple(data) + #Three dimensional data + elif len(data[0][0]) == 3: + self.variable_radius = True + + #List with three dimensional tuples + elif len(data[0]) == 3: + self.variable_radius = True + Plot.load_series(self, data, x_labels, y_labels, series_colors) + self.calc_boundaries() + self.calc_labels() + + def load_errors(self, errorx, errory): + self.errors = None + if errorx == None and errory == None: + return + self.errors = {} + self.errors[HORZ] = None + self.errors[VERT] = None + #asimetric errors + if errorx and hasattr(errorx[0], "__delitem__"): + self.errors[HORZ] = errorx + #simetric errors + elif errorx: + self.errors[HORZ] = [errorx] + #asimetric errors + if errory and hasattr(errory[0], "__delitem__"): + self.errors[VERT] = errory + #simetric errors + elif errory: + self.errors[VERT] = [errory] + + def calc_labels(self): + if not self.labels[HORZ]: + amplitude = self.bounds[HORZ][1] - self.bounds[HORZ][0] + if amplitude % 10: #if horizontal labels need floating points + self.labels[HORZ] = ["%.2lf" % (float(self.bounds[HORZ][0] + (amplitude * i / 10.0))) for i in range(11) ] + else: + self.labels[HORZ] = ["%d" % (int(self.bounds[HORZ][0] + (amplitude * i / 10.0))) for i in range(11) ] + if not self.labels[VERT]: + amplitude = self.bounds[VERT][1] - self.bounds[VERT][0] + if amplitude % 10: #if vertical labels need floating points + self.labels[VERT] = ["%.2lf" % (float(self.bounds[VERT][0] + (amplitude * i / 10.0))) for i in range(11) ] + else: + self.labels[VERT] = ["%d" % (int(self.bounds[VERT][0] + (amplitude * i / 10.0))) for i in range(11) ] + + def calc_extents(self, direction): + self.context.set_font_size(self.font_size * 0.8) + self.max_value[direction] = max(self.context.text_extents(item)[2] for item in self.labels[direction]) + self.borders[other_direction(direction)] = self.max_value[direction] + self.border + 20 + + def calc_boundaries(self): + #HORZ = 0, VERT = 1, NORM = 2 + min_data_value = [0,0,0] + max_data_value = [0,0,0] + + for group in self.series: + if type(group[0].content) in (int, float, long): + group = [Data((index, item.content)) for index,item in enumerate(group)] + + for point in group: + for index, item in enumerate(point.content): + if item > max_data_value[index]: + max_data_value[index] = item + elif item < min_data_value[index]: + min_data_value[index] = item + + if not self.bounds[HORZ]: + self.bounds[HORZ] = (min_data_value[HORZ], max_data_value[HORZ]) + if not self.bounds[VERT]: + self.bounds[VERT] = (min_data_value[VERT], max_data_value[VERT]) + if not self.bounds[NORM]: + self.bounds[NORM] = (min_data_value[NORM], max_data_value[NORM]) + + def calc_all_extents(self): + self.calc_extents(HORZ) + self.calc_extents(VERT) + + self.plot_height = self.dimensions[VERT] - 2 * self.borders[VERT] + self.plot_width = self.dimensions[HORZ] - 2* self.borders[HORZ] + + self.plot_top = self.dimensions[VERT] - self.borders[VERT] + + def calc_steps(self): + #Calculates all the x, y, z and color steps + series_amplitude = [self.bounds[index][1] - self.bounds[index][0] for index in range(3)] + + if series_amplitude[HORZ]: + self.horizontal_step = float (self.plot_width) / series_amplitude[HORZ] + else: + self.horizontal_step = 0.00 + + if series_amplitude[VERT]: + self.vertical_step = float (self.plot_height) / series_amplitude[VERT] + else: + self.vertical_step = 0.00 + + if series_amplitude[NORM]: + if self.variable_radius: + self.z_step = float (self.bounds[NORM][1]) / series_amplitude[NORM] + if self.circle_colors: + self.circle_color_step = tuple([float(self.circle_colors[1][i]-self.circle_colors[0][i])/series_amplitude[NORM] for i in range(4)]) + else: + self.z_step = 0.00 + self.circle_color_step = ( 0.0, 0.0, 0.0, 0.0 ) + + def get_circle_color(self, value): + return tuple( [self.circle_colors[0][i] + value*self.circle_color_step[i] for i in range(4)] ) + + def render(self): + self.calc_all_extents() + self.calc_steps() + self.render_background() + self.render_bounding_box() + if self.axis: + self.render_axis() + if self.grid: + self.render_grid() + self.render_labels() + self.render_plot() + if self.errors: + self.render_errors() + if self.series_legend and self.series_labels: + self.render_legend() + + def render_axis(self): + #Draws both the axis lines and their titles + cr = self.context + cr.set_source_rgba(*self.line_color) + cr.move_to(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) + cr.line_to(self.borders[HORZ], self.borders[VERT]) + cr.stroke() + + cr.move_to(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) + cr.line_to(self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) + cr.stroke() + + cr.set_source_rgba(*self.label_color) + self.context.set_font_size( 1.2 * self.font_size ) + if self.titles[HORZ]: + title_width,title_height = cr.text_extents(self.titles[HORZ])[2:4] + cr.move_to( self.dimensions[HORZ]/2 - title_width/2, self.borders[VERT] - title_height/2 ) + cr.show_text( self.titles[HORZ] ) + + if self.titles[VERT]: + title_width,title_height = cr.text_extents(self.titles[VERT])[2:4] + cr.move_to( self.dimensions[HORZ] - self.borders[HORZ] + title_height/2, self.dimensions[VERT]/2 - title_width/2) + cr.save() + cr.rotate( math.pi/2 ) + cr.show_text( self.titles[VERT] ) + cr.restore() + + def render_grid(self): + cr = self.context + horizontal_step = float( self.plot_height ) / ( len( self.labels[VERT] ) - 1 ) + vertical_step = float( self.plot_width ) / ( len( self.labels[HORZ] ) - 1 ) + + x = self.borders[HORZ] + vertical_step + y = self.plot_top - horizontal_step + + for label in self.labels[HORZ][:-1]: + cr.set_source_rgba(*self.grid_color) + cr.move_to(x, self.dimensions[VERT] - self.borders[VERT]) + cr.line_to(x, self.borders[VERT]) + cr.stroke() + x += vertical_step + for label in self.labels[VERT][:-1]: + cr.set_source_rgba(*self.grid_color) + cr.move_to(self.borders[HORZ], y) + cr.line_to(self.dimensions[HORZ] - self.borders[HORZ], y) + cr.stroke() + y -= horizontal_step + + def render_labels(self): + self.context.set_font_size(self.font_size * 0.8) + self.render_horz_labels() + self.render_vert_labels() + + def render_horz_labels(self): + cr = self.context + step = float( self.plot_width ) / ( len( self.labels[HORZ] ) - 1 ) + x = self.borders[HORZ] + y = self.dimensions[VERT] - self.borders[VERT] + 5 + + # store rotation matrix from the initial state + rotation_matrix = cr.get_matrix() + rotation_matrix.rotate(self.x_label_angle) + + cr.set_source_rgba(*self.label_color) + + for item in self.labels[HORZ]: + width = cr.text_extents(item)[2] + cr.move_to(x, y) + cr.save() + cr.set_matrix(rotation_matrix) + cr.show_text(item) + cr.restore() + x += step + + def render_vert_labels(self): + cr = self.context + step = ( self.plot_height ) / ( len( self.labels[VERT] ) - 1 ) + y = self.plot_top + cr.set_source_rgba(*self.label_color) + for item in self.labels[VERT]: + width = cr.text_extents(item)[2] + cr.move_to(self.borders[HORZ] - width - 5,y) + cr.show_text(item) + y -= step + + def render_legend(self): + cr = self.context + cr.set_font_size(self.font_size) + cr.set_line_width(self.line_width) + + widest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[2]) + tallest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[3]) + max_width = self.context.text_extents(widest_word)[2] + max_height = self.context.text_extents(tallest_word)[3] * 1.1 + + color_box_height = max_height / 2 + color_box_width = color_box_height * 2 + + #Draw a bounding box + bounding_box_width = max_width + color_box_width + 15 + bounding_box_height = (len(self.series_labels)+0.5) * max_height + cr.set_source_rgba(1,1,1) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - bounding_box_width, self.borders[VERT], + bounding_box_width, bounding_box_height) + cr.fill() + + cr.set_source_rgba(*self.line_color) + cr.set_line_width(self.line_width) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - bounding_box_width, self.borders[VERT], + bounding_box_width, bounding_box_height) + cr.stroke() + + for idx,key in enumerate(self.series_labels): + #Draw color box + cr.set_source_rgba(*self.series_colors[idx][:4]) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - max_width - color_box_width - 10, + self.borders[VERT] + color_box_height + (idx*max_height) , + color_box_width, color_box_height) + cr.fill() + + cr.set_source_rgba(0, 0, 0) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - max_width - color_box_width - 10, + self.borders[VERT] + color_box_height + (idx*max_height), + color_box_width, color_box_height) + cr.stroke() + + #Draw series labels + cr.set_source_rgba(0, 0, 0) + cr.move_to(self.dimensions[HORZ] - self.borders[HORZ] - max_width - 5, self.borders[VERT] + ((idx+1)*max_height)) + cr.show_text(key) + + def render_errors(self): + cr = self.context + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) + cr.clip() + radius = self.dots + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + for index, group in enumerate(self.series): + cr.set_source_rgba(*self.series_colors[index][:4]) + for number, data in enumerate(group): + x = x0 + self.horizontal_step * data.content[0] + y = self.dimensions[VERT] - y0 - self.vertical_step * data.content[1] + if self.errors[HORZ]: + cr.move_to(x, y) + x1 = x - self.horizontal_step * self.errors[HORZ][0][number] + cr.line_to(x1, y) + cr.line_to(x1, y - radius) + cr.line_to(x1, y + radius) + cr.stroke() + if self.errors[HORZ] and len(self.errors[HORZ]) == 2: + cr.move_to(x, y) + x1 = x + self.horizontal_step * self.errors[HORZ][1][number] + cr.line_to(x1, y) + cr.line_to(x1, y - radius) + cr.line_to(x1, y + radius) + cr.stroke() + if self.errors[VERT]: + cr.move_to(x, y) + y1 = y + self.vertical_step * self.errors[VERT][0][number] + cr.line_to(x, y1) + cr.line_to(x - radius, y1) + cr.line_to(x + radius, y1) + cr.stroke() + if self.errors[VERT] and len(self.errors[VERT]) == 2: + cr.move_to(x, y) + y1 = y - self.vertical_step * self.errors[VERT][1][number] + cr.line_to(x, y1) + cr.line_to(x - radius, y1) + cr.line_to(x + radius, y1) + cr.stroke() + + + def render_plot(self): + cr = self.context + if self.discrete: + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) + cr.clip() + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + radius = self.dots + for number, group in enumerate (self.series): + cr.set_source_rgba(*self.series_colors[number][:4]) + for data in group : + if self.variable_radius: + radius = data.content[2]*self.z_step + if self.circle_colors: + cr.set_source_rgba( *self.get_circle_color( data.content[2]) ) + x = x0 + self.horizontal_step*data.content[0] + y = y0 + self.vertical_step*data.content[1] + cr.arc(x, self.dimensions[VERT] - y, radius, 0, 2*math.pi) + cr.fill() + else: + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) + cr.clip() + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + radius = self.dots + for number, group in enumerate (self.series): + last_data = None + cr.set_source_rgba(*self.series_colors[number][:4]) + for data in group : + x = x0 + self.horizontal_step*data.content[0] + y = y0 + self.vertical_step*data.content[1] + if self.dots: + if self.variable_radius: + radius = data.content[2]*self.z_step + cr.arc(x, self.dimensions[VERT] - y, radius, 0, 2*math.pi) + cr.fill() + if last_data : + old_x = x0 + self.horizontal_step*last_data.content[0] + old_y = y0 + self.vertical_step*last_data.content[1] + cr.move_to( old_x, self.dimensions[VERT] - old_y ) + cr.line_to( x, self.dimensions[VERT] - y) + cr.set_line_width(self.series_widths[number]) + + #?Display line as dash line + if self.dash and self.dash[number]: + s = self.series_widths[number] + cr.set_dash([s*3, s*3], 0) + + cr.stroke() + cr.set_dash([]) + last_data = data + +class DotLinePlot(ScatterPlot): + def __init__(self, + surface=None, + data=None, + width=640, + height=480, + background=None, + border=0, + axis = False, + dash = False, + dots = 0, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None): + + ScatterPlot.__init__(self, surface, data, None, None, width, height, background, border, + axis, dash, False, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, None, x_title, y_title, series_colors, None ) + + + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + for group in self.series : + for index,data in enumerate(group): + group[index].content = (index, data.content) + + self.calc_boundaries() + self.calc_labels() + +class FunctionPlot(ScatterPlot): + def __init__(self, + surface=None, + data=None, + width=640, + height=480, + background=None, + border=0, + axis = False, + discrete = False, + dots = 0, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + step = 1): + + self.function = data + self.step = step + self.discrete = discrete + + data, x_bounds = self.load_series_from_function( self.function, x_bounds ) + + ScatterPlot.__init__(self, surface, data, None, None, width, height, background, border, + axis, False, discrete, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, None, x_title, y_title, series_colors, None ) + + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + + if len(self.series[0][0]) is 1: + for group_id, group in enumerate(self.series) : + for index,data in enumerate(group): + group[index].content = (self.bounds[HORZ][0] + self.step*index, data.content) + + self.calc_boundaries() + self.calc_labels() + + def load_series_from_function( self, function, x_bounds ): + #TODO: Add the possibility for the user to define multiple functions with different discretization parameters + + #This function converts a function, a list of functions or a dictionary + #of functions into its corresponding array of data + series = Series() + + if isinstance(function, Group) or isinstance(function, Data): + function = Series(function) + + # If is instance of Series + if isinstance(function, Series): + # Overwrite any bounds passed by the function + x_bounds = (function.range[0],function.range[-1]) + + #if no bounds are provided + if x_bounds == None: + x_bounds = (0,10) + + + #TODO: Finish the dict translation + if hasattr(function, "keys"): #dictionary: + for key in function.keys(): + group = Group(name=key) + #data[ key ] = [] + i = x_bounds[0] + while i <= x_bounds[1] : + group.add_data(function[ key ](i)) + #data[ key ].append( function[ key ](i) ) + i += self.step + series.add_group(group) + + elif hasattr(function, "__delitem__"): #list of functions + for index,f in enumerate( function ) : + group = Group() + #data.append( [] ) + i = x_bounds[0] + while i <= x_bounds[1] : + group.add_data(f(i)) + #data[ index ].append( f(i) ) + i += self.step + series.add_group(group) + + elif isinstance(function, Series): # instance of Series + series = function + + else: #function + group = Group() + i = x_bounds[0] + while i <= x_bounds[1] : + group.add_data(function(i)) + i += self.step + series.add_group(group) + + + return series, x_bounds + + def calc_labels(self): + if not self.labels[HORZ]: + self.labels[HORZ] = [] + i = self.bounds[HORZ][0] + while i<=self.bounds[HORZ][1]: + self.labels[HORZ].append(str(i)) + i += float(self.bounds[HORZ][1] - self.bounds[HORZ][0])/10 + ScatterPlot.calc_labels(self) + + def render_plot(self): + if not self.discrete: + ScatterPlot.render_plot(self) + else: + last = None + cr = self.context + for number, group in enumerate (self.series): + cr.set_source_rgba(*self.series_colors[number][:4]) + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + for data in group: + x = x0 + self.horizontal_step * data.content[0] + y = y0 + self.vertical_step * data.content[1] + cr.move_to(x, self.dimensions[VERT] - y) + cr.line_to(x, self.plot_top) + cr.set_line_width(self.series_widths[number]) + cr.stroke() + if self.dots: + cr.new_path() + cr.arc(x, self.dimensions[VERT] - y, 3, 0, 2.1 * math.pi) + cr.close_path() + cr.fill() + +class BarPlot(Plot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None, + main_dir = None): + + self.bounds = {} + self.bounds[HORZ] = x_bounds + self.bounds[VERT] = y_bounds + self.display_values = display_values + self.grid = grid + self.rounded_corners = rounded_corners + self.stack = stack + self.three_dimension = three_dimension + self.x_label_angle = math.pi / 2.5 + self.main_dir = main_dir + self.max_value = {} + self.plot_dimensions = {} + self.steps = {} + self.value_label_color = (0.5,0.5,0.5,1.0) + + Plot.__init__(self, surface, data, width, height, background, border, x_labels, y_labels, series_colors) + + def load_series(self, data, x_labels = None, y_labels = None, series_colors = None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + self.calc_boundaries() + + def process_colors(self, series_colors): + #Data for a BarPlot might be a List or a List of Lists. + #On the first case, colors must be generated for all bars, + #On the second, colors must be generated for each of the inner lists. + + #TODO: Didn't get it... + #if hasattr(self.data[0], '__getitem__'): + # length = max(len(series) for series in self.data) + #else: + # length = len( self.data ) + + length = max(len(group) for group in self.series) + + Plot.process_colors( self, series_colors, length, 'linear') + + def calc_boundaries(self): + if not self.bounds[self.main_dir]: + if self.stack: + max_data_value = max(sum(group.to_list()) for group in self.series) + else: + max_data_value = max(max(group.to_list()) for group in self.series) + self.bounds[self.main_dir] = (0, max_data_value) + if not self.bounds[other_direction(self.main_dir)]: + self.bounds[other_direction(self.main_dir)] = (0, len(self.series)) + + def calc_extents(self, direction): + self.max_value[direction] = 0 + if self.labels[direction]: + widest_word = max(self.labels[direction], key = lambda item: self.context.text_extents(item)[2]) + self.max_value[direction] = self.context.text_extents(widest_word)[3 - direction] + self.borders[other_direction(direction)] = (2-direction)*self.max_value[direction] + self.border + direction*(5) + else: + self.borders[other_direction(direction)] = self.border + + def calc_horz_extents(self): + self.calc_extents(HORZ) + + def calc_vert_extents(self): + self.calc_extents(VERT) + + def calc_all_extents(self): + self.calc_horz_extents() + self.calc_vert_extents() + other_dir = other_direction(self.main_dir) + self.value_label = 0 + if self.display_values: + if self.stack: + self.value_label = self.context.text_extents(str(max(sum(group.to_list()) for group in self.series)))[2 + self.main_dir] + else: + self.value_label = self.context.text_extents(str(max(max(group.to_list()) for group in self.series)))[2 + self.main_dir] + if self.labels[self.main_dir]: + self.plot_dimensions[self.main_dir] = self.dimensions[self.main_dir] - 2*self.borders[self.main_dir] - self.value_label + else: + self.plot_dimensions[self.main_dir] = self.dimensions[self.main_dir] - self.borders[self.main_dir] - 1.2*self.border - self.value_label + self.plot_dimensions[other_dir] = self.dimensions[other_dir] - self.borders[other_dir] - self.border + self.plot_top = self.dimensions[VERT] - self.borders[VERT] + + def calc_steps(self): + other_dir = other_direction(self.main_dir) + self.series_amplitude = self.bounds[self.main_dir][1] - self.bounds[self.main_dir][0] + if self.series_amplitude: + self.steps[self.main_dir] = float(self.plot_dimensions[self.main_dir])/self.series_amplitude + else: + self.steps[self.main_dir] = 0.00 + series_length = len(self.series) + self.steps[other_dir] = float(self.plot_dimensions[other_dir])/(series_length + 0.1*(series_length + 1)) + self.space = 0.1*self.steps[other_dir] + + def render(self): + self.calc_all_extents() + self.calc_steps() + self.render_background() + self.render_bounding_box() + if self.grid: + self.render_grid() + if self.three_dimension: + self.render_ground() + if self.display_values: + self.render_values() + self.render_labels() + self.render_plot() + if self.series_labels: + self.render_legend() + + def draw_3d_rectangle_front(self, x0, y0, x1, y1, shift): + self.context.rectangle(x0-shift, y0+shift, x1-x0, y1-y0) + + def draw_3d_rectangle_side(self, x0, y0, x1, y1, shift): + self.context.move_to(x1-shift,y0+shift) + self.context.line_to(x1, y0) + self.context.line_to(x1, y1) + self.context.line_to(x1-shift, y1+shift) + self.context.line_to(x1-shift, y0+shift) + self.context.close_path() + + def draw_3d_rectangle_top(self, x0, y0, x1, y1, shift): + self.context.move_to(x0-shift,y0+shift) + self.context.line_to(x0, y0) + self.context.line_to(x1, y0) + self.context.line_to(x1-shift, y0+shift) + self.context.line_to(x0-shift, y0+shift) + self.context.close_path() + + def draw_round_rectangle(self, x0, y0, x1, y1): + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) + self.context.line_to(x1-5, y0) + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) + self.context.line_to(x1, y1-5) + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) + self.context.line_to(x0+5, y1) + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) + self.context.line_to(x0, y0+5) + self.context.close_path() + + def render_ground(self): + self.draw_3d_rectangle_front(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_side (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_top (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + def render_labels(self): + self.context.set_font_size(self.font_size * 0.8) + if self.labels[HORZ]: + self.render_horz_labels() + if self.labels[VERT]: + self.render_vert_labels() + + def render_legend(self): + cr = self.context + cr.set_font_size(self.font_size) + cr.set_line_width(self.line_width) + + widest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[2]) + tallest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[3]) + max_width = self.context.text_extents(widest_word)[2] + max_height = self.context.text_extents(tallest_word)[3] * 1.1 + 5 + + color_box_height = max_height / 2 + color_box_width = color_box_height * 2 + + #Draw a bounding box + bounding_box_width = max_width + color_box_width + 15 + bounding_box_height = (len(self.series_labels)+0.5) * max_height + cr.set_source_rgba(1,1,1) + cr.rectangle(self.dimensions[HORZ] - self.border - bounding_box_width, self.border, + bounding_box_width, bounding_box_height) + cr.fill() + + cr.set_source_rgba(*self.line_color) + cr.set_line_width(self.line_width) + cr.rectangle(self.dimensions[HORZ] - self.border - bounding_box_width, self.border, + bounding_box_width, bounding_box_height) + cr.stroke() + + for idx,key in enumerate(self.series_labels): + #Draw color box + cr.set_source_rgba(*self.series_colors[idx][:4]) + cr.rectangle(self.dimensions[HORZ] - self.border - max_width - color_box_width - 10, + self.border + color_box_height + (idx*max_height) , + color_box_width, color_box_height) + cr.fill() + + cr.set_source_rgba(0, 0, 0) + cr.rectangle(self.dimensions[HORZ] - self.border - max_width - color_box_width - 10, + self.border + color_box_height + (idx*max_height), + color_box_width, color_box_height) + cr.stroke() + + #Draw series labels + cr.set_source_rgba(0, 0, 0) + cr.move_to(self.dimensions[HORZ] - self.border - max_width - 5, self.border + ((idx+1)*max_height)) + cr.show_text(key) + + +class HorizontalBarPlot(BarPlot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None): + + BarPlot.__init__(self, surface, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, series_colors, HORZ) + self.series_labels = series_labels + + def calc_vert_extents(self): + self.calc_extents(VERT) + if self.labels[HORZ] and not self.labels[VERT]: + self.borders[HORZ] += 10 + + def draw_rectangle_bottom(self, x0, y0, x1, y1): + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) + self.context.line_to(x0, y0+5) + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) + self.context.line_to(x1, y0) + self.context.line_to(x1, y1) + self.context.line_to(x0+5, y1) + self.context.close_path() + + def draw_rectangle_top(self, x0, y0, x1, y1): + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) + self.context.line_to(x1, y1-5) + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) + self.context.line_to(x0, y1) + self.context.line_to(x0, y0) + self.context.line_to(x1, y0) + self.context.close_path() + + def draw_rectangle(self, index, length, x0, y0, x1, y1): + if length == 1: + BarPlot.draw_rectangle(self, x0, y0, x1, y1) + elif index == 0: + self.draw_rectangle_bottom(x0, y0, x1, y1) + elif index == length-1: + self.draw_rectangle_top(x0, y0, x1, y1) + else: + self.context.rectangle(x0, y0, x1-x0, y1-y0) + + #TODO: Review BarPlot.render_grid code + def render_grid(self): + self.context.set_source_rgba(0.8, 0.8, 0.8) + if self.labels[HORZ]: + self.context.set_font_size(self.font_size * 0.8) + step = (self.dimensions[HORZ] - 2*self.borders[HORZ] - self.value_label)/(len(self.labels[HORZ])-1) + x = self.borders[HORZ] + next_x = 0 + for item in self.labels[HORZ]: + width = self.context.text_extents(item)[2] + if x - width/2 > next_x and x - width/2 > self.border: + self.context.move_to(x, self.border) + self.context.line_to(x, self.dimensions[VERT] - self.borders[VERT]) + self.context.stroke() + next_x = x + width/2 + x += step + else: + lines = 11 + horizontal_step = float(self.plot_dimensions[HORZ])/(lines-1) + x = self.borders[HORZ] + for y in xrange(0, lines): + self.context.move_to(x, self.border) + self.context.line_to(x, self.dimensions[VERT] - self.borders[VERT]) + self.context.stroke() + x += horizontal_step + + def render_horz_labels(self): + step = (self.dimensions[HORZ] - 2*self.borders[HORZ])/(len(self.labels[HORZ])-1) + x = self.borders[HORZ] + next_x = 0 + + for item in self.labels[HORZ]: + self.context.set_source_rgba(*self.label_color) + width = self.context.text_extents(item)[2] + if x - width/2 > next_x and x - width/2 > self.border: + self.context.move_to(x - width/2, self.dimensions[VERT] - self.borders[VERT] + self.max_value[HORZ] + 3) + self.context.show_text(item) + next_x = x + width/2 + x += step + + def render_vert_labels(self): + series_length = len(self.labels[VERT]) + step = (self.plot_dimensions[VERT] - (series_length + 1)*self.space)/(len(self.labels[VERT])) + y = self.border + step/2 + self.space + + for item in self.labels[VERT]: + self.context.set_source_rgba(*self.label_color) + width, height = self.context.text_extents(item)[2:4] + self.context.move_to(self.borders[HORZ] - width - 5, y + height/2) + self.context.show_text(item) + y += step + self.space + self.labels[VERT].reverse() + + def render_values(self): + self.context.set_source_rgba(*self.value_label_color) + self.context.set_font_size(self.font_size * 0.8) + if self.stack: + for i,group in enumerate(self.series): + value = sum(group.to_list()) + height = self.context.text_extents(str(value))[3] + x = self.borders[HORZ] + value*self.steps[HORZ] + 2 + y = self.borders[VERT] + (i+0.5)*self.steps[VERT] + (i+1)*self.space + height/2 + self.context.move_to(x, y) + self.context.show_text(str(value)) + else: + for i,group in enumerate(self.series): + inner_step = self.steps[VERT]/len(group) + y0 = self.border + i*self.steps[VERT] + (i+1)*self.space + for number,data in enumerate(group): + height = self.context.text_extents(str(data.content))[3] + self.context.move_to(self.borders[HORZ] + data.content*self.steps[HORZ] + 2, y0 + 0.5*inner_step + height/2, ) + self.context.show_text(str(data.content)) + y0 += inner_step + + def render_plot(self): + if self.stack: + for i,group in enumerate(self.series): + x0 = self.borders[HORZ] + y0 = self.borders[VERT] + i*self.steps[VERT] + (i+1)*self.space + for number,data in enumerate(group): + if self.series_colors[number][4] in ('radial','linear') : + linear = cairo.LinearGradient( data.content*self.steps[HORZ]/2, y0, data.content*self.steps[HORZ]/2, y0 + self.steps[VERT] ) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + elif self.series_colors[number][4] == 'solid': + self.context.set_source_rgba(*self.series_colors[number][:4]) + if self.rounded_corners: + self.draw_rectangle(number, len(group), x0, y0, x0+data.content*self.steps[HORZ], y0+self.steps[VERT]) + self.context.fill() + else: + self.context.rectangle(x0, y0, data.content*self.steps[HORZ], self.steps[VERT]) + self.context.fill() + x0 += data.content*self.steps[HORZ] + else: + for i,group in enumerate(self.series): + inner_step = self.steps[VERT]/len(group) + x0 = self.borders[HORZ] + y0 = self.border + i*self.steps[VERT] + (i+1)*self.space + for number,data in enumerate(group): + linear = cairo.LinearGradient(data.content*self.steps[HORZ]/2, y0, data.content*self.steps[HORZ]/2, y0 + inner_step) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + if self.rounded_corners and data.content != 0: + BarPlot.draw_round_rectangle(self,x0, y0, x0 + data.content*self.steps[HORZ], y0 + inner_step) + self.context.fill() + else: + self.context.rectangle(x0, y0, data.content*self.steps[HORZ], inner_step) + self.context.fill() + y0 += inner_step + +class VerticalBarPlot(BarPlot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None): + + BarPlot.__init__(self, surface, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, series_colors, VERT) + self.series_labels = series_labels + + def calc_vert_extents(self): + self.calc_extents(VERT) + if self.labels[VERT] and not self.labels[HORZ]: + self.borders[VERT] += 10 + + def draw_rectangle_bottom(self, x0, y0, x1, y1): + self.context.move_to(x1,y1) + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) + self.context.line_to(x0+5, y1) + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) + self.context.line_to(x0, y0) + self.context.line_to(x1, y0) + self.context.line_to(x1, y1) + self.context.close_path() + + def draw_rectangle_top(self, x0, y0, x1, y1): + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) + self.context.line_to(x1-5, y0) + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) + self.context.line_to(x1, y1) + self.context.line_to(x0, y1) + self.context.line_to(x0, y0) + self.context.close_path() + + def draw_rectangle(self, index, length, x0, y0, x1, y1): + if length == 1: + BarPlot.draw_rectangle(self, x0, y0, x1, y1) + elif index == 0: + self.draw_rectangle_bottom(x0, y0, x1, y1) + elif index == length-1: + self.draw_rectangle_top(x0, y0, x1, y1) + else: + self.context.rectangle(x0, y0, x1-x0, y1-y0) + + def render_grid(self): + self.context.set_source_rgba(0.8, 0.8, 0.8) + if self.labels[VERT]: + lines = len(self.labels[VERT]) + vertical_step = float(self.plot_dimensions[self.main_dir])/(lines-1) + y = self.borders[VERT] + self.value_label + else: + lines = 11 + vertical_step = float(self.plot_dimensions[self.main_dir])/(lines-1) + y = 1.2*self.border + self.value_label + for x in xrange(0, lines): + self.context.move_to(self.borders[HORZ], y) + self.context.line_to(self.dimensions[HORZ] - self.border, y) + self.context.stroke() + y += vertical_step + + def render_ground(self): + self.draw_3d_rectangle_front(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_side (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_top (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + def render_horz_labels(self): + series_length = len(self.labels[HORZ]) + step = float (self.plot_dimensions[HORZ] - (series_length + 1)*self.space)/len(self.labels[HORZ]) + x = self.borders[HORZ] + step/2 + self.space + next_x = 0 + + for item in self.labels[HORZ]: + self.context.set_source_rgba(*self.label_color) + width = self.context.text_extents(item)[2] + if x - width/2 > next_x and x - width/2 > self.borders[HORZ]: + self.context.move_to(x - width/2, self.dimensions[VERT] - self.borders[VERT] + self.max_value[HORZ] + 3) + self.context.show_text(item) + next_x = x + width/2 + x += step + self.space + + def render_vert_labels(self): + self.context.set_source_rgba(*self.label_color) + y = self.borders[VERT] + self.value_label + step = (self.dimensions[VERT] - 2*self.borders[VERT] - self.value_label)/(len(self.labels[VERT]) - 1) + self.labels[VERT].reverse() + for item in self.labels[VERT]: + width, height = self.context.text_extents(item)[2:4] + self.context.move_to(self.borders[HORZ] - width - 5, y + height/2) + self.context.show_text(item) + y += step + self.labels[VERT].reverse() + + def render_values(self): + self.context.set_source_rgba(*self.value_label_color) + self.context.set_font_size(self.font_size * 0.8) + if self.stack: + for i,group in enumerate(self.series): + value = sum(group.to_list()) + width = self.context.text_extents(str(value))[2] + x = self.borders[HORZ] + (i+0.5)*self.steps[HORZ] + (i+1)*self.space - width/2 + y = value*self.steps[VERT] + 2 + self.context.move_to(x, self.plot_top-y) + self.context.show_text(str(value)) + else: + for i,group in enumerate(self.series): + inner_step = self.steps[HORZ]/len(group) + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space + for number,data in enumerate(group): + width = self.context.text_extents(str(data.content))[2] + self.context.move_to(x0 + 0.5*inner_step - width/2, self.plot_top - data.content*self.steps[VERT] - 2) + self.context.show_text(str(data.content)) + x0 += inner_step + + def render_plot(self): + if self.stack: + for i,group in enumerate(self.series): + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space + y0 = 0 + for number,data in enumerate(group): + if self.series_colors[number][4] in ('linear','radial'): + linear = cairo.LinearGradient( x0, data.content*self.steps[VERT]/2, x0 + self.steps[HORZ], data.content*self.steps[VERT]/2 ) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + elif self.series_colors[number][4] == 'solid': + self.context.set_source_rgba(*self.series_colors[number][:4]) + if self.rounded_corners: + self.draw_rectangle(number, len(group), x0, self.plot_top - y0 - data.content*self.steps[VERT], x0 + self.steps[HORZ], self.plot_top - y0) + self.context.fill() + else: + self.context.rectangle(x0, self.plot_top - y0 - data.content*self.steps[VERT], self.steps[HORZ], data.content*self.steps[VERT]) + self.context.fill() + y0 += data.content*self.steps[VERT] + else: + for i,group in enumerate(self.series): + inner_step = self.steps[HORZ]/len(group) + y0 = self.borders[VERT] + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space + for number,data in enumerate(group): + if self.series_colors[number][4] == 'linear': + linear = cairo.LinearGradient( x0, data.content*self.steps[VERT]/2, x0 + inner_step, data.content*self.steps[VERT]/2 ) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + elif self.series_colors[number][4] == 'solid': + self.context.set_source_rgba(*self.series_colors[number][:4]) + if self.rounded_corners and data.content != 0: + BarPlot.draw_round_rectangle(self, x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top) + self.context.fill() + elif self.three_dimension: + self.draw_3d_rectangle_front(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) + self.context.fill() + self.draw_3d_rectangle_side(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) + self.context.fill() + self.draw_3d_rectangle_top(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) + self.context.fill() + else: + self.context.rectangle(x0, self.plot_top - data.content*self.steps[VERT], inner_step, data.content*self.steps[VERT]) + self.context.fill() + + x0 += inner_step + +class StreamChart(VerticalBarPlot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + grid = False, + series_legend = None, + x_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None): + + VerticalBarPlot.__init__(self, surface, data, width, height, background, border, + False, grid, False, True, False, + None, x_labels, None, x_bounds, y_bounds, series_colors) + + def calc_steps(self): + other_dir = other_direction(self.main_dir) + self.series_amplitude = self.bounds[self.main_dir][1] - self.bounds[self.main_dir][0] + if self.series_amplitude: + self.steps[self.main_dir] = float(self.plot_dimensions[self.main_dir])/self.series_amplitude + else: + self.steps[self.main_dir] = 0.00 + series_length = len(self.data) + self.steps[other_dir] = float(self.plot_dimensions[other_dir])/series_length + + def render_legend(self): + pass + + def ground(self, index): + sum_values = sum(self.data[index]) + return -0.5*sum_values + + def calc_angles(self): + middle = self.plot_top - self.plot_dimensions[VERT]/2.0 + self.angles = [tuple([0.0 for x in range(len(self.data)+1)])] + for x_index in range(1, len(self.data)-1): + t = [] + x0 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y0 = middle - self.ground(x_index-1)*self.steps[VERT] + y2 = middle - self.ground(x_index+1)*self.steps[VERT] + t.append(math.atan(float(y0-y2)/(x0-x2))) + for data_index in range(len(self.data[x_index])): + x0 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y0 = middle - self.ground(x_index-1)*self.steps[VERT] - self.data[x_index-1][data_index]*self.steps[VERT] + y2 = middle - self.ground(x_index+1)*self.steps[VERT] - self.data[x_index+1][data_index]*self.steps[VERT] + + for i in range(0,data_index): + y0 -= self.data[x_index-1][i]*self.steps[VERT] + y2 -= self.data[x_index+1][i]*self.steps[VERT] + + if data_index == len(self.data[0])-1 and False: + self.context.set_source_rgba(0.0,0.0,0.0,0.3) + self.context.move_to(x0,y0) + self.context.line_to(x2,y2) + self.context.stroke() + self.context.arc(x0,y0,2,0,2*math.pi) + self.context.fill() + t.append(math.atan(float(y0-y2)/(x0-x2))) + self.angles.append(tuple(t)) + self.angles.append(tuple([0.0 for x in range(len(self.data)+1)])) + + def render_plot(self): + self.calc_angles() + middle = self.plot_top - self.plot_dimensions[VERT]/2.0 + p = 0.4*self.steps[HORZ] + for data_index in range(len(self.data[0])-1,-1,-1): + self.context.set_source_rgba(*self.series_colors[data_index][:4]) + + #draw the upper line + for x_index in range(len(self.data)-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] - self.data[x_index][data_index]*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index + 1)*self.steps[VERT] - self.data[x_index + 1][data_index]*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index+1][i]*self.steps[VERT] + + if x_index == 0: + self.context.move_to(x1,y1) + + ang1 = self.angles[x_index][data_index+1] + ang2 = self.angles[x_index+1][data_index+1] + math.pi + self.context.curve_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1), + x2+p*math.cos(ang2),y2+p*math.sin(ang2), + x2,y2) + + for x_index in range(len(self.data)-1,0,-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index - 1)*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index-1][i]*self.steps[VERT] + + if x_index == len(self.data)-1: + self.context.line_to(x1,y1+2) + + #revert angles by pi degrees to take the turn back + ang1 = self.angles[x_index][data_index] + math.pi + ang2 = self.angles[x_index-1][data_index] + self.context.curve_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1), + x2+p*math.cos(ang2),y2+p*math.sin(ang2), + x2,y2+2) + + self.context.close_path() + self.context.fill() + + if False: + self.context.move_to(self.borders[HORZ] + 0.5*self.steps[HORZ], middle) + for x_index in range(len(self.data)-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] - self.data[x_index][data_index]*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index + 1)*self.steps[VERT] - self.data[x_index + 1][data_index]*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index+1][i]*self.steps[VERT] + + ang1 = self.angles[x_index][data_index+1] + ang2 = self.angles[x_index+1][data_index+1] + math.pi + self.context.set_source_rgba(1.0,0.0,0.0) + self.context.arc(x1+p*math.cos(ang1),y1+p*math.sin(ang1),2,0,2*math.pi) + self.context.fill() + self.context.set_source_rgba(0.0,0.0,0.0) + self.context.arc(x2+p*math.cos(ang2),y2+p*math.sin(ang2),2,0,2*math.pi) + self.context.fill() + '''self.context.set_source_rgba(0.0,0.0,0.0,0.3) + self.context.arc(x2,y2,2,0,2*math.pi) + self.context.fill()''' + self.context.move_to(x1,y1) + self.context.line_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1)) + self.context.stroke() + self.context.move_to(x2,y2) + self.context.line_to(x2+p*math.cos(ang2),y2+p*math.sin(ang2)) + self.context.stroke() + if False: + for x_index in range(len(self.data)-1,0,-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index - 1)*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index-1][i]*self.steps[VERT] + + #revert angles by pi degrees to take the turn back + ang1 = self.angles[x_index][data_index] + math.pi + ang2 = self.angles[x_index-1][data_index] + self.context.set_source_rgba(0.0,1.0,0.0) + self.context.arc(x1+p*math.cos(ang1),y1+p*math.sin(ang1),2,0,2*math.pi) + self.context.fill() + self.context.set_source_rgba(0.0,0.0,1.0) + self.context.arc(x2+p*math.cos(ang2),y2+p*math.sin(ang2),2,0,2*math.pi) + self.context.fill() + '''self.context.set_source_rgba(0.0,0.0,0.0,0.3) + self.context.arc(x2,y2,2,0,2*math.pi) + self.context.fill()''' + self.context.move_to(x1,y1) + self.context.line_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1)) + self.context.stroke() + self.context.move_to(x2,y2) + self.context.line_to(x2+p*math.cos(ang2),y2+p*math.sin(ang2)) + self.context.stroke() + #break + + #self.context.arc(self.dimensions[HORZ]/2, self.dimensions[VERT]/2,50,0,3*math.pi/2) + #self.context.fill() + + +class PiePlot(Plot): + #TODO: Check the old cairoplot, graphs aren't matching + def __init__ (self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + gradient = False, + shadow = False, + colors = None): + + Plot.__init__( self, surface, data, width, height, background, series_colors = colors ) + self.center = (self.dimensions[HORZ]/2, self.dimensions[VERT]/2) + self.total = sum( self.series.to_list() ) + self.radius = min(self.dimensions[HORZ]/3,self.dimensions[VERT]/3) + self.gradient = gradient + self.shadow = shadow + + def sort_function(x,y): + return x.content - y.content + + def load_series(self, data, x_labels=None, y_labels=None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + # Already done inside series + #self.data = sorted(self.data) + + def draw_piece(self, angle, next_angle): + self.context.move_to(self.center[0],self.center[1]) + self.context.line_to(self.center[0] + self.radius*math.cos(angle), self.center[1] + self.radius*math.sin(angle)) + self.context.arc(self.center[0], self.center[1], self.radius, angle, next_angle) + self.context.line_to(self.center[0], self.center[1]) + self.context.close_path() + + def render(self): + self.render_background() + self.render_bounding_box() + if self.shadow: + self.render_shadow() + self.render_plot() + self.render_series_labels() + + def render_shadow(self): + horizontal_shift = 3 + vertical_shift = 3 + self.context.set_source_rgba(0, 0, 0, 0.5) + self.context.arc(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.radius, 0, 2*math.pi) + self.context.fill() + + def render_series_labels(self): + angle = 0 + next_angle = 0 + x0,y0 = self.center + cr = self.context + for number,key in enumerate(self.series_labels): + # self.data[number] should be just a number + data = sum(self.series[number].to_list()) + + next_angle = angle + 2.0*math.pi*data/self.total + cr.set_source_rgba(*self.series_colors[number][:4]) + w = cr.text_extents(key)[2] + if (angle + next_angle)/2 < math.pi/2 or (angle + next_angle)/2 > 3*math.pi/2: + cr.move_to(x0 + (self.radius+10)*math.cos((angle+next_angle)/2), y0 + (self.radius+10)*math.sin((angle+next_angle)/2) ) + else: + cr.move_to(x0 + (self.radius+10)*math.cos((angle+next_angle)/2) - w, y0 + (self.radius+10)*math.sin((angle+next_angle)/2) ) + cr.show_text(key) + angle = next_angle + + def render_plot(self): + angle = 0 + next_angle = 0 + x0,y0 = self.center + cr = self.context + for number,group in enumerate(self.series): + # Group should be just a number + data = sum(group.to_list()) + next_angle = angle + 2.0*math.pi*data/self.total + if self.gradient or self.series_colors[number][4] in ('linear','radial'): + gradient_color = cairo.RadialGradient(self.center[0], self.center[1], 0, self.center[0], self.center[1], self.radius) + gradient_color.add_color_stop_rgba(0.3, *self.series_colors[number][:4]) + gradient_color.add_color_stop_rgba(1, self.series_colors[number][0]*0.7, + self.series_colors[number][1]*0.7, + self.series_colors[number][2]*0.7, + self.series_colors[number][3]) + cr.set_source(gradient_color) + else: + cr.set_source_rgba(*self.series_colors[number][:4]) + + self.draw_piece(angle, next_angle) + cr.fill() + + cr.set_source_rgba(1.0, 1.0, 1.0) + self.draw_piece(angle, next_angle) + cr.stroke() + + angle = next_angle + +class DonutPlot(PiePlot): + def __init__ (self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + gradient = False, + shadow = False, + colors = None, + inner_radius=-1): + + Plot.__init__( self, surface, data, width, height, background, series_colors = colors ) + + self.center = ( self.dimensions[HORZ]/2, self.dimensions[VERT]/2 ) + self.total = sum( self.series.to_list() ) + self.radius = min( self.dimensions[HORZ]/3,self.dimensions[VERT]/3 ) + self.inner_radius = inner_radius*self.radius + + if inner_radius == -1: + self.inner_radius = self.radius/3 + + self.gradient = gradient + self.shadow = shadow + + def draw_piece(self, angle, next_angle): + self.context.move_to(self.center[0] + (self.inner_radius)*math.cos(angle), self.center[1] + (self.inner_radius)*math.sin(angle)) + self.context.line_to(self.center[0] + self.radius*math.cos(angle), self.center[1] + self.radius*math.sin(angle)) + self.context.arc(self.center[0], self.center[1], self.radius, angle, next_angle) + self.context.line_to(self.center[0] + (self.inner_radius)*math.cos(next_angle), self.center[1] + (self.inner_radius)*math.sin(next_angle)) + self.context.arc_negative(self.center[0], self.center[1], self.inner_radius, next_angle, angle) + self.context.close_path() + + def render_shadow(self): + horizontal_shift = 3 + vertical_shift = 3 + self.context.set_source_rgba(0, 0, 0, 0.5) + self.context.arc(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.inner_radius, 0, 2*math.pi) + self.context.arc_negative(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.radius, 0, -2*math.pi) + self.context.fill() + +class GanttChart (Plot) : + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + x_labels = None, + y_labels = None, + colors = None): + self.bounds = {} + self.max_value = {} + Plot.__init__(self, surface, data, width, height, x_labels = x_labels, y_labels = y_labels, series_colors = colors) + + def load_series(self, data, x_labels=None, y_labels=None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + self.calc_boundaries() + + def calc_boundaries(self): + self.bounds[HORZ] = (0,len(self.series)) + end_pos = max(self.series.to_list()) + + #for group in self.series: + # if hasattr(item, "__delitem__"): + # for sub_item in item: + # end_pos = max(sub_item) + # else: + # end_pos = max(item) + self.bounds[VERT] = (0,end_pos) + + def calc_extents(self, direction): + self.max_value[direction] = 0 + if self.labels[direction]: + self.max_value[direction] = max(self.context.text_extents(item)[2] for item in self.labels[direction]) + else: + self.max_value[direction] = self.context.text_extents( str(self.bounds[direction][1] + 1) )[2] + + def calc_horz_extents(self): + self.calc_extents(HORZ) + self.borders[HORZ] = 100 + self.max_value[HORZ] + + def calc_vert_extents(self): + self.calc_extents(VERT) + self.borders[VERT] = self.dimensions[VERT]/(self.bounds[HORZ][1] + 1) + + def calc_steps(self): + self.horizontal_step = (self.dimensions[HORZ] - self.borders[HORZ])/(len(self.labels[VERT])) + self.vertical_step = self.borders[VERT] + + def render(self): + self.calc_horz_extents() + self.calc_vert_extents() + self.calc_steps() + self.render_background() + + self.render_labels() + self.render_grid() + self.render_plot() + + def render_background(self): + cr = self.context + cr.set_source_rgba(255,255,255) + cr.rectangle(0,0,self.dimensions[HORZ], self.dimensions[VERT]) + cr.fill() + for number,group in enumerate(self.series): + linear = cairo.LinearGradient(self.dimensions[HORZ]/2, self.borders[VERT] + number*self.vertical_step, + self.dimensions[HORZ]/2, self.borders[VERT] + (number+1)*self.vertical_step) + linear.add_color_stop_rgba(0,1.0,1.0,1.0,1.0) + linear.add_color_stop_rgba(1.0,0.9,0.9,0.9,1.0) + cr.set_source(linear) + cr.rectangle(0,self.borders[VERT] + number*self.vertical_step,self.dimensions[HORZ],self.vertical_step) + cr.fill() + + def render_grid(self): + cr = self.context + cr.set_source_rgba(0.7, 0.7, 0.7) + cr.set_dash((1,0,0,0,0,0,1)) + cr.set_line_width(0.5) + for number,label in enumerate(self.labels[VERT]): + h = cr.text_extents(label)[3] + cr.move_to(self.borders[HORZ] + number*self.horizontal_step, self.vertical_step/2 + h) + cr.line_to(self.borders[HORZ] + number*self.horizontal_step, self.dimensions[VERT]) + cr.stroke() + + def render_labels(self): + self.context.set_font_size(0.02 * self.dimensions[HORZ]) + + self.render_horz_labels() + self.render_vert_labels() + + def render_horz_labels(self): + cr = self.context + labels = self.labels[HORZ] + if not labels: + labels = [str(i) for i in range(1, self.bounds[HORZ][1] + 1) ] + for number,label in enumerate(labels): + if label != None: + cr.set_source_rgba(0.5, 0.5, 0.5) + w,h = cr.text_extents(label)[2], cr.text_extents(label)[3] + cr.move_to(40,self.borders[VERT] + number*self.vertical_step + self.vertical_step/2 + h/2) + cr.show_text(label) + + def render_vert_labels(self): + cr = self.context + labels = self.labels[VERT] + if not labels: + labels = [str(i) for i in range(1, self.bounds[VERT][1] + 1) ] + for number,label in enumerate(labels): + w,h = cr.text_extents(label)[2], cr.text_extents(label)[3] + cr.move_to(self.borders[HORZ] + number*self.horizontal_step - w/2, self.vertical_step/2) + cr.show_text(label) + + def render_rectangle(self, x0, y0, x1, y1, color): + self.draw_shadow(x0, y0, x1, y1) + self.draw_rectangle(x0, y0, x1, y1, color) + + def draw_rectangular_shadow(self, gradient, x0, y0, w, h): + self.context.set_source(gradient) + self.context.rectangle(x0,y0,w,h) + self.context.fill() + + def draw_circular_shadow(self, x, y, radius, ang_start, ang_end, mult, shadow): + gradient = cairo.RadialGradient(x, y, 0, x, y, 2*radius) + gradient.add_color_stop_rgba(0, 0, 0, 0, shadow) + gradient.add_color_stop_rgba(1, 0, 0, 0, 0) + self.context.set_source(gradient) + self.context.move_to(x,y) + self.context.line_to(x + mult[0]*radius,y + mult[1]*radius) + self.context.arc(x, y, 8, ang_start, ang_end) + self.context.line_to(x,y) + self.context.close_path() + self.context.fill() + + def draw_rectangle(self, x0, y0, x1, y1, color): + cr = self.context + middle = (x0+x1)/2 + linear = cairo.LinearGradient(middle,y0,middle,y1) + linear.add_color_stop_rgba(0,3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1,*color[:4]) + cr.set_source(linear) + + cr.arc(x0+5, y0+5, 5, 0, 2*math.pi) + cr.arc(x1-5, y0+5, 5, 0, 2*math.pi) + cr.arc(x0+5, y1-5, 5, 0, 2*math.pi) + cr.arc(x1-5, y1-5, 5, 0, 2*math.pi) + cr.rectangle(x0+5,y0,x1-x0-10,y1-y0) + cr.rectangle(x0,y0+5,x1-x0,y1-y0-10) + cr.fill() + + def draw_shadow(self, x0, y0, x1, y1): + shadow = 0.4 + h_mid = (x0+x1)/2 + v_mid = (y0+y1)/2 + h_linear_1 = cairo.LinearGradient(h_mid,y0-4,h_mid,y0+4) + h_linear_2 = cairo.LinearGradient(h_mid,y1-4,h_mid,y1+4) + v_linear_1 = cairo.LinearGradient(x0-4,v_mid,x0+4,v_mid) + v_linear_2 = cairo.LinearGradient(x1-4,v_mid,x1+4,v_mid) + + h_linear_1.add_color_stop_rgba( 0, 0, 0, 0, 0) + h_linear_1.add_color_stop_rgba( 1, 0, 0, 0, shadow) + h_linear_2.add_color_stop_rgba( 0, 0, 0, 0, shadow) + h_linear_2.add_color_stop_rgba( 1, 0, 0, 0, 0) + v_linear_1.add_color_stop_rgba( 0, 0, 0, 0, 0) + v_linear_1.add_color_stop_rgba( 1, 0, 0, 0, shadow) + v_linear_2.add_color_stop_rgba( 0, 0, 0, 0, shadow) + v_linear_2.add_color_stop_rgba( 1, 0, 0, 0, 0) + + self.draw_rectangular_shadow(h_linear_1,x0+4,y0-4,x1-x0-8,8) + self.draw_rectangular_shadow(h_linear_2,x0+4,y1-4,x1-x0-8,8) + self.draw_rectangular_shadow(v_linear_1,x0-4,y0+4,8,y1-y0-8) + self.draw_rectangular_shadow(v_linear_2,x1-4,y0+4,8,y1-y0-8) + + self.draw_circular_shadow(x0+4, y0+4, 4, math.pi, 3*math.pi/2, (-1,0), shadow) + self.draw_circular_shadow(x1-4, y0+4, 4, 3*math.pi/2, 2*math.pi, (0,-1), shadow) + self.draw_circular_shadow(x0+4, y1-4, 4, math.pi/2, math.pi, (0,1), shadow) + self.draw_circular_shadow(x1-4, y1-4, 4, 0, math.pi/2, (1,0), shadow) + + def render_plot(self): + for index,group in enumerate(self.series): + for data in group: + self.render_rectangle(self.borders[HORZ] + data.content[0]*self.horizontal_step, + self.borders[VERT] + index*self.vertical_step + self.vertical_step/4.0, + self.borders[HORZ] + data.content[1]*self.horizontal_step, + self.borders[VERT] + index*self.vertical_step + 3.0*self.vertical_step/4.0, + self.series_colors[index]) + +# Function definition + +def scatter_plot(name, + data = None, + errorx = None, + errory = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + axis = False, + dash = False, + discrete = False, + dots = False, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + z_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + circle_colors = None): + + ''' + - Function to plot scatter data. + + - Parameters + + data - The values to be ploted might be passed in a two basic: + list of points: [(0,0), (0,1), (0,2)] or [(0,0,1), (0,1,4), (0,2,1)] + lists of coordinates: [ [0,0,0] , [0,1,2] ] or [ [0,0,0] , [0,1,2] , [1,4,1] ] + Notice that these kinds of that can be grouped in order to form more complex data + using lists of lists or dictionaries; + series_colors - Define color values for each of the series + circle_colors - Define a lower and an upper bound for the circle colors for variable radius + (3 dimensions) series + ''' + + plot = ScatterPlot( name, data, errorx, errory, width, height, background, border, + axis, dash, discrete, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, z_bounds, x_title, y_title, series_colors, circle_colors ) + plot.render() + plot.commit() + +def dot_line_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + axis = False, + dash = False, + dots = False, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None): + ''' + - Function to plot graphics using dots and lines. + + dot_line_plot (name, data, width, height, background = "white light_gray", border = 0, axis = False, grid = False, x_labels = None, y_labels = None, x_bounds = None, y_bounds = None) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + axis - Whether or not the axis are to be drawn; + dash - Boolean or a list or a dictionary of booleans indicating whether or not the associated series should be drawn in dashed mode; + dots - Whether or not dots should be drawn on each point; + grid - Whether or not the gris is to be drawn; + series_legend - Whether or not the legend is to be drawn; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + x_title - Whether or not to plot a title over the x axis. + y_title - Whether or not to plot a title over the y axis. + + - Examples of use + + data = [0, 1, 3, 8, 9, 0, 10, 10, 2, 1] + CairoPlot.dot_line_plot('teste', data, 400, 300) + + data = { "john" : [10, 10, 10, 10, 30], "mary" : [0, 0, 3, 5, 15], "philip" : [13, 32, 11, 25, 2] } + x_labels = ["jan/2008", "feb/2008", "mar/2008", "apr/2008", "may/2008" ] + CairoPlot.dot_line_plot( 'test', data, 400, 300, axis = True, grid = True, + series_legend = True, x_labels = x_labels ) + ''' + plot = DotLinePlot( name, data, width, height, background, border, + axis, dash, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, x_title, y_title, series_colors ) + plot.render() + plot.commit() + +def function_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + axis = True, + dots = False, + discrete = False, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + step = 1): + + ''' + - Function to plot functions. + + function_plot(name, data, width, height, background = "white light_gray", border = 0, axis = True, grid = False, dots = False, x_labels = None, y_labels = None, x_bounds = None, y_bounds = None, step = 1, discrete = False) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + axis - Whether or not the axis are to be drawn; + grid - Whether or not the gris is to be drawn; + dots - Whether or not dots should be shown at each point; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + step - the horizontal distance from one point to the other. The smaller, the smoother the curve will be; + discrete - whether or not the function should be plotted in discrete format. + + - Example of use + + data = lambda x : x**2 + CairoPlot.function_plot('function4', data, 400, 300, grid = True, x_bounds=(-10,10), step = 0.1) + ''' + + plot = FunctionPlot( name, data, width, height, background, border, + axis, discrete, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, x_title, y_title, series_colors, step ) + plot.render() + plot.commit() + +def pie_plot( name, data, width, height, background = "white light_gray", gradient = False, shadow = False, colors = None ): + + ''' + - Function to plot pie graphics. + + pie_plot(name, data, width, height, background = "white light_gray", gradient = False, colors = None) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + gradient - Whether or not the pie color will be painted with a gradient; + shadow - Whether or not there will be a shadow behind the pie; + colors - List of slices colors. + + - Example of use + + teste_data = {"john" : 123, "mary" : 489, "philip" : 890 , "suzy" : 235} + CairoPlot.pie_plot("pie_teste", teste_data, 500, 500) + ''' + + plot = PiePlot( name, data, width, height, background, gradient, shadow, colors ) + plot.render() + plot.commit() + +def donut_plot(name, data, width, height, background = "white light_gray", gradient = False, shadow = False, colors = None, inner_radius = -1): + + ''' + - Function to plot donut graphics. + + donut_plot(name, data, width, height, background = "white light_gray", gradient = False, inner_radius = -1) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + shadow - Whether or not there will be a shadow behind the donut; + gradient - Whether or not the donut color will be painted with a gradient; + colors - List of slices colors; + inner_radius - The radius of the donut's inner circle. + + - Example of use + + teste_data = {"john" : 123, "mary" : 489, "philip" : 890 , "suzy" : 235} + CairoPlot.donut_plot("donut_teste", teste_data, 500, 500) + ''' + + plot = DonutPlot(name, data, width, height, background, gradient, shadow, colors, inner_radius) + plot.render() + plot.commit() + +def gantt_chart(name, pieces, width, height, x_labels, y_labels, colors): + + ''' + - Function to generate Gantt Charts. + + gantt_chart(name, pieces, width, height, x_labels, y_labels, colors): + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + pieces - A list defining the spaces to be drawn. The user must pass, for each line, the index of its start and the index of its end. If a line must have two or more spaces, they must be passed inside a list; + width, height - Dimensions of the output image; + x_labels - A list of names for each of the vertical lines; + y_labels - A list of names for each of the horizontal spaces; + colors - List containing the colors expected for each of the horizontal spaces + + - Example of use + + pieces = [ (0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,8)] + x_labels = [ 'teste01', 'teste02', 'teste03', 'teste04'] + y_labels = [ '0001', '0002', '0003', '0004', '0005', '0006', '0007', '0008', '0009', '0010' ] + colors = [ (1.0, 0.0, 0.0), (1.0, 0.7, 0.0), (1.0, 1.0, 0.0), (0.0, 1.0, 0.0) ] + CairoPlot.gantt_chart('gantt_teste', pieces, 600, 300, x_labels, y_labels, colors) + ''' + + plot = GanttChart(name, pieces, width, height, x_labels, y_labels, colors) + plot.render() + plot.commit() + +def vertical_bar_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + colors = None): + #TODO: Fix docstring for vertical_bar_plot + ''' + - Function to generate vertical Bar Plot Charts. + + bar_plot(name, data, width, height, background, border, grid, rounded_corners, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, colors): + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtime; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + grid - Whether or not the gris is to be drawn; + rounded_corners - Whether or not the bars should have rounded corners; + three_dimension - Whether or not the bars should be drawn in pseudo 3D; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + colors - List containing the colors expected for each of the bars. + + - Example of use + + data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] + CairoPlot.vertical_bar_plot ('bar2', data, 400, 300, border = 20, grid = True, rounded_corners = False) + ''' + + plot = VerticalBarPlot(name, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + series_labels, x_labels, y_labels, x_bounds, y_bounds, colors) + plot.render() + plot.commit() + +def horizontal_bar_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + colors = None): + + #TODO: Fix docstring for horizontal_bar_plot + ''' + - Function to generate Horizontal Bar Plot Charts. + + bar_plot(name, data, width, height, background, border, grid, rounded_corners, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, colors): + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtime; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + grid - Whether or not the gris is to be drawn; + rounded_corners - Whether or not the bars should have rounded corners; + three_dimension - Whether or not the bars should be drawn in pseudo 3D; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + colors - List containing the colors expected for each of the bars. + + - Example of use + + data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] + CairoPlot.bar_plot ('bar2', data, 400, 300, border = 20, grid = True, rounded_corners = False) + ''' + + plot = HorizontalBarPlot(name, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + series_labels, x_labels, y_labels, x_bounds, y_bounds, colors) + plot.render() + plot.commit() + +def stream_chart(name, + data, + width, + height, + background = "white light_gray", + border = 0, + grid = False, + series_legend = None, + x_labels = None, + x_bounds = None, + y_bounds = None, + colors = None): + + #TODO: Fix docstring for horizontal_bar_plot + plot = StreamChart(name, data, width, height, background, border, + grid, series_legend, x_labels, x_bounds, y_bounds, colors) + plot.render() + plot.commit() + + +if __name__ == "__main__": + import tests + import seriestests diff --git a/bindings/python/examples/output_format_modules/pprint_table.py b/bindings/python/examples/output_format_modules/pprint_table.py new file mode 100644 index 0000000..1cd8620 --- /dev/null +++ b/bindings/python/examples/output_format_modules/pprint_table.py @@ -0,0 +1,35 @@ +# This module is used to pretty-print a table +# Adapted from +# http://ginstrom.com/scribbles/2007/09/04/pretty-printing-a-table-in-python/ + +import sys + +def get_max_width(table, index): + """Get the maximum width of the given column index""" + + return max([len(str(row[index])) for row in table]) + + +def pprint_table(table, nbLeft=1, out=sys.stdout): + """ + Prints out a table of data, padded for alignment + @param table: The table to print. A list of lists. + Each row must have the same number of columns. + @param nbLeft: The number of columns aligned left + @param out: Output stream (file-like object) + """ + + col_paddings = [] + + for i in range(len(table[0])): + col_paddings.append(get_max_width(table, i)) + + for row in table: + # left cols + for i in range(nbLeft): + print >> out, str(row[i]).ljust(col_paddings[i] + 1), + # rest of the cols + for i in range(nbLeft, len(row)): + col = str(row[i]).rjust(col_paddings[i] + 2) + print >> out, col, + print >> out diff --git a/bindings/python/examples/output_format_modules/series.py b/bindings/python/examples/output_format_modules/series.py new file mode 100755 index 0000000..8e8b236 --- /dev/null +++ b/bindings/python/examples/output_format_modules/series.py @@ -0,0 +1,1140 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# Serie.py +# +# Copyright (c) 2008 Magnun Leno da Silva +# +# Author: Magnun Leno da Silva +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public License +# as published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 +# USA + +# Contributor: Rodrigo Moreiro Araujo + +#import cairoplot +import doctest + +NUMTYPES = (int, float, long) +LISTTYPES = (list, tuple) +STRTYPES = (str, unicode) +FILLING_TYPES = ['linear', 'solid', 'gradient'] +DEFAULT_COLOR_FILLING = 'solid' +#TODO: Define default color list +DEFAULT_COLOR_LIST = None + +class Data(object): + ''' + Class that models the main data structure. + It can hold: + - a number type (int, float or long) + - a tuple, witch represents a point and can have 2 or 3 items (x,y,z) + - if a list is passed it will be converted to a tuple. + + obs: In case a tuple is passed it will convert to tuple + ''' + def __init__(self, data=None, name=None, parent=None): + ''' + Starts main atributes from the Data class + @name - Name for each point; + @content - The real data, can be an int, float, long or tuple, which + represents a point (x,y) or (x,y,z); + @parent - A pointer that give the data access to it's parent. + + Usage: + >>> d = Data(name='empty'); print d + empty: () + >>> d = Data((1,1),'point a'); print d + point a: (1, 1) + >>> d = Data((1,2,3),'point b'); print d + point b: (1, 2, 3) + >>> d = Data([2,3],'point c'); print d + point c: (2, 3) + >>> d = Data(12, 'simple value'); print d + simple value: 12 + ''' + # Initial values + self.__content = None + self.__name = None + + # Setting passed values + self.parent = parent + self.name = name + self.content = data + + # Name property + @apply + def name(): + doc = ''' + Name is a read/write property that controls the input of name. + - If passed an invalid value it cleans the name with None + + Usage: + >>> d = Data(13); d.name = 'name_test'; print d + name_test: 13 + >>> d.name = 11; print d + 13 + >>> d.name = 'other_name'; print d + other_name: 13 + >>> d.name = None; print d + 13 + >>> d.name = 'last_name'; print d + last_name: 13 + >>> d.name = ''; print d + 13 + ''' + def fget(self): + ''' + returns the name as a string + ''' + return self.__name + + def fset(self, name): + ''' + Sets the name of the Data + ''' + if type(name) in STRTYPES and len(name) > 0: + self.__name = name + else: + self.__name = None + + + + return property(**locals()) + + # Content property + @apply + def content(): + doc = ''' + Content is a read/write property that validate the data passed + and return it. + + Usage: + >>> d = Data(); d.content = 13; d.content + 13 + >>> d = Data(); d.content = (1,2); d.content + (1, 2) + >>> d = Data(); d.content = (1,2,3); d.content + (1, 2, 3) + >>> d = Data(); d.content = [1,2,3]; d.content + (1, 2, 3) + >>> d = Data(); d.content = [1.5,.2,3.3]; d.content + (1.5, 0.20000000000000001, 3.2999999999999998) + ''' + def fget(self): + ''' + Return the content of Data + ''' + return self.__content + + def fset(self, data): + ''' + Ensures that data is a valid tuple/list or a number (int, float + or long) + ''' + # Type: None + if data is None: + self.__content = None + return + + # Type: Int or Float + elif type(data) in NUMTYPES: + self.__content = data + + # Type: List or Tuple + elif type(data) in LISTTYPES: + # Ensures the correct size + if len(data) not in (2, 3): + raise TypeError, "Data (as list/tuple) must have 2 or 3 items" + return + + # Ensures that all items in list/tuple is a number + isnum = lambda x : type(x) not in NUMTYPES + + if max(map(isnum, data)): + # An item in data isn't an int or a float + raise TypeError, "All content of data must be a number (int or float)" + + # Convert the tuple to list + if type(data) is list: + data = tuple(data) + + # Append a copy and sets the type + self.__content = data[:] + + # Unknown type! + else: + self.__content = None + raise TypeError, "Data must be an int, float or a tuple with two or three items" + return + + return property(**locals()) + + + def clear(self): + ''' + Clear the all Data (content, name and parent) + ''' + self.content = None + self.name = None + self.parent = None + + def copy(self): + ''' + Returns a copy of the Data structure + ''' + # The copy + new_data = Data() + if self.content is not None: + # If content is a point + if type(self.content) is tuple: + new_data.__content = self.content[:] + + # If content is a number + else: + new_data.__content = self.content + + # If it has a name + if self.name is not None: + new_data.__name = self.name + + return new_data + + def __str__(self): + ''' + Return a string representation of the Data structure + ''' + if self.name is None: + if self.content is None: + return '' + return str(self.content) + else: + if self.content is None: + return self.name+": ()" + return self.name+": "+str(self.content) + + def __len__(self): + ''' + Return the length of the Data. + - If it's a number return 1; + - If it's a list return it's length; + - If its None return 0. + ''' + if self.content is None: + return 0 + elif type(self.content) in NUMTYPES: + return 1 + return len(self.content) + + + + +class Group(object): + ''' + Class that models a group of data. Every value (int, float, long, tuple + or list) passed is converted to a list of Data. + It can receive: + - A single number (int, float, long); + - A list of numbers; + - A tuple of numbers; + - An instance of Data; + - A list of Data; + + Obs: If a tuple with 2 or 3 items is passed it is converted to a point. + If a tuple with only 1 item is passed it's converted to a number; + If a tuple with more than 2 items is passed it's converted to a + list of numbers + ''' + def __init__(self, group=None, name=None, parent=None): + ''' + Starts main atributes in Group instance. + @data_list - a list of data which forms the group; + @range - a range that represent the x axis of possible functions; + @name - name of the data group; + @parent - the Serie parent of this group. + + Usage: + >>> g = Group(13, 'simple number'); print g + simple number ['13'] + >>> g = Group((1,2), 'simple point'); print g + simple point ['(1, 2)'] + >>> g = Group([1,2,3,4], 'list of numbers'); print g + list of numbers ['1', '2', '3', '4'] + >>> g = Group((1,2,3,4),'int in tuple'); print g + int in tuple ['1', '2', '3', '4'] + >>> g = Group([(1,2),(2,3),(3,4)], 'list of points'); print g + list of points ['(1, 2)', '(2, 3)', '(3, 4)'] + >>> g = Group([[1,2,3],[1,2,3]], '2D coordinate lists'); print g + 2D coordinated lists ['(1, 1)', '(2, 2)', '(3, 3)'] + >>> g = Group([[1,2],[1,2],[1,2]], '3D coordinate lists'); print g + 3D coordinated lists ['(1, 1, 1)', '(2, 2, 2)'] + ''' + # Initial values + self.__data_list = [] + self.__range = [] + self.__name = None + + + self.parent = parent + self.name = name + self.data_list = group + + # Name property + @apply + def name(): + doc = ''' + Name is a read/write property that controls the input of name. + - If passed an invalid value it cleans the name with None + + Usage: + >>> g = Group(13); g.name = 'name_test'; print g + name_test ['13'] + >>> g.name = 11; print g + ['13'] + >>> g.name = 'other_name'; print g + other_name ['13'] + >>> g.name = None; print g + ['13'] + >>> g.name = 'last_name'; print g + last_name ['13'] + >>> g.name = ''; print g + ['13'] + ''' + def fget(self): + ''' + Returns the name as a string + ''' + return self.__name + + def fset(self, name): + ''' + Sets the name of the Group + ''' + if type(name) in STRTYPES and len(name) > 0: + self.__name = name + else: + self.__name = None + + return property(**locals()) + + # data_list property + @apply + def data_list(): + doc = ''' + The data_list is a read/write property that can be a list of + numbers, a list of points or a list of 2 or 3 coordinate lists. This + property uses mainly the self.add_data method. + + Usage: + >>> g = Group(); g.data_list = 13; print g + ['13'] + >>> g.data_list = (1,2); print g + ['(1, 2)'] + >>> g.data_list = Data((1,2),'point a'); print g + ['point a: (1, 2)'] + >>> g.data_list = [1,2,3]; print g + ['1', '2', '3'] + >>> g.data_list = (1,2,3,4); print g + ['1', '2', '3', '4'] + >>> g.data_list = [(1,2),(2,3),(3,4)]; print g + ['(1, 2)', '(2, 3)', '(3, 4)'] + >>> g.data_list = [[1,2],[1,2]]; print g + ['(1, 1)', '(2, 2)'] + >>> g.data_list = [[1,2],[1,2],[1,2]]; print g + ['(1, 1, 1)', '(2, 2, 2)'] + >>> g.range = (10); g.data_list = lambda x:x**2; print g + ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)'] + ''' + def fget(self): + ''' + Returns the value of data_list + ''' + return self.__data_list + + def fset(self, group): + ''' + Ensures that group is valid. + ''' + # None + if group is None: + self.__data_list = [] + + # Int/float/long or Instance of Data + elif type(group) in NUMTYPES or isinstance(group, Data): + # Clean data_list + self.__data_list = [] + self.add_data(group) + + # One point + elif type(group) is tuple and len(group) in (2,3): + self.__data_list = [] + self.add_data(group) + + # list of items + elif type(group) in LISTTYPES and type(group[0]) is not list: + # Clean data_list + self.__data_list = [] + for item in group: + # try to append and catch an exception + self.add_data(item) + + # function lambda + elif callable(group): + # Explicit is better than implicit + function = group + # Has range + if len(self.range) is not 0: + # Clean data_list + self.__data_list = [] + # Generate values for the lambda function + for x in self.range: + #self.add_data((x,round(group(x),2))) + self.add_data((x,function(x))) + + # Only have range in parent + elif self.parent is not None and len(self.parent.range) is not 0: + # Copy parent range + self.__range = self.parent.range[:] + # Clean data_list + self.__data_list = [] + # Generate values for the lambda function + for x in self.range: + #self.add_data((x,round(group(x),2))) + self.add_data((x,function(x))) + + # Don't have range anywhere + else: + # x_data don't exist + raise Exception, "Data argument is valid but to use function type please set x_range first" + + # Coordinate Lists + elif type(group) in LISTTYPES and type(group[0]) is list: + # Clean data_list + self.__data_list = [] + data = [] + if len(group) == 3: + data = zip(group[0], group[1], group[2]) + elif len(group) == 2: + data = zip(group[0], group[1]) + else: + raise TypeError, "Only one list of coordinates was received." + + for item in data: + self.add_data(item) + + else: + raise TypeError, "Group type not supported" + + return property(**locals()) + + @apply + def range(): + doc = ''' + The range is a read/write property that generates a range of values + for the x axis of the functions. When passed a tuple it almost works + like the built-in range funtion: + - 1 item, represent the end of the range started from 0; + - 2 items, represents the start and the end, respectively; + - 3 items, the last one represents the step; + + When passed a list the range function understands as a valid range. + + Usage: + >>> g = Group(); g.range = 10; print g.range + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0] + >>> g = Group(); g.range = (5); print g.range + [0.0, 1.0, 2.0, 3.0, 4.0] + >>> g = Group(); g.range = (1,7); print g.range + [1.0, 2.0, 3.0, 4.0, 5.0, 6.0] + >>> g = Group(); g.range = (0,10,2); print g.range + [0.0, 2.0, 4.0, 6.0, 8.0] + >>> + >>> g = Group(); g.range = [0]; print g.range + [0.0] + >>> g = Group(); g.range = [0,10,20]; print g.range + [0.0, 10.0, 20.0] + ''' + def fget(self): + ''' + Returns the range + ''' + return self.__range + + def fset(self, x_range): + ''' + Controls the input of a valid type and generate the range + ''' + # if passed a simple number convert to tuple + if type(x_range) in NUMTYPES: + x_range = (x_range,) + + # A list, just convert to float + if type(x_range) is list and len(x_range) > 0: + # Convert all to float + x_range = map(float, x_range) + # Prevents repeated values and convert back to list + self.__range = list(set(x_range[:])) + # Sort the list to ascending order + self.__range.sort() + + # A tuple, must check the lengths and generate the values + elif type(x_range) is tuple and len(x_range) in (1,2,3): + # Convert all to float + x_range = map(float, x_range) + + # Inital values + start = 0.0 + step = 1.0 + end = 0.0 + + # Only the end and it can't be less or iqual to 0 + if len(x_range) is 1 and x_range > 0: + end = x_range[0] + + # The start and the end but the start must be less then the end + elif len(x_range) is 2 and x_range[0] < x_range[1]: + start = x_range[0] + end = x_range[1] + + # All 3, but the start must be less then the end + elif x_range[0] <= x_range[1]: + start = x_range[0] + end = x_range[1] + step = x_range[2] + + # Starts the range + self.__range = [] + # Generate the range + # Can't use the range function because it doesn't support float values + while start < end: + self.__range.append(start) + start += step + + # Incorrect type + else: + raise Exception, "x_range must be a list with one or more items or a tuple with 2 or 3 items" + + return property(**locals()) + + def add_data(self, data, name=None): + ''' + Append a new data to the data_list. + - If data is an instance of Data, append it + - If it's an int, float, tuple or list create an instance of Data and append it + + Usage: + >>> g = Group() + >>> g.add_data(12); print g + ['12'] + >>> g.add_data(7,'other'); print g + ['12', 'other: 7'] + >>> + >>> g = Group() + >>> g.add_data((1,1),'a'); print g + ['a: (1, 1)'] + >>> g.add_data((2,2),'b'); print g + ['a: (1, 1)', 'b: (2, 2)'] + >>> + >>> g.add_data(Data((1,2),'c')); print g + ['a: (1, 1)', 'b: (2, 2)', 'c: (1, 2)'] + ''' + if not isinstance(data, Data): + # Try to convert + data = Data(data,name,self) + + if data.content is not None: + self.__data_list.append(data.copy()) + self.__data_list[-1].parent = self + + + def to_list(self): + ''' + Returns the group as a list of numbers (int, float or long) or a + list of tuples (points 2D or 3D). + + Usage: + >>> g = Group([1,2,3,4],'g1'); g.to_list() + [1, 2, 3, 4] + >>> g = Group([(1,2),(2,3),(3,4)],'g2'); g.to_list() + [(1, 2), (2, 3), (3, 4)] + >>> g = Group([(1,2,3),(3,4,5)],'g2'); g.to_list() + [(1, 2, 3), (3, 4, 5)] + ''' + return [data.content for data in self] + + def copy(self): + ''' + Returns a copy of this group + ''' + new_group = Group() + new_group.__name = self.__name + if self.__range is not None: + new_group.__range = self.__range[:] + for data in self: + new_group.add_data(data.copy()) + return new_group + + def get_names(self): + ''' + Return a list with the names of all data in this group + ''' + names = [] + for data in self: + if data.name is None: + names.append('Data '+str(data.index()+1)) + else: + names.append(data.name) + return names + + + def __str__ (self): + ''' + Returns a string representing the Group + ''' + ret = "" + if self.name is not None: + ret += self.name + " " + if len(self) > 0: + list_str = [str(item) for item in self] + ret += str(list_str) + else: + ret += "[]" + return ret + + def __getitem__(self, key): + ''' + Makes a Group iterable, based in the data_list property + ''' + return self.data_list[key] + + def __len__(self): + ''' + Returns the length of the Group, based in the data_list property + ''' + return len(self.data_list) + + +class Colors(object): + ''' + Class that models the colors its labels (names) and its properties, RGB + and filling type. + + It can receive: + - A list where each item is a list with 3 or 4 items. The + first 3 items represent the RGB values and the last argument + defines the filling type. The list will be converted to a dict + and each color will receve a name based in its position in the + list. + - A dictionary where each key will be the color name and its item + can be a list with 3 or 4 items. The first 3 items represent + the RGB colors and the last argument defines the filling type. + ''' + def __init__(self, color_list=None): + ''' + Start the color_list property + @ color_list - the list or dict contaning the colors properties. + ''' + self.__color_list = None + + self.color_list = color_list + + @apply + def color_list(): + doc = ''' + >>> c = Colors([[1,1,1],[2,2,2,'linear'],[3,3,3,'gradient']]) + >>> print c.color_list + {'Color 2': [2, 2, 2, 'linear'], 'Color 3': [3, 3, 3, 'gradient'], 'Color 1': [1, 1, 1, 'solid']} + >>> c.color_list = [[1,1,1],(2,2,2,'solid'),(3,3,3,'linear')] + >>> print c.color_list + {'Color 2': [2, 2, 2, 'solid'], 'Color 3': [3, 3, 3, 'linear'], 'Color 1': [1, 1, 1, 'solid']} + >>> c.color_list = {'a':[1,1,1],'b':(2,2,2,'solid'),'c':(3,3,3,'linear'), 'd':(4,4,4)} + >>> print c.color_list + {'a': [1, 1, 1, 'solid'], 'c': [3, 3, 3, 'linear'], 'b': [2, 2, 2, 'solid'], 'd': [4, 4, 4, 'solid']} + ''' + def fget(self): + ''' + Return the color list + ''' + return self.__color_list + + def fset(self, color_list): + ''' + Format the color list to a dictionary + ''' + if color_list is None: + self.__color_list = None + return + + if type(color_list) in LISTTYPES and type(color_list[0]) in LISTTYPES: + old_color_list = color_list[:] + color_list = {} + for index, color in enumerate(old_color_list): + if len(color) is 3 and max(map(type, color)) in NUMTYPES: + color_list['Color '+str(index+1)] = list(color)+[DEFAULT_COLOR_FILLING] + elif len(color) is 4 and max(map(type, color[:-1])) in NUMTYPES and color[-1] in FILLING_TYPES: + color_list['Color '+str(index+1)] = list(color) + else: + raise TypeError, "Unsuported color format" + elif type(color_list) is not dict: + raise TypeError, "Unsuported color format" + + for name, color in color_list.items(): + if len(color) is 3: + if max(map(type, color)) in NUMTYPES: + color_list[name] = list(color)+[DEFAULT_COLOR_FILLING] + else: + raise TypeError, "Unsuported color format" + elif len(color) is 4: + if max(map(type, color[:-1])) in NUMTYPES and color[-1] in FILLING_TYPES: + color_list[name] = list(color) + else: + raise TypeError, "Unsuported color format" + self.__color_list = color_list.copy() + + return property(**locals()) + + +class Series(object): + ''' + Class that models a Series (group of groups). Every value (int, float, + long, tuple or list) passed is converted to a list of Group or Data. + It can receive: + - a single number or point, will be converted to a Group of one Data; + - a list of numbers, will be converted to a group of numbers; + - a list of tuples, will converted to a single Group of points; + - a list of lists of numbers, each 'sublist' will be converted to a + group of numbers; + - a list of lists of tuples, each 'sublist' will be converted to a + group of points; + - a list of lists of lists, the content of the 'sublist' will be + processed as coordinated lists and the result will be converted to + a group of points; + - a Dictionary where each item can be the same of the list: number, + point, list of numbers, list of points or list of lists (coordinated + lists); + - an instance of Data; + - an instance of group. + ''' + def __init__(self, series=None, name=None, property=[], colors=None): + ''' + Starts main atributes in Group instance. + @series - a list, dict of data of which the series is composed; + @name - name of the series; + @property - a list/dict of properties to be used in the plots of + this Series + + Usage: + >>> print Series([1,2,3,4]) + ["Group 1 ['1', '2', '3', '4']"] + >>> print Series([[1,2,3],[4,5,6]]) + ["Group 1 ['1', '2', '3']", "Group 2 ['4', '5', '6']"] + >>> print Series((1,2)) + ["Group 1 ['(1, 2)']"] + >>> print Series([(1,2),(2,3)]) + ["Group 1 ['(1, 2)', '(2, 3)']"] + >>> print Series([[(1,2),(2,3)],[(4,5),(5,6)]]) + ["Group 1 ['(1, 2)', '(2, 3)']", "Group 2 ['(4, 5)', '(5, 6)']"] + >>> print Series([[[1,2,3],[1,2,3],[1,2,3]]]) + ["Group 1 ['(1, 1, 1)', '(2, 2, 2)', '(3, 3, 3)']"] + >>> print Series({'g1':[1,2,3], 'g2':[4,5,6]}) + ["g1 ['1', '2', '3']", "g2 ['4', '5', '6']"] + >>> print Series({'g1':[(1,2),(2,3)], 'g2':[(4,5),(5,6)]}) + ["g1 ['(1, 2)', '(2, 3)']", "g2 ['(4, 5)', '(5, 6)']"] + >>> print Series({'g1':[[1,2],[1,2]], 'g2':[[4,5],[4,5]]}) + ["g1 ['(1, 1)', '(2, 2)']", "g2 ['(4, 4)', '(5, 5)']"] + >>> print Series(Data(1,'d1')) + ["Group 1 ['d1: 1']"] + >>> print Series(Group([(1,2),(2,3)],'g1')) + ["g1 ['(1, 2)', '(2, 3)']"] + ''' + # Intial values + self.__group_list = [] + self.__name = None + self.__range = None + + # TODO: Implement colors with filling + self.__colors = None + + self.name = name + self.group_list = series + self.colors = colors + + # Name property + @apply + def name(): + doc = ''' + Name is a read/write property that controls the input of name. + - If passed an invalid value it cleans the name with None + + Usage: + >>> s = Series(13); s.name = 'name_test'; print s + name_test ["Group 1 ['13']"] + >>> s.name = 11; print s + ["Group 1 ['13']"] + >>> s.name = 'other_name'; print s + other_name ["Group 1 ['13']"] + >>> s.name = None; print s + ["Group 1 ['13']"] + >>> s.name = 'last_name'; print s + last_name ["Group 1 ['13']"] + >>> s.name = ''; print s + ["Group 1 ['13']"] + ''' + def fget(self): + ''' + Returns the name as a string + ''' + return self.__name + + def fset(self, name): + ''' + Sets the name of the Group + ''' + if type(name) in STRTYPES and len(name) > 0: + self.__name = name + else: + self.__name = None + + return property(**locals()) + + + + # Colors property + @apply + def colors(): + doc = ''' + >>> s = Series() + >>> s.colors = [[1,1,1],[2,2,2,'linear'],[3,3,3,'gradient']] + >>> print s.colors + {'Color 2': [2, 2, 2, 'linear'], 'Color 3': [3, 3, 3, 'gradient'], 'Color 1': [1, 1, 1, 'solid']} + >>> s.colors = [[1,1,1],(2,2,2,'solid'),(3,3,3,'linear')] + >>> print s.colors + {'Color 2': [2, 2, 2, 'solid'], 'Color 3': [3, 3, 3, 'linear'], 'Color 1': [1, 1, 1, 'solid']} + >>> s.colors = {'a':[1,1,1],'b':(2,2,2,'solid'),'c':(3,3,3,'linear'), 'd':(4,4,4)} + >>> print s.colors + {'a': [1, 1, 1, 'solid'], 'c': [3, 3, 3, 'linear'], 'b': [2, 2, 2, 'solid'], 'd': [4, 4, 4, 'solid']} + ''' + def fget(self): + ''' + Return the color list + ''' + return self.__colors.color_list + + def fset(self, colors): + ''' + Format the color list to a dictionary + ''' + self.__colors = Colors(colors) + + return property(**locals()) + + @apply + def range(): + doc = ''' + The range is a read/write property that generates a range of values + for the x axis of the functions. When passed a tuple it almost works + like the built-in range funtion: + - 1 item, represent the end of the range started from 0; + - 2 items, represents the start and the end, respectively; + - 3 items, the last one represents the step; + + When passed a list the range function understands as a valid range. + + Usage: + >>> s = Series(); s.range = 10; print s.range + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0] + >>> s = Series(); s.range = (5); print s.range + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0] + >>> s = Series(); s.range = (1,7); print s.range + [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0] + >>> s = Series(); s.range = (0,10,2); print s.range + [0.0, 2.0, 4.0, 6.0, 8.0, 10.0] + >>> + >>> s = Series(); s.range = [0]; print s.range + [0.0] + >>> s = Series(); s.range = [0,10,20]; print s.range + [0.0, 10.0, 20.0] + ''' + def fget(self): + ''' + Returns the range + ''' + return self.__range + + def fset(self, x_range): + ''' + Controls the input of a valid type and generate the range + ''' + # if passed a simple number convert to tuple + if type(x_range) in NUMTYPES: + x_range = (x_range,) + + # A list, just convert to float + if type(x_range) is list and len(x_range) > 0: + # Convert all to float + x_range = map(float, x_range) + # Prevents repeated values and convert back to list + self.__range = list(set(x_range[:])) + # Sort the list to ascending order + self.__range.sort() + + # A tuple, must check the lengths and generate the values + elif type(x_range) is tuple and len(x_range) in (1,2,3): + # Convert all to float + x_range = map(float, x_range) + + # Inital values + start = 0.0 + step = 1.0 + end = 0.0 + + # Only the end and it can't be less or iqual to 0 + if len(x_range) is 1 and x_range > 0: + end = x_range[0] + + # The start and the end but the start must be lesser then the end + elif len(x_range) is 2 and x_range[0] < x_range[1]: + start = x_range[0] + end = x_range[1] + + # All 3, but the start must be lesser then the end + elif x_range[0] < x_range[1]: + start = x_range[0] + end = x_range[1] + step = x_range[2] + + # Starts the range + self.__range = [] + # Generate the range + # Cnat use the range function becouse it don't suport float values + while start <= end: + self.__range.append(start) + start += step + + # Incorrect type + else: + raise Exception, "x_range must be a list with one or more item or a tuple with 2 or 3 items" + + return property(**locals()) + + @apply + def group_list(): + doc = ''' + The group_list is a read/write property used to pre-process the list + of Groups. + It can be: + - a single number, point or lambda, will be converted to a single + Group of one Data; + - a list of numbers, will be converted to a group of numbers; + - a list of tuples, will converted to a single Group of points; + - a list of lists of numbers, each 'sublist' will be converted to + a group of numbers; + - a list of lists of tuples, each 'sublist' will be converted to a + group of points; + - a list of lists of lists, the content of the 'sublist' will be + processed as coordinated lists and the result will be converted + to a group of points; + - a list of lambdas, each lambda represents a Group; + - a Dictionary where each item can be the same of the list: number, + point, list of numbers, list of points, list of lists + (coordinated lists) or lambdas + - an instance of Data; + - an instance of group. + + Usage: + >>> s = Series() + >>> s.group_list = [1,2,3,4]; print s + ["Group 1 ['1', '2', '3', '4']"] + >>> s.group_list = [[1,2,3],[4,5,6]]; print s + ["Group 1 ['1', '2', '3']", "Group 2 ['4', '5', '6']"] + >>> s.group_list = (1,2); print s + ["Group 1 ['(1, 2)']"] + >>> s.group_list = [(1,2),(2,3)]; print s + ["Group 1 ['(1, 2)', '(2, 3)']"] + >>> s.group_list = [[(1,2),(2,3)],[(4,5),(5,6)]]; print s + ["Group 1 ['(1, 2)', '(2, 3)']", "Group 2 ['(4, 5)', '(5, 6)']"] + >>> s.group_list = [[[1,2,3],[1,2,3],[1,2,3]]]; print s + ["Group 1 ['(1, 1, 1)', '(2, 2, 2)', '(3, 3, 3)']"] + >>> s.group_list = [(0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,9)]; print s + ["Group 1 ['(0.5, 5.5)']", "Group 2 ['(0, 4)', '(6, 8)']", "Group 3 ['(5.5, 7)']", "Group 4 ['(7, 9)']"] + >>> s.group_list = {'g1':[1,2,3], 'g2':[4,5,6]}; print s + ["g1 ['1', '2', '3']", "g2 ['4', '5', '6']"] + >>> s.group_list = {'g1':[(1,2),(2,3)], 'g2':[(4,5),(5,6)]}; print s + ["g1 ['(1, 2)', '(2, 3)']", "g2 ['(4, 5)', '(5, 6)']"] + >>> s.group_list = {'g1':[[1,2],[1,2]], 'g2':[[4,5],[4,5]]}; print s + ["g1 ['(1, 1)', '(2, 2)']", "g2 ['(4, 4)', '(5, 5)']"] + >>> s.range = 10 + >>> s.group_list = lambda x:x*2 + >>> s.group_list = [lambda x:x*2, lambda x:x**2, lambda x:x**3]; print s + ["Group 1 ['(0.0, 0.0)', '(1.0, 2.0)', '(2.0, 4.0)', '(3.0, 6.0)', '(4.0, 8.0)', '(5.0, 10.0)', '(6.0, 12.0)', '(7.0, 14.0)', '(8.0, 16.0)', '(9.0, 18.0)', '(10.0, 20.0)']", "Group 2 ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)', '(10.0, 100.0)']", "Group 3 ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 8.0)', '(3.0, 27.0)', '(4.0, 64.0)', '(5.0, 125.0)', '(6.0, 216.0)', '(7.0, 343.0)', '(8.0, 512.0)', '(9.0, 729.0)', '(10.0, 1000.0)']"] + >>> s.group_list = {'linear':lambda x:x*2, 'square':lambda x:x**2, 'cubic':lambda x:x**3}; print s + ["cubic ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 8.0)', '(3.0, 27.0)', '(4.0, 64.0)', '(5.0, 125.0)', '(6.0, 216.0)', '(7.0, 343.0)', '(8.0, 512.0)', '(9.0, 729.0)', '(10.0, 1000.0)']", "linear ['(0.0, 0.0)', '(1.0, 2.0)', '(2.0, 4.0)', '(3.0, 6.0)', '(4.0, 8.0)', '(5.0, 10.0)', '(6.0, 12.0)', '(7.0, 14.0)', '(8.0, 16.0)', '(9.0, 18.0)', '(10.0, 20.0)']", "square ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)', '(10.0, 100.0)']"] + >>> s.group_list = Data(1,'d1'); print s + ["Group 1 ['d1: 1']"] + >>> s.group_list = Group([(1,2),(2,3)],'g1'); print s + ["g1 ['(1, 2)', '(2, 3)']"] + ''' + def fget(self): + ''' + Return the group list. + ''' + return self.__group_list + + def fset(self, series): + ''' + Controls the input of a valid group list. + ''' + #TODO: Add support to the following strem of data: [ (0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,9)] + + # Type: None + if series is None: + self.__group_list = [] + + # List or Tuple + elif type(series) in LISTTYPES: + self.__group_list = [] + + is_function = lambda x: callable(x) + # Groups + if list in map(type, series) or max(map(is_function, series)): + for group in series: + self.add_group(group) + + # single group + else: + self.add_group(series) + + #old code + ## List of numbers + #if type(series[0]) in NUMTYPES or type(series[0]) is tuple: + # print series + # self.add_group(series) + # + ## List of anything else + #else: + # for group in series: + # self.add_group(group) + + # Dict representing series of groups + elif type(series) is dict: + self.__group_list = [] + names = series.keys() + names.sort() + for name in names: + self.add_group(Group(series[name],name,self)) + + # A single lambda + elif callable(series): + self.__group_list = [] + self.add_group(series) + + # Int/float, instance of Group or Data + elif type(series) in NUMTYPES or isinstance(series, Group) or isinstance(series, Data): + self.__group_list = [] + self.add_group(series) + + # Default + else: + raise TypeError, "Serie type not supported" + + return property(**locals()) + + def add_group(self, group, name=None): + ''' + Append a new group in group_list + ''' + if not isinstance(group, Group): + #Try to convert + group = Group(group, name, self) + + if len(group.data_list) is not 0: + # Auto naming groups + if group.name is None: + group.name = "Group "+str(len(self.__group_list)+1) + + self.__group_list.append(group) + self.__group_list[-1].parent = self + + def copy(self): + ''' + Returns a copy of the Series + ''' + new_series = Series() + new_series.__name = self.__name + if self.__range is not None: + new_series.__range = self.__range[:] + #Add color property in the copy method + #self.__colors = None + + for group in self: + new_series.add_group(group.copy()) + + return new_series + + def get_names(self): + ''' + Returns a list of the names of all groups in the Serie + ''' + names = [] + for group in self: + if group.name is None: + names.append('Group '+str(group.index()+1)) + else: + names.append(group.name) + + return names + + def to_list(self): + ''' + Returns a list with the content of all groups and data + ''' + big_list = [] + for group in self: + for data in group: + if type(data.content) in NUMTYPES: + big_list.append(data.content) + else: + big_list = big_list + list(data.content) + return big_list + + def __getitem__(self, key): + ''' + Makes the Series iterable, based in the group_list property + ''' + return self.__group_list[key] + + def __str__(self): + ''' + Returns a string that represents the Series + ''' + ret = "" + if self.name is not None: + ret += self.name + " " + if len(self) > 0: + list_str = [str(item) for item in self] + ret += str(list_str) + else: + ret += "[]" + return ret + + def __len__(self): + ''' + Returns the length of the Series, based in the group_lsit property + ''' + return len(self.group_list) + + +if __name__ == '__main__': + doctest.testmod() diff --git a/bindings/python/examples/sched_switch.py b/bindings/python/examples/sched_switch.py new file mode 100644 index 0000000..f252ab5 --- /dev/null +++ b/bindings/python/examples/sched_switch.py @@ -0,0 +1,110 @@ +# The script takes one optional argument (pid) +# The script will read events based on pid and +# print the scheduler switches happening with the process. +# If no arguments are passed, it displays all the scheduler switches. +# This can be used to understand which tasks schedule out the current +# process being traced, and when it gets scheduled in again. +# The trace needs PID context (lttng add-context -k -t pid) + +import sys +from babeltrace import * + +if len(sys.argv) < 2 or len(sys.argv) > 3: + raise TypeError("Usage: python sched_switch.py [pid] path/to/trace") +elif len(sys.argv) == 3: + usePID = True +else: + usePID = False + + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +while event is not None: + while True: + if event.get_name() == "sched_switch": + # Getting scope definition + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) + if sco is None: + print("ERROR: Cannot get definition scope for sched_switch") + break # Next event + + # Getting PID + pid_field = event.get_field(sco, "_pid") + pid = pid_field.get_int64() + + if ctf.field_error(): + print("ERROR: Missing PID info for sched_switch") + break # Next event + + if usePID and (pid != long(sys.argv[1])): + break # Next event + + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + + # prev_comm + field = event.get_field(sco, "_prev_comm") + prev_comm = field.get_char_array() + if ctf.field_error(): + print("ERROR: Missing prev_comm context info") + + # prev_tid + field = event.get_field(sco, "_prev_tid") + prev_tid = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing prev_tid context info") + + # prev_prio + field = event.get_field(sco, "_prev_prio") + prev_prio = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing prev_prio context info") + + # prev_state + field = event.get_field(sco, "_prev_state") + prev_state = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing prev_state context info") + + # next_comm + field = event.get_field(sco, "_next_comm") + next_comm = field.get_char_array() + if ctf.field_error(): + print("ERROR: Missing next_comm context info") + + # next_tid + field = event.get_field(sco, "_next_tid") + next_tid = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing next_tid context info") + + # next_prio + field = event.get_field(sco, "_next_prio") + next_prio = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing next_prio context info") + + # Output + print("sched_switch, pid = {}, TS = {}, prev_comm = {},\n\t" + "prev_tid = {}, prev_prio = {}, prev_state = {},\n\t" + "next_comm = {}, next_tid = {}, next_prio = {}".format( + pid, event.get_timestamp(), prev_comm, prev_tid, + prev_prio, prev_state, next_comm, next_tid, next_prio)) + + break # Next event + + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it diff --git a/bindings/python/examples/softirqtimes.py b/bindings/python/examples/softirqtimes.py new file mode 100644 index 0000000..19b2deb --- /dev/null +++ b/bindings/python/examples/softirqtimes.py @@ -0,0 +1,130 @@ +# The script checks the trace for the amount of time +# spent from each softirq_raise to softirq_exit. +# It prints out the min, max (with timestamp), +# average times, the standard deviation and the total count. +# Using the cairoplot module, a .svg graph is also outputted +# showing the taken time in function of the time since the +# beginning of the trace. + +import sys, math +from output_format_modules import cairoplot +from babeltrace import * + +if len(sys.argv) < 2: + raise TypeError("Usage: python softirqtimes.py path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +time_taken = [] +graph_data = [] +max_time = (0.0, 0.0) # (val, ts) + +# tmp template: {(cpu_id, vec):TS raise} +tmp = {} +largest_val = 0 + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +start_time = event.get_timestamp() +while(event is not None): + + event_name = event.get_name() + error = True + appendNext = False + + if event_name == 'softirq_raise' or event_name == 'softirq_exit': + # Recover cpu_id and vec values to make a key to tmp + error = False + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) + field = event.get_field(scope, "cpu_id") + cpu_id = field.get_uint64() + if ctf.field_error(): + print("ERROR: Missing cpu_id info for {}".format( + event.get_name())) + error = True + + scope = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + field = event.get_field(scope, "_vec") + vec = field.get_uint64() + if ctf.field_error(): + print("ERROR: Missing vec info for {}".format( + event.get_name())) + error = True + key = (cpu_id, vec) + + if event_name == 'softirq_raise' and not error: + # Add timestamp to tmp + if key in tmp: + # If key already exists + i = 0 + while True: + # Add index + key = (cpu_id, vec, i) + if key in tmp: + i += 1 + continue + if i > largest_val: + largest_val = i + break + + tmp[key] = event.get_timestamp() + + if event_name == 'softirq_exit' and not error: + # Saving data for output + # Key check + if not (key in tmp): + i = 0 + while i <= largest_val: + key = (key[0], key[1], i) + if key in tmp: + break + i += 1 + + raise_timestamp = tmp[key] + time_data = event.get_timestamp() - tmp.pop(key) + if time_data > max_time[0]: + # max_time = (val, ts) + max_time = (time_data, raise_timestamp) + time_taken.append(time_data) + graph_data.append((raise_timestamp - start_time, time_data)) + + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + + +del ctf_it + +# Standard dev. calc. +try: + mean = sum(time_taken)/float(len(time_taken)) +except ZeroDivisionError: + raise TypeError("empty data") +deviations_squared = [] +for x in time_taken: + deviations_squared.append(math.pow((x - mean), 2)) +try: + stddev = math.sqrt(sum(deviations_squared) / (len(deviations_squared) - 1)) +except ZeroDivisionError: + stddev = '-' + +# Terminal output +print("AVG TIME: {} ns".format(mean)) +print("MIN TIME: {} ns".format(min(time_taken))) +print("MAX TIME: {} ns, TS: {}".format(max_time[0], max_time[1])) +print("STD DEV: {}".format(stddev)) +print("TOTAL COUNT: {}".format(len(time_taken))) + +# Graph output +cairoplot.scatter_plot ( 'softirqtimes.svg', data = graph_data, + width = 5000, height = 4000, border = 20, axis = True, + grid = True, series_colors = ["red"] ) diff --git a/bindings/python/examples/syscalls_by_pid.py b/bindings/python/examples/syscalls_by_pid.py new file mode 100644 index 0000000..f6127ed --- /dev/null +++ b/bindings/python/examples/syscalls_by_pid.py @@ -0,0 +1,61 @@ +# The script checks all syscall in the trace and prints a list +# showing the number of systemcalls executed by each PID +# ordered from greatest to least number of syscalls. +# The trace needs PID context (lttng add-context -k -t pid) + +import sys +from babeltrace import * +from output_format_modules.pprint_table import pprint_table as pprint + +if len(sys.argv) < 2 : + raise TypeError("Usage: python syscalls_by_pid.py path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +data = {} + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +while event is not None: + if event.get_name().find("sys") >= 0: + # Getting scope definition + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) + if sco is None: + print("ERROR: Cannot get definition scope for {}".format( + event.get_name())) + else: + # Getting PID + pid_field = event.get_field(sco, "_pid") + pid = pid_field.get_int64() + + if ctf.field_error(): + print("ERROR: Missing PID info for sched_switch".format( + event.get_name())) + elif pid in data: + data[pid] += 1 + else: + data[pid] = 1 + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it + +# Setting table for output +table = [] +for item in data: + table.append([data[item], item]) # [count, pid] +table.sort(reverse = True) # [big count first, pid] +for i in range(len(table)): + table[i].reverse() # [pid, big count first] +table.insert(0, ["PID", "SYSCALL COUNT"]) +pprint(table) diff --git a/bindings/python/python-complements.c b/bindings/python/python-complements.c new file mode 100644 index 0000000..8c4d811 --- /dev/null +++ b/bindings/python/python-complements.c @@ -0,0 +1,105 @@ +/* python-complements.c + Needed functions for python binding +*/ + +#include "python-complements.h" + +/* FILE functions + ---------------------------------------------------- +*/ + +FILE *_bt_file_open(char *file_path, char *mode) +{ + FILE *fp = stdout; + if (file_path != NULL) + fp = fopen(file_path, mode); + return fp; +} + +void _bt_file_close(FILE *fp) +{ + if (fp != NULL) + fclose(fp); +} + + +/* List-related functions + ---------------------------------------------------- +*/ + +/* ctf-field-list */ +struct definition **_bt_python_field_listcaller( + const struct bt_ctf_event *ctf_event, + const struct definition *scope) +{ + struct definition **list; + unsigned int count; + int ret; + + ret = bt_ctf_get_field_list(ctf_event, scope, + (const struct definition * const **)&list, &count); + + if (ret < 0) /* For python to know an error occured */ + list = NULL; + else /* For python to know the end is reached */ + list[count] = NULL; + + return list; +} + +struct definition *_bt_python_field_one_from_list( + struct definition **list, int index) +{ + return list[index]; +} + +/* event_decl_list */ +struct bt_ctf_event_decl **_bt_python_event_decl_listcaller( + int handle_id, struct bt_context *ctx) +{ + struct bt_ctf_event_decl **list; + unsigned int count; + int ret; + + ret = bt_ctf_get_event_decl_list(handle_id, ctx, + (struct bt_ctf_event_decl * const **)&list, &count); + + if (ret < 0) /* For python to know an error occured */ + list = NULL; + else /* For python to know the end is reached */ + list[count] = NULL; + + return list; +} + +struct bt_ctf_event_decl *_bt_python_decl_one_from_list( + struct bt_ctf_event_decl **list, int index) +{ + return list[index]; +} + +/* decl_fields */ +struct bt_ctf_field_decl **_by_python_field_decl_listcaller( + struct bt_ctf_event_decl *event_decl, + enum bt_ctf_scope scope) +{ + struct bt_ctf_field_decl **list; + unsigned int count; + int ret; + + ret = bt_ctf_get_decl_fields(event_decl, scope, + (const struct bt_ctf_field_decl * const **)&list, &count); + + if (ret < 0) /* For python to know an error occured */ + list = NULL; + else /* For python to know the end is reached */ + list[count] = NULL; + + return list; +} + +struct bt_ctf_field_decl *_bt_python_field_decl_one_from_list( + struct bt_ctf_field_decl **list, int index) +{ + return list[index]; +} diff --git a/bindings/python/python-complements.h b/bindings/python/python-complements.h new file mode 100644 index 0000000..cdd5528 --- /dev/null +++ b/bindings/python/python-complements.h @@ -0,0 +1,36 @@ +/* python-complements.h + Needed functions for python binding +*/ + +#include +#include +#include +#include +#include +#include +#include +#include + +/* File */ +FILE *_bt_file_open(char *file_path, char *mode); +void _bt_file_close(FILE *fp); + +/* ctf-field-list */ +struct definition **_bt_python_field_listcaller( + const struct bt_ctf_event *ctf_event, + const struct definition *scope); +struct definition *_bt_python_field_one_from_list( + struct definition **list, int index); + +/* event_decl_list */ +struct bt_ctf_event_decl **_bt_python_event_decl_listcaller( + int handle_id, struct bt_context *ctx); +struct bt_ctf_event_decl *_bt_python_decl_one_from_list( + struct bt_ctf_event_decl **list, int index); + +/* decl_fields */ +struct bt_ctf_field_decl **_by_python_field_decl_listcaller( + struct bt_ctf_event_decl *event_decl, + enum bt_ctf_scope scope); +struct bt_ctf_field_decl *_bt_python_field_decl_one_from_list( + struct bt_ctf_field_decl **list, int index); diff --git a/bootstrap b/bootstrap index c507425..f6926ca 100755 --- a/bootstrap +++ b/bootstrap @@ -4,7 +4,7 @@ set -x if [ ! -e config ]; then mkdir config fi -aclocal +aclocal -I m4 libtoolize --force --copy autoheader automake --add-missing --copy diff --git a/configure.ac b/configure.ac index d90479d..f9cff9d 100644 --- a/configure.ac +++ b/configure.ac @@ -74,6 +74,41 @@ AC_CHECK_LIB([popt], [poptGetContext], [], [AC_MSG_ERROR([Cannot find popt.])] ) + +# For Python +# SWIG version needed or newer: +swig_version=2.0.0 + +AC_ARG_ENABLE([python], + [AC_HELP_STRING([--disable-python], + [do not compile Python bindings])], + [], [enable_python=yes]) + +AM_CONDITIONAL([USE_PYTHON], [test "x${enable_python:-yes}" = xyes]) + +if test "x${enable_python:-yes}" = xyes; then + AC_MSG_NOTICE([You may configure with --disable-python ]dnl +[if you do not want Python bindings.]) + + AX_PKG_SWIG($swig_version, [], [ AC_MSG_ERROR([SWIG $swig_version or newer is needed]) ]) + AM_PATH_PYTHON + + AC_ARG_VAR([PYTHON_INCLUDE], [Include flags for python, bypassing python-config]) + AC_ARG_VAR([PYTHON_CONFIG], [Path to python-config]) + AS_IF([test -z "$PYTHON_INCLUDE"], [ + AS_IF([test -z "$PYTHON_CONFIG"], [ + AC_PATH_PROGS([PYTHON_CONFIG], + [python$PYTHON_VERSION-config python-config], + [no], + [`dirname $PYTHON`]) + AS_IF([test "$PYTHON_CONFIG" = no], [AC_MSG_ERROR([cannot find python-config for $PYTHON.])]) + ]) + AC_MSG_CHECKING([python include flags]) + PYTHON_INCLUDE=`$PYTHON_CONFIG --includes` + AC_MSG_RESULT([$PYTHON_INCLUDE]) + ]) +fi + pkg_modules="gmodule-2.0 >= 2.0.0" PKG_CHECK_MODULES(GMODULE, [$pkg_modules]) AC_SUBST(PACKAGE_LIBS) @@ -103,6 +138,8 @@ AC_CONFIG_FILES([ lib/Makefile lib/prio_heap/Makefile include/Makefile + bindings/Makefile + bindings/python/Makefile tests/Makefile ]) AC_OUTPUT diff --git a/doc/python-howto.txt b/doc/python-howto.txt new file mode 100644 index 0000000..e2ed751 --- /dev/null +++ b/doc/python-howto.txt @@ -0,0 +1,70 @@ +PYTHON BINDINGS +---------------- + +This is a brief howto for using the Babeltrace Python module. + + +INSTALLATION: + +By default, the Python bindings are installed. +If you do not wish the Python bindings, you can configure with the +--disable-python option during the installation procedure: + + $ ./configure --disable-python + +The Python module is automatically generated using SWIG, therefore the +swig2.0 package on Debian/Ubuntu is requied. + + +USAGE: + +Once installed, the Python module can be used by importing it in Python. +In the Python interpreter: + + >>> import babeltrace + +Then the starting point is to create a context and add a trace to it. + + >>> ctx = babeltrace.Context() + >>> ctx.add_trace("path/to/trace", ) + +Where is a string containing the format name in which the trace +was produced. To print a list of available formats to the standard +output, it is possible to use the print_format_list function. + + >>> out = babeltrace.File(None) # This returns stdout + >>> babeltrace.print_format_list(out) + +When a trace is added to a context, it is opened and ready to read using +an iterator. While creating an iterator, optional starting and ending +position may be specified. So far, only ctf iterator are supported. + + >>> begin_pos = babeltrace.IterPos(babeltrace.SEEK_BEGIN) + >>> iterator = babeltrace.ctf.Iterator(ctx, begin_pos) + +From there, it is possible to read the events. + + >>> event = iterator.read_event() + +It is simple to obtain the timestamp of that event. + + >>> timestamp = event.get_timestamp() + +Let's say that we want to extract the prev_comm context info for a +sched_switch event. To do so, it is needed to set an event scope +with which we can obtain the field wanted. + + >>> if event.get_name == "sched_switch": + ... #prev_comm only for sched_switch events + ... scope = event.get_top_level_scope(babeltrace.ctf.scope.EVENT_FIELDS) + ... field = event.get_field(scope, "_prev_comm") + ... prev_comm = field.get_char_array() + +It is also possible to move on to the next event. + + >>> ret = iterator.next() # Move the iterator + >>> if ret == 0: # No error occured + ... event = iterator.read_event() # Read the next event + +For many usage script examples of the Babeltrace Python module, see the +bindings/python/examples directory. diff --git a/m4/ax_pkg_swig.m4 b/m4/ax_pkg_swig.m4 new file mode 100644 index 0000000..e112f3d --- /dev/null +++ b/m4/ax_pkg_swig.m4 @@ -0,0 +1,135 @@ +# =========================================================================== +# http://www.gnu.org/software/autoconf-archive/ax_pkg_swig.html +# =========================================================================== +# +# SYNOPSIS +# +# AX_PKG_SWIG([major.minor.micro], [action-if-found], [action-if-not-found]) +# +# DESCRIPTION +# +# This macro searches for a SWIG installation on your system. If found, +# then SWIG is AC_SUBST'd; if not found, then $SWIG is empty. If SWIG is +# found, then SWIG_LIB is set to the SWIG library path, and AC_SUBST'd. +# +# You can use the optional first argument to check if the version of the +# available SWIG is greater than or equal to the value of the argument. It +# should have the format: N[.N[.N]] (N is a number between 0 and 999. Only +# the first N is mandatory.) If the version argument is given (e.g. +# 1.3.17), AX_PKG_SWIG checks that the swig package is this version number +# or higher. +# +# As usual, action-if-found is executed if SWIG is found, otherwise +# action-if-not-found is executed. +# +# In configure.in, use as: +# +# AX_PKG_SWIG(1.3.17, [], [ AC_MSG_ERROR([SWIG is required to build..]) ]) +# AX_SWIG_ENABLE_CXX +# AX_SWIG_MULTI_MODULE_SUPPORT +# AX_SWIG_PYTHON +# +# LICENSE +# +# Copyright (c) 2008 Sebastian Huber +# Copyright (c) 2008 Alan W. Irwin +# Copyright (c) 2008 Rafael Laboissiere +# Copyright (c) 2008 Andrew Collier +# Copyright (c) 2011 Murray Cumming +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation; either version 2 of the License, or (at your +# option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program. If not, see . +# +# As a special exception, the respective Autoconf Macro's copyright owner +# gives unlimited permission to copy, distribute and modify the configure +# scripts that are the output of Autoconf when processing the Macro. You +# need not follow the terms of the GNU General Public License when using +# or distributing such scripts, even though portions of the text of the +# Macro appear in them. The GNU General Public License (GPL) does govern +# all other use of the material that constitutes the Autoconf Macro. +# +# This special exception to the GPL applies to versions of the Autoconf +# Macro released by the Autoconf Archive. When you make and distribute a +# modified version of the Autoconf Macro, you may extend this special +# exception to the GPL to apply to your modified version as well. + +#serial 8 + +AC_DEFUN([AX_PKG_SWIG],[ + # Ubuntu has swig 2.0 as /usr/bin/swig2.0 + AC_PATH_PROGS([SWIG],[swig swig2.0]) + if test -z "$SWIG" ; then + m4_ifval([$3],[$3],[:]) + elif test -n "$1" ; then + AC_MSG_CHECKING([SWIG version]) + [swig_version=`$SWIG -version 2>&1 | grep 'SWIG Version' | sed 's/.*\([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\).*/\1/g'`] + AC_MSG_RESULT([$swig_version]) + if test -n "$swig_version" ; then + # Calculate the required version number components + [required=$1] + [required_major=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_major" ; then + [required_major=0] + fi + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] + [required_minor=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_minor" ; then + [required_minor=0] + fi + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] + [required_patch=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_patch" ; then + [required_patch=0] + fi + # Calculate the available version number components + [available=$swig_version] + [available_major=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_major" ; then + [available_major=0] + fi + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] + [available_minor=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_minor" ; then + [available_minor=0] + fi + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] + [available_patch=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_patch" ; then + [available_patch=0] + fi + # Convert the version tuple into a single number for easier comparison. + # Using base 100 should be safe since SWIG internally uses BCD values + # to encode its version number. + required_swig_vernum=`expr $required_major \* 10000 \ + \+ $required_minor \* 100 \+ $required_patch` + available_swig_vernum=`expr $available_major \* 10000 \ + \+ $available_minor \* 100 \+ $available_patch` + + if test $available_swig_vernum -lt $required_swig_vernum; then + AC_MSG_WARN([SWIG version >= $1 is required. You have $swig_version.]) + SWIG='' + m4_ifval([$3],[$3],[]) + else + AC_MSG_CHECKING([for SWIG library]) + SWIG_LIB=`$SWIG -swiglib` + AC_MSG_RESULT([$SWIG_LIB]) + m4_ifval([$2],[$2],[]) + fi + else + AC_MSG_WARN([cannot determine SWIG version]) + SWIG='' + m4_ifval([$3],[$3],[]) + fi + fi + AC_SUBST([SWIG_LIB]) +]) diff --git a/tests/tests-python.py b/tests/tests-python.py new file mode 100644 index 0000000..0bd71c2 --- /dev/null +++ b/tests/tests-python.py @@ -0,0 +1,115 @@ +import unittest +import sys +from babeltrace import * + +class TestBabeltracePythonModule(unittest.TestCase): + + def test_handle_decl(self): + #Context creation, adding trace + ctx = Context() + trace_handle = ctx.add_trace( + "ctf-traces/succeed/lttng-modules-2.0-pre5", + "ctf") + self.assertIsNotNone(trace_handle, "Error adding trace") + + #TraceHandle test + ts_begin = trace_handle.get_timestamp_begin(ctx, CLOCK_REAL) + ts_end = trace_handle.get_timestamp_end(ctx, CLOCK_REAL) + self.assertGreater(ts_end, ts_begin, "Error get_timestamp from trace_handle") + + lst = ctf.get_event_decl_list(trace_handle, ctx) + self.assertIsNotNone(lst, "Error get_event_decl_list") + + name = lst[0].get_name() + self.assertIsNotNone(name) + + fields = lst[0].get_decl_fields(ctf.scope.EVENT_FIELDS) + self.assertIsNotNone(fields, "Error getting FieldDecl list") + + if len(fields) > 0: + self.assertIsNotNone(fields[0].get_name(), + "Error getting name from FieldDecl") + + #Remove trace + ctx.remove_trace(trace_handle) + del ctx + del trace_handle + + + def test_iterator_event(self): + #Context creation, adding trace + ctx = Context() + trace_handle = ctx.add_trace( + "ctf-traces/succeed/lttng-modules-2.0-pre5", + "ctf") + self.assertIsNotNone(trace_handle, "Error adding trace") + + begin_pos = IterPos(SEEK_BEGIN) + it = ctf.Iterator(ctx, begin_pos) + self.assertIsNotNone(it, "Error creating iterator") + + event = it.read_event() + self.assertIsNotNone(event, "Error reading event") + + handle = event.get_handle() + self.assertIsNotNone(handle, "Error getting handle") + + context = event.get_context() + self.assertIsNotNone(context, "Error getting context") + + name = "" + while event is not None and name != "sched_switch": + name = event.get_name() + self.assertIsNotNone(name, "Error getting event name") + + ts = event.get_timestamp() + self.assertGreaterEqual(ts, 0, "Error getting timestamp") + + cycles = event.get_cycles() + self.assertGreaterEqual(cycles, 0, "Error getting cycles") + + if name == "sched_switch": + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) + self.assertIsNotNone(scope, "Error getting scope definition") + + field = event.get_field(scope, "cpu_id") + prev_comm_str = field.get_uint64() + self.assertEqual(ctf.field_error(), 0, "Error getting vec info") + + field_lst = event.get_field_list(scope) + self.assertIsNotNone(field_lst, "Error getting field list") + + fname = field.field_name() + self.assertIsNotNone(fname, "Error getting field name") + + ftype = field.field_type() + self.assertIsNot(ftype, -1, "Error getting field type (unknown)") + + ret = it.next() + self.assertGreaterEqual(ret, 0, "Error moving iterator") + + event = it.read_event() + + pos = it.get_pos() + self.assertIsNotNone(pos._pos, "Error getting iterator position") + + ret = it.set_pos(pos) + self.assertEqual(ret, 0, "Error setting iterator position") + + pos = it.create_time_pos(ts) + self.assertIsNotNone(pos._pos, "Error creating time-based iterator position") + + del it, pos + set_pos = IterPos(SEEK_TIME, ts) + it = ctf.Iterator(ctx, begin_pos, set_pos) + self.assertIsNotNone(it, "Error creating iterator with end_pos") + + + def test_file_class(self): + f = File("test-bitfield.c") + self.assertIsNotNone(f, "Error opening file") + f.close() + + +if __name__ == "__main__": + unittest.main() -- 1.7.9.5 From danny.serres at efficios.com Thu Aug 9 14:42:59 2012 From: danny.serres at efficios.com (Danny Serres) Date: Thu, 9 Aug 2012 14:42:59 -0400 Subject: [lttng-dev] [babeltrace PATCH] Remove trace-collection.h from include_headers Message-ID: <1344537779-19779-1-git-send-email-danny.serres@efficios.com> Signed-off-by: Danny Serres --- include/Makefile.am | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/Makefile.am b/include/Makefile.am index f7488da..79247a0 100644 --- a/include/Makefile.am +++ b/include/Makefile.am @@ -3,7 +3,6 @@ babeltraceinclude_HEADERS = \ babeltrace/format.h \ babeltrace/context.h \ babeltrace/iterator.h \ - babeltrace/trace-collection.h \ babeltrace/trace-handle.h \ babeltrace/list.h \ babeltrace/clock-types.h @@ -21,6 +20,7 @@ noinst_HEADERS = \ babeltrace/context-internal.h \ babeltrace/iterator-internal.h \ babeltrace/prio_heap.h \ + babeltrace/trace-collection.h \ babeltrace/types.h \ babeltrace/ctf-ir/metadata.h \ babeltrace/ctf/events-internal.h \ -- 1.7.9.5 From mathieu.desnoyers at efficios.com Thu Aug 9 16:02:11 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 16:02:11 -0400 Subject: [lttng-dev] [PATCH 2/2] urcu: add notice to URCU_TLS() for it is not async-signal-safe In-Reply-To: References: <1344414673-14714-1-git-send-email-laijs@cn.fujitsu.com> <1344414673-14714-2-git-send-email-laijs@cn.fujitsu.com> <20120809141417.GC9064@Krystal> Message-ID: <20120809200211.GA14409@Krystal> Looking at the result of a quick google search: http://curl.haxx.se/mail/lib-2006-09/0224.html http://www.slamb.org/projects/sigsafe/api/patternref.html "Additionally, it makes the same assumption as all other methods for handling thread-directed signals (with the exception of kevent(2) handling), that pthread_getspecific(2) is async signal-safe. This is not guaranteed by SUSv3." and https://groups.google.com/forum/?fromgroups#!topic/comp.os.linux.development/nZfmndKbzJw[1-25] it looks like using pthread_getspecific from a signal handler is not always safe, mainly due to possible use of sigaltstack. So disabling signals works for the "pthread_key_create" part, but we still have an issue with pthread_getspecific. Ideas are welcome on how to best deal with this issue. Thanks! Mathieu * Lai Jiangshan (eag0628 at gmail.com) wrote: > OK for me. Please do it. > > On Thu, Aug 9, 2012 at 10:14 PM, Mathieu Desnoyers > wrote: > > * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > >> Signed-off-by: Lai Jiangshan > >> --- > >> urcu/tls-compat.h | 14 ++++++++++++++ > >> 1 files changed, 14 insertions(+), 0 deletions(-) > >> > >> diff --git a/urcu/tls-compat.h b/urcu/tls-compat.h > >> index 192a536..b7bf363 100644 > >> --- a/urcu/tls-compat.h > >> +++ b/urcu/tls-compat.h > >> @@ -59,6 +59,20 @@ extern "C" { > >> > >> #else /* #ifndef CONFIG_RCU_TLS */ > >> > >> +/* > >> + * NOTE: URCU_TLS() is NOT async-signal-safe, you can't use it > >> + * inside any function which can be called from signal handler. > >> + * > >> + * But if pthread_getspecific() is async-signal-safe in your > >> + * platform, you can make URCU_TLS() async-signal-safe via: > >> + * ensuring the first call to URCU_TLS() of a given TLS variable of > >> + * all threads is called earliest from a non-signal handler function. > >> + * > >> + * Exmaple: In any thread, the first call of URCU_TLS(rcu_reader) > >> + * is called from rcu_register_thread(), so we can ensure all later > >> + * URCU_TLS(rcu_reader) in any thread is async-signal-safe. > > > > Hrm. We could also just block all signals within type *__tls_access_ ## > > name(void) (in tls-compat.h) and make sure it is async-signal-safe I > > guess ? I would prefer that solution: it would make the code more robust > > for a rarely taken performance hit. > > > > Thoughts ? > > > > Thanks, > > > > Mathieu > > > >> + */ > >> + > >> # include > >> > >> struct urcu_tls { > >> -- > >> 1.7.4.4 > >> > > > > -- > > Mathieu Desnoyers > > Operating System Efficiency R&D Consultant > > EfficiOS Inc. > > http://www.efficios.com > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 9 16:40:45 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 9 Aug 2012 16:40:45 -0400 Subject: [lttng-dev] [Babeltrace-patch] Add BT_SEEK_LAST type to bt_iter_pos. In-Reply-To: <1344542423-31441-1-git-send-email-francis.deslauriers@polymtl.ca> References: <1344542423-31441-1-git-send-email-francis.deslauriers@polymtl.ca> Message-ID: <20120809204045.GA15109@Krystal> * Francis Deslauriers (francis.deslauriers at polymtl.ca) wrote: > Signed-off-by: Francis Deslauriers > --- > include/babeltrace/iterator.h | 1 + > lib/iterator.c | 199 ++++++++++++++++++++++++++++++++++++++++- > 2 files changed, 199 insertions(+), 1 deletion(-) > > diff --git a/include/babeltrace/iterator.h b/include/babeltrace/iterator.h > index aa6470e..c8edbd1 100644 > --- a/include/babeltrace/iterator.h > +++ b/include/babeltrace/iterator.h > @@ -48,6 +48,7 @@ struct bt_iter_pos { > BT_SEEK_CUR, > BT_SEEK_BEGIN, > BT_SEEK_END, > + BT_SEEK_LAST please add comma to last entry. This is a coding style I always follow to make automated regexps easier. > } type; > union { > uint64_t seek_time; > diff --git a/lib/iterator.c b/lib/iterator.c > index bcb77d8..5125afb 100644 > --- a/lib/iterator.c > +++ b/lib/iterator.c > @@ -3,7 +3,7 @@ > * > * Babeltrace Library > * > - * Copyright 2010-2011 EfficiOS Inc. and Linux Foundation > +/ * Copyright 2010-2011 EfficiOS Inc. and Linux Foundation This seems wrong. > * > * Author: Mathieu Desnoyers > * > @@ -186,7 +186,185 @@ static int seek_ctf_trace_by_timestamp(struct ctf_trace *tin, > > return found ? 0 : EOF; > } > +int seek_last_element_ctf_file_stream(struct trace_collection *tc, > + struct ctf_file_stream **cfs, int max_packet, > + int max_stream_id, int max_stream_class_id, > + int max_trace_descriptor_id, uint64_t max_timestamp) > +{ > + int ret; > + struct trace_descriptor *max_td_read; > + struct ctf_trace *max_tin; > + struct ctf_stream_declaration *max_stream_class; > + struct ctf_stream_definition *max_stream; > + struct ctf_stream_pos *max_stream_pos; > + > + /* get the targeted trace descriptor */ > + max_td_read = g_ptr_array_index(tc->array, > + max_trace_descriptor_id); > + max_tin = container_of(max_td_read, > + struct ctf_trace, parent); > + /* get targeted stream class */ > + max_stream_class = g_ptr_array_index(max_tin->streams, > + max_stream_class_id); > + /* get targeted stream */ > + max_stream = g_ptr_array_index(max_stream_class->streams, > + max_stream_id); One comment by line might be a bit much. I prefer to comment the important things that must not be forgotten in the code, and sometimes, if needed, a commend about what the function does before the function. > + *cfs = container_of(max_stream, > + struct ctf_file_stream, parent); > + max_stream_pos = &(*cfs)->pos; > + /* we seek to the last packet of the stream.*/ ah, this comment (above) is slightly more relevant. > + max_stream_pos->packet_seek(&max_stream_pos->parent, > + max_packet, > + SEEK_SET); > + /* > + * iterate over all the event until we reach on that all the event -> every event on that his.. -> an event such that its ... > + * his timestamp correspond with the max saved previously. > + */ > + do { > + ret = stream_read_event(*cfs); > + } while ((*cfs)->parent.real_timestamp != max_timestamp && ret == 0); > + > + /* We insert the stream in the heap.*/ in -> into missing space before */ > + return ret; > +} missing whiteline. please use checkpatch.pl provided in the linux kernel scripts against your patch to identify style errors. > +int find_last_event(struct ctf_file_stream *cfs, > + uint64_t *timestamp_end, > + int *packet) > +{ > + int ret = 0; > + int count = 0; > + int event_read = 0; put all integers on the same line. int ret = 0, count = 0, ...... > + uint64_t tmp = 0; > + int i; > + struct ctf_stream_pos *stream_pos; > + stream_pos = &cfs->pos; missing whiteline after variable declarations. > + /* > + * we start by the last packet as the current one. > + * If the current one is empty we go back one packet if possible. > + */ > + i = stream_pos->packet_real_index->len; > + /* > + * Check if we have iterated on all the packets. > + * If we are not short on packets, we check what > + * made us leave the reading event loop. > + */ > + while (i >= 1 && count == 0) { what a strangely-looking loop construct. Please use a straightforward for () loop. > + i--; > + stream_pos->packet_seek(&stream_pos->parent, i, SEEK_SET); > + count = 0; > + /* read each event until we reach the end of the packet */ > + do { > + tmp = cfs->parent.real_timestamp; > + ret = stream_read_event(cfs); > + if (ret == 0) { > + count++; > + } > + } while (ret == 0); > + > + /*Error*/ missing spaces around /* */ > + if (ret > 0) { > + goto end; > + } > + event_read += count; > + } > + *packet = i; > + *timestamp_end = tmp; > + > + /* Check if we read at least one event on the stream */ > + if (event_read <= 1) { > + ret = 1; > + goto end; > + } > + ret = 0; > + > +end: > + return ret; > +} > > +int find_max_timestamp_ctf_file_stream( > + struct ctf_stream_declaration *stream_class, int *max_stream_id, > + int *packet, uint64_t *max_timestamp, int *new_max) > +{ > + struct ctf_file_stream *cfs; > + int i; > + int max_packet = 0; > + int ret = 0; > + int error = 1; please combine those int into the same line. > + uint64_t savedtime; > + > + for (i = 0; i < stream_class->streams->len; i++) { > + struct ctf_stream_definition *stream; > + stream = g_ptr_array_index( > + stream_class->streams, i); > + if (!stream) > + continue; > + cfs = container_of(stream, struct ctf_file_stream, parent); > + ret = find_last_event(cfs, &savedtime, &max_packet); > + /* Can return either EOF, 0, or error (> 0). */ > + if (ret == 0 && savedtime >= *max_timestamp) { > + *new_max = 1; > + *max_stream_id = i; > + *packet = max_packet; > + *max_timestamp = savedtime; > + } > + error = error & ret; > + } > + return error ; extra space. > +} > + > +int seek_last_ctf_file_stream(struct trace_collection *tc, > + struct ctf_file_stream **cfs) > +{ > + int i, j; > + int ret; > + int found = 0; > + int new_max_found; > + uint64_t max_timestamp = 0; > + int max_packet; > + int max_stream_id; > + int max_stream_class_id; > + int max_trace_descriptor_id; combine int. missing whileline. I look forward to see the next round. Thanks, Mathieu > + /* For each trace in the trace_collection*/ > + for (i = 0; i < tc->array->len; i++) { > + struct ctf_trace *tin; > + struct trace_descriptor *td_read; > + td_read = g_ptr_array_index(tc->array, i); > + if (!td_read) > + continue; > + tin = container_of(td_read, struct ctf_trace, parent); > + /* For each stream_class in the trace*/ > + for (j = 0; j < tin->streams->len; j++) { > + struct ctf_stream_declaration *stream_class; > + > + stream_class = g_ptr_array_index(tin->streams, j); > + if (!stream_class) > + continue; > + /* For each file_stream in the stream_class */ > + new_max_found = 0; > + ret = find_max_timestamp_ctf_file_stream(stream_class, > + &max_stream_id, &max_packet, > + &max_timestamp, &new_max_found); > + if (!ret && new_max_found) { > + found = 1; > + max_trace_descriptor_id = i; > + max_stream_class_id = j; > + } > + } > + } > + /* > + * Now we know in which trace, stream_class and stream is the > + * last event of the trace_collection. > + * We can seek to this targeted event. > + */ > + if (!found) { > + ret = -1; > + } else { > + ret = seek_last_element_ctf_file_stream(tc, cfs, max_packet, > + max_stream_id, max_stream_class_id, > + max_trace_descriptor_id, max_timestamp); > + } > + return ret; > +} > int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) > { > struct trace_collection *tc; > @@ -326,6 +504,25 @@ int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) > } > } > break; > + case BT_SEEK_LAST: > + { > + struct ctf_file_stream *cfs; > + tc = iter->ctx->tc; > + > + ret = seek_last_ctf_file_stream(tc, &cfs); > + if (ret < 0) > + goto error; > + /* remove all stream from the heap*/ > + heap_free(iter->stream_heap); > + /* Create a new empty heap*/ > + ret = heap_init(iter->stream_heap, 0, stream_compare); > + if (ret < 0) > + goto error; > + /* Insert the stream that contains the last event. */ > + heap_insert(iter->stream_heap, cfs); > + > + return 0; > + } > default: > /* not implemented */ > return -EINVAL; > -- > 1.7.9.5 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From danny.serres at efficios.com Thu Aug 9 17:39:38 2012 From: danny.serres at efficios.com (Danny Serres) Date: Thu, 9 Aug 2012 17:39:38 -0400 Subject: [lttng-dev] =?utf-8?q?=5Bbabeltrace_PATCH=5D_API_documentation_fi?= =?utf-8?b?bGXigI8=?= Message-ID: <1344548378-22039-1-git-send-email-danny.serres@efficios.com> Signed-off-by: Danny Serres --- doc/API.txt | 237 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100644 doc/API.txt diff --git a/doc/API.txt b/doc/API.txt new file mode 100644 index 0000000..8cb7e31 --- /dev/null +++ b/doc/API.txt @@ -0,0 +1,237 @@ +Babeltrace API documentation + +Babeltrace is a trace viewer and converter reading and writing the +Common Trace Format (CTF). Its main use is to pretty-print CTF traces +into a human-readable text output. + +This document describes the main concepts to use the libbabeltrace, +which exposes the Babeltrace trace reading capability. + + +TODO: add explanation for each scope (line 116) + + +TERMINOLOGY +??????????????? + +* A "callback" is a reference to a piece of executable code (such as a + function) that is passed as an argument to another piece of code + (like another function). + +* A "context" is a structure that represents an object in which a trace + collection is openned. + +* An "iterator" is a structure that enables the user to traverse a trace. + +* A "trace handle" is a unique identifier representing a trace file. + It allows the user to manipulate a trace file directly. + + + +USAGE +??????????????? + +Context: + +In order to use libbabeltrace to read a trace, the first step is to create a +context structure and to add a trace to it. This is done using the +bt_context_create() and bt_context_add_trace() functions. As long as this +context structure is allocated, the trace collection is open. + +The context can be destroyed by calling one more bt_context_put() +than bt_context_get(), functions which respectively decrement and increment +the refcount of the context. These functions ensures that the context won't +be destroyed when it is in use. + +Once a trace is added to the context, it can be read and seeked using iterators +or callbacks. + + +Iterator: + +An iterator can be created using the bt_iter_create() function. +As for now, only ctf iterator are supported. These are used to traverse a ctf- +formatted trace. Such iterator can be created with bt_ctf_iter_create(). + +While creating an iterator, a begin and an end position may be specified. +To do so, one or two struct bt_iter_pos must be passed. Such struct have 2 +attributes: type and u. "type" is the seek type, can be either: + BT_SEEK_TIME + BT_SEEK_RESTORE + BT_SEEK_CUR + BT_SEEK_BEGIN + BT_SEEK_END +and "u" is a union of the seek time (if using BT_SEEK_TIME) and the restore +position (if using BT_SEEK_RESTORE). + +Once the iterator is created, various functions become available. We have +bt_ctf_iter_read_event() which returns the ctf event of the trace where the +iterator is set. There is also bt_ctf_iter_destroy() which free the iterator. +Note that only one iterator can be created against a context at the same time. +If more than one iterator is being created for the same context, the second +creation will return NULL. The previous iterator must be destroyed before +creation of the new iterator. In the future, creation of multiples iterators +will probably be allowed. + +Finally, we have the bt_ctf_get_iter() function which returns a struct bt_iter +with which the iterator can be moved using one of these functions: + bt_iter_next(), which moves the position to the next event + bt_iter_set_pos(), which moves the position to the specified one + +To get the current position (struct bt_iter_pos) of the iterator, the function +bt_iter_get_pos() can be used. To get an arbitrary position based on a +specific time, bt_itet_create_time_pos() is the function to use. The position +returned by one of these two functions must be freed with bt_iter_free_pos() +after use. + + +CTF Event: + +A CTF event is usually obtained from an iterator via the +bt_ctf_iter_read_event() function, which returns a struct bt_ctf_event. Having +the event, we can collect its data: + * bt_ctf_event_name() will return the name of the event; + * bt_ctf_get_timestamp() will return the timestamp of the event + offsetted with the system clock source (in ns); + * bt_ctf_get_cycles will return the timestamp of the event as written + in the packet (in cycles). + +To access the event fields which contain various information, the first step is +to obtain a scope definition for the desired field. This can be done using the +bt_ctf_get_top_level_scope() function. This function provides the mapping +between the enum and the actual definition of top-level scopes and returns +a definition of the top-level scope. +Top-level scopes are defined in the bt_ctf_scope enum: + BT_TRACE_PACKET_HEADER = 0, + BT_STREAM_PACKET_CONTEXT = 1, + BT_STREAM_EVENT_HEADER = 2, + BT_STREAM_EVENT_CONTEXT = 3, + BT_EVENT_CONTEXT = 4, + BT_EVENT_FIELDS = 5. + + +In order to get a field or a field list, the user needs to pass a scope as +argument, this scope can be a top-level scope or a scope relative to an +arbitrary field. + +To get a field list, the bt_ctf_get_field_list() function is to be used. +To get the definition of a specific field, the appropriate function would be +bt_ctf_get_field(). + +Once the field is obtained, we can obtain its name and type using the +bt_ctf_field_name() and bt_ctf_field_type() functions respectively. The +possible types are defined in the ctf_type_id enum: + CTF_TYPE_UNKNOWN = 0, + CTF_TYPE_INTEGER, + CTF_TYPE_FLOAT, + CTF_TYPE_ENUM, + CTF_TYPE_STRING, + CTF_TYPE_STRUCT, + CTF_TYPE_UNTAGGED_VARIANT, + CTF_TYPE_VARIANT, + CTF_TYPE_ARRAY, + CTF_TYPE_SEQUENCE, + NR_CTF_TYPES. + +Depending on the field type, we can get data on its value using the appropriate +function: + * bt_ctf_get_index() return the element at the index + position of an array of a sequence; + + * bt_ctf_get_array_len() return the len of an array; + + * bt_ctf_get_int_signedness() return the signedness of an integer; + + * bt_ctf_get_int_base() return the base of an integer; + + * bt_ctf_get_int_byte_order() return the byte order of an integer; + + * bt_ctf_get_int_len() return the size in bits of an integer; + + * bt_ctf_get_encoding() return the encoding of an int or a + string defined in the + ctf_string_encoding enum: + CTF_STRING_NONE = 0, + CTF_STRING_UTF8, + CTF_STRING_ASCII, + CTF_STRING_UNKNOWN. + +To obtain the value associated with a field, the appropriate function of these +is to be used: + + * bt_ctf_get_uint64(); + * bt_ctf_get_int64(); + * bt_ctf_get_char_array(); + * bt_ctf_get_string(). + +If the field does not exist or is not of the type requested, the value returned +with these four functions is undefined. To check if an error occured, use the +bt_ctf_field_get_error() function after accessing a field. If no error occured, +the function will return 0. + +It is also possible to access the declaration fields, the same way as the +definition ones. bt_ctf_get_event_decl_list() sets a list to an array of +bt_ctf_event_decl pointers and bt_ctf_get_event_decl_fields() sets a list to an +array of bt_ctf_field_decl pointers. From the first type, the name of the +event can be obtained with bt_ctf_get_decl_event_name(). For the second type, +the field decl name is obtained with bt_ctf_get_decl_field_name(). + + +Callback: + +The other alternative to reading a trace is with the use of callbacks. This is +done with the bt_ctf_iter_add_callback() function. It still requires a valid +ctf iterator as the first argument. Here are all arguments: + + @iter: trace collection iterator (input) + @event: event to target. 0 for all events. + @private_data: private data pointer to pass to the callback + @flags: specific flags controlling the behavior of this callback + (or'd). + @callback: function pointer to call + @depends: struct bt_dependency detailing the required computation + results. Ends with 0. + @weak_depends: struct bt_dependency detailing the optional computation + results that can be optionally consumed by this + callback. + @provides: struct bt_dependency detailing the computation results + provided by this callback. + Ends with 0. + +"depends", "weak_depends" and "provides" memory is handled by the babeltrace +library after this call succeeds or fails. These objects can still be used by +the caller until the babeltrace iterator is destroyed, but they belong to the +babeltrace library. + +Note: the dependency graph is calculated when bt_ctf_iter_read_event() is +executed after a bt_ctf_iter_add_callback(). Beware that it is valid to +create/add callbacks/read/add more callbacks/read some more. + +The callback function passed to bt_ctf_iter_add_callback() must return a +bt_cb_ret value: + BT_CB_OK = 0, + BT_CB_OK_STOP = 1, + BT_CB_ERROR_STOP = 2, + BT_CB_ERROR_CONTINUE = 3. + + +Trace handle: + +When a trace is added to a context, bt_context_add_trace() returns a trace +handle id. This id is associated with its corresponding trace handle. With +that id, it is possible to manipulate directly the trace file. + + * bt_trace_handle_get_path() + -> returns the path of the trace handle (path to the trace). + + * bt_trace_handle_get_timestamp_begin() + * bt_trace_handle_get_timestamp_end() + -> return the creation/destruction timestamps (in ns or cycles + depending on the type specified) of the buffers of a + trace. + + * bt_ctf_event_get_handle_id() + -> returns the handle id associated with an event. + + +For more information on CTF, see the CTF documentation (git://git.efficios.com/ctf.git). -- 1.7.9.5 From laijs at cn.fujitsu.com Thu Aug 9 23:23:02 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Fri, 10 Aug 2012 11:23:02 +0800 Subject: [lttng-dev] [PATCH 2/2] urcu: add notice to URCU_TLS() for it is not async-signal-safe In-Reply-To: <20120809200211.GA14409@Krystal> References: <1344414673-14714-1-git-send-email-laijs@cn.fujitsu.com> <1344414673-14714-2-git-send-email-laijs@cn.fujitsu.com> <20120809141417.GC9064@Krystal> <20120809200211.GA14409@Krystal> Message-ID: <50247E96.9080909@cn.fujitsu.com> On 08/10/2012 04:02 AM, Mathieu Desnoyers wrote: > Looking at the result of a quick google search: > > http://curl.haxx.se/mail/lib-2006-09/0224.html > http://www.slamb.org/projects/sigsafe/api/patternref.html > > "Additionally, it makes the same assumption as all other methods for > handling thread-directed signals (with the exception of kevent(2) > handling), that pthread_getspecific(2) is async signal-safe. This is not > guaranteed by SUSv3." > > and > > https://groups.google.com/forum/?fromgroups#!topic/comp.os.linux.development/nZfmndKbzJw[1-25] > > it looks like using pthread_getspecific from a signal handler is not > always safe, mainly due to possible use of sigaltstack. So disabling > signals works for the "pthread_key_create" part, but we still have an > issue with pthread_getspecific. > > Ideas are welcome on how to best deal with this issue. What's the problem with disabling signals + pthread_getspecific()? Waht's the problem with disabling signals + __tls_access_ ## name()? From paulbarrette at gmail.com Fri Aug 10 10:02:16 2012 From: paulbarrette at gmail.com (Paul Barrette) Date: Fri, 10 Aug 2012 10:02:16 -0400 Subject: [lttng-dev] qemuarm: Session dameon died (exit status 0) Message-ID: Hi all, lttng-modules 2.0.4 lttng-tools 2.0.4 linux kernel 3.4.x I am running qemuarm and seeing the following issue when running "lttng create": Error: Session daemon died (exit status 0) Is there a patch I am missing for arm? Thanks Pb -- -- Paul Barrette Linked In Profile http://sitarplayer.net ? \\=^% (*)---(*) ^^~~^~~~^^^~~^ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mathieu.desnoyers at efficios.com Fri Aug 10 10:16:15 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 10 Aug 2012 10:16:15 -0400 Subject: [lttng-dev] [PATCH 2/2] urcu: add notice to URCU_TLS() for it is not async-signal-safe In-Reply-To: <50247E96.9080909@cn.fujitsu.com> References: <1344414673-14714-1-git-send-email-laijs@cn.fujitsu.com> <1344414673-14714-2-git-send-email-laijs@cn.fujitsu.com> <20120809141417.GC9064@Krystal> <20120809200211.GA14409@Krystal> <50247E96.9080909@cn.fujitsu.com> Message-ID: <20120810141615.GA27969@Krystal> (for the records: this discussion is about userspace RCU tls-compat.h, which uses TLS on systems supporting it, and fall back to pthread_key_create/getspecific/setspecific if not. The culprit of the issue is that we want to allow reading the tls-compat variable from signal handlers, and thus async-signal-safety of pthread_key_* and TLS. The code we refer to is: http://git.lttng.org/?p=userspace-rcu.git;a=blob;f=urcu/tls-compat.h;h=192a53609fb5f6bc445f98fdd6bc26918126687e;hb=HEAD) * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > On 08/10/2012 04:02 AM, Mathieu Desnoyers wrote: > > Looking at the result of a quick google search: > > > > http://curl.haxx.se/mail/lib-2006-09/0224.html > > http://www.slamb.org/projects/sigsafe/api/patternref.html > > > > "Additionally, it makes the same assumption as all other methods for > > handling thread-directed signals (with the exception of kevent(2) > > handling), that pthread_getspecific(2) is async signal-safe. This is not > > guaranteed by SUSv3." > > > > and > > > > https://groups.google.com/forum/?fromgroups#!topic/comp.os.linux.development/nZfmndKbzJw[1-25] > > > > it looks like using pthread_getspecific from a signal handler is not > > always safe, mainly due to possible use of sigaltstack. So disabling > > signals works for the "pthread_key_create" part, but we still have an > > issue with pthread_getspecific. > > > > Ideas are welcome on how to best deal with this issue. > > What's the problem with disabling signals + pthread_getspecific()? > Waht's the problem with disabling signals + __tls_access_ ## name()? * pthread_key_* fallback Disabling signals would allow us to be reentrant with respect to signals, which actually solves part of the problem (reentrancy) for the pthread_key_create part (protected by lock). Disabling signals, AFAIU, (and if we disregard the SUSv3 standard for a minute) should not be strictly required around the pthread_getspecific call on most architectures (no lock taken). We should carefully review the implementations for each architecture we support if we want to assume this though. If the getspecific returns NULL. we should disable signals and call pthread_getspecific again with signals disabled, and then call pthread_setspecific if necessary, before re-enabling signals. The benefit of not _always_ disabling signals around pthread_getspecific() is significant gain in performance in the common case: the entire hot path of rcu_read_lock/unlock all happens in userspace, without any system call, and uses tls-compat variables. The other part of the problem with pthread_getspecific and signal handlers is that it does not seem to be SUSv3-compliant to use pthread_getspecific() from within a signal handler. One example that can lead to problems is if the signal handler is setup with sigaltstack(2). We might want to simply document this limitation: "RCU read-side critical sections can be used in signals handlers, except those setup with sigaltstack(2)." * TLS Userspace RCU always touch the TLS variables from thread context (from within rcu_register_thread()) before they are allowed to be touched by signal handlers nested over threads. This ensures that issues with lazy binding and dynamic linker lock are not encountered (ref. http://sourceware.org/ml/libc-alpha/2012-06/msg00372.html). I did the same within my LTTng-UST use of TLS variables: they are touched by a constructor once so we don't run into deadlocks between UST lock and the libc lock protecting dynamic linking (recursive mutex also taken around the constructor calls, within which we needed to take the UST lock, thus causing deadlocks). Feedback is welcome, Thanks, Mathieu -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From yannick.brosseau at gmail.com Fri Aug 10 10:31:20 2012 From: yannick.brosseau at gmail.com (Yannick Brosseau) Date: Fri, 10 Aug 2012 10:31:20 -0400 Subject: [lttng-dev] qemuarm: Session dameon died (exit status 0) In-Reply-To: References: Message-ID: <50251B38.8040205@gmail.com> Hi, Can you try to start the lttng-sessiond manually and see if there is error message (with full logging enabled: lttng-sessiond -vvv) Yannick On 2012-08-10 10:02, Paul Barrette wrote: > Hi all, > lttng-modules 2.0.4 > lttng-tools 2.0.4 > linux kernel 3.4.x > > I am running qemuarm and seeing the following issue when running > "lttng create": > Error: Session daemon died (exit status 0) > > Is there a patch I am missing for arm? > > Thanks > Pb > > -- > -- > Paul Barrette > Linked In Profile > http://sitarplayer.net > O" > \\=^% > (*)---(*) > ^^~~^~~~^^^~~^ > > > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From paulbarrette at gmail.com Fri Aug 10 10:40:03 2012 From: paulbarrette at gmail.com (Paul Barrette) Date: Fri, 10 Aug 2012 10:40:03 -0400 Subject: [lttng-dev] qemuarm: Session dameon died (exit status 0) In-Reply-To: <50251B38.8040205@gmail.com> References: <50251B38.8040205@gmail.com> Message-ID: On Fri, Aug 10, 2012 at 10:31 AM, Yannick Brosseau < yannick.brosseau at gmail.com> wrote: > Hi, > > Can you try to start the lttng-sessiond manually and see if there is error > message (with full logging enabled: lttng-sessiond -vvv) > > Hi, # lttng-sessiond -ww Segmentation fault. Pb > Yannick > > > On 2012-08-10 10:02, Paul Barrette wrote: > > Hi all, > lttng-modules 2.0.4 > lttng-tools 2.0.4 > linux kernel 3.4.x > > I am running qemuarm and seeing the following issue when running "lttng > create": > Error: Session daemon died (exit status 0) > > Is there a patch I am missing for arm? > > Thanks > Pb > > -- > -- > Paul Barrette > Linked In Profile > http://sitarplayer.net > ? > \\=^% > (*)---(*) > ^^~~^~~~^^^~~^ > > > > > _______________________________________________ > lttng-dev mailing listlttng-dev at lists.lttng.orghttp://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > > -- -- Paul Barrette Linked In Profile http://sitarplayer.net ? \\=^% (*)---(*) ^^~~^~~~^^^~~^ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dgoulet at efficios.com Fri Aug 10 10:43:21 2012 From: dgoulet at efficios.com (David Goulet) Date: Fri, 10 Aug 2012 10:43:21 -0400 Subject: [lttng-dev] qemuarm: Session dameon died (exit status 0) In-Reply-To: References: <50251B38.8040205@gmail.com> Message-ID: <50251E09.6010703@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 That's interesting! Could you send us the gdb backtrace $ gdb lttng-sessiond gdb> r gdb> bt full And seems you use "-ww" instead of "-vvv" ... I doubt the segfault is in the option parsing but still. Thanks! David Paul Barrette: > > > On Fri, Aug 10, 2012 at 10:31 AM, Yannick Brosseau > > > wrote: > > Hi, > > Can you try to start the lttng-sessiond manually and see if there > is error message (with full logging enabled: lttng-sessiond -vvv) > > Hi, # lttng-sessiond -ww Segmentation fault. > > > Pb > > Yannick > > > On 2012-08-10 10:02, Paul Barrette wrote: >> Hi all, lttng-modules 2.0.4 lttng-tools 2.0.4 linux kernel 3.4.x >> >> I am running qemuarm and seeing the following issue when >> running "lttng create": Error: Session daemon died (exit status >> 0) >> >> Is there a patch I am missing for arm? >> >> Thanks Pb >> >> -- -- Paul Barrette Linked In Profile >> >> http://sitarplayer.net ? \\=^% (*)---(*) ^^~~^~~~^^^~~^ >> >> >> >> >> _______________________________________________ lttng-dev mailing >> list lttng-dev at lists.lttng.org >> >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > > > > -- -- Paul Barrette Linked In Profile > > http://sitarplayer.net ? \\=^% (*)---(*) ^^~~^~~~^^^~~^ > > > > > This body part will be downloaded on demand. -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQJR4JAAoJEELoaioR9I02n+IIALs8mfV9uxPqTqn7w0aEWhlG M1RYNwP5nPcaqYeZxptaMR3N5jziPTQZPiVgUZWGlapcguTHmi8gvYkcrYuQnBPs 58P57eCAwo43SX1j5JHNWq6NfP+TsslrlyUESDXNlzLLk2YcZ2uOPrpIby1q0TyP wpSLGtpWhWXRD71KdY0kKChQ6UOWQZiDNzrbtz9wc4UWiOqcLr9Mf8BXu+qN/Czh sUrU3HTN2nMn6pG6RopXcqVEHvSw1KKrIXqqcSVCnYsmveEbtIkXvYdYeiQmEFlO L99VBC8CBva1zh3nkA8KriE94CA0j5WL7oFbjM7F6q3EAVaNw+g668zHkWdocJE= =9I9j -----END PGP SIGNATURE----- From paulmck at linux.vnet.ibm.com Fri Aug 10 11:49:23 2012 From: paulmck at linux.vnet.ibm.com (Paul E. McKenney) Date: Fri, 10 Aug 2012 08:49:23 -0700 Subject: [lttng-dev] [PATCH 2/2] urcu: new wfqueue implementation In-Reply-To: <1344501986-23151-2-git-send-email-laijs@cn.fujitsu.com> References: <1344501986-23151-1-git-send-email-laijs@cn.fujitsu.com> <1344501986-23151-2-git-send-email-laijs@cn.fujitsu.com> Message-ID: <20120810154923.GA9303@linux.vnet.ibm.com> On Thu, Aug 09, 2012 at 04:46:26PM +0800, Lai Jiangshan wrote: > Some guys would be surprised by this fact: > There are already TWO implementations of wfqueue in urcu. > > The first one is in urcu/static/wfqueue.h: > 1) enqueue: exchange the tail and then update previous->next > 2) dequeue: wait for first node's next pointer and them shift, a dummy node > is introduced to avoid the queue->tail become NULL when shift. > > The second one shares some code with the first one, and the left code > are spreading in urcu-call-rcu-impl.h: > 1) enqueue: share with the first one > 2) no dequeue operation: and no shift, so it don't need dummy node, > Although the dummy node is queued when initialization, but it is removed > after the first dequeue_all operation in call_rcu_thread(). > call_rcu_data_free() forgets to handle the dummy node if it is not removed. > 3)dequeue_all: record the old head and tail, and queue->head become the special > tail node.(atomic record the tail and change the tail). > > The second implementation's code are spreading, bad for review, and it is not > tested by tests/test_urcu_wfq. > > So we need a better implementation avoid the dummy node dancing and can service > both generic wfqueue APIs and dequeue_all API for call rcu. > > The new implementation: > 1) enqueue: share with the first one/original implementation. > 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. > no dummy node, save memory. > 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail > and return the old head.next. Looks good at first glance! The reason for xchg and the dummy node is old history involving parallel systems based on 80386, which did not have a cmpxchg instruction. ;-) Thanx, Paul > More implementation details are in the code. > tests/test_urcu_wfq will be update in future for testing new APIs. > > > Signed-off-by: Lai Jiangshan > --- > urcu-call-rcu-impl.h | 50 ++++++++++-------------- > urcu/static/wfqueue.h | 104 ++++++++++++++++++++++++++++++++++++------------ > urcu/wfqueue.h | 25 ++++++++++-- > wfqueue.c | 29 ++++++++++++++ > 4 files changed, 149 insertions(+), 59 deletions(-) > > diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h > index 13b24ff..dbfb410 100644 > --- a/urcu-call-rcu-impl.h > +++ b/urcu-call-rcu-impl.h > @@ -221,7 +221,7 @@ static void *call_rcu_thread(void *arg) > { > unsigned long cbcount; > struct cds_wfq_node *cbs; > - struct cds_wfq_node **cbs_tail; > + struct cds_wfq_node *cbs_tail; > struct call_rcu_data *crdp = (struct call_rcu_data *)arg; > struct rcu_head *rhp; > int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); > @@ -243,24 +243,18 @@ static void *call_rcu_thread(void *arg) > cmm_smp_mb(); > } > for (;;) { > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > - poll(NULL, 0, 1); > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > - cbs_tail = (struct cds_wfq_node **) > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > + cbs = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &cbs_tail); > + if (cbs) { > synchronize_rcu(); > cbcount = 0; > do { > - while (cbs->next == NULL && > - &cbs->next != cbs_tail) > - poll(NULL, 0, 1); > - if (cbs == &crdp->cbs.dummy) { > - cbs = cbs->next; > - continue; > - } > rhp = (struct rcu_head *)cbs; > - cbs = cbs->next; > + > + if (cbs != cbs_tail) > + cbs = __cds_wfq_node_sync_next(cbs); > + else > + cbs = NULL; > + > rhp->func(rhp); > cbcount++; > } while (cbs != NULL); > @@ -270,8 +264,7 @@ static void *call_rcu_thread(void *arg) > break; > rcu_thread_offline(); > if (!rt) { > - if (&crdp->cbs.head > - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { > + if (cds_wfq_empty(&crdp->cbs)) { > call_rcu_wait(crdp); > poll(NULL, 0, 10); > uatomic_dec(&crdp->futex); > @@ -625,32 +618,31 @@ void call_rcu(struct rcu_head *head, > */ > void call_rcu_data_free(struct call_rcu_data *crdp) > { > - struct cds_wfq_node *cbs; > - struct cds_wfq_node **cbs_tail; > - struct cds_wfq_node **cbs_endprev; > + struct cds_wfq_node *head, *tail; > > if (crdp == NULL || crdp == default_call_rcu_data) { > return; > } > + > if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { > uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); > wake_call_rcu_thread(crdp); > while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) > poll(NULL, 0, 1); > } > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > - poll(NULL, 0, 1); > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > - cbs_tail = (struct cds_wfq_node **) > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > + > + if (!cds_wfq_empty(&crdp->cbs)) { > + head = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &tail); > + assert(head); > + > /* Create default call rcu data if need be */ > (void) get_default_call_rcu_data(); > - cbs_endprev = (struct cds_wfq_node **) > - uatomic_xchg(&default_call_rcu_data, cbs_tail); > - *cbs_endprev = cbs; > + > + __cds_wfq_append_list(&default_call_rcu_data->cbs, head, tail); > + > uatomic_add(&default_call_rcu_data->qlen, > uatomic_read(&crdp->qlen)); > + > wake_call_rcu_thread(default_call_rcu_data); > } > > diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h > index 636e1af..15ea9fc 100644 > --- a/urcu/static/wfqueue.h > +++ b/urcu/static/wfqueue.h > @@ -10,6 +10,7 @@ > * dynamically with the userspace rcu library. > * > * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -29,6 +30,7 @@ > #include > #include > #include > +#include > #include > #include > > @@ -38,8 +40,6 @@ extern "C" { > > /* > * Queue with wait-free enqueue/blocking dequeue. > - * This implementation adds a dummy head node when the queue is empty to ensure > - * we can always update the queue locklessly. > * > * Inspired from half-wait-free/half-blocking queue implementation done by > * Paul E. McKenney. > @@ -57,31 +57,43 @@ static inline void _cds_wfq_init(struct cds_wfq_queue *q) > { > int ret; > > - _cds_wfq_node_init(&q->dummy); > /* Set queue head and tail */ > - q->head = &q->dummy; > - q->tail = &q->dummy.next; > + _cds_wfq_node_init(&q->head); > + q->tail = &q->head; > ret = pthread_mutex_init(&q->lock, NULL); > assert(!ret); > } > > -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > - struct cds_wfq_node *node) > +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) > { > - struct cds_wfq_node **old_tail; > + /* > + * Queue is empty if no node is pointed by q->head.next nor q->tail. > + */ > + return q->head.next == NULL && CMM_LOAD_SHARED(q->tail) == &q->head; > +} > > +static inline void ___cds_wfq_append_list(struct cds_wfq_queue *q, > + struct cds_wfq_node *head, struct cds_wfq_node *tail) > +{ > /* > * uatomic_xchg() implicit memory barrier orders earlier stores to data > * structure containing node and setting node->next to NULL before > * publication. > */ > - old_tail = uatomic_xchg(&q->tail, &node->next); > + tail = uatomic_xchg(&q->tail, tail); > + > /* > - * At this point, dequeuers see a NULL old_tail->next, which indicates > + * At this point, dequeuers see a NULL tail->next, which indicates > * that the queue is being appended to. The following store will append > * "node" to the queue from a dequeuer perspective. > */ > - CMM_STORE_SHARED(*old_tail, node); > + CMM_STORE_SHARED(tail->next, head); > +} > + > +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > + struct cds_wfq_node *node) > +{ > + ___cds_wfq_append_list(q, node, node); > } > > /* > @@ -120,27 +132,46 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > { > struct cds_wfq_node *node, *next; > > - /* > - * Queue is empty if it only contains the dummy node. > - */ > - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) > + if (_cds_wfq_empty(q)) > return NULL; > - node = q->head; > > - next = ___cds_wfq_node_sync_next(node); > + node = ___cds_wfq_node_sync_next(&q->head); > + > + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { > + if (CMM_LOAD_SHARED(q->tail) == node) { > + /* > + * @node is the only node in the queue. > + * Try to move the tail to &q->head > + */ > + _cds_wfq_node_init(&q->head); > + if (uatomic_cmpxchg(&q->tail, node, &q->head) == node) > + return node; > + } > + next = ___cds_wfq_node_sync_next(node); > + } > > /* > * Move queue head forward. > */ > - q->head = next; > - /* > - * Requeue dummy node if we just dequeued it. > - */ > - if (node == &q->dummy) { > - _cds_wfq_node_init(node); > - _cds_wfq_enqueue(q, node); > - return ___cds_wfq_dequeue_blocking(q); > - } > + q->head.next = next; > + > + return node; > +} > + > +/* dequeue all nodes, the nodes are not synchronized for the next pointer */ > +static inline struct cds_wfq_node * > +___cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, > + struct cds_wfq_node **tail) > +{ > + struct cds_wfq_node *node; > + > + if (_cds_wfq_empty(q)) > + return NULL; > + > + node = ___cds_wfq_node_sync_next(&q->head); > + _cds_wfq_node_init(&q->head); > + *tail = uatomic_xchg(&q->tail, &q->head); > + > return node; > } > > @@ -158,6 +189,27 @@ _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > return retnode; > } > > +static inline struct cds_wfq_node * > +_cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, > + struct cds_wfq_node **tail) > +{ > + struct cds_wfq_node *node, *next; > + int ret; > + > + ret = pthread_mutex_lock(&q->lock); > + assert(!ret); > + node = ___cds_wfq_dequeue_all_blocking(q, tail); > + ret = pthread_mutex_unlock(&q->lock); > + assert(!ret); > + > + /* synchronize all nodes' next pointer */ > + next = node; > + while (next != *tail) > + next = ___cds_wfq_node_sync_next(next); > + > + return node; > +} > + > #ifdef __cplusplus > } > #endif > diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h > index 03a73f1..985f540 100644 > --- a/urcu/wfqueue.h > +++ b/urcu/wfqueue.h > @@ -7,6 +7,7 @@ > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > * > * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -25,6 +26,7 @@ > > #include > #include > +#include > #include > > #ifdef __cplusplus > @@ -33,8 +35,6 @@ extern "C" { > > /* > * Queue with wait-free enqueue/blocking dequeue. > - * This implementation adds a dummy head node when the queue is empty to ensure > - * we can always update the queue locklessly. > * > * Inspired from half-wait-free/half-blocking queue implementation done by > * Paul E. McKenney. > @@ -45,8 +45,7 @@ struct cds_wfq_node { > }; > > struct cds_wfq_queue { > - struct cds_wfq_node *head, **tail; > - struct cds_wfq_node dummy; /* Dummy node */ > + struct cds_wfq_node head, *tail; > pthread_mutex_t lock; > }; > > @@ -56,18 +55,36 @@ struct cds_wfq_queue { > > #define cds_wfq_node_init _cds_wfq_node_init > #define cds_wfq_init _cds_wfq_init > +#define cds_wfq_empty _cds_wfq_empty > +#define __cds_wfq_append_list ___cds_wfq_append_list > #define cds_wfq_enqueue _cds_wfq_enqueue > #define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking > #define cds_wfq_dequeue_blocking _cds_wfq_dequeue_blocking > +#define __cds_wfq_node_sync_next ___cds_wfq_node_sync_next > +#define __cds_wfq_dequeue_all_blocking ___cds_wfq_dequeue_all_blocking > +#define cds_wfq_dequeue_all_blocking _cds_wfq_dequeue_all_blocking > > #else /* !_LGPL_SOURCE */ > > extern void cds_wfq_node_init(struct cds_wfq_node *node); > extern void cds_wfq_init(struct cds_wfq_queue *q); > +extern bool cds_wfq_empty(struct cds_wfq_queue *q); > +/* __cds_wfq_append_list: caller ensures mutual exclusion between dequeues */ > +extern void __cds_wfq_append_list(struct cds_wfq_queue *q, > + struct cds_wfq_node *head, struct cds_wfq_node *tail); > extern void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node); > /* __cds_wfq_dequeue_blocking: caller ensures mutual exclusion between dequeues */ > extern struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); > extern struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); > +extern struct cds_wfq_node *__cds_wfq_node_sync_next(struct cds_wfq_node *node); > +/* > + * __cds_wfq_dequeue_all_blocking: caller ensures mutual exclusion between > + * dequeues, and need synchronize next pointer berfore use it. > + */ > +extern struct cds_wfq_node *__cds_wfq_dequeue_all_blocking( > + struct cds_wfq_queue *q, struct cds_wfq_node **tail); > +extern struct cds_wfq_node *cds_wfq_dequeue_all_blocking( > + struct cds_wfq_queue *q, struct cds_wfq_node **tail); > > #endif /* !_LGPL_SOURCE */ > > diff --git a/wfqueue.c b/wfqueue.c > index 3337171..28a7b58 100644 > --- a/wfqueue.c > +++ b/wfqueue.c > @@ -4,6 +4,7 @@ > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > * > * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -38,6 +39,17 @@ void cds_wfq_init(struct cds_wfq_queue *q) > _cds_wfq_init(q); > } > > +bool cds_wfq_empty(struct cds_wfq_queue *q) > +{ > + return _cds_wfq_empty(q); > +} > + > +void __cds_wfq_append_list(struct cds_wfq_queue *q, > + struct cds_wfq_node *head, struct cds_wfq_node *tail) > +{ > + return ___cds_wfq_append_list(q, head, tail); > +} > + > void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node) > { > _cds_wfq_enqueue(q, node); > @@ -52,3 +64,20 @@ struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > { > return _cds_wfq_dequeue_blocking(q); > } > + > +struct cds_wfq_node *__cds_wfq_node_sync_next(struct cds_wfq_node *node) > +{ > + return ___cds_wfq_node_sync_next(node); > +} > + > +struct cds_wfq_node *__cds_wfq_dequeue_all_blocking( > + struct cds_wfq_queue *q, struct cds_wfq_node **tail) > +{ > + return ___cds_wfq_dequeue_all_blocking(q, tail); > +} > + > +struct cds_wfq_node *cds_wfq_dequeue_all_blocking( > + struct cds_wfq_queue *q, struct cds_wfq_node **tail) > +{ > + return _cds_wfq_dequeue_all_blocking(q, tail); > +} > -- > 1.7.7 > From francis.deslauriers at polymtl.ca Fri Aug 10 11:49:54 2012 From: francis.deslauriers at polymtl.ca (Francis Deslauriers) Date: Fri, 10 Aug 2012 11:49:54 -0400 Subject: [lttng-dev] [Babeltrace PATCH] Add BT_SEEK_LAST type to bt_iter_pos. Message-ID: <1344613794-17669-1-git-send-email-francis.deslauriers@polymtl.ca> Signed-off-by: Francis Deslauriers --- include/babeltrace/iterator.h | 1 + lib/iterator.c | 189 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 190 insertions(+) diff --git a/include/babeltrace/iterator.h b/include/babeltrace/iterator.h index aa6470e..c13055d 100644 --- a/include/babeltrace/iterator.h +++ b/include/babeltrace/iterator.h @@ -48,6 +48,7 @@ struct bt_iter_pos { BT_SEEK_CUR, BT_SEEK_BEGIN, BT_SEEK_END, + BT_SEEK_LAST, } type; union { uint64_t seek_time; diff --git a/lib/iterator.c b/lib/iterator.c index bcb77d8..875c93d 100644 --- a/lib/iterator.c +++ b/lib/iterator.c @@ -187,6 +187,176 @@ static int seek_ctf_trace_by_timestamp(struct ctf_trace *tin, return found ? 0 : EOF; } +int seek_last_element_ctf_file_stream(struct trace_collection *tc, + struct ctf_file_stream **cfs, int max_packet, + int max_stream_id, int max_stream_class_id, + int max_trace_descriptor_id, uint64_t max_timestamp) +{ + int ret; + struct trace_descriptor *max_td_read; + struct ctf_trace *max_tin; + struct ctf_stream_declaration *max_stream_class; + struct ctf_stream_definition *max_stream; + struct ctf_stream_pos *max_stream_pos; + + max_td_read = g_ptr_array_index(tc->array, + max_trace_descriptor_id); + max_tin = container_of(max_td_read, + struct ctf_trace, parent); + max_stream_class = g_ptr_array_index(max_tin->streams, + max_stream_class_id); + max_stream = g_ptr_array_index(max_stream_class->streams, + max_stream_id); + *cfs = container_of(max_stream, + struct ctf_file_stream, parent); + max_stream_pos = &(*cfs)->pos; + /* we seek to the last packet of the stream.*/ + max_stream_pos->packet_seek(&max_stream_pos->parent, + max_packet, + SEEK_SET); + /* + * iterate over every event until we reach on an event that + * its timestamp correspond with the max saved previously. + */ + do { + ret = stream_read_event(*cfs); + } while ((*cfs)->parent.real_timestamp != max_timestamp && ret == 0); + + /* We insert the stream into the heap. */ + return ret; +} + +int find_last_event(struct ctf_file_stream *cfs, + uint64_t *timestamp_end, + int *packet) +{ + int ret = 0, count = 0, event_read = 0; + int i; + uint64_t tmp = 0; + struct ctf_stream_pos *stream_pos; + stream_pos = &cfs->pos; + + /* + * we start by the last packet as the current one. + * If the current one is empty we go back one packet if possible. + */ + i = stream_pos->packet_real_index->len - 1; + /* + * Check if we have iterated on all the packets. + * If we are not short on packets, we check what + * made us leave the reading event loop. + */ + for (; i >= 0 && count == 0; i--) { + stream_pos->packet_seek(&stream_pos->parent, i, SEEK_SET); + count = 0; + /* read each event until we reach the end of the packet */ + do { + tmp = cfs->parent.real_timestamp; + ret = stream_read_event(cfs); + if (ret == 0) { + count++; + } + } while (ret == 0); + + /* Error */ + if (ret > 0) { + goto end; + } + event_read += count; + } + *packet = i; + *timestamp_end = tmp; + + /* Check if we read at least one event on the stream */ + if (event_read <= 1) { + ret = 1; + goto end; + } + ret = 0; + +end: + return ret; +} + +int find_max_timestamp_ctf_file_stream( + struct ctf_stream_declaration *stream_class, int *max_stream_id, + int *packet, uint64_t *max_timestamp, int *new_max) +{ + struct ctf_file_stream *cfs; + int max_packet = 0, ret = 0, error = 1; + int i; + uint64_t savedtime; + + for (i = 0; i < stream_class->streams->len; i++) { + struct ctf_stream_definition *stream; + stream = g_ptr_array_index( + stream_class->streams, i); + if (!stream) + continue; + cfs = container_of(stream, struct ctf_file_stream, parent); + ret = find_last_event(cfs, &savedtime, &max_packet); + /* Can return either EOF, 0, or error (> 0). */ + if (ret == 0 && savedtime >= *max_timestamp) { + *new_max = 1; + *max_stream_id = i; + *packet = max_packet; + *max_timestamp = savedtime; + } + error = error & ret; + } + return error; +} + +int seek_last_ctf_file_stream(struct trace_collection *tc, + struct ctf_file_stream **cfs) +{ + int i, j, ret, new_max_found, max_packet, max_stream_id, + max_stream_class_id, max_trace_descriptor_id; + int found = 0; + uint64_t max_timestamp = 0; + + /* For each trace in the trace_collection*/ + for (i = 0; i < tc->array->len; i++) { + struct ctf_trace *tin; + struct trace_descriptor *td_read; + td_read = g_ptr_array_index(tc->array, i); + if (!td_read) + continue; + tin = container_of(td_read, struct ctf_trace, parent); + /* For each stream_class in the trace*/ + for (j = 0; j < tin->streams->len; j++) { + struct ctf_stream_declaration *stream_class; + + stream_class = g_ptr_array_index(tin->streams, j); + if (!stream_class) + continue; + /* For each file_stream in the stream_class */ + new_max_found = 0; + ret = find_max_timestamp_ctf_file_stream(stream_class, + &max_stream_id, &max_packet, + &max_timestamp, &new_max_found); + if (!ret && new_max_found) { + found = 1; + max_trace_descriptor_id = i; + max_stream_class_id = j; + } + } + } + /* + * Now we know in which trace, stream_class and stream is the + * last event of the trace_collection. + * We can seek to this targeted event. + */ + if (!found) { + ret = -1; + } else { + ret = seek_last_element_ctf_file_stream(tc, cfs, max_packet, + max_stream_id, max_stream_class_id, + max_trace_descriptor_id, max_timestamp); + } + return ret; +} + int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) { struct trace_collection *tc; @@ -326,6 +496,25 @@ int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) } } break; + case BT_SEEK_LAST: + { + struct ctf_file_stream *cfs; + tc = iter->ctx->tc; + + ret = seek_last_ctf_file_stream(tc, &cfs); + if (ret < 0) + goto error; + /* remove all stream from the heap*/ + heap_free(iter->stream_heap); + /* Create a new empty heap*/ + ret = heap_init(iter->stream_heap, 0, stream_compare); + if (ret < 0) + goto error; + /* Insert the stream that contains the last event. */ + heap_insert(iter->stream_heap, cfs); + + return 0; + } default: /* not implemented */ return -EINVAL; -- 1.7.9.5 From jdesfossez at efficios.com Fri Aug 10 12:29:02 2012 From: jdesfossez at efficios.com (Julien Desfossez) Date: Fri, 10 Aug 2012 12:29:02 -0400 Subject: [lttng-dev] =?utf-8?q?=5Bbabeltrace_PATCH=5D_API_documentation_fi?= =?utf-8?b?bGXigI8=?= In-Reply-To: <1344548378-22039-1-git-send-email-danny.serres@efficios.com> References: <1344548378-22039-1-git-send-email-danny.serres@efficios.com> Message-ID: <502536CE.6060408@efficios.com> Hi Danny, Thanks for the documentation, it is great ! I added some comments below. On 09/08/12 05:39 PM, Danny Serres wrote: > Signed-off-by: Danny Serres > --- > doc/API.txt | 237 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 237 insertions(+) > create mode 100644 doc/API.txt > > diff --git a/doc/API.txt b/doc/API.txt > new file mode 100644 > index 0000000..8cb7e31 > --- /dev/null > +++ b/doc/API.txt > @@ -0,0 +1,237 @@ > +Babeltrace API documentation > + > +Babeltrace is a trace viewer and converter reading and writing the > +Common Trace Format (CTF). Its main use is to pretty-print CTF traces > +into a human-readable text output. (taken from the babeltrace README) : Babeltrace provides trace read and write libraries, as well as a trace converter. A plugin can be created for any trace format to allow its conversion to/from another trace format. The main format expected to be converted to/from is the Common Trace Format (CTF). The latest version of the CTF specification can be found at: git tree: git://git.efficios.com/ctf.git gitweb: http://git.efficios.com/?p=ctf.git > + > +This document describes the main concepts to use the libbabeltrace, > +which exposes the Babeltrace trace reading capability. > + > + > +TODO: add explanation for each scope (line 116) You can just add a reference to the CTF spec for that. > + > + > +TERMINOLOGY > +??????????????? > + > +* A "callback" is a reference to a piece of executable code (such as a > + function) that is passed as an argument to another piece of code > + (like another function). > + > +* A "context" is a structure that represents an object in which a trace > + collection is openned. opened > + > +* An "iterator" is a structure that enables the user to traverse a trace. > + > +* A "trace handle" is a unique identifier representing a trace file. > + It allows the user to manipulate a trace file directly. a trace, not trace file. > + > + > + > +USAGE > +??????????????? > + > +Context: > + > +In order to use libbabeltrace to read a trace, the first step is to create a > +context structure and to add a trace to it. This is done using the > +bt_context_create() and bt_context_add_trace() functions. As long as this > +context structure is allocated, the trace collection is open. (and the trace is valid) trace, not trace collection > + > +The context can be destroyed by calling one more bt_context_put() > +than bt_context_get(), functions which respectively decrement and increment > +the refcount of the context. These functions ensures that the context won't > +be destroyed when it is in use. > + > +Once a trace is added to the context, it can be read and seeked using iterators > +or callbacks. "and callbacks" (iterators are needed to read the trace, the callbacks are only there to read the events) > + > + > +Iterator: > + > +An iterator can be created using the bt_iter_create() function. > +As for now, only ctf iterator are supported. These are used to traverse a ctf- > +formatted trace. Such iterator can be created with bt_ctf_iter_create(). > + > +While creating an iterator, a begin and an end position may be specified. > +To do so, one or two struct bt_iter_pos must be passed. Such struct have 2 > +attributes: type and u. "type" is the seek type, can be either: > + BT_SEEK_TIME > + BT_SEEK_RESTORE > + BT_SEEK_CUR > + BT_SEEK_BEGIN > + BT_SEEK_END > +and "u" is a union of the seek time (if using BT_SEEK_TIME) and the restore > +position (if using BT_SEEK_RESTORE). > + > +Once the iterator is created, various functions become available. We have > +bt_ctf_iter_read_event() which returns the ctf event of the trace where the > +iterator is set. There is also bt_ctf_iter_destroy() which free the iterator. > +Note that only one iterator can be created against a context at the same time. > +If more than one iterator is being created for the same context, the second > +creation will return NULL. The previous iterator must be destroyed before > +creation of the new iterator. In the future, creation of multiples iterators > +will probably be allowed. > + > +Finally, we have the bt_ctf_get_iter() function which returns a struct bt_iter > +with which the iterator can be moved using one of these functions: > + bt_iter_next(), which moves the position to the next event > + bt_iter_set_pos(), which moves the position to the specified one > + > +To get the current position (struct bt_iter_pos) of the iterator, the function > +bt_iter_get_pos() can be used. To get an arbitrary position based on a > +specific time, bt_itet_create_time_pos() is the function to use. The position > +returned by one of these two functions must be freed with bt_iter_free_pos() > +after use. > + > + > +CTF Event: > + > +A CTF event is usually obtained from an iterator via the > +bt_ctf_iter_read_event() function, which returns a struct bt_ctf_event. Having > +the event, we can collect its data: > + * bt_ctf_event_name() will return the name of the event; > + * bt_ctf_get_timestamp() will return the timestamp of the event > + offsetted with the system clock source (in ns); > + * bt_ctf_get_cycles will return the timestamp of the event as written > + in the packet (in cycles). > + > +To access the event fields which contain various information, the first step is > +to obtain a scope definition for the desired field. This can be done using the > +bt_ctf_get_top_level_scope() function. This function provides the mapping > +between the enum and the actual definition of top-level scopes and returns > +a definition of the top-level scope. > +Top-level scopes are defined in the bt_ctf_scope enum: > + BT_TRACE_PACKET_HEADER = 0, > + BT_STREAM_PACKET_CONTEXT = 1, > + BT_STREAM_EVENT_HEADER = 2, > + BT_STREAM_EVENT_CONTEXT = 3, > + BT_EVENT_CONTEXT = 4, > + BT_EVENT_FIELDS = 5. > + > + > +In order to get a field or a field list, the user needs to pass a scope as > +argument, this scope can be a top-level scope or a scope relative to an > +arbitrary field. > + > +To get a field list, the bt_ctf_get_field_list() function is to be used. > +To get the definition of a specific field, the appropriate function would be > +bt_ctf_get_field(). > + > +Once the field is obtained, we can obtain its name and type using the > +bt_ctf_field_name() and bt_ctf_field_type() functions respectively. The > +possible types are defined in the ctf_type_id enum: > + CTF_TYPE_UNKNOWN = 0, > + CTF_TYPE_INTEGER, > + CTF_TYPE_FLOAT, > + CTF_TYPE_ENUM, > + CTF_TYPE_STRING, > + CTF_TYPE_STRUCT, > + CTF_TYPE_UNTAGGED_VARIANT, > + CTF_TYPE_VARIANT, > + CTF_TYPE_ARRAY, > + CTF_TYPE_SEQUENCE, > + NR_CTF_TYPES. > + > +Depending on the field type, we can get data on its value using the appropriate > +function: > + * bt_ctf_get_index() return the element at the index > + position of an array of a sequence; > + > + * bt_ctf_get_array_len() return the len of an array; > + > + * bt_ctf_get_int_signedness() return the signedness of an integer; > + > + * bt_ctf_get_int_base() return the base of an integer; > + > + * bt_ctf_get_int_byte_order() return the byte order of an integer; > + > + * bt_ctf_get_int_len() return the size in bits of an integer; > + > + * bt_ctf_get_encoding() return the encoding of an int or a > + string defined in the > + ctf_string_encoding enum: > + CTF_STRING_NONE = 0, > + CTF_STRING_UTF8, > + CTF_STRING_ASCII, > + CTF_STRING_UNKNOWN. > + > +To obtain the value associated with a field, the appropriate function of these > +is to be used: > + > + * bt_ctf_get_uint64(); > + * bt_ctf_get_int64(); > + * bt_ctf_get_char_array(); > + * bt_ctf_get_string(). > + > +If the field does not exist or is not of the type requested, the value returned > +with these four functions is undefined. To check if an error occured, use the > +bt_ctf_field_get_error() function after accessing a field. If no error occured, > +the function will return 0. > + > +It is also possible to access the declaration fields, the same way as the > +definition ones. bt_ctf_get_event_decl_list() sets a list to an array of > +bt_ctf_event_decl pointers and bt_ctf_get_event_decl_fields() sets a list to an > +array of bt_ctf_field_decl pointers. From the first type, the name of the > +event can be obtained with bt_ctf_get_decl_event_name(). For the second type, > +the field decl name is obtained with bt_ctf_get_decl_field_name(). The declaration functions allow the user to list the events, fields and contexts fields enabled in the trace once it is opened, whereas the definition functions apply on the current event being read. > + > + > +Callback: > + The iterator allow the user to read the trace, in order to access the events and fields, the user can either call the functions listed previously on each event, or register callbacks functions that are called when specific (or all) events are read. > +The other alternative to reading a trace is with the use of callbacks. This is > +done with the bt_ctf_iter_add_callback() function. It still requires a valid > +ctf iterator as the first argument. Here are all arguments: > + > + @iter: trace collection iterator (input) > + @event: event to target. 0 for all events. > + @private_data: private data pointer to pass to the callback > + @flags: specific flags controlling the behavior of this callback > + (or'd). > + @callback: function pointer to call > + @depends: struct bt_dependency detailing the required computation > + results. Ends with 0. > + @weak_depends: struct bt_dependency detailing the optional computation > + results that can be optionally consumed by this > + callback. > + @provides: struct bt_dependency detailing the computation results > + provided by this callback. > + Ends with 0. > + > +"depends", "weak_depends" and "provides" memory is handled by the babeltrace > +library after this call succeeds or fails. These objects can still be used by > +the caller until the babeltrace iterator is destroyed, but they belong to the > +babeltrace library. > + > +Note: the dependency graph is calculated when bt_ctf_iter_read_event() is > +executed after a bt_ctf_iter_add_callback(). Beware that it is valid to > +create/add callbacks/read/add more callbacks/read some more. As of now the flags and dependencies are not used, the callbacks are processed in FIFO order. > + > +The callback function passed to bt_ctf_iter_add_callback() must return a > +bt_cb_ret value: > + BT_CB_OK = 0, > + BT_CB_OK_STOP = 1, > + BT_CB_ERROR_STOP = 2, > + BT_CB_ERROR_CONTINUE = 3. > + > + > +Trace handle: > + > +When a trace is added to a context, bt_context_add_trace() returns a trace > +handle id. This id is associated with its corresponding trace handle. With > +that id, it is possible to manipulate directly the trace file. trace > + > + * bt_trace_handle_get_path() > + -> returns the path of the trace handle (path to the trace). > + > + * bt_trace_handle_get_timestamp_begin() > + * bt_trace_handle_get_timestamp_end() > + -> return the creation/destruction timestamps (in ns or cycles > + depending on the type specified) of the buffers of a > + trace. > + > + * bt_ctf_event_get_handle_id() > + -> returns the handle id associated with an event. > + > + > +For more information on CTF, see the CTF documentation (git://git.efficios.com/ctf.git). Thanks ! Julien From christian.babeux at efficios.com Fri Aug 10 13:22:50 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Fri, 10 Aug 2012 13:22:50 -0400 Subject: [lttng-dev] [PATCH lttng-tools 0/2] relayd IPv6 fixes Message-ID: <1344619372-21976-1-git-send-email-christian.babeux@efficios.com> Hi all, Here are a few fixes to IPv6 support in relayd. When trying to start the relayd using an IPv6 address, the relayd simply segfaulted. The first patch fix this issue. The second patch fix a wrong domain value used in the socket() call for the IPv6 socket causing a "Address family not supported by protocol" error when calling bind(). Thanks, Christian Christian Babeux (2): Fix: Off by one in lttcomm_sock_domain enum Fix: Wrong domain used when initializing IPv6 sockets src/common/sessiond-comm/inet6.c | 2 +- src/common/sessiond-comm/sessiond-comm.h | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) -- 1.7.11.4 From christian.babeux at efficios.com Fri Aug 10 13:22:51 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Fri, 10 Aug 2012 13:22:51 -0400 Subject: [lttng-dev] [PATCH lttng-tools 1/2] Fix: Off by one in lttcomm_sock_domain enum In-Reply-To: <1344619372-21976-1-git-send-email-christian.babeux@efficios.com> References: <1344619372-21976-1-git-send-email-christian.babeux@efficios.com> Message-ID: <1344619372-21976-2-git-send-email-christian.babeux@efficios.com> The wrong value is used to lookup the socket creation function in net_families causing a segfault when using IPv6. Signed-off-by: Christian Babeux --- src/common/sessiond-comm/sessiond-comm.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/common/sessiond-comm/sessiond-comm.h b/src/common/sessiond-comm/sessiond-comm.h index aebe30c..8d1400c 100644 --- a/src/common/sessiond-comm/sessiond-comm.h +++ b/src/common/sessiond-comm/sessiond-comm.h @@ -208,8 +208,8 @@ enum lttcomm_sock_proto { * Index in the net_families array below. Please keep in sync! */ enum lttcomm_sock_domain { - LTTCOMM_INET = 1, - LTTCOMM_INET6 = 2, + LTTCOMM_INET = 0, + LTTCOMM_INET6 = 1, }; struct lttcomm_sockaddr { -- 1.7.11.4 From christian.babeux at efficios.com Fri Aug 10 13:22:52 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Fri, 10 Aug 2012 13:22:52 -0400 Subject: [lttng-dev] [PATCH lttng-tools 2/2] Fix: Wrong domain used when initializing IPv6 sockets In-Reply-To: <1344619372-21976-1-git-send-email-christian.babeux@efficios.com> References: <1344619372-21976-1-git-send-email-christian.babeux@efficios.com> Message-ID: <1344619372-21976-3-git-send-email-christian.babeux@efficios.com> Signed-off-by: Christian Babeux --- src/common/sessiond-comm/inet6.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/common/sessiond-comm/inet6.c b/src/common/sessiond-comm/inet6.c index 0d96c31..98aba04 100644 --- a/src/common/sessiond-comm/inet6.c +++ b/src/common/sessiond-comm/inet6.c @@ -52,7 +52,7 @@ int lttcomm_create_inet6_sock(struct lttcomm_sock *sock, int type, int proto) int val = 1, ret; /* Create server socket */ - if ((sock->fd = socket(PF_INET, type, proto)) < 0) { + if ((sock->fd = socket(PF_INET6, type, proto)) < 0) { PERROR("socket inet6"); goto error; } -- 1.7.11.4 From danny.serres at efficios.com Fri Aug 10 13:25:23 2012 From: danny.serres at efficios.com (Danny Serres) Date: Fri, 10 Aug 2012 13:25:23 -0400 Subject: [lttng-dev] =?utf-8?q?=5Bbabeltrace_PATCH=5D_API_documentation_fi?= =?utf-8?q?le_v2?= Message-ID: <1344619524-4481-1-git-send-email-danny.serres@efficios.com> Signed-off-by: Danny Serres --- doc/API.txt | 257 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 257 insertions(+) create mode 100644 doc/API.txt diff --git a/doc/API.txt b/doc/API.txt new file mode 100644 index 0000000..4688a75 --- /dev/null +++ b/doc/API.txt @@ -0,0 +1,257 @@ +Babeltrace API documentation + +Babeltrace is a trace viewer and converter reading and writing the +Common Trace Format (CTF). Its main use is to pretty-print CTF traces +into a human-readable text output. + +Babeltrace provides trace read and write libraries, as well as a trace +converter. A plugin can be created for any trace format to allow its +conversion to/from another trace format. + +The main format expected to be converted to/from is the Common Trace +Format (CTF). The latest version of the CTF specification can be found at: + git tree: git://git.efficios.com/ctf.git + gitweb: http://git.efficios.com/?p=ctf.git + +This document describes the main concepts to use the libbabeltrace, +which exposes the Babeltrace trace reading capability. + + +TERMINOLOGY +??????????????? + +* A "callback" is a reference to a piece of executable code (such as a + function) that is passed as an argument to another piece of code + (like another function). + +* A "context" is a structure that represents an object in which a trace + collection is opened. + +* An "iterator" is a structure that enables the user to traverse a trace. + +* A "trace handle" is a unique identifier representing a trace file. + It allows the user to manipulate a trace directly. + + + +USAGE +??????????????? + +Context: + +In order to use libbabeltrace to read a trace, the first step is to create a +context structure and to add a trace to it. This is done using the +bt_context_create() and bt_context_add_trace() functions. As long as this +context structure is allocated, the trace is open and valid. + +The context can be destroyed by calling one more bt_context_put() +than bt_context_get(), functions which respectively decrement and increment +the refcount of the context. These functions ensures that the context won't +be destroyed when it is in use. + +Once a trace is added to the context, it can be read and seeked using iterators +and callbacks. + + +Iterator: + +An iterator can be created using the bt_iter_create() function. +As for now, only ctf iterator are supported. These are used to traverse a ctf- +formatted trace. Such iterator can be created with bt_ctf_iter_create(). + +While creating an iterator, a begin and an end position may be specified. +To do so, one or two struct bt_iter_pos must be passed. Such struct have 2 +attributes: type and u. "type" is the seek type, can be either: + BT_SEEK_TIME + BT_SEEK_RESTORE + BT_SEEK_CUR + BT_SEEK_BEGIN + BT_SEEK_END +and "u" is a union of the seek time (if using BT_SEEK_TIME) and the restore +position (if using BT_SEEK_RESTORE). + +Once the iterator is created, various functions become available. We have +bt_ctf_iter_read_event() which returns the ctf event of the trace where the +iterator is set. There is also bt_ctf_iter_destroy() which free the iterator. +Note that only one iterator can be created against a context at the same time. +If more than one iterator is being created for the same context, the second +creation will return NULL. The previous iterator must be destroyed before +creation of the new iterator. In the future, creation of multiples iterators +will probably be allowed. + +Finally, we have the bt_ctf_get_iter() function which returns a struct bt_iter +with which the iterator can be moved using one of these functions: + bt_iter_next(), which moves the position to the next event + bt_iter_set_pos(), which moves the position to the specified one + +To get the current position (struct bt_iter_pos) of the iterator, the function +bt_iter_get_pos() can be used. To get an arbitrary position based on a +specific time, bt_itet_create_time_pos() is the function to use. The position +returned by one of these two functions must be freed with bt_iter_free_pos() +after use. + + +CTF Event: + +A CTF event is usually obtained from an iterator via the +bt_ctf_iter_read_event() function, which returns a struct bt_ctf_event. Having +the event, we can collect its data: + * bt_ctf_event_name() will return the name of the event; + * bt_ctf_get_timestamp() will return the timestamp of the event + offsetted with the system clock source (in ns); + * bt_ctf_get_cycles will return the timestamp of the event as written + in the packet (in cycles). + +To access the event fields which contain various information, the first step is +to obtain a scope definition for the desired field. This can be done using the +bt_ctf_get_top_level_scope() function. This function provides the mapping +between the enum and the actual definition of top-level scopes and returns +a definition of the top-level scope. +Top-level scopes are defined in the bt_ctf_scope enum: + BT_TRACE_PACKET_HEADER = 0, + BT_STREAM_PACKET_CONTEXT = 1, + BT_STREAM_EVENT_HEADER = 2, + BT_STREAM_EVENT_CONTEXT = 3, + BT_EVENT_CONTEXT = 4, + BT_EVENT_FIELDS = 5. + + +In order to get a field or a field list, the user needs to pass a scope as +argument, this scope can be a top-level scope or a scope relative to an +arbitrary field. + +For more information on each scope, see the CTF specifications. + +To get a field list, the bt_ctf_get_field_list() function is to be used. +To get the definition of a specific field, the appropriate function would be +bt_ctf_get_field(). + +Once the field is obtained, we can obtain its name and type using the +bt_ctf_field_name() and bt_ctf_field_type() functions respectively. The +possible types are defined in the ctf_type_id enum: + CTF_TYPE_UNKNOWN = 0, + CTF_TYPE_INTEGER, + CTF_TYPE_FLOAT, + CTF_TYPE_ENUM, + CTF_TYPE_STRING, + CTF_TYPE_STRUCT, + CTF_TYPE_UNTAGGED_VARIANT, + CTF_TYPE_VARIANT, + CTF_TYPE_ARRAY, + CTF_TYPE_SEQUENCE, + NR_CTF_TYPES. + +Depending on the field type, we can get data on its value using the appropriate +function: + * bt_ctf_get_index() return the element at the index + position of an array of a sequence; + + * bt_ctf_get_array_len() return the len of an array; + + * bt_ctf_get_int_signedness() return the signedness of an integer; + + * bt_ctf_get_int_base() return the base of an integer; + + * bt_ctf_get_int_byte_order() return the byte order of an integer; + + * bt_ctf_get_int_len() return the size in bits of an integer; + + * bt_ctf_get_encoding() return the encoding of an int or a + string defined in the + ctf_string_encoding enum: + CTF_STRING_NONE = 0, + CTF_STRING_UTF8, + CTF_STRING_ASCII, + CTF_STRING_UNKNOWN. + +To obtain the value associated with a field, the appropriate function of these +is to be used: + + * bt_ctf_get_uint64(); + * bt_ctf_get_int64(); + * bt_ctf_get_char_array(); + * bt_ctf_get_string(). + +If the field does not exist or is not of the type requested, the value returned +with these four functions is undefined. To check if an error occured, use the +bt_ctf_field_get_error() function after accessing a field. If no error +occured, the function will return 0. + +It is also possible to access the declaration fields, the same way as the +definition ones. bt_ctf_get_event_decl_list() sets a list to an array of +bt_ctf_event_decl pointers and bt_ctf_get_event_decl_fields() sets a list to an +array of bt_ctf_field_decl pointers. From the first type, the name of the +event can be obtained with bt_ctf_get_decl_event_name(). For the second type, +the field decl name is obtained with bt_ctf_get_decl_field_name(). + +The declaration functions allow the user to list the events, fields and +contexts fields enabled in the trace once it is opened, whereas the definition +functions apply on the current event being read. + + +Callback: + +The iterator allow the user to read the trace, in order to access the events +and fields, the user can either call the functions listed previously on each +event, or register callbacks functions that are called when specific (or all) +events are read. + +The other alternative to reading a trace is with the use of callbacks. This is +done with the bt_ctf_iter_add_callback() function. It still requires a valid +ctf iterator as the first argument. Here are all arguments: + + @iter: trace collection iterator (input) + @event: event to target. 0 for all events. + @private_data: private data pointer to pass to the callback + @flags: specific flags controlling the behavior of this callback + (or'd). + @callback: function pointer to call + @depends: struct bt_dependency detailing the required computation + results. Ends with 0. + @weak_depends: struct bt_dependency detailing the optional computation + results that can be optionally consumed by this + callback. + @provides: struct bt_dependency detailing the computation results + provided by this callback. + Ends with 0. + +"depends", "weak_depends" and "provides" memory is handled by the babeltrace +library after this call succeeds or fails. These objects can still be used by +the caller until the babeltrace iterator is destroyed, but they belong to the +babeltrace library. + +Note: the dependency graph is calculated when bt_ctf_iter_read_event() is +executed after a bt_ctf_iter_add_callback(). Beware that it is valid to +create/add callbacks/read/add more callbacks/read some more. + +As of now the flags and dependencies are not used, the callbacks are +processed in FIFO order. + +The callback function passed to bt_ctf_iter_add_callback() must return a +bt_cb_ret value: + BT_CB_OK = 0, + BT_CB_OK_STOP = 1, + BT_CB_ERROR_STOP = 2, + BT_CB_ERROR_CONTINUE = 3. + + +Trace handle: + +When a trace is added to a context, bt_context_add_trace() returns a trace +handle id. This id is associated with its corresponding trace handle. With +that id, it is possible to manipulate directly the trace. + + * bt_trace_handle_get_path() + -> returns the path of the trace handle (path to the trace). + + * bt_trace_handle_get_timestamp_begin() + * bt_trace_handle_get_timestamp_end() + -> return the creation/destruction timestamps (in ns or cycles + depending on the type specified) of the buffers of a + trace. + + * bt_ctf_event_get_handle_id() + -> returns the handle id associated with an event. + + +For more information on CTF, see the CTF documentation. -- 1.7.9.5 From danny.serres at efficios.com Fri Aug 10 13:42:35 2012 From: danny.serres at efficios.com (Danny Serres) Date: Fri, 10 Aug 2012 13:42:35 -0400 Subject: [lttng-dev] [lttng-ust PATCH] Python module Message-ID: <1344620555-4776-1-git-send-email-danny.serres@efficios.com> Proof of concept: A simple library to create tracepoints and allow tracing in Python programs. To build, configure with Python: ./configure --with-python To use, import the module in Python: import lttngust Signed-off-by: Danny Serres --- Makefile.am | 4 + README | 4 + configure.ac | 31 ++++++ liblttng-ust-python/Makefile.am | 13 +++ liblttng-ust-python/README | 8 ++ liblttng-ust-python/lttngust.c | 198 +++++++++++++++++++++++++++++++++++++++ liblttng-ust-python/lttngust.h | 103 ++++++++++++++++++++ 7 files changed, 361 insertions(+) create mode 100644 liblttng-ust-python/Makefile.am create mode 100644 liblttng-ust-python/README create mode 100644 liblttng-ust-python/lttngust.c create mode 100644 liblttng-ust-python/lttngust.h diff --git a/Makefile.am b/Makefile.am index e61735d..d43aeab 100644 --- a/Makefile.am +++ b/Makefile.am @@ -13,6 +13,10 @@ if BUILD_JNI_INTERFACE SUBDIRS += liblttng-ust-java endif +if USE_PYTHON +SUBDIRS += liblttng-ust-python +endif + #temporarily disabled # liblttng-ust-malloc diff --git a/README b/README index 55c72df..c9a455d 100644 --- a/README +++ b/README @@ -170,3 +170,7 @@ PACKAGE CONTENTS: - liblttng-ust-java A simple library that uses JNI to allow tracing in java programs. See liblttng-ust-java/README for build instructions. + + - liblttng-ust-python + A simple library to allow tracing in Python programs. + See liblttng-ust-python/README for build instructions. diff --git a/configure.ac b/configure.ac index 4d40f75..e85bd0d 100644 --- a/configure.ac +++ b/configure.ac @@ -144,6 +144,33 @@ AC_CHECK_LIB([urcu-bp], [synchronize_rcu_bp], [], [AC_MSG_ERROR([Cannot find lib AC_CHECK_LIB([urcu-bp], [call_rcu_bp], [], [AC_MSG_ERROR([liburcu 0.6 or newer is needed, please update your version or use [LDFLAGS]=-Ldir to specify the right location.])]) +# For Python +AC_ARG_WITH([python], + [AS_HELP_STRING([--with-python], + [compile with Python bindings])], + [enable_python=yes], [enable_python=no]) + +AM_CONDITIONAL([USE_PYTHON], [test "x${enable_python:-yes}" = xyes]) + +if test "x${enable_python:-yes}" = xyes; then + AM_PATH_PYTHON + + AC_ARG_VAR([PYTHON_INCLUDE], [Include flags for python, bypassing python-config]) + AC_ARG_VAR([PYTHON_CONFIG], [Path to python-config]) + AS_IF([test -z "$PYTHON_INCLUDE"], [ + AS_IF([test -z "$PYTHON_CONFIG"], [ + AC_PATH_PROGS([PYTHON_CONFIG], + [python$PYTHON_VERSION-config python-config], + [no], + [`dirname $PYTHON`]) + AS_IF([test "$PYTHON_CONFIG" = no], [AC_MSG_ERROR([cannot find python-config for $PYTHON.])]) + ]) + AC_MSG_CHECKING([python include flags]) + PYTHON_INCLUDE=`$PYTHON_CONFIG --includes` + AC_MSG_RESULT([$PYTHON_INCLUDE]) + ]) +fi + # Check for various supplementary host information (beyond the # triplet) which might affect the library format choices. E.g., we # can be building with `i686-unknown-linux-gnu-gcc -m64' @@ -300,6 +327,7 @@ AC_CONFIG_FILES([ liblttng-ust-fork/Makefile liblttng-ust-java/Makefile liblttng-ust-libc-wrapper/Makefile + liblttng-ust-python/Makefile tools/Makefile tests/Makefile tests/hello/Makefile @@ -327,6 +355,9 @@ AS_ECHO() AS_ECHO_N("Java support (JNI): ") AS_IF([test "x$jni_interface" = "xyes"], [AS_ECHO("Enabled")], [AS_ECHO("Disabled")]) +AS_ECHO_N("Python support: ") +AS_IF([test "x$enable_python" = "xyes"], [AS_ECHO("Enabled")], [AS_ECHO("Disabled")]) + AS_ECHO_N("sdt.h integration: ") AS_IF([test "x$with_sdt" = "xyes"], [AS_ECHO("Enabled")], [AS_ECHO("Disabled")]) diff --git a/liblttng-ust-python/Makefile.am b/liblttng-ust-python/Makefile.am new file mode 100644 index 0000000..993e0b9 --- /dev/null +++ b/liblttng-ust-python/Makefile.am @@ -0,0 +1,13 @@ +AM_CFLAGS = -I$(PYTHON_INCLUDE) -I$(top_srcdir)/include/ + +pyexec_LTLIBRARIES = lttngust.la + +MAINTAINERCLEANFILES = lttngust.c + +lttngust_la_SOURCES = lttngust.c + +lttngust_la_LDFLAGS = -module + +lttngust_la_CFLAGS = $(AM_CFLAGS) + +lttngust_la_LIBADD = -lc -L$(top_builddir)/liblttng-ust/.libs ../liblttng-ust/liblttng-ust.la diff --git a/liblttng-ust-python/README b/liblttng-ust-python/README new file mode 100644 index 0000000..405ef62 --- /dev/null +++ b/liblttng-ust-python/README @@ -0,0 +1,8 @@ +This directory contains a simple API for instrumenting Python applications. + +Configuration to build this library: + + ./configure --with-python + +After building, you can use the generated module in a +Python project ('import lttngust'). diff --git a/liblttng-ust-python/lttngust.c b/liblttng-ust-python/lttngust.c new file mode 100644 index 0000000..39ecb1d --- /dev/null +++ b/liblttng-ust-python/lttngust.c @@ -0,0 +1,198 @@ +#define TRACEPOINT_DEFINE +#define TRACEPOINT_CREATE_PROBES + +#include +#include "lttngust.h" + +/* Tracepoint with int value */ +static PyObject * +lttngust_tracepointInt(PyObject *self, PyObject *args) +{ + const char *name; + int val; + + if (!PyArg_ParseTuple(args, "si", &name, &val)) + return NULL; + tracepoint(lttngust_python, int_event, name, val); + + Py_INCREF(Py_None); + return Py_None; +} + +/* Tracepoint with long value */ +static PyObject * +lttngust_tracepointLong(PyObject *self, PyObject *args) +{ + const char *name; + long val; + + if (!PyArg_ParseTuple(args, "sl", &name, &val)) + return NULL; + tracepoint(lttngust_python, long_event, name, val); + + Py_INCREF(Py_None); + return Py_None; +} + +/* Tracepoint with string value */ +static PyObject * +lttngust_tracepointString(PyObject *self, PyObject *args) +{ + const char *name; + const char *val; + + if (!PyArg_ParseTuple(args, "ss", &name, &val)) + return NULL; + tracepoint(lttngust_python, string_event, name, val); + + Py_INCREF(Py_None); + return Py_None; +} + +/* Tracepoint with double value */ +static PyObject * +lttngust_tracepointDouble(PyObject *self, PyObject *args) +{ + const char *name; + double val; + + if (!PyArg_ParseTuple(args, "sd", &name, &val)) + return NULL; + tracepoint(lttngust_python, double_event, name, val); + + Py_INCREF(Py_None); + return Py_None; +} + +/* Tracepoint with complex object value */ +static PyObject * +lttngust_tracepointComplex(PyObject *self, PyObject *args) +{ + const char *name; + Py_complex complex; + char val[256]; + + if (!PyArg_ParseTuple(args, "sD", &name, &complex)) + return NULL; + sprintf(val, "%g+%gj", complex.real, complex.imag); + tracepoint(lttngust_python, complex_event, name, val); + + Py_INCREF(Py_None); + return Py_None; +} + +/* Tracepoint with bool(object) value */ +static PyObject * +lttngust_tracepointBool(PyObject *self, PyObject *args) +{ + const char *name; + PyObject *obj; + char *val; + + + if (!PyArg_ParseTuple(args, "sO", &name, &obj)) + return NULL; + + if (PyObject_IsTrue(obj)) + val = "True"; + else + val = "False"; + + tracepoint(lttngust_python, bool_event, name, val); + + Py_INCREF(Py_None); + return Py_None; +} + +/* Tracepoint with repr(object) value */ +static PyObject * +lttngust_tracepointRepr(PyObject *self, PyObject *args) +{ + const char *name; + PyObject *obj; + const char *repr; + + if (!PyArg_ParseTuple(args, "sO", &name, &obj)) + return NULL; + + repr = PyString_AsString(PyObject_Repr(obj)); + tracepoint(lttngust_python, obj_repr, name, repr); + + Py_INCREF(Py_None); + return Py_None; +} + +/* Tracepoint with message, loglevel TRACE_INFO */ +static PyObject * +lttngust_tracepointMessage(PyObject *self, PyObject *args) +{ + const char *text; + + if (!PyArg_ParseTuple(args, "s", &text)) + return NULL; + tracepoint(lttngust_python, message, text); + + Py_INCREF(Py_None); + return Py_None; +} + +/* Tracepoint with warning, loglevel TRACE_WARNING */ +static PyObject * +lttngust_tracepointWarning(PyObject *self, PyObject *args) +{ + const char *text; + + if (!PyArg_ParseTuple(args, "s", &text)) + return NULL; + tracepoint(lttngust_python, warning, text); + + Py_INCREF(Py_None); + return Py_None; +} + + +/* Definition of Python methods */ +static PyMethodDef UstMethods[] = { + + {"tracepointInt", lttngust_tracepointInt, METH_VARARGS, + "tracepointInt(name, value)\n\nTracepoint with Int value."}, + + {"tracepointLong", lttngust_tracepointLong, METH_VARARGS, + "tracepointLong(name, value)\n\nTracepoint with Long value."}, + + {"tracepointDouble", lttngust_tracepointDouble, METH_VARARGS, + "tracepointDouble(name, value)\n\n" + "Tracepoint with Double value."}, + + {"tracepointComplex", lttngust_tracepointComplex, METH_VARARGS, + "tracepointComplex(name, complex_number)\n\n" + "Tracepoint with Complex number value."}, + + {"tracepointString", lttngust_tracepointString, METH_VARARGS, + "tracepointString(name, value)\n\n" + "Tracepoint with String value."}, + + {"tracepointBool", lttngust_tracepointBool, METH_VARARGS, + "tracepointBool(name, object)\n\n" + "Tracepoint with Object value, uses bool()."}, + + {"tracepointRepr", lttngust_tracepointRepr, METH_VARARGS, + "tracepointRepr(name, object)\n\n" + "Tracepoint with Object value, uses repr()."}, + + {"tracepointMessage", lttngust_tracepointMessage, METH_VARARGS, + "tracepointMessage(text)\n\nTracepoint with string info."}, + + {"tracepointWarning", lttngust_tracepointWarning, METH_VARARGS, + "tracepointWarning(text)\n\nTracepoint with string warning."}, + + {NULL, NULL, 0, NULL} /* Sentinel */ +}; + + +/* Init module */ +PyMODINIT_FUNC +initlttngust(void) +{ + (void) Py_InitModule("lttngust", UstMethods); +} diff --git a/liblttng-ust-python/lttngust.h b/liblttng-ust-python/lttngust.h new file mode 100644 index 0000000..f668ee1 --- /dev/null +++ b/liblttng-ust-python/lttngust.h @@ -0,0 +1,103 @@ +#undef TRACEPOINT_PROVIDER +#define TRACEPOINT_PROVIDER lttngust_python + +#undef TRACEPOINT_INCLUDE_FILE +#define TRACEPOINT_INCLUDE_FILE ./lttngust.h + +#ifdef __cplusplus +extern "C"{ +#endif /* __cplusplus */ + +#if !defined(_lttngust_H) || defined(TRACEPOINT_HEADER_MULTI_READ) +#define _lttngust_H + +#include + +/* Tracepoint with int value */ +TRACEPOINT_EVENT(lttngust_python, int_event, + TP_ARGS(const char *, name, int, val), + TP_FIELDS( + ctf_string(name, name) + ctf_integer(int, value, val) + ) +) + +/* Tracepoint with long value */ +TRACEPOINT_EVENT(lttngust_python, long_event, + TP_ARGS(const char *, name, long, val), + TP_FIELDS( + ctf_string(name, name) + ctf_integer(int, value, val) + ) +) + +/* Tracepoint with string value */ +TRACEPOINT_EVENT(lttngust_python, string_event, + TP_ARGS(const char *, name, const char *, val), + TP_FIELDS( + ctf_string(name, name) + ctf_string(value, val) + ) +) + +/* Tracepoint with double value */ +TRACEPOINT_EVENT(lttngust_python, double_event, + TP_ARGS(const char *, name, double, val), + TP_FIELDS( + ctf_string(name, name) + ctf_float(double, value, val) + ) +) + +/* Tracepoint with complex object value */ +TRACEPOINT_EVENT(lttngust_python, complex_event, + TP_ARGS(const char *, name, char *, val), + TP_FIELDS( + ctf_string(name, name) + ctf_string(value, val) + ) +) + +/* Tracepoint with bool(object) value */ +TRACEPOINT_EVENT(lttngust_python, bool_event, + TP_ARGS(const char *, name, char *, val), + TP_FIELDS( + ctf_string(name, name) + ctf_string(value, val) + ) +) + +/* Tracepoint with repr(object) value */ +TRACEPOINT_EVENT(lttngust_python, obj_repr, + TP_ARGS(const char *, name, const char *, repr), + TP_FIELDS( + ctf_string(name, name) + ctf_string(repr, repr) + ) +) + +/* Tracepoint with message, loglevel TRACE_INFO */ +TRACEPOINT_EVENT(lttngust_python, message, + TP_ARGS(const char *, text), + TP_FIELDS( + ctf_string(message, text) + ) +) +TRACEPOINT_LOGLEVEL(lttngust_python, message, TRACE_INFO) + +/* Tracepoint with warning, loglevel TRACE_WARNING */ +TRACEPOINT_EVENT(lttngust_python, warning, + TP_ARGS(const char *, text), + TP_FIELDS( + ctf_string(message, text) + ) +) +TRACEPOINT_LOGLEVEL(lttngust_python, warning, TRACE_WARNING) + +#endif /* _lttngust_H */ + +#include + +#ifdef __cplusplus +} +#endif /* __cplusplus */ -- 1.7.9.5 From paulmck at linux.vnet.ibm.com Fri Aug 10 14:23:05 2012 From: paulmck at linux.vnet.ibm.com (Paul E. McKenney) Date: Fri, 10 Aug 2012 11:23:05 -0700 Subject: [lttng-dev] [PATCH 2/2] urcu: add notice to URCU_TLS() for it is not async-signal-safe In-Reply-To: <20120810141615.GA27969@Krystal> References: <1344414673-14714-1-git-send-email-laijs@cn.fujitsu.com> <1344414673-14714-2-git-send-email-laijs@cn.fujitsu.com> <20120809141417.GC9064@Krystal> <20120809200211.GA14409@Krystal> <50247E96.9080909@cn.fujitsu.com> <20120810141615.GA27969@Krystal> Message-ID: <20120810182305.GH2371@linux.vnet.ibm.com> On Fri, Aug 10, 2012 at 10:16:15AM -0400, Mathieu Desnoyers wrote: > (for the records: this discussion is about userspace RCU tls-compat.h, > which uses TLS on systems supporting it, and fall back to > pthread_key_create/getspecific/setspecific if not. The culprit of the > issue is that we want to allow reading the tls-compat variable from > signal handlers, and thus async-signal-safety of pthread_key_* and TLS. > > The code we refer to is: > http://git.lttng.org/?p=userspace-rcu.git;a=blob;f=urcu/tls-compat.h;h=192a53609fb5f6bc445f98fdd6bc26918126687e;hb=HEAD) > > * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > > On 08/10/2012 04:02 AM, Mathieu Desnoyers wrote: > > > Looking at the result of a quick google search: > > > > > > http://curl.haxx.se/mail/lib-2006-09/0224.html > > > http://www.slamb.org/projects/sigsafe/api/patternref.html > > > > > > "Additionally, it makes the same assumption as all other methods for > > > handling thread-directed signals (with the exception of kevent(2) > > > handling), that pthread_getspecific(2) is async signal-safe. This is not > > > guaranteed by SUSv3." > > > > > > and > > > > > > https://groups.google.com/forum/?fromgroups#!topic/comp.os.linux.development/nZfmndKbzJw[1-25] > > > > > > it looks like using pthread_getspecific from a signal handler is not > > > always safe, mainly due to possible use of sigaltstack. So disabling > > > signals works for the "pthread_key_create" part, but we still have an > > > issue with pthread_getspecific. > > > > > > Ideas are welcome on how to best deal with this issue. > > > > What's the problem with disabling signals + pthread_getspecific()? > > Waht's the problem with disabling signals + __tls_access_ ## name()? > > * pthread_key_* fallback > > Disabling signals would allow us to be reentrant with respect to > signals, which actually solves part of the problem (reentrancy) for the > pthread_key_create part (protected by lock). > > Disabling signals, AFAIU, (and if we disregard the SUSv3 standard for a > minute) should not be strictly required around the pthread_getspecific > call on most architectures (no lock taken). We should carefully review > the implementations for each architecture we support if we want to > assume this though. If the getspecific returns NULL. we should disable > signals and call pthread_getspecific again with signals disabled, and > then call pthread_setspecific if necessary, before re-enabling signals. > The benefit of not _always_ disabling signals around > pthread_getspecific() is significant gain in performance in the common > case: the entire hot path of rcu_read_lock/unlock all happens in > userspace, without any system call, and uses tls-compat variables. > > The other part of the problem with pthread_getspecific and signal > handlers is that it does not seem to be SUSv3-compliant to use > pthread_getspecific() from within a signal handler. One example that can > lead to problems is if the signal handler is setup with sigaltstack(2). > > We might want to simply document this limitation: > > "RCU read-side critical sections can be used in signals handlers, except > those setup with sigaltstack(2)." Seems to me to be a reasonable limitation. Easier to relax limitations than to add new ones! > * TLS > > Userspace RCU always touch the TLS variables from thread context (from > within rcu_register_thread()) before they are allowed to be touched by > signal handlers nested over threads. This ensures that issues with lazy > binding and dynamic linker lock are not encountered s/ensures that/avoids/ s/and dynamic/and the dynamic/ s/ are not encountered// > (ref. http://sourceware.org/ml/libc-alpha/2012-06/msg00372.html). And stop here. > I did > the same within my LTTng-UST use of TLS variables: they are touched by a > constructor once so we don't run into deadlocks between UST lock and the > libc lock protecting dynamic linking (recursive mutex also taken around > the constructor calls, within which we needed to take the UST lock, thus > causing deadlocks). > > > Feedback is welcome, Looks good! Thanx, Paul > Thanks, > > Mathieu > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com > From mathieu.desnoyers at efficios.com Fri Aug 10 14:28:40 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 10 Aug 2012 14:28:40 -0400 Subject: [lttng-dev] [PATCH 2/2] urcu: new wfqueue implementation In-Reply-To: <1344501986-23151-2-git-send-email-laijs@cn.fujitsu.com> References: <1344501986-23151-1-git-send-email-laijs@cn.fujitsu.com> <1344501986-23151-2-git-send-email-laijs@cn.fujitsu.com> Message-ID: <20120810182840.GA31156@Krystal> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > Some guys would be surprised by this fact: > There are already TWO implementations of wfqueue in urcu. > > The first one is in urcu/static/wfqueue.h: > 1) enqueue: exchange the tail and then update previous->next > 2) dequeue: wait for first node's next pointer and them shift, a dummy node > is introduced to avoid the queue->tail become NULL when shift. > > The second one shares some code with the first one, and the left code > are spreading in urcu-call-rcu-impl.h: > 1) enqueue: share with the first one > 2) no dequeue operation: and no shift, so it don't need dummy node, > Although the dummy node is queued when initialization, but it is removed > after the first dequeue_all operation in call_rcu_thread(). > call_rcu_data_free() forgets to handle the dummy node if it is not removed. > 3)dequeue_all: record the old head and tail, and queue->head become the special > tail node.(atomic record the tail and change the tail). > > The second implementation's code are spreading, bad for review, and it is not > tested by tests/test_urcu_wfq. > > So we need a better implementation avoid the dummy node dancing and can service > both generic wfqueue APIs and dequeue_all API for call rcu. > > The new implementation: > 1) enqueue: share with the first one/original implementation. > 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. > no dummy node, save memory. > 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail > and return the old head.next. Hi Lai, Your approach is very interesting! It is indeed good for testing and maintenance if we can keep all the queue code within the API. I am concerned about the following scenario in your new implementation, I would like to know your thoughts on this. It could happen on architectures reordering loads (DEC Alpha, AMD64, IA64, PA-RISC, POWER, SPARC RMO, x86 TSO, and x86 OOStore): init state: list is empty CPU 0 CPU 1 ___cds_wfq_append_list() (append newtail) oldtail = uatomic_xchg(&q->tail, newtail); (A) CMM_STORE_SHARED(oldtail->next, head); (B) (B) is observable by cpu 1, but not (A) yet ___cds_wfq_dequeue_blocking() _cds_wfq_empty(q) return q->head.next == NULL && CMM_LOAD_SHARED(q->tail) == &q->head; -> false (q->tail != &q->head) node = ___cds_wfq_node_sync_next(&q->head); -> node is newtail if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { -> taken, newtail->next is indeed NULL * (see note below) if (CMM_LOAD_SHARED(q->tail) == node) { -> not taken, since q->tail still appears as &q->head } next = ___cds_wfq_node_sync_next(node); -> endless loop if no other enqueue is performed. (BUG) } (A) is observable by cpu 1 * note: I think we should add a cmm_smp_rmb() here to fix this issue. It would force CPU 1 to necessarily see store (A) if store (B) is seen. This would be matching the full memory barrier implied after uatomic_xchg(). Thanks, Mathieu > > More implementation details are in the code. > tests/test_urcu_wfq will be update in future for testing new APIs. > > > Signed-off-by: Lai Jiangshan > --- > urcu-call-rcu-impl.h | 50 ++++++++++-------------- > urcu/static/wfqueue.h | 104 ++++++++++++++++++++++++++++++++++++------------ > urcu/wfqueue.h | 25 ++++++++++-- > wfqueue.c | 29 ++++++++++++++ > 4 files changed, 149 insertions(+), 59 deletions(-) > > diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h > index 13b24ff..dbfb410 100644 > --- a/urcu-call-rcu-impl.h > +++ b/urcu-call-rcu-impl.h > @@ -221,7 +221,7 @@ static void *call_rcu_thread(void *arg) > { > unsigned long cbcount; > struct cds_wfq_node *cbs; > - struct cds_wfq_node **cbs_tail; > + struct cds_wfq_node *cbs_tail; > struct call_rcu_data *crdp = (struct call_rcu_data *)arg; > struct rcu_head *rhp; > int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); > @@ -243,24 +243,18 @@ static void *call_rcu_thread(void *arg) > cmm_smp_mb(); > } > for (;;) { > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > - poll(NULL, 0, 1); > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > - cbs_tail = (struct cds_wfq_node **) > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > + cbs = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &cbs_tail); > + if (cbs) { > synchronize_rcu(); > cbcount = 0; > do { > - while (cbs->next == NULL && > - &cbs->next != cbs_tail) > - poll(NULL, 0, 1); > - if (cbs == &crdp->cbs.dummy) { > - cbs = cbs->next; > - continue; > - } > rhp = (struct rcu_head *)cbs; > - cbs = cbs->next; > + > + if (cbs != cbs_tail) > + cbs = __cds_wfq_node_sync_next(cbs); > + else > + cbs = NULL; > + > rhp->func(rhp); > cbcount++; > } while (cbs != NULL); > @@ -270,8 +264,7 @@ static void *call_rcu_thread(void *arg) > break; > rcu_thread_offline(); > if (!rt) { > - if (&crdp->cbs.head > - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { > + if (cds_wfq_empty(&crdp->cbs)) { > call_rcu_wait(crdp); > poll(NULL, 0, 10); > uatomic_dec(&crdp->futex); > @@ -625,32 +618,31 @@ void call_rcu(struct rcu_head *head, > */ > void call_rcu_data_free(struct call_rcu_data *crdp) > { > - struct cds_wfq_node *cbs; > - struct cds_wfq_node **cbs_tail; > - struct cds_wfq_node **cbs_endprev; > + struct cds_wfq_node *head, *tail; > > if (crdp == NULL || crdp == default_call_rcu_data) { > return; > } > + > if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { > uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); > wake_call_rcu_thread(crdp); > while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) > poll(NULL, 0, 1); > } > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > - poll(NULL, 0, 1); > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > - cbs_tail = (struct cds_wfq_node **) > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > + > + if (!cds_wfq_empty(&crdp->cbs)) { > + head = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &tail); > + assert(head); > + > /* Create default call rcu data if need be */ > (void) get_default_call_rcu_data(); > - cbs_endprev = (struct cds_wfq_node **) > - uatomic_xchg(&default_call_rcu_data, cbs_tail); > - *cbs_endprev = cbs; > + > + __cds_wfq_append_list(&default_call_rcu_data->cbs, head, tail); > + > uatomic_add(&default_call_rcu_data->qlen, > uatomic_read(&crdp->qlen)); > + > wake_call_rcu_thread(default_call_rcu_data); > } > > diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h > index 636e1af..15ea9fc 100644 > --- a/urcu/static/wfqueue.h > +++ b/urcu/static/wfqueue.h > @@ -10,6 +10,7 @@ > * dynamically with the userspace rcu library. > * > * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -29,6 +30,7 @@ > #include > #include > #include > +#include > #include > #include > > @@ -38,8 +40,6 @@ extern "C" { > > /* > * Queue with wait-free enqueue/blocking dequeue. > - * This implementation adds a dummy head node when the queue is empty to ensure > - * we can always update the queue locklessly. > * > * Inspired from half-wait-free/half-blocking queue implementation done by > * Paul E. McKenney. > @@ -57,31 +57,43 @@ static inline void _cds_wfq_init(struct cds_wfq_queue *q) > { > int ret; > > - _cds_wfq_node_init(&q->dummy); > /* Set queue head and tail */ > - q->head = &q->dummy; > - q->tail = &q->dummy.next; > + _cds_wfq_node_init(&q->head); > + q->tail = &q->head; > ret = pthread_mutex_init(&q->lock, NULL); > assert(!ret); > } > > -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > - struct cds_wfq_node *node) > +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) > { > - struct cds_wfq_node **old_tail; > + /* > + * Queue is empty if no node is pointed by q->head.next nor q->tail. > + */ > + return q->head.next == NULL && CMM_LOAD_SHARED(q->tail) == &q->head; > +} > > +static inline void ___cds_wfq_append_list(struct cds_wfq_queue *q, > + struct cds_wfq_node *head, struct cds_wfq_node *tail) > +{ > /* > * uatomic_xchg() implicit memory barrier orders earlier stores to data > * structure containing node and setting node->next to NULL before > * publication. > */ > - old_tail = uatomic_xchg(&q->tail, &node->next); > + tail = uatomic_xchg(&q->tail, tail); > + > /* > - * At this point, dequeuers see a NULL old_tail->next, which indicates > + * At this point, dequeuers see a NULL tail->next, which indicates > * that the queue is being appended to. The following store will append > * "node" to the queue from a dequeuer perspective. > */ > - CMM_STORE_SHARED(*old_tail, node); > + CMM_STORE_SHARED(tail->next, head); > +} > + > +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > + struct cds_wfq_node *node) > +{ > + ___cds_wfq_append_list(q, node, node); > } > > /* > @@ -120,27 +132,46 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > { > struct cds_wfq_node *node, *next; > > - /* > - * Queue is empty if it only contains the dummy node. > - */ > - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) > + if (_cds_wfq_empty(q)) > return NULL; > - node = q->head; > > - next = ___cds_wfq_node_sync_next(node); > + node = ___cds_wfq_node_sync_next(&q->head); > + > + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { > + if (CMM_LOAD_SHARED(q->tail) == node) { > + /* > + * @node is the only node in the queue. > + * Try to move the tail to &q->head > + */ > + _cds_wfq_node_init(&q->head); > + if (uatomic_cmpxchg(&q->tail, node, &q->head) == node) > + return node; > + } > + next = ___cds_wfq_node_sync_next(node); > + } > > /* > * Move queue head forward. > */ > - q->head = next; > - /* > - * Requeue dummy node if we just dequeued it. > - */ > - if (node == &q->dummy) { > - _cds_wfq_node_init(node); > - _cds_wfq_enqueue(q, node); > - return ___cds_wfq_dequeue_blocking(q); > - } > + q->head.next = next; > + > + return node; > +} > + > +/* dequeue all nodes, the nodes are not synchronized for the next pointer */ > +static inline struct cds_wfq_node * > +___cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, > + struct cds_wfq_node **tail) > +{ > + struct cds_wfq_node *node; > + > + if (_cds_wfq_empty(q)) > + return NULL; > + > + node = ___cds_wfq_node_sync_next(&q->head); > + _cds_wfq_node_init(&q->head); > + *tail = uatomic_xchg(&q->tail, &q->head); > + > return node; > } > > @@ -158,6 +189,27 @@ _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > return retnode; > } > > +static inline struct cds_wfq_node * > +_cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, > + struct cds_wfq_node **tail) > +{ > + struct cds_wfq_node *node, *next; > + int ret; > + > + ret = pthread_mutex_lock(&q->lock); > + assert(!ret); > + node = ___cds_wfq_dequeue_all_blocking(q, tail); > + ret = pthread_mutex_unlock(&q->lock); > + assert(!ret); > + > + /* synchronize all nodes' next pointer */ > + next = node; > + while (next != *tail) > + next = ___cds_wfq_node_sync_next(next); > + > + return node; > +} > + > #ifdef __cplusplus > } > #endif > diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h > index 03a73f1..985f540 100644 > --- a/urcu/wfqueue.h > +++ b/urcu/wfqueue.h > @@ -7,6 +7,7 @@ > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > * > * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -25,6 +26,7 @@ > > #include > #include > +#include > #include > > #ifdef __cplusplus > @@ -33,8 +35,6 @@ extern "C" { > > /* > * Queue with wait-free enqueue/blocking dequeue. > - * This implementation adds a dummy head node when the queue is empty to ensure > - * we can always update the queue locklessly. > * > * Inspired from half-wait-free/half-blocking queue implementation done by > * Paul E. McKenney. > @@ -45,8 +45,7 @@ struct cds_wfq_node { > }; > > struct cds_wfq_queue { > - struct cds_wfq_node *head, **tail; > - struct cds_wfq_node dummy; /* Dummy node */ > + struct cds_wfq_node head, *tail; > pthread_mutex_t lock; > }; > > @@ -56,18 +55,36 @@ struct cds_wfq_queue { > > #define cds_wfq_node_init _cds_wfq_node_init > #define cds_wfq_init _cds_wfq_init > +#define cds_wfq_empty _cds_wfq_empty > +#define __cds_wfq_append_list ___cds_wfq_append_list > #define cds_wfq_enqueue _cds_wfq_enqueue > #define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking > #define cds_wfq_dequeue_blocking _cds_wfq_dequeue_blocking > +#define __cds_wfq_node_sync_next ___cds_wfq_node_sync_next > +#define __cds_wfq_dequeue_all_blocking ___cds_wfq_dequeue_all_blocking > +#define cds_wfq_dequeue_all_blocking _cds_wfq_dequeue_all_blocking > > #else /* !_LGPL_SOURCE */ > > extern void cds_wfq_node_init(struct cds_wfq_node *node); > extern void cds_wfq_init(struct cds_wfq_queue *q); > +extern bool cds_wfq_empty(struct cds_wfq_queue *q); > +/* __cds_wfq_append_list: caller ensures mutual exclusion between dequeues */ > +extern void __cds_wfq_append_list(struct cds_wfq_queue *q, > + struct cds_wfq_node *head, struct cds_wfq_node *tail); > extern void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node); > /* __cds_wfq_dequeue_blocking: caller ensures mutual exclusion between dequeues */ > extern struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); > extern struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); > +extern struct cds_wfq_node *__cds_wfq_node_sync_next(struct cds_wfq_node *node); > +/* > + * __cds_wfq_dequeue_all_blocking: caller ensures mutual exclusion between > + * dequeues, and need synchronize next pointer berfore use it. > + */ > +extern struct cds_wfq_node *__cds_wfq_dequeue_all_blocking( > + struct cds_wfq_queue *q, struct cds_wfq_node **tail); > +extern struct cds_wfq_node *cds_wfq_dequeue_all_blocking( > + struct cds_wfq_queue *q, struct cds_wfq_node **tail); > > #endif /* !_LGPL_SOURCE */ > > diff --git a/wfqueue.c b/wfqueue.c > index 3337171..28a7b58 100644 > --- a/wfqueue.c > +++ b/wfqueue.c > @@ -4,6 +4,7 @@ > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > * > * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -38,6 +39,17 @@ void cds_wfq_init(struct cds_wfq_queue *q) > _cds_wfq_init(q); > } > > +bool cds_wfq_empty(struct cds_wfq_queue *q) > +{ > + return _cds_wfq_empty(q); > +} > + > +void __cds_wfq_append_list(struct cds_wfq_queue *q, > + struct cds_wfq_node *head, struct cds_wfq_node *tail) > +{ > + return ___cds_wfq_append_list(q, head, tail); > +} > + > void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node) > { > _cds_wfq_enqueue(q, node); > @@ -52,3 +64,20 @@ struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > { > return _cds_wfq_dequeue_blocking(q); > } > + > +struct cds_wfq_node *__cds_wfq_node_sync_next(struct cds_wfq_node *node) > +{ > + return ___cds_wfq_node_sync_next(node); > +} > + > +struct cds_wfq_node *__cds_wfq_dequeue_all_blocking( > + struct cds_wfq_queue *q, struct cds_wfq_node **tail) > +{ > + return ___cds_wfq_dequeue_all_blocking(q, tail); > +} > + > +struct cds_wfq_node *cds_wfq_dequeue_all_blocking( > + struct cds_wfq_queue *q, struct cds_wfq_node **tail) > +{ > + return _cds_wfq_dequeue_all_blocking(q, tail); > +} > -- > 1.7.7 > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From dgoulet at efficios.com Fri Aug 10 16:36:59 2012 From: dgoulet at efficios.com (David Goulet) Date: Fri, 10 Aug 2012 16:36:59 -0400 Subject: [lttng-dev] [PATCH lttng-tools 0/2] relayd IPv6 fixes In-Reply-To: <1344619372-21976-1-git-send-email-christian.babeux@efficios.com> References: <1344619372-21976-1-git-send-email-christian.babeux@efficios.com> Message-ID: <502570EB.8080303@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Merged! Thanks Christian Babeux: > Hi all, > > Here are a few fixes to IPv6 support in relayd. > > When trying to start the relayd using an IPv6 address, the relayd > simply segfaulted. The first patch fix this issue. > > The second patch fix a wrong domain value used in the socket() call > for the IPv6 socket causing a "Address family not supported by > protocol" error when calling bind(). > > Thanks, > > Christian > > Christian Babeux (2): Fix: Off by one in lttcomm_sock_domain enum > Fix: Wrong domain used when initializing IPv6 sockets > > src/common/sessiond-comm/inet6.c | 2 +- > src/common/sessiond-comm/sessiond-comm.h | 4 ++-- 2 files changed, > 3 insertions(+), 3 deletions(-) > -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQJXDnAAoJEELoaioR9I029KwIALM5s5qYwKhmrMpXxY87VXsV Bqp7AUi4gDcHCC7oU74kCYG4QDwRhOAmFYAQ43R2PmkydRUemlwHM7emFgCycnJ9 ot5uOBqCZtdnP0BqilFyLldm+a4h/LKCOEEqYlPDJeMRzcnbiKWAexyGsvqow9pi 5QH8jaflXfMWX+LEDRdevVB4lIvuURIPWd7SZuIRqWcxCgC2r921KWhgl7yZQd1R 4hWXeCaVLP6kD98Kzj9D98Ex1cRgEsJPUu79uHKU+mGQaueC9c2J9c/nPbcMs6Bz lc/WQrGrEhbENgkGu1M0KQ1Iwn5U6XKpacGHAhUrfo+5mlI6iYiT4G/Z7F+09r0= =VLFk -----END PGP SIGNATURE----- From danny.serres at efficios.com Fri Aug 10 17:06:55 2012 From: danny.serres at efficios.com (Danny Serres) Date: Fri, 10 Aug 2012 17:06:55 -0400 Subject: [lttng-dev] =?utf-8?q?=5Bbabeltrace_PATCH=5D_Babeltrace_python_mo?= =?utf-8?q?dule_v3?= Message-ID: <1344632815-6541-1-git-send-email-danny.serres@efficios.com> The Babeltrace Python module can be used to directly control the Babeltrace API inside Python, using 'import babeltrace'. Therefore, it becomes possible to create a Context, add a trace to it, iterate on it, read events and so on from within Python. SWIG >= 2.0 is used to create the wrapper and its 'warning md variable unused' bug is patched in Makefile.am In the interface file, struct and enum are directly copied from the include files. All changes to struct bt_iter_pos and to enums in ctf/events.h and clock-types.h must also be made in the interface file. Signed-off-by: Danny Serres Signed-off-by: Yannick Brosseau --- .gitignore | 4 + Makefile.am | 2 +- README | 6 + bindings/Makefile.am | 3 + bindings/python/Makefile.am | 28 + bindings/python/babeltrace.i.in | 1079 +++++++++ bindings/python/examples/babeltrace_and_lttng.py | 107 + bindings/python/examples/eventcount.py | 66 + bindings/python/examples/eventcountlist.py | 65 + bindings/python/examples/events_per_cpu.py | 81 + bindings/python/examples/example-api-test.py | 59 + bindings/python/examples/histogram.py | 121 + .../examples/output_format_modules/cairoplot.py | 2336 ++++++++++++++++++++ .../examples/output_format_modules/pprint_table.py | 35 + .../examples/output_format_modules/series.py | 1140 ++++++++++ bindings/python/examples/sched_switch.py | 110 + bindings/python/examples/softirqtimes.py | 130 ++ bindings/python/examples/syscalls_by_pid.py | 61 + bindings/python/python-complements.c | 105 + bindings/python/python-complements.h | 36 + bootstrap | 2 +- configure.ac | 37 + doc/python-howto.txt | 70 + m4/ax_pkg_swig.m4 | 135 ++ tests/tests-python.py | 115 + 25 files changed, 5931 insertions(+), 2 deletions(-) create mode 100644 bindings/Makefile.am create mode 100644 bindings/python/Makefile.am create mode 100644 bindings/python/babeltrace.i.in create mode 100644 bindings/python/examples/babeltrace_and_lttng.py create mode 100644 bindings/python/examples/eventcount.py create mode 100644 bindings/python/examples/eventcountlist.py create mode 100644 bindings/python/examples/events_per_cpu.py create mode 100644 bindings/python/examples/example-api-test.py create mode 100644 bindings/python/examples/histogram.py create mode 100644 bindings/python/examples/output_format_modules/__init__.py create mode 100755 bindings/python/examples/output_format_modules/cairoplot.py create mode 100644 bindings/python/examples/output_format_modules/pprint_table.py create mode 100755 bindings/python/examples/output_format_modules/series.py create mode 100644 bindings/python/examples/sched_switch.py create mode 100644 bindings/python/examples/softirqtimes.py create mode 100644 bindings/python/examples/syscalls_by_pid.py create mode 100644 bindings/python/python-complements.c create mode 100644 bindings/python/python-complements.h create mode 100644 doc/python-howto.txt create mode 100644 m4/ax_pkg_swig.m4 create mode 100644 tests/tests-python.py diff --git a/.gitignore b/.gitignore index d6098ac..3fe60e7 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ /tests/test-bitfield +*~ *.o *.a *.la @@ -26,3 +27,6 @@ converter/babeltrace-log core formats/ctf/metadata/ctf-parser.output stamp-h1 +bindings/python/babeltrace.i +bindings/python/babeltrace.py +bindings/python/babeltrace_wrap.c diff --git a/Makefile.am b/Makefile.am index 308ee16..6584c5d 100644 --- a/Makefile.am +++ b/Makefile.am @@ -2,7 +2,7 @@ AM_CFLAGS = $(PACKAGE_CFLAGS) -I$(top_srcdir)/include ACLOCAL_AMFLAGS = -I m4 -SUBDIRS = include types lib formats converter tests doc +SUBDIRS = include types lib formats converter bindings tests doc dist_doc_DATA = ChangeLog LICENSE mit-license.txt gpl-2.0.txt \ std-ext-lib.txt diff --git a/README b/README index 75bf0cf..1687075 100644 --- a/README +++ b/README @@ -25,6 +25,7 @@ BUILDING make install ldconfig + If you do not want Python bindings, run ./configure --disable-python. DEPENDENCIES ------------ @@ -44,6 +45,11 @@ To compile Babeltrace, you will need: libpopt >= 1.13 development libraries (Debian : libpopt-dev) (Fedora : popt) + python headers (optional) + (Debian/Ubuntu : python-dev) + swig >= 2.0 (optional) + (Debian/Ubuntu : swig2.0) + For developers using the git tree: diff --git a/bindings/Makefile.am b/bindings/Makefile.am new file mode 100644 index 0000000..dcd868d --- /dev/null +++ b/bindings/Makefile.am @@ -0,0 +1,3 @@ +if USE_PYTHON +SUBDIRS = python +endif diff --git a/bindings/python/Makefile.am b/bindings/python/Makefile.am new file mode 100644 index 0000000..579759f --- /dev/null +++ b/bindings/python/Makefile.am @@ -0,0 +1,28 @@ +babeltrace.i: babeltrace.i.in + sed "s/BABELTRACE_VERSION_STR/Babeltrace $(PACKAGE_VERSION)/g" babeltrace.i + +AM_CFLAGS = -I$(PYTHON_INCLUDE) -I$(top_srcdir)/include/ + +EXTRA_DIST = babeltrace.i +python_PYTHON = babeltrace.py +pyexec_LTLIBRARIES = _babeltrace.la + +MAINTAINERCLEANFILES = babeltrace_wrap.c babeltrace.py + +_babeltrace_la_SOURCES = babeltrace_wrap.c python-complements.c + +_babeltrace_la_LDFLAGS = -module + +_babeltrace_la_CFLAGS = $(GLIB_CFLAGS) $(AM_CFLAGS) + +_babeltrace_la_LIBS = $(GLIB_LIBS) + +_babeltrace_la_LIBADD = $(top_srcdir)/formats/ctf/libbabeltrace-ctf.la \ + $(top_srcdir)/formats/ctf-text/libbabeltrace-ctf-text.la + +# SWIG 'warning md variable unused' fixed after SWIG build: +babeltrace_wrap.c: babeltrace.i + $(SWIG) -python -Wall -I. -I$(top_srcdir)/include babeltrace.i + sed -i "s/PyObject \*m, \*d, \*md;/PyObject \*m, \*d;\n#if defined(SWIGPYTHON_BUILTIN)\nPyObject *md;\n#endif/g" babeltrace_wrap.c + sed -i "s/md = d/d/g" babeltrace_wrap.c + sed -i "s/(void)public_symbol;/(void)public_symbol;\n md = d;/g" babeltrace_wrap.c diff --git a/bindings/python/babeltrace.i.in b/bindings/python/babeltrace.i.in new file mode 100644 index 0000000..49636d1 --- /dev/null +++ b/bindings/python/babeltrace.i.in @@ -0,0 +1,1079 @@ +/* BABELTRACE PYTHON MODULE interface file */ + +%define DOCSTRING +"BABELTRACE_VERSION_STR + +Babeltrace is a trace viewer and converter reading and writing the +Common Trace Format (CTF). Its main use is to pretty-print CTF +traces into a human-readable text output. + +To use this module, the first step is to create a Context and add a +trace to it." +%enddef + +%module(docstring=DOCSTRING) babeltrace + +%include "typemaps.i" +%{ +#define SWIG_FILE_WITH_INIT +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "python-complements.h" +%} + +typedef unsigned long long uint64_t; +typedef long long int64_t; +typedef int bt_intern_str; + +/* ================================================================= + CONTEXT.H, CONTEXT-INTERNAL.H + ????????????????????????????? +*/ + +%rename("_bt_context_create") bt_context_create(void); +%rename("_bt_context_add_trace") bt_context_add_trace( + struct bt_context *ctx, const char *path, const char *format, + void (*packet_seek)(struct stream_pos *pos, size_t index, int whence), + struct mmap_stream_list *stream_list, FILE *metadata); +%rename("_bt_context_remove_trace") bt_context_remove_trace( + struct bt_context *ctx, int trace_id); +%rename("_bt_context_get") bt_context_get(struct bt_context *ctx); +%rename("_bt_context_put") bt_context_put(struct bt_context *ctx); +%rename("_bt_ctf_event_get_context") bt_ctf_event_get_context( + const struct bt_ctf_event *event); + +struct bt_context *bt_context_create(void); +int bt_context_add_trace(struct bt_context *ctx, const char *path, const char *format, + void (*packet_seek)(struct stream_pos *pos, size_t index, int whence), + struct mmap_stream_list *stream_list, FILE *metadata); +void bt_context_remove_trace(struct bt_context *ctx, int trace_id); +void bt_context_get(struct bt_context *ctx); +void bt_context_put(struct bt_context *ctx); +struct bt_context *bt_ctf_event_get_context(const struct bt_ctf_event *event); + +// class Context to prevent direct access to struct bt_context +%pythoncode%{ +class Context: + """ + The context represents the object in which a trace_collection is + open. As long as this structure is allocated, the trace_collection + is open and the traces it contains can be read and seeked by the + iterators and callbacks. + """ + + def __init__(self): + self._c = _bt_context_create() + + def __del__(self): + _bt_context_put(self._c) + + def add_trace(self, path, format_str, + packet_seek=None, stream_list=None, metadata=None): + """ + Add a trace by path to the context. + + Open a trace. + + path is the path to the trace, it is not recursive. + If "path" is None, stream_list is used instead as a list + of mmap streams to open for the trace. + + format is a string containing the format name in which the trace was + produced. + + packet_seek is not implemented for Python. Should be left None to + use the default packet_seek handler provided by the trace format. + + stream_list is a linked list of streams, it is used to open a trace + where the trace data is located in memory mapped areas instead of + trace files, this argument should be None when path is not None. + + The metadata parameter acts as a metadata override when not None, + otherwise the format handles the metadata opening. + + Return: the corresponding TraceHandle on success or None on error. + """ + if metadata is not None: + metadata = metadata._file + + ret = _bt_context_add_trace(self._c, path, format_str, packet_seek, + stream_list, metadata) + if ret < 0: + return None + + th = TraceHandle.__new__(TraceHandle) + th._id = ret + return th + + def add_traces_recursive(self, path, format_str): + """ + Open a trace recursively. + + Find each trace present in the subdirectory starting from the given + path, and add them to the context. + + Return a dict of TraceHandle instances (the full path is the key). + Return None on error. + """ + + import os + + trace_handles = {} + + noTrace = True + error = False + + for fullpath, dirs, files in os.walk(path): + if "metadata" in files: + trace_handle = self.add_trace(fullpath, format_str) + if trace_handle is None: + error = True + continue + + trace_handles[fullpath] = trace_handle + noTrace = False + + if noTrace and error: + return None + return trace_handles + + def remove_trace(self, trace_handle): + """ + Remove a trace from the context. + Effectively closing the trace. + """ + try: + _bt_context_remove_trace(self._c, trace_handle._id) + except AttributeError: + raise TypeError("in remove_trace, " + "argument 2 must be a TraceHandle instance") +%} + + + +/* ================================================================= + FORMAT.H, REGISTRY + ?????????????????? +*/ + +%rename("lookup_format") bt_lookup_format(bt_intern_str qname); +%rename("_bt_print_format_list") bt_fprintf_format_list(FILE *fp); +%rename("register_format") bt_register_format(struct format *format); + +extern struct format *bt_lookup_format(bt_intern_str qname); +extern void bt_fprintf_format_list(FILE *fp); +extern int bt_register_format(struct format *format); + +void format_init(void); +void format_finalize(void); + +%pythoncode %{ + +def print_format_list(babeltrace_file): + """ + Print a list of available formats to file. + + babeltrace_file must be a File instance opened in write mode. + """ + try: + if babeltrace_file._file is not None: + _bt_print_format_list(babeltrace_file._file) + except AttributeError: + raise TypeError("in print_format_list, " + "argument 1 must be a File instance") + +%} + + +/* ================================================================= + ITERATOR.H, ITERATOR-INTERNAL.H + ??????????????????????????????? +*/ + +%rename("_bt_iter_create") bt_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, const struct bt_iter_pos *end_pos); +%rename("_bt_iter_destroy") bt_iter_destroy(struct bt_iter *iter); +%rename("_bt_iter_next") bt_iter_next(struct bt_iter *iter); +%rename("_bt_iter_get_pos") bt_iter_get_pos(struct bt_iter *iter); +%rename("_bt_iter_free_pos") bt_iter_free_pos(struct bt_iter_pos *pos); +%rename("_bt_iter_set_pos") bt_iter_set_pos(struct bt_iter *iter, + const struct bt_iter_pos *pos); +%rename("_bt_iter_create_time_pos") bt_iter_create_time_pos(struct bt_iter *iter, + uint64_t timestamp); + +struct bt_iter *bt_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, const struct bt_iter_pos *end_pos); +void bt_iter_destroy(struct bt_iter *iter); +int bt_iter_next(struct bt_iter *iter); +struct bt_iter_pos *bt_iter_get_pos(struct bt_iter *iter); +void bt_iter_free_pos(struct bt_iter_pos *pos); +int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *pos); +struct bt_iter_pos *bt_iter_create_time_pos(struct bt_iter *iter, uint64_t timestamp); + +%rename("_bt_iter_pos") bt_iter_pos; +%rename("SEEK_TIME") BT_SEEK_TIME; +%rename("SEEK_RESTORE") BT_SEEK_RESTORE; +%rename("SEEK_CUR") BT_SEEK_CUR; +%rename("SEEK_BEGIN") BT_SEEK_BEGIN; +%rename("SEEK_END") BT_SEEK_END; + + +// This struct is taken from iterator.h +// All changes to the struct must also be made here +struct bt_iter_pos { + enum { + BT_SEEK_TIME, /* uses u.seek_time */ + BT_SEEK_RESTORE, /* uses u.restore */ + BT_SEEK_CUR, + BT_SEEK_BEGIN, + BT_SEEK_END, + } type; + union { + uint64_t seek_time; + struct bt_saved_pos *restore; + } u; +}; + + +%pythoncode%{ + +class IterPos: + """This class represents the position where to set an iterator.""" + + __can_access = False + + def __init__(self, seek_type, seek_time = None): + """ + seek_type represents the type of seek to use. + seek_time is the timestamp to seek to when using SEEK_TIME, it + is expressed in nanoseconds + Only use SEEK_RESTORE on IterPos obtained from the get_pos function + in Iter class. + """ + + self._pos = _bt_iter_pos() + self._pos.type = seek_type + if seek_time and seek_type == SEEK_TIME: + self._pos.u.seek_time = seek_time + self.__can_access = True + + def __del__(self): + if not self.__can_access: + _bt_iter_free_pos(self._pos) + + def _get_type(self): + if not __can_access: + raise AttributeError("seek_type is not available") + return self._pos.type + + def _set_type(self, seek_type): + if not __can_access: + raise AttributeError("seek_type is not available") + self._pos.type = seek_type + + def _get_time(self): + if not __can_access: + raise AttributeError("seek_time is not available") + + elif self._pos.type is not SEEK_TIME: + raise TypeError("seek_type is not SEEK_TIME") + + return self._pos.u.seek_time + + def _set_time(self, time): + if not __can_access: + raise AttributeError("seek_time is not available") + + elif self._pos.type is not SEEK_TIME: + raise TypeError("seek_type is not SEEK_TIME") + + self._pos.u.seek_time = time + + def _get_pos(self): + return self._pos + + + seek_type = property(_get_type, _set_type) + seek_time = property(_get_time, _set_time) + + +class Iterator: + + __with_init = False + + def __init__(self, context, begin_pos = None, end_pos = None, _no_init = None): + """ + Allocate a trace collection iterator. + + begin_pos and end_pos are optional parameters to specify the + position at which the trace collection should be seeked upon + iterator creation, and the position at which iteration will + start returning "EOF". + + By default, if begin_pos is None, a BT_SEEK_CUR is performed at + creation. By default, if end_pos is None, a BT_SEEK_END (end of + trace) is the EOF criterion. + """ + if _no_init is None: + if begin_pos is None: + bp = None + else: + try: + bp = begin_pos._pos + except AttributeError: + raise TypeError("in __init__, " + "argument 3 must be a IterPos instance") + + if end_pos is None: + ep = None + else: + try: + ep = end_pos._pos + except AttributeError: + raise TypeError("in __init__, " + "argument 4 must be a IterPos instance") + + try: + self._bi = _bt_iter_create(context._c, bp, ep) + except AttributeError: + raise TypeError("in __init__, " + "argument 2 must be a Context instance") + + self.__with_init = True + + else: + self._bi = _no_init + + def __del__(self): + if self.__with_init: + _bt_iter_destroy(self._bi) + + def next(self): + """ + Move trace collection position to the next event. + Returns 0 on success, a negative value on error. + """ + return _bt_iter_next(self._bi) + + def get_pos(self): + """Return a IterPos class of the current iterator position.""" + ret = IterPos(0) + ret.__can_access = False + ret._pos = _bt_iter_get_pos(self._bi) + return ret + + def set_pos(self, pos): + """ + Move the iterator to a given position. + + On error, the stream_heap is reinitialized and returned empty. + Return 0 for success. + Return EOF if the position requested is after the last event of the + trace collection. + Return -EINVAL when called with invalid parameter. + Return -ENOMEM if the stream_heap could not be properly initialized. + """ + try: + return _bt_iter_set_pos(self._bi, pos._pos) + except AttributeError: + raise TypeError("in set_pos, " + "argument 2 must be a IterPos instance") + + def create_time_pos(self, timestamp): + """ + Create a position based on time + This function allocates and returns a new IterPos to be able to + restore an iterator position based on a timestamp. + """ + + if timestamp < 0: + raise TypeError("timestamp must be an unsigned int") + + ret = IterPos(0) + ret.__can_access = False + ret._pos = _bt_iter_create_time_pos(self._bi, timestamp) + return ret +%} + + +/* ================================================================= + CLOCK-TYPE.H + ???????????? + *** Enum copied from clock-type.h? + All changes must also be made here +*/ +%rename("CLOCK_CYCLES") BT_CLOCK_CYCLES; +%rename("CLOCK_REAL") BT_CLOCK_REAL; + +enum bt_clock_type { + BT_CLOCK_CYCLES = 0, + BT_CLOCK_REAL +}; + +/* ================================================================= + TRACE-HANDLE.H, TRACE-HANDLE-INTERNAL.H + ??????????????????????????????????????? +*/ + +%rename("_bt_trace_handle_create") bt_trace_handle_create(struct bt_context *ctx); +%rename("_bt_trace_handle_destroy") bt_trace_handle_destroy(struct bt_trace_handle *bt); +struct bt_trace_handle *bt_trace_handle_create(struct bt_context *ctx); +void bt_trace_handle_destroy(struct bt_trace_handle *bt); + +%rename("_bt_trace_handle_get_path") bt_trace_handle_get_path(struct bt_context *ctx, + int handle_id); +%rename("_bt_trace_handle_get_timestamp_begin") bt_trace_handle_get_timestamp_begin( + struct bt_context *ctx, int handle_id, enum bt_clock_type type); +%rename("_bt_trace_handle_get_timestamp_end") bt_trace_handle_get_timestamp_end( + struct bt_context *ctx, int handle_id, enum bt_clock_type type); +%rename("_bt_trace_handle_get_id") bt_trace_handle_get_id(struct bt_trace_handle *th); +int bt_trace_handle_get_id(struct bt_trace_handle *th); +const char *bt_trace_handle_get_path(struct bt_context *ctx, int handle_id); +uint64_t bt_trace_handle_get_timestamp_begin(struct bt_context *ctx, int handle_id, + enum bt_clock_type type); +uint64_t bt_trace_handle_get_timestamp_end(struct bt_context *ctx, int handle_id, + enum bt_clock_type type); + +%rename("_bt_ctf_event_get_handle_id") bt_ctf_event_get_handle_id( + const struct bt_ctf_event *event); +int bt_ctf_event_get_handle_id(const struct bt_ctf_event *event); + + +%pythoncode%{ + +class TraceHandle(object): + """ + The TraceHandle allows the user to manipulate a trace file directly. + It is a unique identifier representing a trace file. + Do not instantiate. + """ + + def __init__(self): + raise NotImplementedError("TraceHandle cannot be instantiated") + + def __repr__(self): + return "Babeltrace TraceHandle: trace_id('{}')".format(self._id) + + def get_id(self): + """Return the TraceHandle id.""" + return self._id + + def get_path(self, context): + """Return the path of a TraceHandle.""" + try: + return _bt_trace_handle_get_path(context._c, self._id) + except AttributeError: + raise TypeError("in get_path, " + "argument 2 must be a Context instance") + + def get_timestamp_begin(self, context, clock_type): + """Return the creation time of the buffers of a trace.""" + try: + return _bt_trace_handle_get_timestamp_begin( + context._c, self._id,clock_type) + except AttributeError: + raise TypeError("in get_timestamp_begin, " + "argument 2 must be a Context instance") + + def get_timestamp_end(self, context, clock_type): + """Return the destruction timestamp of the buffers of a trace.""" + try: + return _bt_trace_handle_get_timestamp_end( + context._c, self._id, clock_type) + except AttributeError: + raise TypeError("in get_timestamp_end, " + "argument 2 must be a Context instance") + +%} + + + +// ================================================================= +// CTF +// ================================================================= + +/* ================================================================= + ITERATOR.H, EVENTS.H + ???????????????????? +*/ + +//Iterator +%rename("_bt_ctf_iter_create") bt_ctf_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, + const struct bt_iter_pos *end_pos); +%rename("_bt_ctf_get_iter") bt_ctf_get_iter(struct bt_ctf_iter *iter); +%rename("_bt_ctf_iter_destroy") bt_ctf_iter_destroy(struct bt_ctf_iter *iter); +%rename("_bt_ctf_iter_read_event") bt_ctf_iter_read_event(struct bt_ctf_iter *iter); + +struct bt_ctf_iter *bt_ctf_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, + const struct bt_iter_pos *end_pos); +struct bt_iter *bt_ctf_get_iter(struct bt_ctf_iter *iter); +void bt_ctf_iter_destroy(struct bt_ctf_iter *iter); +struct bt_ctf_event *bt_ctf_iter_read_event(struct bt_ctf_iter *iter); + + +//Events + +%rename("_bt_ctf_get_top_level_scope") bt_ctf_get_top_level_scope(const struct + bt_ctf_event *event, enum bt_ctf_scope scope); +%rename("_bt_ctf_event_name") bt_ctf_event_name(const struct bt_ctf_event *ctf_event); +%rename("_bt_ctf_get_timestamp") bt_ctf_get_timestamp( + const struct bt_ctf_event *ctf_event); +%rename("_bt_ctf_get_cycles") bt_ctf_get_cycles( + const struct bt_ctf_event *ctf_event); + +%rename("_bt_ctf_get_field") bt_ctf_get_field(const struct bt_ctf_event *ctf_event, + const struct definition *scope, const char *field); +%rename("_bt_ctf_get_index") bt_ctf_get_index(const struct bt_ctf_event *ctf_event, + const struct definition *field, unsigned int index); +%rename("_bt_ctf_field_name") bt_ctf_field_name(const struct definition *def); +%rename("_bt_ctf_field_type") bt_ctf_field_type(const struct definition *def); +%rename("_bt_ctf_get_int_signedness") bt_ctf_get_int_signedness( + const struct definition *field); +%rename("_bt_ctf_get_int_base") bt_ctf_get_int_base(const struct definition *field); +%rename("_bt_ctf_get_int_byte_order") bt_ctf_get_int_byte_order( + const struct definition *field); +%rename("_bt_ctf_get_int_len") bt_ctf_get_int_len(const struct definition *field); +%rename("_bt_ctf_get_encoding") bt_ctf_get_encoding(const struct definition *field); +%rename("_bt_ctf_get_array_len") bt_ctf_get_array_len(const struct definition *field); +%rename("_bt_ctf_get_uint64") bt_ctf_get_uint64(const struct definition *field); +%rename("_bt_ctf_get_int64") bt_ctf_get_int64(const struct definition *field); +%rename("_bt_ctf_get_char_array") bt_ctf_get_char_array(const struct definition *field); +%rename("_bt_ctf_get_string") bt_ctf_get_string(const struct definition *field); +%rename("_bt_ctf_field_get_error") bt_ctf_field_get_error(void); +%rename("_bt_ctf_get_decl_event_name") bt_ctf_get_decl_event_name(const struct + bt_ctf_event_decl *event); +%rename("_bt_ctf_get_decl_field_name") bt_ctf_get_decl_field_name( + const struct bt_ctf_field_decl *field); + +const struct definition *bt_ctf_get_top_level_scope(const struct bt_ctf_event *ctf_event, + enum bt_ctf_scope scope); +const char *bt_ctf_event_name(const struct bt_ctf_event *ctf_event); +uint64_t bt_ctf_get_timestamp(const struct bt_ctf_event *ctf_event); +uint64_t bt_ctf_get_cycles(const struct bt_ctf_event *ctf_event); +const struct definition *bt_ctf_get_field(const struct bt_ctf_event *ctf_event, + const struct definition *scope, + const char *field); +const struct definition *bt_ctf_get_index(const struct bt_ctf_event *ctf_event, + const struct definition *field, + unsigned int index); +const char *bt_ctf_field_name(const struct definition *def); +enum ctf_type_id bt_ctf_field_type(const struct definition *def); +int bt_ctf_get_int_signedness(const struct definition *field); +int bt_ctf_get_int_base(const struct definition *field); +int bt_ctf_get_int_byte_order(const struct definition *field); +ssize_t bt_ctf_get_int_len(const struct definition *field); +enum ctf_string_encoding bt_ctf_get_encoding(const struct definition *field); +int bt_ctf_get_array_len(const struct definition *field); +uint64_t bt_ctf_get_uint64(const struct definition *field); +int64_t bt_ctf_get_int64(const struct definition *field); +char *bt_ctf_get_char_array(const struct definition *field); +char *bt_ctf_get_string(const struct definition *field); +int bt_ctf_field_get_error(void); +const char *bt_ctf_get_decl_event_name(const struct bt_ctf_event_decl *event); +const char *bt_ctf_get_decl_field_name(const struct bt_ctf_field_decl *field); + +%pythoncode%{ + +class ctf: + + #enum equivalent, accessible constants + #These are taken directly from ctf/events.h + #All changes to enums must also be made here + class type_id: + UNKNOWN = 0 + INTEGER = 1 + FLOAT = 2 + ENUM = 3 + STRING = 4 + STRUCT = 5 + UNTAGGED_VARIANT = 6 + VARIANT = 7 + ARRAY = 8 + SEQUENCE = 9 + NR_CTF_TYPES = 10 + + class scope: + TRACE_PACKET_HEADER = 0 + STREAM_PACKET_CONTEXT = 1 + STREAM_EVENT_HEADER = 2 + STREAM_EVENT_CONTEXT = 3 + EVENT_CONTEXT = 4 + EVENT_FIELDS = 5 + + class string_encoding: + NONE = 0 + UTF8 = 1 + ASCII = 2 + UNKNOWN = 3 + + class Iterator(Iterator, object): + """ + Allocate a CTF trace collection iterator. + + begin_pos and end_pos are optional parameters to specify the + position at which the trace collection should be seeked upon + iterator creation, and the position at which iteration will + start returning "EOF". + + By default, if begin_pos is None, a SEEK_CUR is performed at + creation. By default, if end_pos is None, a SEEK_END (end of + trace) is the EOF criterion. + + Only one iterator can be created against a context. If more than one + iterator is being created for the same context, the second creation + will return None. The previous iterator must be destroyed before + creation of the new iterator for this function to succeed. + """ + + def __new__(cls, context, begin_pos = None, end_pos = None): + # __new__ is used to control the return value + # as the ctf.Iterator class should return None + # if bt_ctf_iter_create returns NULL + + if begin_pos is None: + bp = None + else: + bp = begin_pos._pos + if end_pos is None: + ep = None + else: + ep = end_pos._pos + try: + it = _bt_ctf_iter_create(context._c, bp, ep) + except AttributeError: + raise TypeError("in __init__, " + "argument 2 must be a Context instance") + if it is None: + return None + + ret_class = super(ctf.Iterator, cls).__new__(cls) + ret_class._i = it + return ret_class + + def __init__(self, context, begin_pos = None, end_pos = None): + Iterator.__init__(self, None, None, None, + _bt_ctf_get_iter(self._i)) + + def __del__(self): + _bt_ctf_iter_destroy(self._i) + + def read_event(self): + """ + Read the iterator's current event data. + Return current event on success, None on end of trace. + """ + ret = _bt_ctf_iter_read_event(self._i) + if ret is None: + return ret + ev = ctf.Event.__new__(ctf.Event) + ev._e = ret + return ev + + + class Event(object): + """ + This class represents an event from the trace. + It is obtained with read_event() from ctf.Iterator. + Do not instantiate. + """ + + def __init__(self): + raise NotImplementedError("ctf.Event cannot be instantiated") + + def get_top_level_scope(self, scope): + """ + Return a definition of the top-level scope + Top-level scopes are defined in ctf.scope. + In order to get a field or a field list, the user needs to pass a + scope as argument, this scope can be a top-level scope or a scope + relative to an arbitrary field. This function provides the mapping + between the scope and the actual definition of top-level scopes. + On error return None. + """ + evDef = ctf.Definition.__new__(ctf.Definition) + evDef._d = _bt_ctf_get_top_level_scope(self._e, scope) + if evDef._d is None: + return None + return evDef + + def get_name(self): + """Return the name of the event or None on error.""" + return _bt_ctf_event_name(self._e) + + def get_cycles(self): + """ + Return the timestamp of the event as written in + the packet (in cycles) or -1ULL on error. + """ + return _bt_ctf_get_cycles(self._e) + + def get_timestamp(self): + """ + Return the timestamp of the event offsetted with the + system clock source or -1ULL on error. + """ + return _bt_ctf_get_timestamp(self._e) + + def get_field(self, scope, field): + """Return the definition of a specific field.""" + evDef = ctf.Definition.__new__(ctf.Definition) + try: + evDef._d = _bt_ctf_get_field(self._e, scope._d, field) + except AttributeError: + raise TypeError("in get_field, argument 2 must be a " + "Definition (scope) instance") + return evDef + + def get_field_list(self, scope): + """ + Return a list of Definitions + Return None on error. + """ + try: + field_lc = _bt_python_field_listcaller(self._e, scope._d) + except AttributeError: + raise TypeError("in get_field_list, argument 2 must be a " + "Definition (scope) instance") + + if field_lc is None: + return None + + def_list = [] + i = 0 + while True: + tmp = ctf.Definition.__new__(ctf.Definition) + tmp._d = _bt_python_field_one_from_list(field_lc, i) + + if tmp._d is None: + #Last item of list is None, assured in + #_bt_python_field_listcaller + break + + def_list.append(tmp) + i += 1 + return def_list + + def get_index(self, field, index): + """ + If the field is an array or a sequence, return the element + at position index, otherwise return None + """ + evDef = ctf.Definition.__new__(ctf.Definition) + try: + evDef._d = _bt_ctf_get_index(self._e, field._d, index) + except AttributeError: + raise TypeError("in get_index, argument 2 must be a " + "Definition (field) instance") + + if evDef._d is None: + return None + return evDef + + def get_handle(self): + """ + Get the TraceHandle associated with an event + Return None on error + """ + ret = _bt_ctf_event_get_handle_id(self._e) + if ret < 0: + return None + + th = TraceHandle.__new__(TraceHandle) + th._id = ret + return th + + def get_context(self): + """ + Get the context associated with an event. + Return None on error. + """ + ctx = Context() + ctx._c = _bt_ctf_event_get_context(self._e); + if ctx._c is None: + return None + else: + return ctx + + + class Definition(object): + """Definition class. Do not instantiate.""" + + def __init__(self): + raise NotImplementedError("ctf.Definition cannot be instantiated") + + def __repr__(self): + return "Babeltrace Definition: name('{}'), type({})".format( + self.field_name(), self.field_type()) + + def field_name(self): + """Return the name of a field or None on error.""" + return _bt_ctf_field_name(self._d) + + def field_type(self): + """Return the type of a field or -1 if unknown.""" + return _bt_ctf_field_type(self._d) + + def get_int_signedness(self): + """ + Return the signedness of an integer: + 0 if unsigned; 1 if signed; -1 on error. + """ + return _bt_ctf_get_int_signedness(self._d) + + def get_int_base(self): + """Return the base of an int or a negative value on error.""" + return _bt_ctf_get_int_base(self._d) + + def get_int_byte_order(self): + """ + Return the byte order of an int or a negative + value on error. + """ + return _bt_ctf_get_int_byte_order(self._d) + + def get_int_len(self): + """ + Return the size, in bits, of an int or a negative + value on error. + """ + return _bt_ctf_get_int_len(self._d) + + def get_encoding(self): + """ + Return the encoding of an int or a string. + Return a negative value on error. + """ + return _bt_ctf_get_encoding(self._d) + + def get_array_len(self): + """ + Return the len of an array or a negative + value on error. + """ + return _bt_ctf_get_array_len(self._d) + + def get_uint64(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_uint64(self._d) + + def get_int64(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_int64(self._d) + + def get_char_array(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_char_array(self._d) + + def get_str(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_string(self._d) + + + class EventDecl(object): + """Event declaration class. Do not instantiate.""" + + def __init__(self): + raise NotImplementedError("ctf.EventDecl cannot be instantiated") + + def __repr__(self): + return "Babeltrace EventDecl: name {}".format(self.get_name()) + + def get_name(self): + """Return the name of the event or None on error""" + return _bt_ctf_get_decl_event_name(self._d) + + def get_decl_fields(self, scope): + """ + Return a list of ctf.FieldDecl + Return None on error. + """ + ptr_list = _by_python_field_decl_listcaller(self._d, scope) + + if ptr_list is None: + return None + + decl_list = [] + i = 0 + while True: + tmp = ctf.FieldDecl.__new__(ctf.FieldDecl) + tmp._d = _bt_python_field_decl_one_from_list( + ptr_list, i) + + if tmp._d is None: + #Last item of list is None + break + + decl_list.append(tmp) + i += 1 + return decl_list + + + class FieldDecl(object): + """Field declaration class. Do not instantiate.""" + + def __init__(self): + raise NotImplementedError("ctf.FieldDecl cannot be instantiated") + + def __repr__(self): + return "Babeltrace FieldDecl: name {}".format(self.get_name()) + + def get_name(self): + """Return the name of a FieldDecl or None on error""" + return _bt_ctf_get_decl_field_name(self._d) + + + @staticmethod + def field_error(): + """ + Return the last error code encountered while + accessing a field and reset the error flag. + Return 0 if no error, a negative value otherwise. + """ + return _bt_ctf_field_get_error() + + @staticmethod + def get_event_decl_list(trace_handle, context): + """ + Return a list of ctf.EventDecl + Return None on error. + """ + try: + handle_id = trace_handle._id + except AttributeError: + raise TypeError("in get_event_decl_list, " + "argument 1 must be a TraceHandle instance") + try: + ptr_list = _bt_python_event_decl_listcaller(handle_id, context._c) + except AttributeError: + raise TypeError("in get_event_decl_list, " + "argument 2 must be a Context instance") + + if ptr_list is None: + return None + + decl_list = [] + i = 0 + while True: + tmp = ctf.EventDecl.__new__(ctf.EventDecl) + tmp._d = _bt_python_decl_one_from_list(ptr_list, i) + + if tmp._d is None: + #Last item of list is None + break + + decl_list.append(tmp) + i += 1 + return decl_list + +%} + + + +// ================================================================= +// NEW FUNCTIONS +// File and list-related +// python-complements.h +// ================================================================= + +%include python-complements.c + +%pythoncode %{ + +class File(object): + """ + Open a file for babeltrace. + + file_path is a string containing the path or None to use the + standard output in writing mode. + + The mode can be 'r', 'w' or 'a' for reading (default), writing or + appending. The file will be created if it doesn't exist when + opened for writing or appending; it will be truncated when opened + for writing. Add a 'b' to the mode for binary files. Add a '+' + to the mode to allow simultaneous reading and writing. + """ + + def __new__(cls, file_path, mode='r'): + # __new__ is used to control the return value + # as the File class should return None + # if _bt_file_open returns NULL + + # Type check + if file_path is not None and type(file_path) is not str: + raise TypeError("in method __init__, argument 2 of type 'str'") + if type(mode) is not str: + raise TypeError("in method __init__, argument 3 of type 'str'") + + # Opening file + file_ptr = _bt_file_open(file_path, mode) + if file_ptr is None: + return None + + # Class instantiation + file_inst = super(File, cls).__new__(cls) + file_inst._file = file_ptr + return file_inst + + def __init__(self, file_path, mode='r'): + self._opened = True + self._use_stdout = False + + if file_path is None: + # use stdout + file_path = "stdout" + mode = 'w' + self._use_stdout = True + + self._file_path = file_path + self._mode = mode + + def __del__(self): + self.close() + + def __repr__(self): + if self._opened: + stat = 'opened' + else: + stat = 'closed' + return "{} babeltrace File; file_path('{}'), mode('{}')".format( + stat, self._file_path, self._mode) + + def close(self): + """Close the file. Is also called using del.""" + if self._opened and not self._use_stdout: + _bt_file_close(self._file) + self._opened = False +%} diff --git a/bindings/python/examples/babeltrace_and_lttng.py b/bindings/python/examples/babeltrace_and_lttng.py new file mode 100644 index 0000000..ef0e35c --- /dev/null +++ b/bindings/python/examples/babeltrace_and_lttng.py @@ -0,0 +1,107 @@ +# This script uses both lttng-tools and babeltrace +# python modules. It creates a session, enables +# events, starts tracing for 2 seconds, stops tracing, +# destroys the session and outputs the trace in the +# specified output file. +# +# WARNING: will destroy any existing trace having +# the same name as ses_name + + +# ------------------------------------------------------ +ses_name = "babeltrace-lttng-test" +trace_path = "/lttng-traces/babeltrace-lttng-trace/" +out_file = "babeltrace-lttng-trace-text-output.txt" +# ------------------------------------------------------ + + +import time +try: + import babeltrace, lttng +except ImportError: + raise ImportError( "both babeltrace and lttng-tools " + "python modules must be installed" ) + + +# Errors to raise if something goes wrong +class LTTngError(Exception): + pass +class BabeltraceError(Exception): + pass + + +# LTTNG-TOOLS + +# Making sure session does not already exist +lttng.destroy(ses_name) + +# Creating a new session and handle +ret = lttng.create(ses_name,trace_path) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + +han = None +han = lttng.Handle(ses_name, lttng.Domain()) +if han is None: + raise LTTngError("Handle not created") + + +# Enabling all events +ret = lttng.enable_event(han, lttng.Event(), None) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + + +# Start, wait, stop +ret = lttng.start(ses_name) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) +print("Tracing...") +time.sleep(2) +print("Stopped.") +ret = lttng.stop(ses_name) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + + +# Destroying tracing session +ret = lttng.destroy(ses_name) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + + +# BABELTRACE + +# Create context and add trace: +ctx = babeltrace.Context() +ret = ctx.add_trace(trace_path + "/kernel", "ctf") +if ret is None: + raise BabeltraceError("Error adding trace") + +# Iterator setup +bp = babeltrace.IterPos(babeltrace.SEEK_BEGIN) +ctf_it = babeltrace.ctf.Iterator(ctx,bp) + +# Reading events from trace +# and outputting timestamps and event names +# in out_file +print("Writing trace file...") +output = open(out_file, "wt") + +event = ctf_it.read_event() +while(event is not None): + output.write("TS: {}, {} : {}\n".format(event.get_timestamp(), + event.get_cycles(), event.get_name())) + + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +# Closing file +output.close() + +# Destroying dynamic elements +del ctf_it, han +print("Done.") diff --git a/bindings/python/examples/eventcount.py b/bindings/python/examples/eventcount.py new file mode 100644 index 0000000..2a63b74 --- /dev/null +++ b/bindings/python/examples/eventcount.py @@ -0,0 +1,66 @@ +# The script prints a count of specified events and +# their related tid's in a given trace. +# The trace needs TID context (lttng add-context -k -t tid) + +import sys +from babeltrace import * +from output_format_modules.pprint_table import pprint_table as pprint + +if len(sys.argv) < 3: + raise TypeError("Usage: python eventcount.py event1 [event2 ...] path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +counts = {} + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +while(event is not None): + for event_type in sys.argv[1:len(sys.argv)-1]: + if event_type == event.get_name(): + + # Getting scope definition + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) + if sco is None: + print("ERROR: Cannot get definition scope for {}".format( + event.get_name())) + continue + + # Getting TID + tid_field = event.get_field(sco, "_tid") + tid = tid_field.get_int64() + + if ctf.field_error(): + print("ERROR: Missing TID info for {}".format( + event.get_name())) + continue + + tmp = (tid, event.get_name()) + + if tmp in counts: + counts[tmp] += 1 + else: + counts[tmp] = 1 + + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it + +# Appending data to table for output +table = [] +for item in counts: + table.append([item[0], item[1], counts[item]]) +table = sorted(table) +table.insert(0,["TID", "EVENT", "COUNT"]) +pprint(table, 2) diff --git a/bindings/python/examples/eventcountlist.py b/bindings/python/examples/eventcountlist.py new file mode 100644 index 0000000..800996c --- /dev/null +++ b/bindings/python/examples/eventcountlist.py @@ -0,0 +1,65 @@ +# The script prints a count and rate of events. +# It also outputs a bar graph of count per event, using the cairoplot module. + +import sys +from babeltrace import * +from output_format_modules import cairoplot +from output_format_modules.pprint_table import pprint_table as pprint + +# Check for path arg: +if len(sys.argv) < 2: + raise TypeError("Usage: python eventcountlist.py path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +# Events and their assossiated count +# will be stored as a dict: +events_count = {} + +# Setting iterator: +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx,bp) + +prev_event = None +event = ctf_it.read_event() + +start_time = event.get_timestamp() + +# Reading events: +while(event is not None): + if event.get_name() in events_count: + events_count[event.get_name()] += 1 + else: + events_count[event.get_name()] = 1 + + ret = ctf_it.next() + if ret < 0: + break + else: + prev_event = event + event = ctf_it.read_event() + +if event: + total_time = event.get_timestamp() - start_time +else: + total_time = prev_event.get_timestamp() - start_time + +del ctf_it + +# Printing encountered events with respective count and rate: +print("Total time: {} ns".format(total_time)) +table = [["EVENT", "COUNT", "RATE (Hz)"]] +for item in sorted(events_count.iterkeys()): + tmp = [item, events_count[item], + events_count[item]/(total_time/1000000000.0)] + table.append(tmp) +pprint(table) + +# Exporting data as bar graph +cairoplot.vertical_bar_plot ( 'eventcountlist.svg', events_count, 50+85*len(events_count), + 800, border = 20, display_values = True, grid = True, + rounded_corners = True, + x_labels = sorted(events_count.keys()) ) diff --git a/bindings/python/examples/events_per_cpu.py b/bindings/python/examples/events_per_cpu.py new file mode 100644 index 0000000..09cf0e3 --- /dev/null +++ b/bindings/python/examples/events_per_cpu.py @@ -0,0 +1,81 @@ +# The script opens a trace and prints out CPU statistics +# for the given trace (event count per CPU, total active +# time and % of time processing events). +# It also outputs a .txt file showing each time interval +# (since the beginning of the trace) in which each CPU +# was active and the corresponding event. + +import sys, multiprocessing +from output_format_modules.pprint_table import pprint_table as pprint +from babeltrace import * + +if len(sys.argv) < 2: + raise TypeError("Usage: python events_per_cpu.py path/to/trace") + +# Adding trace +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +cpu_usage = [] +nbEvents = 0 +i = 0 +while i < multiprocessing.cpu_count(): + cpu_usage.append([]) + i += 1 + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +start_time = event.get_timestamp() + +while(event is not None): + + event_name = event.get_name() + ts = event.get_timestamp() + + # Getting cpu_id + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) + field = event.get_field(scope, "cpu_id") + cpu_id = field.get_uint64() + if ctf.field_error(): + print("ERROR: Missing cpu_id info for {}".format(event.get_name())) + else: + cpu_usage[cpu_id].append( (int(ts), event_name) ) + nbEvents += 1 + + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + + +# Outputting +table = [] +output = open("events_per_cpu.txt", "wt") +output.write("(timestamp, event)\n") + +for cpu in range(len(cpu_usage)): + # Setting table + event_str = str(100.0 * len(cpu_usage[cpu]) / nbEvents) + '000' + # % is printed with 2 decimals + table.append([cpu, len(cpu_usage[cpu]), event_str[0:event_str.find('.') + 3] + ' %']) + + # Writing to file + output.write("\n\n\n----------------------\n") + output.write("CPU {}\n\n".format(cpu)) + for event in cpu_usage[cpu]: + output.write(str(event) + '\n') + +# Printing table +table.insert(0, ["CPU ID", "EVENT COUNT", "TRACE EVENT %"]) +pprint(table) +print("Total event count: {}".format(nbEvents)) +print("Total trace time: {} ns".format(ts - start_time)) + +output.close() diff --git a/bindings/python/examples/example-api-test.py b/bindings/python/examples/example-api-test.py new file mode 100644 index 0000000..0cfc3ed --- /dev/null +++ b/bindings/python/examples/example-api-test.py @@ -0,0 +1,59 @@ +# This example uses the babeltrace python module +# to partially test the api. + +import sys +from babeltrace import * + +# Check for path arg: +if len(sys.argv) < 2: + raise TypeError("Usage: python example-api-test.py path/to/file") + +# Create context and add trace: +ctx = Context() +trace_handle = ctx.add_trace(sys.argv[1], "ctf") +if trace_handle is None: + raise IOError("Error adding trace") + +# Listing events +lst = ctf.get_event_decl_list(trace_handle, ctx) +print("--- Event list ---") +for item in lst: + print("event : {}".format(item.get_name())) +print("--- Done ---") + +# Iter trace +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx,bp) +event = ctf_it.read_event() + +while(event is not None): + print("TS: {}, {} : {}".format(event.get_timestamp(), + event.get_cycles(), event.get_name())) + + if event.get_name() == "sched_switch": + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + prev_field = event.get_field(sco, "_prev_comm") + prev_comm = prev_field.get_char_array() + + if ctf.field_error(): + print("ERROR: Missing prev_comm context info") + else: + print("sched_switch prev_comm: {}".format(prev_comm)) + + if event.get_name() == "exit_syscall": + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + ret_field = event.get_field(sco, "_ret") + ret_code = ret_field.get_int64() + + if ctf.field_error(): + print("ERROR: Unable to extract ret") + else: + print("exit_syscall ret: {}".format(ret_code)) + + ret = ctf_it.next() + if ret < 0: + break + else: + event = ctf_it.read_event() + +del ctf_it diff --git a/bindings/python/examples/histogram.py b/bindings/python/examples/histogram.py new file mode 100644 index 0000000..a24c00d --- /dev/null +++ b/bindings/python/examples/histogram.py @@ -0,0 +1,121 @@ +# The script checks the number of events in the trace +# and outputs a table and a .svg histogram for the specified +# range (microseconds) or the total trace if no range specified. +# The graph is generated using the cairoplot module. + +import sys +from babeltrace import * +from output_format_modules import cairoplot +from output_format_modules.pprint_table import pprint_table as pprint + +# ------------------------------------------------ +# Output settings + +# number of intervals: +nbDiv = 25 # Should not be over 150 + # for usable graph output + +# table output stream (file-like object): +out = sys.stdout +# ------------------------------------------------- + +if len(sys.argv) < 2 or len(sys.argv) > 4: + raise TypeError("Usage: python histogram.py [ start_time [end_time] ] path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +# Check when to start/stop graphing +sinceBegin = True +beginTime = 0.0 +if len(sys.argv) > 2: + sinceBegin = False + beginTime = float(sys.argv[1]) +untilEnd = True +if len(sys.argv) == 4: + untilEnd = False + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +start_time = event.get_timestamp() +time = 0 +count = {} + +while(event is not None): + # Microsec. + time = (event.get_timestamp() - start_time)/1000.0 + + # Check if in range + if not sinceBegin: + if time < beginTime: + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + continue + if not untilEnd: + if time > float(sys.argv[2]): + break + + # Counting events per timestamp: + if time in count: + count[time] += 1 + else: + count[time] = 1 + + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it + +# Setting data for output +interval = (time - beginTime)/nbDiv +div_begin_time = beginTime +div_end_time = beginTime + interval +data = {} + +# Prefix for string sorting, considering +# there should not be over 150 intervals. +# This would work up to 9999 intervals. +# If needed, add zeros. +prefix = 0.0001 + +while div_end_time <= time: + key = str(prefix) + '[' + str(div_begin_time) + ';' + str(div_end_time) + '[' + for tmp in count: + if tmp >= div_begin_time and tmp < div_end_time: + if key in data: + data[key] += count[tmp] + else: + data[key] = count[tmp] + if not key in data: + data[key] = 0 + div_begin_time = div_end_time + div_end_time += interval + # Prefix increment + prefix += 0.001 + +table = [] +x_labels = [] +for key in sorted(data): + table.append([key[key.find('['):], data[key]]) + x_labels.append(key[key.find('['):]) + +# Table output +table.insert(0, ["INTERVAL (us)", "COUNT"]) +pprint(table, 1, out) + +# Graph output +cairoplot.vertical_bar_plot ( 'histogram.svg', data, 50 + 150*nbDiv, 50*nbDiv, + border = 20, display_values = True, grid = True, + x_labels = x_labels, rounded_corners = True ) diff --git a/bindings/python/examples/output_format_modules/__init__.py b/bindings/python/examples/output_format_modules/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/bindings/python/examples/output_format_modules/cairoplot.py b/bindings/python/examples/output_format_modules/cairoplot.py new file mode 100755 index 0000000..a27113f --- /dev/null +++ b/bindings/python/examples/output_format_modules/cairoplot.py @@ -0,0 +1,2336 @@ +?#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# CairoPlot.py +# +# Copyright (c) 2008 Rodrigo Moreira Ara?jo +# +# Author: Rodrigo Moreiro Araujo +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public License +# as published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 +# USA + +#Contributor: Jo?o S. O. Bueno + +#TODO: review BarPlot Code +#TODO: x_label colision problem on Horizontal Bar Plot +#TODO: y_label's eat too much space on HBP + + +__version__ = 1.2 + +import cairo +import math +import random +from series import Series, Group, Data + +HORZ = 0 +VERT = 1 +NORM = 2 + +COLORS = {"red" : (1.0,0.0,0.0,1.0), "lime" : (0.0,1.0,0.0,1.0), "blue" : (0.0,0.0,1.0,1.0), + "maroon" : (0.5,0.0,0.0,1.0), "green" : (0.0,0.5,0.0,1.0), "navy" : (0.0,0.0,0.5,1.0), + "yellow" : (1.0,1.0,0.0,1.0), "magenta" : (1.0,0.0,1.0,1.0), "cyan" : (0.0,1.0,1.0,1.0), + "orange" : (1.0,0.5,0.0,1.0), "white" : (1.0,1.0,1.0,1.0), "black" : (0.0,0.0,0.0,1.0), + "gray" : (0.5,0.5,0.5,1.0), "light_gray" : (0.9,0.9,0.9,1.0), + "transparent" : (0.0,0.0,0.0,0.0)} + +THEMES = {"black_red" : [(0.0,0.0,0.0,1.0), (1.0,0.0,0.0,1.0)], + "red_green_blue" : [(1.0,0.0,0.0,1.0), (0.0,1.0,0.0,1.0), (0.0,0.0,1.0,1.0)], + "red_orange_yellow" : [(1.0,0.2,0.0,1.0), (1.0,0.7,0.0,1.0), (1.0,1.0,0.0,1.0)], + "yellow_orange_red" : [(1.0,1.0,0.0,1.0), (1.0,0.7,0.0,1.0), (1.0,0.2,0.0,1.0)], + "rainbow" : [(1.0,0.0,0.0,1.0), (1.0,0.5,0.0,1.0), (1.0,1.0,0.0,1.0), (0.0,1.0,0.0,1.0), (0.0,0.0,1.0,1.0), (0.3, 0.0, 0.5,1.0), (0.5, 0.0, 1.0, 1.0)]} + +def colors_from_theme( theme, series_length, mode = 'solid' ): + colors = [] + if theme not in THEMES.keys() : + raise Exception, "Theme not defined" + color_steps = THEMES[theme] + n_colors = len(color_steps) + if series_length <= n_colors: + colors = [color + tuple([mode]) for color in color_steps[0:n_colors]] + else: + iterations = [(series_length - n_colors)/(n_colors - 1) for i in color_steps[:-1]] + over_iterations = (series_length - n_colors) % (n_colors - 1) + for i in range(n_colors - 1): + if over_iterations <= 0: + break + iterations[i] += 1 + over_iterations -= 1 + for index,color in enumerate(color_steps[:-1]): + colors.append(color + tuple([mode])) + if iterations[index] == 0: + continue + next_color = color_steps[index+1] + color_step = ((next_color[0] - color[0])/(iterations[index] + 1), + (next_color[1] - color[1])/(iterations[index] + 1), + (next_color[2] - color[2])/(iterations[index] + 1), + (next_color[3] - color[3])/(iterations[index] + 1)) + for i in range( iterations[index] ): + colors.append((color[0] + color_step[0]*(i+1), + color[1] + color_step[1]*(i+1), + color[2] + color_step[2]*(i+1), + color[3] + color_step[3]*(i+1), + mode)) + colors.append(color_steps[-1] + tuple([mode])) + return colors + + +def other_direction(direction): + "explicit is better than implicit" + if direction == HORZ: + return VERT + else: + return HORZ + +#Class definition + +class Plot(object): + def __init__(self, + surface=None, + data=None, + width=640, + height=480, + background=None, + border = 0, + x_labels = None, + y_labels = None, + series_colors = None): + random.seed(2) + self.create_surface(surface, width, height) + self.dimensions = {} + self.dimensions[HORZ] = width + self.dimensions[VERT] = height + self.context = cairo.Context(self.surface) + self.labels={} + self.labels[HORZ] = x_labels + self.labels[VERT] = y_labels + self.load_series(data, x_labels, y_labels, series_colors) + self.font_size = 10 + self.set_background (background) + self.border = border + self.borders = {} + self.line_color = (0.5, 0.5, 0.5) + self.line_width = 0.5 + self.label_color = (0.0, 0.0, 0.0) + self.grid_color = (0.8, 0.8, 0.8) + + def create_surface(self, surface, width=None, height=None): + self.filename = None + if isinstance(surface, cairo.Surface): + self.surface = surface + return + if not type(surface) in (str, unicode): + raise TypeError("Surface should be either a Cairo surface or a filename, not %s" % surface) + sufix = surface.rsplit(".")[-1].lower() + self.filename = surface + if sufix == "png": + self.surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, width, height) + elif sufix == "ps": + self.surface = cairo.PSSurface(surface, width, height) + elif sufix == "pdf": + self.surface = cairo.PSSurface(surface, width, height) + else: + if sufix != "svg": + self.filename += ".svg" + self.surface = cairo.SVGSurface(self.filename, width, height) + + def commit(self): + try: + self.context.show_page() + if self.filename and self.filename.endswith(".png"): + self.surface.write_to_png(self.filename) + else: + self.surface.finish() + except cairo.Error: + pass + + def load_series (self, data, x_labels=None, y_labels=None, series_colors=None): + self.series_labels = [] + self.series = None + + #The pretty way + #if not isinstance(data, Series): + # # Not an instance of Series + # self.series = Series(data) + #else: + # self.series = data + # + #self.series_labels = self.series.get_names() + + #TODO: Remove on next version + # The ugly way, keeping retrocompatibility... + if callable(data) or type(data) is list and callable(data[0]): # Lambda or List of lambdas + self.series = data + self.series_labels = None + elif isinstance(data, Series): # Instance of Series + self.series = data + self.series_labels = data.get_names() + else: # Anything else + self.series = Series(data) + self.series_labels = self.series.get_names() + + #TODO: allow user passed series_widths + self.series_widths = [1.0 for group in self.series] + + #TODO: Remove on next version + self.process_colors( series_colors ) + + def process_colors( self, series_colors, length = None, mode = 'solid' ): + #series_colors might be None, a theme, a string of colors names or a list of color tuples + if length is None : + length = len( self.series.to_list() ) + + #no colors passed + if not series_colors: + #Randomize colors + self.series_colors = [ [random.random() for i in range(3)] + [1.0, mode] for series in range( length ) ] + else: + #Just theme pattern + if not hasattr( series_colors, "__iter__" ): + theme = series_colors + self.series_colors = colors_from_theme( theme.lower(), length ) + + #Theme pattern and mode + elif not hasattr(series_colors, '__delitem__') and not hasattr( series_colors[0], "__iter__" ): + theme = series_colors[0] + mode = series_colors[1] + self.series_colors = colors_from_theme( theme.lower(), length, mode ) + + #List + else: + self.series_colors = series_colors + for index, color in enumerate( self.series_colors ): + #element is a color name + if not hasattr(color, "__iter__"): + self.series_colors[index] = COLORS[color.lower()] + tuple([mode]) + #element is rgb tuple instead of rgba + elif len( color ) == 3 : + self.series_colors[index] += (1.0,mode) + #element has 4 elements, might be rgba tuple or rgb tuple with mode + elif len( color ) == 4 : + #last element is mode + if not hasattr(color[3], "__iter__"): + self.series_colors[index] += tuple([color[3]]) + self.series_colors[index][3] = 1.0 + #last element is alpha + else: + self.series_colors[index] += tuple([mode]) + + def get_width(self): + return self.surface.get_width() + + def get_height(self): + return self.surface.get_height() + + def set_background(self, background): + if background is None: + self.background = (0.0,0.0,0.0,0.0) + elif type(background) in (cairo.LinearGradient, tuple): + self.background = background + elif not hasattr(background,"__iter__"): + colors = background.split(" ") + if len(colors) == 1 and colors[0] in COLORS: + self.background = COLORS[background] + elif len(colors) > 1: + self.background = cairo.LinearGradient(self.dimensions[HORZ] / 2, 0, self.dimensions[HORZ] / 2, self.dimensions[VERT]) + for index,color in enumerate(colors): + self.background.add_color_stop_rgba(float(index)/(len(colors)-1),*COLORS[color]) + else: + raise TypeError ("Background should be either cairo.LinearGradient or a 3/4-tuple, not %s" % type(background)) + + def render_background(self): + if isinstance(self.background, cairo.LinearGradient): + self.context.set_source(self.background) + else: + self.context.set_source_rgba(*self.background) + self.context.rectangle(0,0, self.dimensions[HORZ], self.dimensions[VERT]) + self.context.fill() + + def render_bounding_box(self): + self.context.set_source_rgba(*self.line_color) + self.context.set_line_width(self.line_width) + self.context.rectangle(self.border, self.border, + self.dimensions[HORZ] - 2 * self.border, + self.dimensions[VERT] - 2 * self.border) + self.context.stroke() + + def render(self): + pass + +class ScatterPlot( Plot ): + def __init__(self, + surface=None, + data=None, + errorx=None, + errory=None, + width=640, + height=480, + background=None, + border=0, + axis = False, + dash = False, + discrete = False, + dots = 0, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + z_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + circle_colors = None ): + + self.bounds = {} + self.bounds[HORZ] = x_bounds + self.bounds[VERT] = y_bounds + self.bounds[NORM] = z_bounds + self.titles = {} + self.titles[HORZ] = x_title + self.titles[VERT] = y_title + self.max_value = {} + self.axis = axis + self.discrete = discrete + self.dots = dots + self.grid = grid + self.series_legend = series_legend + self.variable_radius = False + self.x_label_angle = math.pi / 2.5 + self.circle_colors = circle_colors + + Plot.__init__(self, surface, data, width, height, background, border, x_labels, y_labels, series_colors) + + self.dash = None + if dash: + if hasattr(dash, "keys"): + self.dash = [dash[key] for key in self.series_labels] + elif max([hasattr(item,'__delitem__') for item in data]) : + self.dash = dash + else: + self.dash = [dash] + + self.load_errors(errorx, errory) + + def convert_list_to_tuple(self, data): + #Data must be converted from lists of coordinates to a single + # list of tuples + out_data = zip(*data) + if len(data) == 3: + self.variable_radius = True + return out_data + + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): + #TODO: In cairoplot 2.0 keep only the Series instances + + # Convert Data and Group to Series + if isinstance(data, Data) or isinstance(data, Group): + data = Series(data) + + # Series + if isinstance(data, Series): + for group in data: + for item in group: + if len(item) is 3: + self.variable_radius = True + + #Dictionary with lists + if hasattr(data, "keys") : + if hasattr( data.values()[0][0], "__delitem__" ) : + for key in data.keys() : + data[key] = self.convert_list_to_tuple(data[key]) + elif len(data.values()[0][0]) == 3: + self.variable_radius = True + #List + elif hasattr(data[0], "__delitem__") : + #List of lists + if hasattr(data[0][0], "__delitem__") : + for index,value in enumerate(data) : + data[index] = self.convert_list_to_tuple(value) + #List + elif type(data[0][0]) != type((0,0)): + data = self.convert_list_to_tuple(data) + #Three dimensional data + elif len(data[0][0]) == 3: + self.variable_radius = True + + #List with three dimensional tuples + elif len(data[0]) == 3: + self.variable_radius = True + Plot.load_series(self, data, x_labels, y_labels, series_colors) + self.calc_boundaries() + self.calc_labels() + + def load_errors(self, errorx, errory): + self.errors = None + if errorx == None and errory == None: + return + self.errors = {} + self.errors[HORZ] = None + self.errors[VERT] = None + #asimetric errors + if errorx and hasattr(errorx[0], "__delitem__"): + self.errors[HORZ] = errorx + #simetric errors + elif errorx: + self.errors[HORZ] = [errorx] + #asimetric errors + if errory and hasattr(errory[0], "__delitem__"): + self.errors[VERT] = errory + #simetric errors + elif errory: + self.errors[VERT] = [errory] + + def calc_labels(self): + if not self.labels[HORZ]: + amplitude = self.bounds[HORZ][1] - self.bounds[HORZ][0] + if amplitude % 10: #if horizontal labels need floating points + self.labels[HORZ] = ["%.2lf" % (float(self.bounds[HORZ][0] + (amplitude * i / 10.0))) for i in range(11) ] + else: + self.labels[HORZ] = ["%d" % (int(self.bounds[HORZ][0] + (amplitude * i / 10.0))) for i in range(11) ] + if not self.labels[VERT]: + amplitude = self.bounds[VERT][1] - self.bounds[VERT][0] + if amplitude % 10: #if vertical labels need floating points + self.labels[VERT] = ["%.2lf" % (float(self.bounds[VERT][0] + (amplitude * i / 10.0))) for i in range(11) ] + else: + self.labels[VERT] = ["%d" % (int(self.bounds[VERT][0] + (amplitude * i / 10.0))) for i in range(11) ] + + def calc_extents(self, direction): + self.context.set_font_size(self.font_size * 0.8) + self.max_value[direction] = max(self.context.text_extents(item)[2] for item in self.labels[direction]) + self.borders[other_direction(direction)] = self.max_value[direction] + self.border + 20 + + def calc_boundaries(self): + #HORZ = 0, VERT = 1, NORM = 2 + min_data_value = [0,0,0] + max_data_value = [0,0,0] + + for group in self.series: + if type(group[0].content) in (int, float, long): + group = [Data((index, item.content)) for index,item in enumerate(group)] + + for point in group: + for index, item in enumerate(point.content): + if item > max_data_value[index]: + max_data_value[index] = item + elif item < min_data_value[index]: + min_data_value[index] = item + + if not self.bounds[HORZ]: + self.bounds[HORZ] = (min_data_value[HORZ], max_data_value[HORZ]) + if not self.bounds[VERT]: + self.bounds[VERT] = (min_data_value[VERT], max_data_value[VERT]) + if not self.bounds[NORM]: + self.bounds[NORM] = (min_data_value[NORM], max_data_value[NORM]) + + def calc_all_extents(self): + self.calc_extents(HORZ) + self.calc_extents(VERT) + + self.plot_height = self.dimensions[VERT] - 2 * self.borders[VERT] + self.plot_width = self.dimensions[HORZ] - 2* self.borders[HORZ] + + self.plot_top = self.dimensions[VERT] - self.borders[VERT] + + def calc_steps(self): + #Calculates all the x, y, z and color steps + series_amplitude = [self.bounds[index][1] - self.bounds[index][0] for index in range(3)] + + if series_amplitude[HORZ]: + self.horizontal_step = float (self.plot_width) / series_amplitude[HORZ] + else: + self.horizontal_step = 0.00 + + if series_amplitude[VERT]: + self.vertical_step = float (self.plot_height) / series_amplitude[VERT] + else: + self.vertical_step = 0.00 + + if series_amplitude[NORM]: + if self.variable_radius: + self.z_step = float (self.bounds[NORM][1]) / series_amplitude[NORM] + if self.circle_colors: + self.circle_color_step = tuple([float(self.circle_colors[1][i]-self.circle_colors[0][i])/series_amplitude[NORM] for i in range(4)]) + else: + self.z_step = 0.00 + self.circle_color_step = ( 0.0, 0.0, 0.0, 0.0 ) + + def get_circle_color(self, value): + return tuple( [self.circle_colors[0][i] + value*self.circle_color_step[i] for i in range(4)] ) + + def render(self): + self.calc_all_extents() + self.calc_steps() + self.render_background() + self.render_bounding_box() + if self.axis: + self.render_axis() + if self.grid: + self.render_grid() + self.render_labels() + self.render_plot() + if self.errors: + self.render_errors() + if self.series_legend and self.series_labels: + self.render_legend() + + def render_axis(self): + #Draws both the axis lines and their titles + cr = self.context + cr.set_source_rgba(*self.line_color) + cr.move_to(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) + cr.line_to(self.borders[HORZ], self.borders[VERT]) + cr.stroke() + + cr.move_to(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) + cr.line_to(self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) + cr.stroke() + + cr.set_source_rgba(*self.label_color) + self.context.set_font_size( 1.2 * self.font_size ) + if self.titles[HORZ]: + title_width,title_height = cr.text_extents(self.titles[HORZ])[2:4] + cr.move_to( self.dimensions[HORZ]/2 - title_width/2, self.borders[VERT] - title_height/2 ) + cr.show_text( self.titles[HORZ] ) + + if self.titles[VERT]: + title_width,title_height = cr.text_extents(self.titles[VERT])[2:4] + cr.move_to( self.dimensions[HORZ] - self.borders[HORZ] + title_height/2, self.dimensions[VERT]/2 - title_width/2) + cr.save() + cr.rotate( math.pi/2 ) + cr.show_text( self.titles[VERT] ) + cr.restore() + + def render_grid(self): + cr = self.context + horizontal_step = float( self.plot_height ) / ( len( self.labels[VERT] ) - 1 ) + vertical_step = float( self.plot_width ) / ( len( self.labels[HORZ] ) - 1 ) + + x = self.borders[HORZ] + vertical_step + y = self.plot_top - horizontal_step + + for label in self.labels[HORZ][:-1]: + cr.set_source_rgba(*self.grid_color) + cr.move_to(x, self.dimensions[VERT] - self.borders[VERT]) + cr.line_to(x, self.borders[VERT]) + cr.stroke() + x += vertical_step + for label in self.labels[VERT][:-1]: + cr.set_source_rgba(*self.grid_color) + cr.move_to(self.borders[HORZ], y) + cr.line_to(self.dimensions[HORZ] - self.borders[HORZ], y) + cr.stroke() + y -= horizontal_step + + def render_labels(self): + self.context.set_font_size(self.font_size * 0.8) + self.render_horz_labels() + self.render_vert_labels() + + def render_horz_labels(self): + cr = self.context + step = float( self.plot_width ) / ( len( self.labels[HORZ] ) - 1 ) + x = self.borders[HORZ] + y = self.dimensions[VERT] - self.borders[VERT] + 5 + + # store rotation matrix from the initial state + rotation_matrix = cr.get_matrix() + rotation_matrix.rotate(self.x_label_angle) + + cr.set_source_rgba(*self.label_color) + + for item in self.labels[HORZ]: + width = cr.text_extents(item)[2] + cr.move_to(x, y) + cr.save() + cr.set_matrix(rotation_matrix) + cr.show_text(item) + cr.restore() + x += step + + def render_vert_labels(self): + cr = self.context + step = ( self.plot_height ) / ( len( self.labels[VERT] ) - 1 ) + y = self.plot_top + cr.set_source_rgba(*self.label_color) + for item in self.labels[VERT]: + width = cr.text_extents(item)[2] + cr.move_to(self.borders[HORZ] - width - 5,y) + cr.show_text(item) + y -= step + + def render_legend(self): + cr = self.context + cr.set_font_size(self.font_size) + cr.set_line_width(self.line_width) + + widest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[2]) + tallest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[3]) + max_width = self.context.text_extents(widest_word)[2] + max_height = self.context.text_extents(tallest_word)[3] * 1.1 + + color_box_height = max_height / 2 + color_box_width = color_box_height * 2 + + #Draw a bounding box + bounding_box_width = max_width + color_box_width + 15 + bounding_box_height = (len(self.series_labels)+0.5) * max_height + cr.set_source_rgba(1,1,1) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - bounding_box_width, self.borders[VERT], + bounding_box_width, bounding_box_height) + cr.fill() + + cr.set_source_rgba(*self.line_color) + cr.set_line_width(self.line_width) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - bounding_box_width, self.borders[VERT], + bounding_box_width, bounding_box_height) + cr.stroke() + + for idx,key in enumerate(self.series_labels): + #Draw color box + cr.set_source_rgba(*self.series_colors[idx][:4]) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - max_width - color_box_width - 10, + self.borders[VERT] + color_box_height + (idx*max_height) , + color_box_width, color_box_height) + cr.fill() + + cr.set_source_rgba(0, 0, 0) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - max_width - color_box_width - 10, + self.borders[VERT] + color_box_height + (idx*max_height), + color_box_width, color_box_height) + cr.stroke() + + #Draw series labels + cr.set_source_rgba(0, 0, 0) + cr.move_to(self.dimensions[HORZ] - self.borders[HORZ] - max_width - 5, self.borders[VERT] + ((idx+1)*max_height)) + cr.show_text(key) + + def render_errors(self): + cr = self.context + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) + cr.clip() + radius = self.dots + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + for index, group in enumerate(self.series): + cr.set_source_rgba(*self.series_colors[index][:4]) + for number, data in enumerate(group): + x = x0 + self.horizontal_step * data.content[0] + y = self.dimensions[VERT] - y0 - self.vertical_step * data.content[1] + if self.errors[HORZ]: + cr.move_to(x, y) + x1 = x - self.horizontal_step * self.errors[HORZ][0][number] + cr.line_to(x1, y) + cr.line_to(x1, y - radius) + cr.line_to(x1, y + radius) + cr.stroke() + if self.errors[HORZ] and len(self.errors[HORZ]) == 2: + cr.move_to(x, y) + x1 = x + self.horizontal_step * self.errors[HORZ][1][number] + cr.line_to(x1, y) + cr.line_to(x1, y - radius) + cr.line_to(x1, y + radius) + cr.stroke() + if self.errors[VERT]: + cr.move_to(x, y) + y1 = y + self.vertical_step * self.errors[VERT][0][number] + cr.line_to(x, y1) + cr.line_to(x - radius, y1) + cr.line_to(x + radius, y1) + cr.stroke() + if self.errors[VERT] and len(self.errors[VERT]) == 2: + cr.move_to(x, y) + y1 = y - self.vertical_step * self.errors[VERT][1][number] + cr.line_to(x, y1) + cr.line_to(x - radius, y1) + cr.line_to(x + radius, y1) + cr.stroke() + + + def render_plot(self): + cr = self.context + if self.discrete: + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) + cr.clip() + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + radius = self.dots + for number, group in enumerate (self.series): + cr.set_source_rgba(*self.series_colors[number][:4]) + for data in group : + if self.variable_radius: + radius = data.content[2]*self.z_step + if self.circle_colors: + cr.set_source_rgba( *self.get_circle_color( data.content[2]) ) + x = x0 + self.horizontal_step*data.content[0] + y = y0 + self.vertical_step*data.content[1] + cr.arc(x, self.dimensions[VERT] - y, radius, 0, 2*math.pi) + cr.fill() + else: + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) + cr.clip() + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + radius = self.dots + for number, group in enumerate (self.series): + last_data = None + cr.set_source_rgba(*self.series_colors[number][:4]) + for data in group : + x = x0 + self.horizontal_step*data.content[0] + y = y0 + self.vertical_step*data.content[1] + if self.dots: + if self.variable_radius: + radius = data.content[2]*self.z_step + cr.arc(x, self.dimensions[VERT] - y, radius, 0, 2*math.pi) + cr.fill() + if last_data : + old_x = x0 + self.horizontal_step*last_data.content[0] + old_y = y0 + self.vertical_step*last_data.content[1] + cr.move_to( old_x, self.dimensions[VERT] - old_y ) + cr.line_to( x, self.dimensions[VERT] - y) + cr.set_line_width(self.series_widths[number]) + + #?Display line as dash line + if self.dash and self.dash[number]: + s = self.series_widths[number] + cr.set_dash([s*3, s*3], 0) + + cr.stroke() + cr.set_dash([]) + last_data = data + +class DotLinePlot(ScatterPlot): + def __init__(self, + surface=None, + data=None, + width=640, + height=480, + background=None, + border=0, + axis = False, + dash = False, + dots = 0, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None): + + ScatterPlot.__init__(self, surface, data, None, None, width, height, background, border, + axis, dash, False, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, None, x_title, y_title, series_colors, None ) + + + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + for group in self.series : + for index,data in enumerate(group): + group[index].content = (index, data.content) + + self.calc_boundaries() + self.calc_labels() + +class FunctionPlot(ScatterPlot): + def __init__(self, + surface=None, + data=None, + width=640, + height=480, + background=None, + border=0, + axis = False, + discrete = False, + dots = 0, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + step = 1): + + self.function = data + self.step = step + self.discrete = discrete + + data, x_bounds = self.load_series_from_function( self.function, x_bounds ) + + ScatterPlot.__init__(self, surface, data, None, None, width, height, background, border, + axis, False, discrete, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, None, x_title, y_title, series_colors, None ) + + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + + if len(self.series[0][0]) is 1: + for group_id, group in enumerate(self.series) : + for index,data in enumerate(group): + group[index].content = (self.bounds[HORZ][0] + self.step*index, data.content) + + self.calc_boundaries() + self.calc_labels() + + def load_series_from_function( self, function, x_bounds ): + #TODO: Add the possibility for the user to define multiple functions with different discretization parameters + + #This function converts a function, a list of functions or a dictionary + #of functions into its corresponding array of data + series = Series() + + if isinstance(function, Group) or isinstance(function, Data): + function = Series(function) + + # If is instance of Series + if isinstance(function, Series): + # Overwrite any bounds passed by the function + x_bounds = (function.range[0],function.range[-1]) + + #if no bounds are provided + if x_bounds == None: + x_bounds = (0,10) + + + #TODO: Finish the dict translation + if hasattr(function, "keys"): #dictionary: + for key in function.keys(): + group = Group(name=key) + #data[ key ] = [] + i = x_bounds[0] + while i <= x_bounds[1] : + group.add_data(function[ key ](i)) + #data[ key ].append( function[ key ](i) ) + i += self.step + series.add_group(group) + + elif hasattr(function, "__delitem__"): #list of functions + for index,f in enumerate( function ) : + group = Group() + #data.append( [] ) + i = x_bounds[0] + while i <= x_bounds[1] : + group.add_data(f(i)) + #data[ index ].append( f(i) ) + i += self.step + series.add_group(group) + + elif isinstance(function, Series): # instance of Series + series = function + + else: #function + group = Group() + i = x_bounds[0] + while i <= x_bounds[1] : + group.add_data(function(i)) + i += self.step + series.add_group(group) + + + return series, x_bounds + + def calc_labels(self): + if not self.labels[HORZ]: + self.labels[HORZ] = [] + i = self.bounds[HORZ][0] + while i<=self.bounds[HORZ][1]: + self.labels[HORZ].append(str(i)) + i += float(self.bounds[HORZ][1] - self.bounds[HORZ][0])/10 + ScatterPlot.calc_labels(self) + + def render_plot(self): + if not self.discrete: + ScatterPlot.render_plot(self) + else: + last = None + cr = self.context + for number, group in enumerate (self.series): + cr.set_source_rgba(*self.series_colors[number][:4]) + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + for data in group: + x = x0 + self.horizontal_step * data.content[0] + y = y0 + self.vertical_step * data.content[1] + cr.move_to(x, self.dimensions[VERT] - y) + cr.line_to(x, self.plot_top) + cr.set_line_width(self.series_widths[number]) + cr.stroke() + if self.dots: + cr.new_path() + cr.arc(x, self.dimensions[VERT] - y, 3, 0, 2.1 * math.pi) + cr.close_path() + cr.fill() + +class BarPlot(Plot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None, + main_dir = None): + + self.bounds = {} + self.bounds[HORZ] = x_bounds + self.bounds[VERT] = y_bounds + self.display_values = display_values + self.grid = grid + self.rounded_corners = rounded_corners + self.stack = stack + self.three_dimension = three_dimension + self.x_label_angle = math.pi / 2.5 + self.main_dir = main_dir + self.max_value = {} + self.plot_dimensions = {} + self.steps = {} + self.value_label_color = (0.5,0.5,0.5,1.0) + + Plot.__init__(self, surface, data, width, height, background, border, x_labels, y_labels, series_colors) + + def load_series(self, data, x_labels = None, y_labels = None, series_colors = None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + self.calc_boundaries() + + def process_colors(self, series_colors): + #Data for a BarPlot might be a List or a List of Lists. + #On the first case, colors must be generated for all bars, + #On the second, colors must be generated for each of the inner lists. + + #TODO: Didn't get it... + #if hasattr(self.data[0], '__getitem__'): + # length = max(len(series) for series in self.data) + #else: + # length = len( self.data ) + + length = max(len(group) for group in self.series) + + Plot.process_colors( self, series_colors, length, 'linear') + + def calc_boundaries(self): + if not self.bounds[self.main_dir]: + if self.stack: + max_data_value = max(sum(group.to_list()) for group in self.series) + else: + max_data_value = max(max(group.to_list()) for group in self.series) + self.bounds[self.main_dir] = (0, max_data_value) + if not self.bounds[other_direction(self.main_dir)]: + self.bounds[other_direction(self.main_dir)] = (0, len(self.series)) + + def calc_extents(self, direction): + self.max_value[direction] = 0 + if self.labels[direction]: + widest_word = max(self.labels[direction], key = lambda item: self.context.text_extents(item)[2]) + self.max_value[direction] = self.context.text_extents(widest_word)[3 - direction] + self.borders[other_direction(direction)] = (2-direction)*self.max_value[direction] + self.border + direction*(5) + else: + self.borders[other_direction(direction)] = self.border + + def calc_horz_extents(self): + self.calc_extents(HORZ) + + def calc_vert_extents(self): + self.calc_extents(VERT) + + def calc_all_extents(self): + self.calc_horz_extents() + self.calc_vert_extents() + other_dir = other_direction(self.main_dir) + self.value_label = 0 + if self.display_values: + if self.stack: + self.value_label = self.context.text_extents(str(max(sum(group.to_list()) for group in self.series)))[2 + self.main_dir] + else: + self.value_label = self.context.text_extents(str(max(max(group.to_list()) for group in self.series)))[2 + self.main_dir] + if self.labels[self.main_dir]: + self.plot_dimensions[self.main_dir] = self.dimensions[self.main_dir] - 2*self.borders[self.main_dir] - self.value_label + else: + self.plot_dimensions[self.main_dir] = self.dimensions[self.main_dir] - self.borders[self.main_dir] - 1.2*self.border - self.value_label + self.plot_dimensions[other_dir] = self.dimensions[other_dir] - self.borders[other_dir] - self.border + self.plot_top = self.dimensions[VERT] - self.borders[VERT] + + def calc_steps(self): + other_dir = other_direction(self.main_dir) + self.series_amplitude = self.bounds[self.main_dir][1] - self.bounds[self.main_dir][0] + if self.series_amplitude: + self.steps[self.main_dir] = float(self.plot_dimensions[self.main_dir])/self.series_amplitude + else: + self.steps[self.main_dir] = 0.00 + series_length = len(self.series) + self.steps[other_dir] = float(self.plot_dimensions[other_dir])/(series_length + 0.1*(series_length + 1)) + self.space = 0.1*self.steps[other_dir] + + def render(self): + self.calc_all_extents() + self.calc_steps() + self.render_background() + self.render_bounding_box() + if self.grid: + self.render_grid() + if self.three_dimension: + self.render_ground() + if self.display_values: + self.render_values() + self.render_labels() + self.render_plot() + if self.series_labels: + self.render_legend() + + def draw_3d_rectangle_front(self, x0, y0, x1, y1, shift): + self.context.rectangle(x0-shift, y0+shift, x1-x0, y1-y0) + + def draw_3d_rectangle_side(self, x0, y0, x1, y1, shift): + self.context.move_to(x1-shift,y0+shift) + self.context.line_to(x1, y0) + self.context.line_to(x1, y1) + self.context.line_to(x1-shift, y1+shift) + self.context.line_to(x1-shift, y0+shift) + self.context.close_path() + + def draw_3d_rectangle_top(self, x0, y0, x1, y1, shift): + self.context.move_to(x0-shift,y0+shift) + self.context.line_to(x0, y0) + self.context.line_to(x1, y0) + self.context.line_to(x1-shift, y0+shift) + self.context.line_to(x0-shift, y0+shift) + self.context.close_path() + + def draw_round_rectangle(self, x0, y0, x1, y1): + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) + self.context.line_to(x1-5, y0) + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) + self.context.line_to(x1, y1-5) + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) + self.context.line_to(x0+5, y1) + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) + self.context.line_to(x0, y0+5) + self.context.close_path() + + def render_ground(self): + self.draw_3d_rectangle_front(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_side (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_top (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + def render_labels(self): + self.context.set_font_size(self.font_size * 0.8) + if self.labels[HORZ]: + self.render_horz_labels() + if self.labels[VERT]: + self.render_vert_labels() + + def render_legend(self): + cr = self.context + cr.set_font_size(self.font_size) + cr.set_line_width(self.line_width) + + widest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[2]) + tallest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[3]) + max_width = self.context.text_extents(widest_word)[2] + max_height = self.context.text_extents(tallest_word)[3] * 1.1 + 5 + + color_box_height = max_height / 2 + color_box_width = color_box_height * 2 + + #Draw a bounding box + bounding_box_width = max_width + color_box_width + 15 + bounding_box_height = (len(self.series_labels)+0.5) * max_height + cr.set_source_rgba(1,1,1) + cr.rectangle(self.dimensions[HORZ] - self.border - bounding_box_width, self.border, + bounding_box_width, bounding_box_height) + cr.fill() + + cr.set_source_rgba(*self.line_color) + cr.set_line_width(self.line_width) + cr.rectangle(self.dimensions[HORZ] - self.border - bounding_box_width, self.border, + bounding_box_width, bounding_box_height) + cr.stroke() + + for idx,key in enumerate(self.series_labels): + #Draw color box + cr.set_source_rgba(*self.series_colors[idx][:4]) + cr.rectangle(self.dimensions[HORZ] - self.border - max_width - color_box_width - 10, + self.border + color_box_height + (idx*max_height) , + color_box_width, color_box_height) + cr.fill() + + cr.set_source_rgba(0, 0, 0) + cr.rectangle(self.dimensions[HORZ] - self.border - max_width - color_box_width - 10, + self.border + color_box_height + (idx*max_height), + color_box_width, color_box_height) + cr.stroke() + + #Draw series labels + cr.set_source_rgba(0, 0, 0) + cr.move_to(self.dimensions[HORZ] - self.border - max_width - 5, self.border + ((idx+1)*max_height)) + cr.show_text(key) + + +class HorizontalBarPlot(BarPlot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None): + + BarPlot.__init__(self, surface, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, series_colors, HORZ) + self.series_labels = series_labels + + def calc_vert_extents(self): + self.calc_extents(VERT) + if self.labels[HORZ] and not self.labels[VERT]: + self.borders[HORZ] += 10 + + def draw_rectangle_bottom(self, x0, y0, x1, y1): + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) + self.context.line_to(x0, y0+5) + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) + self.context.line_to(x1, y0) + self.context.line_to(x1, y1) + self.context.line_to(x0+5, y1) + self.context.close_path() + + def draw_rectangle_top(self, x0, y0, x1, y1): + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) + self.context.line_to(x1, y1-5) + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) + self.context.line_to(x0, y1) + self.context.line_to(x0, y0) + self.context.line_to(x1, y0) + self.context.close_path() + + def draw_rectangle(self, index, length, x0, y0, x1, y1): + if length == 1: + BarPlot.draw_rectangle(self, x0, y0, x1, y1) + elif index == 0: + self.draw_rectangle_bottom(x0, y0, x1, y1) + elif index == length-1: + self.draw_rectangle_top(x0, y0, x1, y1) + else: + self.context.rectangle(x0, y0, x1-x0, y1-y0) + + #TODO: Review BarPlot.render_grid code + def render_grid(self): + self.context.set_source_rgba(0.8, 0.8, 0.8) + if self.labels[HORZ]: + self.context.set_font_size(self.font_size * 0.8) + step = (self.dimensions[HORZ] - 2*self.borders[HORZ] - self.value_label)/(len(self.labels[HORZ])-1) + x = self.borders[HORZ] + next_x = 0 + for item in self.labels[HORZ]: + width = self.context.text_extents(item)[2] + if x - width/2 > next_x and x - width/2 > self.border: + self.context.move_to(x, self.border) + self.context.line_to(x, self.dimensions[VERT] - self.borders[VERT]) + self.context.stroke() + next_x = x + width/2 + x += step + else: + lines = 11 + horizontal_step = float(self.plot_dimensions[HORZ])/(lines-1) + x = self.borders[HORZ] + for y in xrange(0, lines): + self.context.move_to(x, self.border) + self.context.line_to(x, self.dimensions[VERT] - self.borders[VERT]) + self.context.stroke() + x += horizontal_step + + def render_horz_labels(self): + step = (self.dimensions[HORZ] - 2*self.borders[HORZ])/(len(self.labels[HORZ])-1) + x = self.borders[HORZ] + next_x = 0 + + for item in self.labels[HORZ]: + self.context.set_source_rgba(*self.label_color) + width = self.context.text_extents(item)[2] + if x - width/2 > next_x and x - width/2 > self.border: + self.context.move_to(x - width/2, self.dimensions[VERT] - self.borders[VERT] + self.max_value[HORZ] + 3) + self.context.show_text(item) + next_x = x + width/2 + x += step + + def render_vert_labels(self): + series_length = len(self.labels[VERT]) + step = (self.plot_dimensions[VERT] - (series_length + 1)*self.space)/(len(self.labels[VERT])) + y = self.border + step/2 + self.space + + for item in self.labels[VERT]: + self.context.set_source_rgba(*self.label_color) + width, height = self.context.text_extents(item)[2:4] + self.context.move_to(self.borders[HORZ] - width - 5, y + height/2) + self.context.show_text(item) + y += step + self.space + self.labels[VERT].reverse() + + def render_values(self): + self.context.set_source_rgba(*self.value_label_color) + self.context.set_font_size(self.font_size * 0.8) + if self.stack: + for i,group in enumerate(self.series): + value = sum(group.to_list()) + height = self.context.text_extents(str(value))[3] + x = self.borders[HORZ] + value*self.steps[HORZ] + 2 + y = self.borders[VERT] + (i+0.5)*self.steps[VERT] + (i+1)*self.space + height/2 + self.context.move_to(x, y) + self.context.show_text(str(value)) + else: + for i,group in enumerate(self.series): + inner_step = self.steps[VERT]/len(group) + y0 = self.border + i*self.steps[VERT] + (i+1)*self.space + for number,data in enumerate(group): + height = self.context.text_extents(str(data.content))[3] + self.context.move_to(self.borders[HORZ] + data.content*self.steps[HORZ] + 2, y0 + 0.5*inner_step + height/2, ) + self.context.show_text(str(data.content)) + y0 += inner_step + + def render_plot(self): + if self.stack: + for i,group in enumerate(self.series): + x0 = self.borders[HORZ] + y0 = self.borders[VERT] + i*self.steps[VERT] + (i+1)*self.space + for number,data in enumerate(group): + if self.series_colors[number][4] in ('radial','linear') : + linear = cairo.LinearGradient( data.content*self.steps[HORZ]/2, y0, data.content*self.steps[HORZ]/2, y0 + self.steps[VERT] ) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + elif self.series_colors[number][4] == 'solid': + self.context.set_source_rgba(*self.series_colors[number][:4]) + if self.rounded_corners: + self.draw_rectangle(number, len(group), x0, y0, x0+data.content*self.steps[HORZ], y0+self.steps[VERT]) + self.context.fill() + else: + self.context.rectangle(x0, y0, data.content*self.steps[HORZ], self.steps[VERT]) + self.context.fill() + x0 += data.content*self.steps[HORZ] + else: + for i,group in enumerate(self.series): + inner_step = self.steps[VERT]/len(group) + x0 = self.borders[HORZ] + y0 = self.border + i*self.steps[VERT] + (i+1)*self.space + for number,data in enumerate(group): + linear = cairo.LinearGradient(data.content*self.steps[HORZ]/2, y0, data.content*self.steps[HORZ]/2, y0 + inner_step) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + if self.rounded_corners and data.content != 0: + BarPlot.draw_round_rectangle(self,x0, y0, x0 + data.content*self.steps[HORZ], y0 + inner_step) + self.context.fill() + else: + self.context.rectangle(x0, y0, data.content*self.steps[HORZ], inner_step) + self.context.fill() + y0 += inner_step + +class VerticalBarPlot(BarPlot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None): + + BarPlot.__init__(self, surface, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, series_colors, VERT) + self.series_labels = series_labels + + def calc_vert_extents(self): + self.calc_extents(VERT) + if self.labels[VERT] and not self.labels[HORZ]: + self.borders[VERT] += 10 + + def draw_rectangle_bottom(self, x0, y0, x1, y1): + self.context.move_to(x1,y1) + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) + self.context.line_to(x0+5, y1) + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) + self.context.line_to(x0, y0) + self.context.line_to(x1, y0) + self.context.line_to(x1, y1) + self.context.close_path() + + def draw_rectangle_top(self, x0, y0, x1, y1): + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) + self.context.line_to(x1-5, y0) + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) + self.context.line_to(x1, y1) + self.context.line_to(x0, y1) + self.context.line_to(x0, y0) + self.context.close_path() + + def draw_rectangle(self, index, length, x0, y0, x1, y1): + if length == 1: + BarPlot.draw_rectangle(self, x0, y0, x1, y1) + elif index == 0: + self.draw_rectangle_bottom(x0, y0, x1, y1) + elif index == length-1: + self.draw_rectangle_top(x0, y0, x1, y1) + else: + self.context.rectangle(x0, y0, x1-x0, y1-y0) + + def render_grid(self): + self.context.set_source_rgba(0.8, 0.8, 0.8) + if self.labels[VERT]: + lines = len(self.labels[VERT]) + vertical_step = float(self.plot_dimensions[self.main_dir])/(lines-1) + y = self.borders[VERT] + self.value_label + else: + lines = 11 + vertical_step = float(self.plot_dimensions[self.main_dir])/(lines-1) + y = 1.2*self.border + self.value_label + for x in xrange(0, lines): + self.context.move_to(self.borders[HORZ], y) + self.context.line_to(self.dimensions[HORZ] - self.border, y) + self.context.stroke() + y += vertical_step + + def render_ground(self): + self.draw_3d_rectangle_front(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_side (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_top (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + def render_horz_labels(self): + series_length = len(self.labels[HORZ]) + step = float (self.plot_dimensions[HORZ] - (series_length + 1)*self.space)/len(self.labels[HORZ]) + x = self.borders[HORZ] + step/2 + self.space + next_x = 0 + + for item in self.labels[HORZ]: + self.context.set_source_rgba(*self.label_color) + width = self.context.text_extents(item)[2] + if x - width/2 > next_x and x - width/2 > self.borders[HORZ]: + self.context.move_to(x - width/2, self.dimensions[VERT] - self.borders[VERT] + self.max_value[HORZ] + 3) + self.context.show_text(item) + next_x = x + width/2 + x += step + self.space + + def render_vert_labels(self): + self.context.set_source_rgba(*self.label_color) + y = self.borders[VERT] + self.value_label + step = (self.dimensions[VERT] - 2*self.borders[VERT] - self.value_label)/(len(self.labels[VERT]) - 1) + self.labels[VERT].reverse() + for item in self.labels[VERT]: + width, height = self.context.text_extents(item)[2:4] + self.context.move_to(self.borders[HORZ] - width - 5, y + height/2) + self.context.show_text(item) + y += step + self.labels[VERT].reverse() + + def render_values(self): + self.context.set_source_rgba(*self.value_label_color) + self.context.set_font_size(self.font_size * 0.8) + if self.stack: + for i,group in enumerate(self.series): + value = sum(group.to_list()) + width = self.context.text_extents(str(value))[2] + x = self.borders[HORZ] + (i+0.5)*self.steps[HORZ] + (i+1)*self.space - width/2 + y = value*self.steps[VERT] + 2 + self.context.move_to(x, self.plot_top-y) + self.context.show_text(str(value)) + else: + for i,group in enumerate(self.series): + inner_step = self.steps[HORZ]/len(group) + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space + for number,data in enumerate(group): + width = self.context.text_extents(str(data.content))[2] + self.context.move_to(x0 + 0.5*inner_step - width/2, self.plot_top - data.content*self.steps[VERT] - 2) + self.context.show_text(str(data.content)) + x0 += inner_step + + def render_plot(self): + if self.stack: + for i,group in enumerate(self.series): + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space + y0 = 0 + for number,data in enumerate(group): + if self.series_colors[number][4] in ('linear','radial'): + linear = cairo.LinearGradient( x0, data.content*self.steps[VERT]/2, x0 + self.steps[HORZ], data.content*self.steps[VERT]/2 ) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + elif self.series_colors[number][4] == 'solid': + self.context.set_source_rgba(*self.series_colors[number][:4]) + if self.rounded_corners: + self.draw_rectangle(number, len(group), x0, self.plot_top - y0 - data.content*self.steps[VERT], x0 + self.steps[HORZ], self.plot_top - y0) + self.context.fill() + else: + self.context.rectangle(x0, self.plot_top - y0 - data.content*self.steps[VERT], self.steps[HORZ], data.content*self.steps[VERT]) + self.context.fill() + y0 += data.content*self.steps[VERT] + else: + for i,group in enumerate(self.series): + inner_step = self.steps[HORZ]/len(group) + y0 = self.borders[VERT] + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space + for number,data in enumerate(group): + if self.series_colors[number][4] == 'linear': + linear = cairo.LinearGradient( x0, data.content*self.steps[VERT]/2, x0 + inner_step, data.content*self.steps[VERT]/2 ) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + elif self.series_colors[number][4] == 'solid': + self.context.set_source_rgba(*self.series_colors[number][:4]) + if self.rounded_corners and data.content != 0: + BarPlot.draw_round_rectangle(self, x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top) + self.context.fill() + elif self.three_dimension: + self.draw_3d_rectangle_front(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) + self.context.fill() + self.draw_3d_rectangle_side(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) + self.context.fill() + self.draw_3d_rectangle_top(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) + self.context.fill() + else: + self.context.rectangle(x0, self.plot_top - data.content*self.steps[VERT], inner_step, data.content*self.steps[VERT]) + self.context.fill() + + x0 += inner_step + +class StreamChart(VerticalBarPlot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + grid = False, + series_legend = None, + x_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None): + + VerticalBarPlot.__init__(self, surface, data, width, height, background, border, + False, grid, False, True, False, + None, x_labels, None, x_bounds, y_bounds, series_colors) + + def calc_steps(self): + other_dir = other_direction(self.main_dir) + self.series_amplitude = self.bounds[self.main_dir][1] - self.bounds[self.main_dir][0] + if self.series_amplitude: + self.steps[self.main_dir] = float(self.plot_dimensions[self.main_dir])/self.series_amplitude + else: + self.steps[self.main_dir] = 0.00 + series_length = len(self.data) + self.steps[other_dir] = float(self.plot_dimensions[other_dir])/series_length + + def render_legend(self): + pass + + def ground(self, index): + sum_values = sum(self.data[index]) + return -0.5*sum_values + + def calc_angles(self): + middle = self.plot_top - self.plot_dimensions[VERT]/2.0 + self.angles = [tuple([0.0 for x in range(len(self.data)+1)])] + for x_index in range(1, len(self.data)-1): + t = [] + x0 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y0 = middle - self.ground(x_index-1)*self.steps[VERT] + y2 = middle - self.ground(x_index+1)*self.steps[VERT] + t.append(math.atan(float(y0-y2)/(x0-x2))) + for data_index in range(len(self.data[x_index])): + x0 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y0 = middle - self.ground(x_index-1)*self.steps[VERT] - self.data[x_index-1][data_index]*self.steps[VERT] + y2 = middle - self.ground(x_index+1)*self.steps[VERT] - self.data[x_index+1][data_index]*self.steps[VERT] + + for i in range(0,data_index): + y0 -= self.data[x_index-1][i]*self.steps[VERT] + y2 -= self.data[x_index+1][i]*self.steps[VERT] + + if data_index == len(self.data[0])-1 and False: + self.context.set_source_rgba(0.0,0.0,0.0,0.3) + self.context.move_to(x0,y0) + self.context.line_to(x2,y2) + self.context.stroke() + self.context.arc(x0,y0,2,0,2*math.pi) + self.context.fill() + t.append(math.atan(float(y0-y2)/(x0-x2))) + self.angles.append(tuple(t)) + self.angles.append(tuple([0.0 for x in range(len(self.data)+1)])) + + def render_plot(self): + self.calc_angles() + middle = self.plot_top - self.plot_dimensions[VERT]/2.0 + p = 0.4*self.steps[HORZ] + for data_index in range(len(self.data[0])-1,-1,-1): + self.context.set_source_rgba(*self.series_colors[data_index][:4]) + + #draw the upper line + for x_index in range(len(self.data)-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] - self.data[x_index][data_index]*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index + 1)*self.steps[VERT] - self.data[x_index + 1][data_index]*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index+1][i]*self.steps[VERT] + + if x_index == 0: + self.context.move_to(x1,y1) + + ang1 = self.angles[x_index][data_index+1] + ang2 = self.angles[x_index+1][data_index+1] + math.pi + self.context.curve_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1), + x2+p*math.cos(ang2),y2+p*math.sin(ang2), + x2,y2) + + for x_index in range(len(self.data)-1,0,-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index - 1)*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index-1][i]*self.steps[VERT] + + if x_index == len(self.data)-1: + self.context.line_to(x1,y1+2) + + #revert angles by pi degrees to take the turn back + ang1 = self.angles[x_index][data_index] + math.pi + ang2 = self.angles[x_index-1][data_index] + self.context.curve_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1), + x2+p*math.cos(ang2),y2+p*math.sin(ang2), + x2,y2+2) + + self.context.close_path() + self.context.fill() + + if False: + self.context.move_to(self.borders[HORZ] + 0.5*self.steps[HORZ], middle) + for x_index in range(len(self.data)-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] - self.data[x_index][data_index]*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index + 1)*self.steps[VERT] - self.data[x_index + 1][data_index]*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index+1][i]*self.steps[VERT] + + ang1 = self.angles[x_index][data_index+1] + ang2 = self.angles[x_index+1][data_index+1] + math.pi + self.context.set_source_rgba(1.0,0.0,0.0) + self.context.arc(x1+p*math.cos(ang1),y1+p*math.sin(ang1),2,0,2*math.pi) + self.context.fill() + self.context.set_source_rgba(0.0,0.0,0.0) + self.context.arc(x2+p*math.cos(ang2),y2+p*math.sin(ang2),2,0,2*math.pi) + self.context.fill() + '''self.context.set_source_rgba(0.0,0.0,0.0,0.3) + self.context.arc(x2,y2,2,0,2*math.pi) + self.context.fill()''' + self.context.move_to(x1,y1) + self.context.line_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1)) + self.context.stroke() + self.context.move_to(x2,y2) + self.context.line_to(x2+p*math.cos(ang2),y2+p*math.sin(ang2)) + self.context.stroke() + if False: + for x_index in range(len(self.data)-1,0,-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index - 1)*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index-1][i]*self.steps[VERT] + + #revert angles by pi degrees to take the turn back + ang1 = self.angles[x_index][data_index] + math.pi + ang2 = self.angles[x_index-1][data_index] + self.context.set_source_rgba(0.0,1.0,0.0) + self.context.arc(x1+p*math.cos(ang1),y1+p*math.sin(ang1),2,0,2*math.pi) + self.context.fill() + self.context.set_source_rgba(0.0,0.0,1.0) + self.context.arc(x2+p*math.cos(ang2),y2+p*math.sin(ang2),2,0,2*math.pi) + self.context.fill() + '''self.context.set_source_rgba(0.0,0.0,0.0,0.3) + self.context.arc(x2,y2,2,0,2*math.pi) + self.context.fill()''' + self.context.move_to(x1,y1) + self.context.line_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1)) + self.context.stroke() + self.context.move_to(x2,y2) + self.context.line_to(x2+p*math.cos(ang2),y2+p*math.sin(ang2)) + self.context.stroke() + #break + + #self.context.arc(self.dimensions[HORZ]/2, self.dimensions[VERT]/2,50,0,3*math.pi/2) + #self.context.fill() + + +class PiePlot(Plot): + #TODO: Check the old cairoplot, graphs aren't matching + def __init__ (self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + gradient = False, + shadow = False, + colors = None): + + Plot.__init__( self, surface, data, width, height, background, series_colors = colors ) + self.center = (self.dimensions[HORZ]/2, self.dimensions[VERT]/2) + self.total = sum( self.series.to_list() ) + self.radius = min(self.dimensions[HORZ]/3,self.dimensions[VERT]/3) + self.gradient = gradient + self.shadow = shadow + + def sort_function(x,y): + return x.content - y.content + + def load_series(self, data, x_labels=None, y_labels=None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + # Already done inside series + #self.data = sorted(self.data) + + def draw_piece(self, angle, next_angle): + self.context.move_to(self.center[0],self.center[1]) + self.context.line_to(self.center[0] + self.radius*math.cos(angle), self.center[1] + self.radius*math.sin(angle)) + self.context.arc(self.center[0], self.center[1], self.radius, angle, next_angle) + self.context.line_to(self.center[0], self.center[1]) + self.context.close_path() + + def render(self): + self.render_background() + self.render_bounding_box() + if self.shadow: + self.render_shadow() + self.render_plot() + self.render_series_labels() + + def render_shadow(self): + horizontal_shift = 3 + vertical_shift = 3 + self.context.set_source_rgba(0, 0, 0, 0.5) + self.context.arc(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.radius, 0, 2*math.pi) + self.context.fill() + + def render_series_labels(self): + angle = 0 + next_angle = 0 + x0,y0 = self.center + cr = self.context + for number,key in enumerate(self.series_labels): + # self.data[number] should be just a number + data = sum(self.series[number].to_list()) + + next_angle = angle + 2.0*math.pi*data/self.total + cr.set_source_rgba(*self.series_colors[number][:4]) + w = cr.text_extents(key)[2] + if (angle + next_angle)/2 < math.pi/2 or (angle + next_angle)/2 > 3*math.pi/2: + cr.move_to(x0 + (self.radius+10)*math.cos((angle+next_angle)/2), y0 + (self.radius+10)*math.sin((angle+next_angle)/2) ) + else: + cr.move_to(x0 + (self.radius+10)*math.cos((angle+next_angle)/2) - w, y0 + (self.radius+10)*math.sin((angle+next_angle)/2) ) + cr.show_text(key) + angle = next_angle + + def render_plot(self): + angle = 0 + next_angle = 0 + x0,y0 = self.center + cr = self.context + for number,group in enumerate(self.series): + # Group should be just a number + data = sum(group.to_list()) + next_angle = angle + 2.0*math.pi*data/self.total + if self.gradient or self.series_colors[number][4] in ('linear','radial'): + gradient_color = cairo.RadialGradient(self.center[0], self.center[1], 0, self.center[0], self.center[1], self.radius) + gradient_color.add_color_stop_rgba(0.3, *self.series_colors[number][:4]) + gradient_color.add_color_stop_rgba(1, self.series_colors[number][0]*0.7, + self.series_colors[number][1]*0.7, + self.series_colors[number][2]*0.7, + self.series_colors[number][3]) + cr.set_source(gradient_color) + else: + cr.set_source_rgba(*self.series_colors[number][:4]) + + self.draw_piece(angle, next_angle) + cr.fill() + + cr.set_source_rgba(1.0, 1.0, 1.0) + self.draw_piece(angle, next_angle) + cr.stroke() + + angle = next_angle + +class DonutPlot(PiePlot): + def __init__ (self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + gradient = False, + shadow = False, + colors = None, + inner_radius=-1): + + Plot.__init__( self, surface, data, width, height, background, series_colors = colors ) + + self.center = ( self.dimensions[HORZ]/2, self.dimensions[VERT]/2 ) + self.total = sum( self.series.to_list() ) + self.radius = min( self.dimensions[HORZ]/3,self.dimensions[VERT]/3 ) + self.inner_radius = inner_radius*self.radius + + if inner_radius == -1: + self.inner_radius = self.radius/3 + + self.gradient = gradient + self.shadow = shadow + + def draw_piece(self, angle, next_angle): + self.context.move_to(self.center[0] + (self.inner_radius)*math.cos(angle), self.center[1] + (self.inner_radius)*math.sin(angle)) + self.context.line_to(self.center[0] + self.radius*math.cos(angle), self.center[1] + self.radius*math.sin(angle)) + self.context.arc(self.center[0], self.center[1], self.radius, angle, next_angle) + self.context.line_to(self.center[0] + (self.inner_radius)*math.cos(next_angle), self.center[1] + (self.inner_radius)*math.sin(next_angle)) + self.context.arc_negative(self.center[0], self.center[1], self.inner_radius, next_angle, angle) + self.context.close_path() + + def render_shadow(self): + horizontal_shift = 3 + vertical_shift = 3 + self.context.set_source_rgba(0, 0, 0, 0.5) + self.context.arc(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.inner_radius, 0, 2*math.pi) + self.context.arc_negative(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.radius, 0, -2*math.pi) + self.context.fill() + +class GanttChart (Plot) : + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + x_labels = None, + y_labels = None, + colors = None): + self.bounds = {} + self.max_value = {} + Plot.__init__(self, surface, data, width, height, x_labels = x_labels, y_labels = y_labels, series_colors = colors) + + def load_series(self, data, x_labels=None, y_labels=None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + self.calc_boundaries() + + def calc_boundaries(self): + self.bounds[HORZ] = (0,len(self.series)) + end_pos = max(self.series.to_list()) + + #for group in self.series: + # if hasattr(item, "__delitem__"): + # for sub_item in item: + # end_pos = max(sub_item) + # else: + # end_pos = max(item) + self.bounds[VERT] = (0,end_pos) + + def calc_extents(self, direction): + self.max_value[direction] = 0 + if self.labels[direction]: + self.max_value[direction] = max(self.context.text_extents(item)[2] for item in self.labels[direction]) + else: + self.max_value[direction] = self.context.text_extents( str(self.bounds[direction][1] + 1) )[2] + + def calc_horz_extents(self): + self.calc_extents(HORZ) + self.borders[HORZ] = 100 + self.max_value[HORZ] + + def calc_vert_extents(self): + self.calc_extents(VERT) + self.borders[VERT] = self.dimensions[VERT]/(self.bounds[HORZ][1] + 1) + + def calc_steps(self): + self.horizontal_step = (self.dimensions[HORZ] - self.borders[HORZ])/(len(self.labels[VERT])) + self.vertical_step = self.borders[VERT] + + def render(self): + self.calc_horz_extents() + self.calc_vert_extents() + self.calc_steps() + self.render_background() + + self.render_labels() + self.render_grid() + self.render_plot() + + def render_background(self): + cr = self.context + cr.set_source_rgba(255,255,255) + cr.rectangle(0,0,self.dimensions[HORZ], self.dimensions[VERT]) + cr.fill() + for number,group in enumerate(self.series): + linear = cairo.LinearGradient(self.dimensions[HORZ]/2, self.borders[VERT] + number*self.vertical_step, + self.dimensions[HORZ]/2, self.borders[VERT] + (number+1)*self.vertical_step) + linear.add_color_stop_rgba(0,1.0,1.0,1.0,1.0) + linear.add_color_stop_rgba(1.0,0.9,0.9,0.9,1.0) + cr.set_source(linear) + cr.rectangle(0,self.borders[VERT] + number*self.vertical_step,self.dimensions[HORZ],self.vertical_step) + cr.fill() + + def render_grid(self): + cr = self.context + cr.set_source_rgba(0.7, 0.7, 0.7) + cr.set_dash((1,0,0,0,0,0,1)) + cr.set_line_width(0.5) + for number,label in enumerate(self.labels[VERT]): + h = cr.text_extents(label)[3] + cr.move_to(self.borders[HORZ] + number*self.horizontal_step, self.vertical_step/2 + h) + cr.line_to(self.borders[HORZ] + number*self.horizontal_step, self.dimensions[VERT]) + cr.stroke() + + def render_labels(self): + self.context.set_font_size(0.02 * self.dimensions[HORZ]) + + self.render_horz_labels() + self.render_vert_labels() + + def render_horz_labels(self): + cr = self.context + labels = self.labels[HORZ] + if not labels: + labels = [str(i) for i in range(1, self.bounds[HORZ][1] + 1) ] + for number,label in enumerate(labels): + if label != None: + cr.set_source_rgba(0.5, 0.5, 0.5) + w,h = cr.text_extents(label)[2], cr.text_extents(label)[3] + cr.move_to(40,self.borders[VERT] + number*self.vertical_step + self.vertical_step/2 + h/2) + cr.show_text(label) + + def render_vert_labels(self): + cr = self.context + labels = self.labels[VERT] + if not labels: + labels = [str(i) for i in range(1, self.bounds[VERT][1] + 1) ] + for number,label in enumerate(labels): + w,h = cr.text_extents(label)[2], cr.text_extents(label)[3] + cr.move_to(self.borders[HORZ] + number*self.horizontal_step - w/2, self.vertical_step/2) + cr.show_text(label) + + def render_rectangle(self, x0, y0, x1, y1, color): + self.draw_shadow(x0, y0, x1, y1) + self.draw_rectangle(x0, y0, x1, y1, color) + + def draw_rectangular_shadow(self, gradient, x0, y0, w, h): + self.context.set_source(gradient) + self.context.rectangle(x0,y0,w,h) + self.context.fill() + + def draw_circular_shadow(self, x, y, radius, ang_start, ang_end, mult, shadow): + gradient = cairo.RadialGradient(x, y, 0, x, y, 2*radius) + gradient.add_color_stop_rgba(0, 0, 0, 0, shadow) + gradient.add_color_stop_rgba(1, 0, 0, 0, 0) + self.context.set_source(gradient) + self.context.move_to(x,y) + self.context.line_to(x + mult[0]*radius,y + mult[1]*radius) + self.context.arc(x, y, 8, ang_start, ang_end) + self.context.line_to(x,y) + self.context.close_path() + self.context.fill() + + def draw_rectangle(self, x0, y0, x1, y1, color): + cr = self.context + middle = (x0+x1)/2 + linear = cairo.LinearGradient(middle,y0,middle,y1) + linear.add_color_stop_rgba(0,3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1,*color[:4]) + cr.set_source(linear) + + cr.arc(x0+5, y0+5, 5, 0, 2*math.pi) + cr.arc(x1-5, y0+5, 5, 0, 2*math.pi) + cr.arc(x0+5, y1-5, 5, 0, 2*math.pi) + cr.arc(x1-5, y1-5, 5, 0, 2*math.pi) + cr.rectangle(x0+5,y0,x1-x0-10,y1-y0) + cr.rectangle(x0,y0+5,x1-x0,y1-y0-10) + cr.fill() + + def draw_shadow(self, x0, y0, x1, y1): + shadow = 0.4 + h_mid = (x0+x1)/2 + v_mid = (y0+y1)/2 + h_linear_1 = cairo.LinearGradient(h_mid,y0-4,h_mid,y0+4) + h_linear_2 = cairo.LinearGradient(h_mid,y1-4,h_mid,y1+4) + v_linear_1 = cairo.LinearGradient(x0-4,v_mid,x0+4,v_mid) + v_linear_2 = cairo.LinearGradient(x1-4,v_mid,x1+4,v_mid) + + h_linear_1.add_color_stop_rgba( 0, 0, 0, 0, 0) + h_linear_1.add_color_stop_rgba( 1, 0, 0, 0, shadow) + h_linear_2.add_color_stop_rgba( 0, 0, 0, 0, shadow) + h_linear_2.add_color_stop_rgba( 1, 0, 0, 0, 0) + v_linear_1.add_color_stop_rgba( 0, 0, 0, 0, 0) + v_linear_1.add_color_stop_rgba( 1, 0, 0, 0, shadow) + v_linear_2.add_color_stop_rgba( 0, 0, 0, 0, shadow) + v_linear_2.add_color_stop_rgba( 1, 0, 0, 0, 0) + + self.draw_rectangular_shadow(h_linear_1,x0+4,y0-4,x1-x0-8,8) + self.draw_rectangular_shadow(h_linear_2,x0+4,y1-4,x1-x0-8,8) + self.draw_rectangular_shadow(v_linear_1,x0-4,y0+4,8,y1-y0-8) + self.draw_rectangular_shadow(v_linear_2,x1-4,y0+4,8,y1-y0-8) + + self.draw_circular_shadow(x0+4, y0+4, 4, math.pi, 3*math.pi/2, (-1,0), shadow) + self.draw_circular_shadow(x1-4, y0+4, 4, 3*math.pi/2, 2*math.pi, (0,-1), shadow) + self.draw_circular_shadow(x0+4, y1-4, 4, math.pi/2, math.pi, (0,1), shadow) + self.draw_circular_shadow(x1-4, y1-4, 4, 0, math.pi/2, (1,0), shadow) + + def render_plot(self): + for index,group in enumerate(self.series): + for data in group: + self.render_rectangle(self.borders[HORZ] + data.content[0]*self.horizontal_step, + self.borders[VERT] + index*self.vertical_step + self.vertical_step/4.0, + self.borders[HORZ] + data.content[1]*self.horizontal_step, + self.borders[VERT] + index*self.vertical_step + 3.0*self.vertical_step/4.0, + self.series_colors[index]) + +# Function definition + +def scatter_plot(name, + data = None, + errorx = None, + errory = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + axis = False, + dash = False, + discrete = False, + dots = False, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + z_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + circle_colors = None): + + ''' + - Function to plot scatter data. + + - Parameters + + data - The values to be ploted might be passed in a two basic: + list of points: [(0,0), (0,1), (0,2)] or [(0,0,1), (0,1,4), (0,2,1)] + lists of coordinates: [ [0,0,0] , [0,1,2] ] or [ [0,0,0] , [0,1,2] , [1,4,1] ] + Notice that these kinds of that can be grouped in order to form more complex data + using lists of lists or dictionaries; + series_colors - Define color values for each of the series + circle_colors - Define a lower and an upper bound for the circle colors for variable radius + (3 dimensions) series + ''' + + plot = ScatterPlot( name, data, errorx, errory, width, height, background, border, + axis, dash, discrete, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, z_bounds, x_title, y_title, series_colors, circle_colors ) + plot.render() + plot.commit() + +def dot_line_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + axis = False, + dash = False, + dots = False, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None): + ''' + - Function to plot graphics using dots and lines. + + dot_line_plot (name, data, width, height, background = "white light_gray", border = 0, axis = False, grid = False, x_labels = None, y_labels = None, x_bounds = None, y_bounds = None) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + axis - Whether or not the axis are to be drawn; + dash - Boolean or a list or a dictionary of booleans indicating whether or not the associated series should be drawn in dashed mode; + dots - Whether or not dots should be drawn on each point; + grid - Whether or not the gris is to be drawn; + series_legend - Whether or not the legend is to be drawn; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + x_title - Whether or not to plot a title over the x axis. + y_title - Whether or not to plot a title over the y axis. + + - Examples of use + + data = [0, 1, 3, 8, 9, 0, 10, 10, 2, 1] + CairoPlot.dot_line_plot('teste', data, 400, 300) + + data = { "john" : [10, 10, 10, 10, 30], "mary" : [0, 0, 3, 5, 15], "philip" : [13, 32, 11, 25, 2] } + x_labels = ["jan/2008", "feb/2008", "mar/2008", "apr/2008", "may/2008" ] + CairoPlot.dot_line_plot( 'test', data, 400, 300, axis = True, grid = True, + series_legend = True, x_labels = x_labels ) + ''' + plot = DotLinePlot( name, data, width, height, background, border, + axis, dash, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, x_title, y_title, series_colors ) + plot.render() + plot.commit() + +def function_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + axis = True, + dots = False, + discrete = False, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + step = 1): + + ''' + - Function to plot functions. + + function_plot(name, data, width, height, background = "white light_gray", border = 0, axis = True, grid = False, dots = False, x_labels = None, y_labels = None, x_bounds = None, y_bounds = None, step = 1, discrete = False) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + axis - Whether or not the axis are to be drawn; + grid - Whether or not the gris is to be drawn; + dots - Whether or not dots should be shown at each point; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + step - the horizontal distance from one point to the other. The smaller, the smoother the curve will be; + discrete - whether or not the function should be plotted in discrete format. + + - Example of use + + data = lambda x : x**2 + CairoPlot.function_plot('function4', data, 400, 300, grid = True, x_bounds=(-10,10), step = 0.1) + ''' + + plot = FunctionPlot( name, data, width, height, background, border, + axis, discrete, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, x_title, y_title, series_colors, step ) + plot.render() + plot.commit() + +def pie_plot( name, data, width, height, background = "white light_gray", gradient = False, shadow = False, colors = None ): + + ''' + - Function to plot pie graphics. + + pie_plot(name, data, width, height, background = "white light_gray", gradient = False, colors = None) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + gradient - Whether or not the pie color will be painted with a gradient; + shadow - Whether or not there will be a shadow behind the pie; + colors - List of slices colors. + + - Example of use + + teste_data = {"john" : 123, "mary" : 489, "philip" : 890 , "suzy" : 235} + CairoPlot.pie_plot("pie_teste", teste_data, 500, 500) + ''' + + plot = PiePlot( name, data, width, height, background, gradient, shadow, colors ) + plot.render() + plot.commit() + +def donut_plot(name, data, width, height, background = "white light_gray", gradient = False, shadow = False, colors = None, inner_radius = -1): + + ''' + - Function to plot donut graphics. + + donut_plot(name, data, width, height, background = "white light_gray", gradient = False, inner_radius = -1) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + shadow - Whether or not there will be a shadow behind the donut; + gradient - Whether or not the donut color will be painted with a gradient; + colors - List of slices colors; + inner_radius - The radius of the donut's inner circle. + + - Example of use + + teste_data = {"john" : 123, "mary" : 489, "philip" : 890 , "suzy" : 235} + CairoPlot.donut_plot("donut_teste", teste_data, 500, 500) + ''' + + plot = DonutPlot(name, data, width, height, background, gradient, shadow, colors, inner_radius) + plot.render() + plot.commit() + +def gantt_chart(name, pieces, width, height, x_labels, y_labels, colors): + + ''' + - Function to generate Gantt Charts. + + gantt_chart(name, pieces, width, height, x_labels, y_labels, colors): + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + pieces - A list defining the spaces to be drawn. The user must pass, for each line, the index of its start and the index of its end. If a line must have two or more spaces, they must be passed inside a list; + width, height - Dimensions of the output image; + x_labels - A list of names for each of the vertical lines; + y_labels - A list of names for each of the horizontal spaces; + colors - List containing the colors expected for each of the horizontal spaces + + - Example of use + + pieces = [ (0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,8)] + x_labels = [ 'teste01', 'teste02', 'teste03', 'teste04'] + y_labels = [ '0001', '0002', '0003', '0004', '0005', '0006', '0007', '0008', '0009', '0010' ] + colors = [ (1.0, 0.0, 0.0), (1.0, 0.7, 0.0), (1.0, 1.0, 0.0), (0.0, 1.0, 0.0) ] + CairoPlot.gantt_chart('gantt_teste', pieces, 600, 300, x_labels, y_labels, colors) + ''' + + plot = GanttChart(name, pieces, width, height, x_labels, y_labels, colors) + plot.render() + plot.commit() + +def vertical_bar_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + colors = None): + #TODO: Fix docstring for vertical_bar_plot + ''' + - Function to generate vertical Bar Plot Charts. + + bar_plot(name, data, width, height, background, border, grid, rounded_corners, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, colors): + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtime; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + grid - Whether or not the gris is to be drawn; + rounded_corners - Whether or not the bars should have rounded corners; + three_dimension - Whether or not the bars should be drawn in pseudo 3D; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + colors - List containing the colors expected for each of the bars. + + - Example of use + + data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] + CairoPlot.vertical_bar_plot ('bar2', data, 400, 300, border = 20, grid = True, rounded_corners = False) + ''' + + plot = VerticalBarPlot(name, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + series_labels, x_labels, y_labels, x_bounds, y_bounds, colors) + plot.render() + plot.commit() + +def horizontal_bar_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + colors = None): + + #TODO: Fix docstring for horizontal_bar_plot + ''' + - Function to generate Horizontal Bar Plot Charts. + + bar_plot(name, data, width, height, background, border, grid, rounded_corners, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, colors): + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtime; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + grid - Whether or not the gris is to be drawn; + rounded_corners - Whether or not the bars should have rounded corners; + three_dimension - Whether or not the bars should be drawn in pseudo 3D; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + colors - List containing the colors expected for each of the bars. + + - Example of use + + data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] + CairoPlot.bar_plot ('bar2', data, 400, 300, border = 20, grid = True, rounded_corners = False) + ''' + + plot = HorizontalBarPlot(name, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + series_labels, x_labels, y_labels, x_bounds, y_bounds, colors) + plot.render() + plot.commit() + +def stream_chart(name, + data, + width, + height, + background = "white light_gray", + border = 0, + grid = False, + series_legend = None, + x_labels = None, + x_bounds = None, + y_bounds = None, + colors = None): + + #TODO: Fix docstring for horizontal_bar_plot + plot = StreamChart(name, data, width, height, background, border, + grid, series_legend, x_labels, x_bounds, y_bounds, colors) + plot.render() + plot.commit() + + +if __name__ == "__main__": + import tests + import seriestests diff --git a/bindings/python/examples/output_format_modules/pprint_table.py b/bindings/python/examples/output_format_modules/pprint_table.py new file mode 100644 index 0000000..1cd8620 --- /dev/null +++ b/bindings/python/examples/output_format_modules/pprint_table.py @@ -0,0 +1,35 @@ +# This module is used to pretty-print a table +# Adapted from +# http://ginstrom.com/scribbles/2007/09/04/pretty-printing-a-table-in-python/ + +import sys + +def get_max_width(table, index): + """Get the maximum width of the given column index""" + + return max([len(str(row[index])) for row in table]) + + +def pprint_table(table, nbLeft=1, out=sys.stdout): + """ + Prints out a table of data, padded for alignment + @param table: The table to print. A list of lists. + Each row must have the same number of columns. + @param nbLeft: The number of columns aligned left + @param out: Output stream (file-like object) + """ + + col_paddings = [] + + for i in range(len(table[0])): + col_paddings.append(get_max_width(table, i)) + + for row in table: + # left cols + for i in range(nbLeft): + print >> out, str(row[i]).ljust(col_paddings[i] + 1), + # rest of the cols + for i in range(nbLeft, len(row)): + col = str(row[i]).rjust(col_paddings[i] + 2) + print >> out, col, + print >> out diff --git a/bindings/python/examples/output_format_modules/series.py b/bindings/python/examples/output_format_modules/series.py new file mode 100755 index 0000000..8e8b236 --- /dev/null +++ b/bindings/python/examples/output_format_modules/series.py @@ -0,0 +1,1140 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# Serie.py +# +# Copyright (c) 2008 Magnun Leno da Silva +# +# Author: Magnun Leno da Silva +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public License +# as published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 +# USA + +# Contributor: Rodrigo Moreiro Araujo + +#import cairoplot +import doctest + +NUMTYPES = (int, float, long) +LISTTYPES = (list, tuple) +STRTYPES = (str, unicode) +FILLING_TYPES = ['linear', 'solid', 'gradient'] +DEFAULT_COLOR_FILLING = 'solid' +#TODO: Define default color list +DEFAULT_COLOR_LIST = None + +class Data(object): + ''' + Class that models the main data structure. + It can hold: + - a number type (int, float or long) + - a tuple, witch represents a point and can have 2 or 3 items (x,y,z) + - if a list is passed it will be converted to a tuple. + + obs: In case a tuple is passed it will convert to tuple + ''' + def __init__(self, data=None, name=None, parent=None): + ''' + Starts main atributes from the Data class + @name - Name for each point; + @content - The real data, can be an int, float, long or tuple, which + represents a point (x,y) or (x,y,z); + @parent - A pointer that give the data access to it's parent. + + Usage: + >>> d = Data(name='empty'); print d + empty: () + >>> d = Data((1,1),'point a'); print d + point a: (1, 1) + >>> d = Data((1,2,3),'point b'); print d + point b: (1, 2, 3) + >>> d = Data([2,3],'point c'); print d + point c: (2, 3) + >>> d = Data(12, 'simple value'); print d + simple value: 12 + ''' + # Initial values + self.__content = None + self.__name = None + + # Setting passed values + self.parent = parent + self.name = name + self.content = data + + # Name property + @apply + def name(): + doc = ''' + Name is a read/write property that controls the input of name. + - If passed an invalid value it cleans the name with None + + Usage: + >>> d = Data(13); d.name = 'name_test'; print d + name_test: 13 + >>> d.name = 11; print d + 13 + >>> d.name = 'other_name'; print d + other_name: 13 + >>> d.name = None; print d + 13 + >>> d.name = 'last_name'; print d + last_name: 13 + >>> d.name = ''; print d + 13 + ''' + def fget(self): + ''' + returns the name as a string + ''' + return self.__name + + def fset(self, name): + ''' + Sets the name of the Data + ''' + if type(name) in STRTYPES and len(name) > 0: + self.__name = name + else: + self.__name = None + + + + return property(**locals()) + + # Content property + @apply + def content(): + doc = ''' + Content is a read/write property that validate the data passed + and return it. + + Usage: + >>> d = Data(); d.content = 13; d.content + 13 + >>> d = Data(); d.content = (1,2); d.content + (1, 2) + >>> d = Data(); d.content = (1,2,3); d.content + (1, 2, 3) + >>> d = Data(); d.content = [1,2,3]; d.content + (1, 2, 3) + >>> d = Data(); d.content = [1.5,.2,3.3]; d.content + (1.5, 0.20000000000000001, 3.2999999999999998) + ''' + def fget(self): + ''' + Return the content of Data + ''' + return self.__content + + def fset(self, data): + ''' + Ensures that data is a valid tuple/list or a number (int, float + or long) + ''' + # Type: None + if data is None: + self.__content = None + return + + # Type: Int or Float + elif type(data) in NUMTYPES: + self.__content = data + + # Type: List or Tuple + elif type(data) in LISTTYPES: + # Ensures the correct size + if len(data) not in (2, 3): + raise TypeError, "Data (as list/tuple) must have 2 or 3 items" + return + + # Ensures that all items in list/tuple is a number + isnum = lambda x : type(x) not in NUMTYPES + + if max(map(isnum, data)): + # An item in data isn't an int or a float + raise TypeError, "All content of data must be a number (int or float)" + + # Convert the tuple to list + if type(data) is list: + data = tuple(data) + + # Append a copy and sets the type + self.__content = data[:] + + # Unknown type! + else: + self.__content = None + raise TypeError, "Data must be an int, float or a tuple with two or three items" + return + + return property(**locals()) + + + def clear(self): + ''' + Clear the all Data (content, name and parent) + ''' + self.content = None + self.name = None + self.parent = None + + def copy(self): + ''' + Returns a copy of the Data structure + ''' + # The copy + new_data = Data() + if self.content is not None: + # If content is a point + if type(self.content) is tuple: + new_data.__content = self.content[:] + + # If content is a number + else: + new_data.__content = self.content + + # If it has a name + if self.name is not None: + new_data.__name = self.name + + return new_data + + def __str__(self): + ''' + Return a string representation of the Data structure + ''' + if self.name is None: + if self.content is None: + return '' + return str(self.content) + else: + if self.content is None: + return self.name+": ()" + return self.name+": "+str(self.content) + + def __len__(self): + ''' + Return the length of the Data. + - If it's a number return 1; + - If it's a list return it's length; + - If its None return 0. + ''' + if self.content is None: + return 0 + elif type(self.content) in NUMTYPES: + return 1 + return len(self.content) + + + + +class Group(object): + ''' + Class that models a group of data. Every value (int, float, long, tuple + or list) passed is converted to a list of Data. + It can receive: + - A single number (int, float, long); + - A list of numbers; + - A tuple of numbers; + - An instance of Data; + - A list of Data; + + Obs: If a tuple with 2 or 3 items is passed it is converted to a point. + If a tuple with only 1 item is passed it's converted to a number; + If a tuple with more than 2 items is passed it's converted to a + list of numbers + ''' + def __init__(self, group=None, name=None, parent=None): + ''' + Starts main atributes in Group instance. + @data_list - a list of data which forms the group; + @range - a range that represent the x axis of possible functions; + @name - name of the data group; + @parent - the Serie parent of this group. + + Usage: + >>> g = Group(13, 'simple number'); print g + simple number ['13'] + >>> g = Group((1,2), 'simple point'); print g + simple point ['(1, 2)'] + >>> g = Group([1,2,3,4], 'list of numbers'); print g + list of numbers ['1', '2', '3', '4'] + >>> g = Group((1,2,3,4),'int in tuple'); print g + int in tuple ['1', '2', '3', '4'] + >>> g = Group([(1,2),(2,3),(3,4)], 'list of points'); print g + list of points ['(1, 2)', '(2, 3)', '(3, 4)'] + >>> g = Group([[1,2,3],[1,2,3]], '2D coordinate lists'); print g + 2D coordinated lists ['(1, 1)', '(2, 2)', '(3, 3)'] + >>> g = Group([[1,2],[1,2],[1,2]], '3D coordinate lists'); print g + 3D coordinated lists ['(1, 1, 1)', '(2, 2, 2)'] + ''' + # Initial values + self.__data_list = [] + self.__range = [] + self.__name = None + + + self.parent = parent + self.name = name + self.data_list = group + + # Name property + @apply + def name(): + doc = ''' + Name is a read/write property that controls the input of name. + - If passed an invalid value it cleans the name with None + + Usage: + >>> g = Group(13); g.name = 'name_test'; print g + name_test ['13'] + >>> g.name = 11; print g + ['13'] + >>> g.name = 'other_name'; print g + other_name ['13'] + >>> g.name = None; print g + ['13'] + >>> g.name = 'last_name'; print g + last_name ['13'] + >>> g.name = ''; print g + ['13'] + ''' + def fget(self): + ''' + Returns the name as a string + ''' + return self.__name + + def fset(self, name): + ''' + Sets the name of the Group + ''' + if type(name) in STRTYPES and len(name) > 0: + self.__name = name + else: + self.__name = None + + return property(**locals()) + + # data_list property + @apply + def data_list(): + doc = ''' + The data_list is a read/write property that can be a list of + numbers, a list of points or a list of 2 or 3 coordinate lists. This + property uses mainly the self.add_data method. + + Usage: + >>> g = Group(); g.data_list = 13; print g + ['13'] + >>> g.data_list = (1,2); print g + ['(1, 2)'] + >>> g.data_list = Data((1,2),'point a'); print g + ['point a: (1, 2)'] + >>> g.data_list = [1,2,3]; print g + ['1', '2', '3'] + >>> g.data_list = (1,2,3,4); print g + ['1', '2', '3', '4'] + >>> g.data_list = [(1,2),(2,3),(3,4)]; print g + ['(1, 2)', '(2, 3)', '(3, 4)'] + >>> g.data_list = [[1,2],[1,2]]; print g + ['(1, 1)', '(2, 2)'] + >>> g.data_list = [[1,2],[1,2],[1,2]]; print g + ['(1, 1, 1)', '(2, 2, 2)'] + >>> g.range = (10); g.data_list = lambda x:x**2; print g + ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)'] + ''' + def fget(self): + ''' + Returns the value of data_list + ''' + return self.__data_list + + def fset(self, group): + ''' + Ensures that group is valid. + ''' + # None + if group is None: + self.__data_list = [] + + # Int/float/long or Instance of Data + elif type(group) in NUMTYPES or isinstance(group, Data): + # Clean data_list + self.__data_list = [] + self.add_data(group) + + # One point + elif type(group) is tuple and len(group) in (2,3): + self.__data_list = [] + self.add_data(group) + + # list of items + elif type(group) in LISTTYPES and type(group[0]) is not list: + # Clean data_list + self.__data_list = [] + for item in group: + # try to append and catch an exception + self.add_data(item) + + # function lambda + elif callable(group): + # Explicit is better than implicit + function = group + # Has range + if len(self.range) is not 0: + # Clean data_list + self.__data_list = [] + # Generate values for the lambda function + for x in self.range: + #self.add_data((x,round(group(x),2))) + self.add_data((x,function(x))) + + # Only have range in parent + elif self.parent is not None and len(self.parent.range) is not 0: + # Copy parent range + self.__range = self.parent.range[:] + # Clean data_list + self.__data_list = [] + # Generate values for the lambda function + for x in self.range: + #self.add_data((x,round(group(x),2))) + self.add_data((x,function(x))) + + # Don't have range anywhere + else: + # x_data don't exist + raise Exception, "Data argument is valid but to use function type please set x_range first" + + # Coordinate Lists + elif type(group) in LISTTYPES and type(group[0]) is list: + # Clean data_list + self.__data_list = [] + data = [] + if len(group) == 3: + data = zip(group[0], group[1], group[2]) + elif len(group) == 2: + data = zip(group[0], group[1]) + else: + raise TypeError, "Only one list of coordinates was received." + + for item in data: + self.add_data(item) + + else: + raise TypeError, "Group type not supported" + + return property(**locals()) + + @apply + def range(): + doc = ''' + The range is a read/write property that generates a range of values + for the x axis of the functions. When passed a tuple it almost works + like the built-in range funtion: + - 1 item, represent the end of the range started from 0; + - 2 items, represents the start and the end, respectively; + - 3 items, the last one represents the step; + + When passed a list the range function understands as a valid range. + + Usage: + >>> g = Group(); g.range = 10; print g.range + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0] + >>> g = Group(); g.range = (5); print g.range + [0.0, 1.0, 2.0, 3.0, 4.0] + >>> g = Group(); g.range = (1,7); print g.range + [1.0, 2.0, 3.0, 4.0, 5.0, 6.0] + >>> g = Group(); g.range = (0,10,2); print g.range + [0.0, 2.0, 4.0, 6.0, 8.0] + >>> + >>> g = Group(); g.range = [0]; print g.range + [0.0] + >>> g = Group(); g.range = [0,10,20]; print g.range + [0.0, 10.0, 20.0] + ''' + def fget(self): + ''' + Returns the range + ''' + return self.__range + + def fset(self, x_range): + ''' + Controls the input of a valid type and generate the range + ''' + # if passed a simple number convert to tuple + if type(x_range) in NUMTYPES: + x_range = (x_range,) + + # A list, just convert to float + if type(x_range) is list and len(x_range) > 0: + # Convert all to float + x_range = map(float, x_range) + # Prevents repeated values and convert back to list + self.__range = list(set(x_range[:])) + # Sort the list to ascending order + self.__range.sort() + + # A tuple, must check the lengths and generate the values + elif type(x_range) is tuple and len(x_range) in (1,2,3): + # Convert all to float + x_range = map(float, x_range) + + # Inital values + start = 0.0 + step = 1.0 + end = 0.0 + + # Only the end and it can't be less or iqual to 0 + if len(x_range) is 1 and x_range > 0: + end = x_range[0] + + # The start and the end but the start must be less then the end + elif len(x_range) is 2 and x_range[0] < x_range[1]: + start = x_range[0] + end = x_range[1] + + # All 3, but the start must be less then the end + elif x_range[0] <= x_range[1]: + start = x_range[0] + end = x_range[1] + step = x_range[2] + + # Starts the range + self.__range = [] + # Generate the range + # Can't use the range function because it doesn't support float values + while start < end: + self.__range.append(start) + start += step + + # Incorrect type + else: + raise Exception, "x_range must be a list with one or more items or a tuple with 2 or 3 items" + + return property(**locals()) + + def add_data(self, data, name=None): + ''' + Append a new data to the data_list. + - If data is an instance of Data, append it + - If it's an int, float, tuple or list create an instance of Data and append it + + Usage: + >>> g = Group() + >>> g.add_data(12); print g + ['12'] + >>> g.add_data(7,'other'); print g + ['12', 'other: 7'] + >>> + >>> g = Group() + >>> g.add_data((1,1),'a'); print g + ['a: (1, 1)'] + >>> g.add_data((2,2),'b'); print g + ['a: (1, 1)', 'b: (2, 2)'] + >>> + >>> g.add_data(Data((1,2),'c')); print g + ['a: (1, 1)', 'b: (2, 2)', 'c: (1, 2)'] + ''' + if not isinstance(data, Data): + # Try to convert + data = Data(data,name,self) + + if data.content is not None: + self.__data_list.append(data.copy()) + self.__data_list[-1].parent = self + + + def to_list(self): + ''' + Returns the group as a list of numbers (int, float or long) or a + list of tuples (points 2D or 3D). + + Usage: + >>> g = Group([1,2,3,4],'g1'); g.to_list() + [1, 2, 3, 4] + >>> g = Group([(1,2),(2,3),(3,4)],'g2'); g.to_list() + [(1, 2), (2, 3), (3, 4)] + >>> g = Group([(1,2,3),(3,4,5)],'g2'); g.to_list() + [(1, 2, 3), (3, 4, 5)] + ''' + return [data.content for data in self] + + def copy(self): + ''' + Returns a copy of this group + ''' + new_group = Group() + new_group.__name = self.__name + if self.__range is not None: + new_group.__range = self.__range[:] + for data in self: + new_group.add_data(data.copy()) + return new_group + + def get_names(self): + ''' + Return a list with the names of all data in this group + ''' + names = [] + for data in self: + if data.name is None: + names.append('Data '+str(data.index()+1)) + else: + names.append(data.name) + return names + + + def __str__ (self): + ''' + Returns a string representing the Group + ''' + ret = "" + if self.name is not None: + ret += self.name + " " + if len(self) > 0: + list_str = [str(item) for item in self] + ret += str(list_str) + else: + ret += "[]" + return ret + + def __getitem__(self, key): + ''' + Makes a Group iterable, based in the data_list property + ''' + return self.data_list[key] + + def __len__(self): + ''' + Returns the length of the Group, based in the data_list property + ''' + return len(self.data_list) + + +class Colors(object): + ''' + Class that models the colors its labels (names) and its properties, RGB + and filling type. + + It can receive: + - A list where each item is a list with 3 or 4 items. The + first 3 items represent the RGB values and the last argument + defines the filling type. The list will be converted to a dict + and each color will receve a name based in its position in the + list. + - A dictionary where each key will be the color name and its item + can be a list with 3 or 4 items. The first 3 items represent + the RGB colors and the last argument defines the filling type. + ''' + def __init__(self, color_list=None): + ''' + Start the color_list property + @ color_list - the list or dict contaning the colors properties. + ''' + self.__color_list = None + + self.color_list = color_list + + @apply + def color_list(): + doc = ''' + >>> c = Colors([[1,1,1],[2,2,2,'linear'],[3,3,3,'gradient']]) + >>> print c.color_list + {'Color 2': [2, 2, 2, 'linear'], 'Color 3': [3, 3, 3, 'gradient'], 'Color 1': [1, 1, 1, 'solid']} + >>> c.color_list = [[1,1,1],(2,2,2,'solid'),(3,3,3,'linear')] + >>> print c.color_list + {'Color 2': [2, 2, 2, 'solid'], 'Color 3': [3, 3, 3, 'linear'], 'Color 1': [1, 1, 1, 'solid']} + >>> c.color_list = {'a':[1,1,1],'b':(2,2,2,'solid'),'c':(3,3,3,'linear'), 'd':(4,4,4)} + >>> print c.color_list + {'a': [1, 1, 1, 'solid'], 'c': [3, 3, 3, 'linear'], 'b': [2, 2, 2, 'solid'], 'd': [4, 4, 4, 'solid']} + ''' + def fget(self): + ''' + Return the color list + ''' + return self.__color_list + + def fset(self, color_list): + ''' + Format the color list to a dictionary + ''' + if color_list is None: + self.__color_list = None + return + + if type(color_list) in LISTTYPES and type(color_list[0]) in LISTTYPES: + old_color_list = color_list[:] + color_list = {} + for index, color in enumerate(old_color_list): + if len(color) is 3 and max(map(type, color)) in NUMTYPES: + color_list['Color '+str(index+1)] = list(color)+[DEFAULT_COLOR_FILLING] + elif len(color) is 4 and max(map(type, color[:-1])) in NUMTYPES and color[-1] in FILLING_TYPES: + color_list['Color '+str(index+1)] = list(color) + else: + raise TypeError, "Unsuported color format" + elif type(color_list) is not dict: + raise TypeError, "Unsuported color format" + + for name, color in color_list.items(): + if len(color) is 3: + if max(map(type, color)) in NUMTYPES: + color_list[name] = list(color)+[DEFAULT_COLOR_FILLING] + else: + raise TypeError, "Unsuported color format" + elif len(color) is 4: + if max(map(type, color[:-1])) in NUMTYPES and color[-1] in FILLING_TYPES: + color_list[name] = list(color) + else: + raise TypeError, "Unsuported color format" + self.__color_list = color_list.copy() + + return property(**locals()) + + +class Series(object): + ''' + Class that models a Series (group of groups). Every value (int, float, + long, tuple or list) passed is converted to a list of Group or Data. + It can receive: + - a single number or point, will be converted to a Group of one Data; + - a list of numbers, will be converted to a group of numbers; + - a list of tuples, will converted to a single Group of points; + - a list of lists of numbers, each 'sublist' will be converted to a + group of numbers; + - a list of lists of tuples, each 'sublist' will be converted to a + group of points; + - a list of lists of lists, the content of the 'sublist' will be + processed as coordinated lists and the result will be converted to + a group of points; + - a Dictionary where each item can be the same of the list: number, + point, list of numbers, list of points or list of lists (coordinated + lists); + - an instance of Data; + - an instance of group. + ''' + def __init__(self, series=None, name=None, property=[], colors=None): + ''' + Starts main atributes in Group instance. + @series - a list, dict of data of which the series is composed; + @name - name of the series; + @property - a list/dict of properties to be used in the plots of + this Series + + Usage: + >>> print Series([1,2,3,4]) + ["Group 1 ['1', '2', '3', '4']"] + >>> print Series([[1,2,3],[4,5,6]]) + ["Group 1 ['1', '2', '3']", "Group 2 ['4', '5', '6']"] + >>> print Series((1,2)) + ["Group 1 ['(1, 2)']"] + >>> print Series([(1,2),(2,3)]) + ["Group 1 ['(1, 2)', '(2, 3)']"] + >>> print Series([[(1,2),(2,3)],[(4,5),(5,6)]]) + ["Group 1 ['(1, 2)', '(2, 3)']", "Group 2 ['(4, 5)', '(5, 6)']"] + >>> print Series([[[1,2,3],[1,2,3],[1,2,3]]]) + ["Group 1 ['(1, 1, 1)', '(2, 2, 2)', '(3, 3, 3)']"] + >>> print Series({'g1':[1,2,3], 'g2':[4,5,6]}) + ["g1 ['1', '2', '3']", "g2 ['4', '5', '6']"] + >>> print Series({'g1':[(1,2),(2,3)], 'g2':[(4,5),(5,6)]}) + ["g1 ['(1, 2)', '(2, 3)']", "g2 ['(4, 5)', '(5, 6)']"] + >>> print Series({'g1':[[1,2],[1,2]], 'g2':[[4,5],[4,5]]}) + ["g1 ['(1, 1)', '(2, 2)']", "g2 ['(4, 4)', '(5, 5)']"] + >>> print Series(Data(1,'d1')) + ["Group 1 ['d1: 1']"] + >>> print Series(Group([(1,2),(2,3)],'g1')) + ["g1 ['(1, 2)', '(2, 3)']"] + ''' + # Intial values + self.__group_list = [] + self.__name = None + self.__range = None + + # TODO: Implement colors with filling + self.__colors = None + + self.name = name + self.group_list = series + self.colors = colors + + # Name property + @apply + def name(): + doc = ''' + Name is a read/write property that controls the input of name. + - If passed an invalid value it cleans the name with None + + Usage: + >>> s = Series(13); s.name = 'name_test'; print s + name_test ["Group 1 ['13']"] + >>> s.name = 11; print s + ["Group 1 ['13']"] + >>> s.name = 'other_name'; print s + other_name ["Group 1 ['13']"] + >>> s.name = None; print s + ["Group 1 ['13']"] + >>> s.name = 'last_name'; print s + last_name ["Group 1 ['13']"] + >>> s.name = ''; print s + ["Group 1 ['13']"] + ''' + def fget(self): + ''' + Returns the name as a string + ''' + return self.__name + + def fset(self, name): + ''' + Sets the name of the Group + ''' + if type(name) in STRTYPES and len(name) > 0: + self.__name = name + else: + self.__name = None + + return property(**locals()) + + + + # Colors property + @apply + def colors(): + doc = ''' + >>> s = Series() + >>> s.colors = [[1,1,1],[2,2,2,'linear'],[3,3,3,'gradient']] + >>> print s.colors + {'Color 2': [2, 2, 2, 'linear'], 'Color 3': [3, 3, 3, 'gradient'], 'Color 1': [1, 1, 1, 'solid']} + >>> s.colors = [[1,1,1],(2,2,2,'solid'),(3,3,3,'linear')] + >>> print s.colors + {'Color 2': [2, 2, 2, 'solid'], 'Color 3': [3, 3, 3, 'linear'], 'Color 1': [1, 1, 1, 'solid']} + >>> s.colors = {'a':[1,1,1],'b':(2,2,2,'solid'),'c':(3,3,3,'linear'), 'd':(4,4,4)} + >>> print s.colors + {'a': [1, 1, 1, 'solid'], 'c': [3, 3, 3, 'linear'], 'b': [2, 2, 2, 'solid'], 'd': [4, 4, 4, 'solid']} + ''' + def fget(self): + ''' + Return the color list + ''' + return self.__colors.color_list + + def fset(self, colors): + ''' + Format the color list to a dictionary + ''' + self.__colors = Colors(colors) + + return property(**locals()) + + @apply + def range(): + doc = ''' + The range is a read/write property that generates a range of values + for the x axis of the functions. When passed a tuple it almost works + like the built-in range funtion: + - 1 item, represent the end of the range started from 0; + - 2 items, represents the start and the end, respectively; + - 3 items, the last one represents the step; + + When passed a list the range function understands as a valid range. + + Usage: + >>> s = Series(); s.range = 10; print s.range + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0] + >>> s = Series(); s.range = (5); print s.range + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0] + >>> s = Series(); s.range = (1,7); print s.range + [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0] + >>> s = Series(); s.range = (0,10,2); print s.range + [0.0, 2.0, 4.0, 6.0, 8.0, 10.0] + >>> + >>> s = Series(); s.range = [0]; print s.range + [0.0] + >>> s = Series(); s.range = [0,10,20]; print s.range + [0.0, 10.0, 20.0] + ''' + def fget(self): + ''' + Returns the range + ''' + return self.__range + + def fset(self, x_range): + ''' + Controls the input of a valid type and generate the range + ''' + # if passed a simple number convert to tuple + if type(x_range) in NUMTYPES: + x_range = (x_range,) + + # A list, just convert to float + if type(x_range) is list and len(x_range) > 0: + # Convert all to float + x_range = map(float, x_range) + # Prevents repeated values and convert back to list + self.__range = list(set(x_range[:])) + # Sort the list to ascending order + self.__range.sort() + + # A tuple, must check the lengths and generate the values + elif type(x_range) is tuple and len(x_range) in (1,2,3): + # Convert all to float + x_range = map(float, x_range) + + # Inital values + start = 0.0 + step = 1.0 + end = 0.0 + + # Only the end and it can't be less or iqual to 0 + if len(x_range) is 1 and x_range > 0: + end = x_range[0] + + # The start and the end but the start must be lesser then the end + elif len(x_range) is 2 and x_range[0] < x_range[1]: + start = x_range[0] + end = x_range[1] + + # All 3, but the start must be lesser then the end + elif x_range[0] < x_range[1]: + start = x_range[0] + end = x_range[1] + step = x_range[2] + + # Starts the range + self.__range = [] + # Generate the range + # Cnat use the range function becouse it don't suport float values + while start <= end: + self.__range.append(start) + start += step + + # Incorrect type + else: + raise Exception, "x_range must be a list with one or more item or a tuple with 2 or 3 items" + + return property(**locals()) + + @apply + def group_list(): + doc = ''' + The group_list is a read/write property used to pre-process the list + of Groups. + It can be: + - a single number, point or lambda, will be converted to a single + Group of one Data; + - a list of numbers, will be converted to a group of numbers; + - a list of tuples, will converted to a single Group of points; + - a list of lists of numbers, each 'sublist' will be converted to + a group of numbers; + - a list of lists of tuples, each 'sublist' will be converted to a + group of points; + - a list of lists of lists, the content of the 'sublist' will be + processed as coordinated lists and the result will be converted + to a group of points; + - a list of lambdas, each lambda represents a Group; + - a Dictionary where each item can be the same of the list: number, + point, list of numbers, list of points, list of lists + (coordinated lists) or lambdas + - an instance of Data; + - an instance of group. + + Usage: + >>> s = Series() + >>> s.group_list = [1,2,3,4]; print s + ["Group 1 ['1', '2', '3', '4']"] + >>> s.group_list = [[1,2,3],[4,5,6]]; print s + ["Group 1 ['1', '2', '3']", "Group 2 ['4', '5', '6']"] + >>> s.group_list = (1,2); print s + ["Group 1 ['(1, 2)']"] + >>> s.group_list = [(1,2),(2,3)]; print s + ["Group 1 ['(1, 2)', '(2, 3)']"] + >>> s.group_list = [[(1,2),(2,3)],[(4,5),(5,6)]]; print s + ["Group 1 ['(1, 2)', '(2, 3)']", "Group 2 ['(4, 5)', '(5, 6)']"] + >>> s.group_list = [[[1,2,3],[1,2,3],[1,2,3]]]; print s + ["Group 1 ['(1, 1, 1)', '(2, 2, 2)', '(3, 3, 3)']"] + >>> s.group_list = [(0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,9)]; print s + ["Group 1 ['(0.5, 5.5)']", "Group 2 ['(0, 4)', '(6, 8)']", "Group 3 ['(5.5, 7)']", "Group 4 ['(7, 9)']"] + >>> s.group_list = {'g1':[1,2,3], 'g2':[4,5,6]}; print s + ["g1 ['1', '2', '3']", "g2 ['4', '5', '6']"] + >>> s.group_list = {'g1':[(1,2),(2,3)], 'g2':[(4,5),(5,6)]}; print s + ["g1 ['(1, 2)', '(2, 3)']", "g2 ['(4, 5)', '(5, 6)']"] + >>> s.group_list = {'g1':[[1,2],[1,2]], 'g2':[[4,5],[4,5]]}; print s + ["g1 ['(1, 1)', '(2, 2)']", "g2 ['(4, 4)', '(5, 5)']"] + >>> s.range = 10 + >>> s.group_list = lambda x:x*2 + >>> s.group_list = [lambda x:x*2, lambda x:x**2, lambda x:x**3]; print s + ["Group 1 ['(0.0, 0.0)', '(1.0, 2.0)', '(2.0, 4.0)', '(3.0, 6.0)', '(4.0, 8.0)', '(5.0, 10.0)', '(6.0, 12.0)', '(7.0, 14.0)', '(8.0, 16.0)', '(9.0, 18.0)', '(10.0, 20.0)']", "Group 2 ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)', '(10.0, 100.0)']", "Group 3 ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 8.0)', '(3.0, 27.0)', '(4.0, 64.0)', '(5.0, 125.0)', '(6.0, 216.0)', '(7.0, 343.0)', '(8.0, 512.0)', '(9.0, 729.0)', '(10.0, 1000.0)']"] + >>> s.group_list = {'linear':lambda x:x*2, 'square':lambda x:x**2, 'cubic':lambda x:x**3}; print s + ["cubic ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 8.0)', '(3.0, 27.0)', '(4.0, 64.0)', '(5.0, 125.0)', '(6.0, 216.0)', '(7.0, 343.0)', '(8.0, 512.0)', '(9.0, 729.0)', '(10.0, 1000.0)']", "linear ['(0.0, 0.0)', '(1.0, 2.0)', '(2.0, 4.0)', '(3.0, 6.0)', '(4.0, 8.0)', '(5.0, 10.0)', '(6.0, 12.0)', '(7.0, 14.0)', '(8.0, 16.0)', '(9.0, 18.0)', '(10.0, 20.0)']", "square ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)', '(10.0, 100.0)']"] + >>> s.group_list = Data(1,'d1'); print s + ["Group 1 ['d1: 1']"] + >>> s.group_list = Group([(1,2),(2,3)],'g1'); print s + ["g1 ['(1, 2)', '(2, 3)']"] + ''' + def fget(self): + ''' + Return the group list. + ''' + return self.__group_list + + def fset(self, series): + ''' + Controls the input of a valid group list. + ''' + #TODO: Add support to the following strem of data: [ (0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,9)] + + # Type: None + if series is None: + self.__group_list = [] + + # List or Tuple + elif type(series) in LISTTYPES: + self.__group_list = [] + + is_function = lambda x: callable(x) + # Groups + if list in map(type, series) or max(map(is_function, series)): + for group in series: + self.add_group(group) + + # single group + else: + self.add_group(series) + + #old code + ## List of numbers + #if type(series[0]) in NUMTYPES or type(series[0]) is tuple: + # print series + # self.add_group(series) + # + ## List of anything else + #else: + # for group in series: + # self.add_group(group) + + # Dict representing series of groups + elif type(series) is dict: + self.__group_list = [] + names = series.keys() + names.sort() + for name in names: + self.add_group(Group(series[name],name,self)) + + # A single lambda + elif callable(series): + self.__group_list = [] + self.add_group(series) + + # Int/float, instance of Group or Data + elif type(series) in NUMTYPES or isinstance(series, Group) or isinstance(series, Data): + self.__group_list = [] + self.add_group(series) + + # Default + else: + raise TypeError, "Serie type not supported" + + return property(**locals()) + + def add_group(self, group, name=None): + ''' + Append a new group in group_list + ''' + if not isinstance(group, Group): + #Try to convert + group = Group(group, name, self) + + if len(group.data_list) is not 0: + # Auto naming groups + if group.name is None: + group.name = "Group "+str(len(self.__group_list)+1) + + self.__group_list.append(group) + self.__group_list[-1].parent = self + + def copy(self): + ''' + Returns a copy of the Series + ''' + new_series = Series() + new_series.__name = self.__name + if self.__range is not None: + new_series.__range = self.__range[:] + #Add color property in the copy method + #self.__colors = None + + for group in self: + new_series.add_group(group.copy()) + + return new_series + + def get_names(self): + ''' + Returns a list of the names of all groups in the Serie + ''' + names = [] + for group in self: + if group.name is None: + names.append('Group '+str(group.index()+1)) + else: + names.append(group.name) + + return names + + def to_list(self): + ''' + Returns a list with the content of all groups and data + ''' + big_list = [] + for group in self: + for data in group: + if type(data.content) in NUMTYPES: + big_list.append(data.content) + else: + big_list = big_list + list(data.content) + return big_list + + def __getitem__(self, key): + ''' + Makes the Series iterable, based in the group_list property + ''' + return self.__group_list[key] + + def __str__(self): + ''' + Returns a string that represents the Series + ''' + ret = "" + if self.name is not None: + ret += self.name + " " + if len(self) > 0: + list_str = [str(item) for item in self] + ret += str(list_str) + else: + ret += "[]" + return ret + + def __len__(self): + ''' + Returns the length of the Series, based in the group_lsit property + ''' + return len(self.group_list) + + +if __name__ == '__main__': + doctest.testmod() diff --git a/bindings/python/examples/sched_switch.py b/bindings/python/examples/sched_switch.py new file mode 100644 index 0000000..f252ab5 --- /dev/null +++ b/bindings/python/examples/sched_switch.py @@ -0,0 +1,110 @@ +# The script takes one optional argument (pid) +# The script will read events based on pid and +# print the scheduler switches happening with the process. +# If no arguments are passed, it displays all the scheduler switches. +# This can be used to understand which tasks schedule out the current +# process being traced, and when it gets scheduled in again. +# The trace needs PID context (lttng add-context -k -t pid) + +import sys +from babeltrace import * + +if len(sys.argv) < 2 or len(sys.argv) > 3: + raise TypeError("Usage: python sched_switch.py [pid] path/to/trace") +elif len(sys.argv) == 3: + usePID = True +else: + usePID = False + + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +while event is not None: + while True: + if event.get_name() == "sched_switch": + # Getting scope definition + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) + if sco is None: + print("ERROR: Cannot get definition scope for sched_switch") + break # Next event + + # Getting PID + pid_field = event.get_field(sco, "_pid") + pid = pid_field.get_int64() + + if ctf.field_error(): + print("ERROR: Missing PID info for sched_switch") + break # Next event + + if usePID and (pid != long(sys.argv[1])): + break # Next event + + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + + # prev_comm + field = event.get_field(sco, "_prev_comm") + prev_comm = field.get_char_array() + if ctf.field_error(): + print("ERROR: Missing prev_comm context info") + + # prev_tid + field = event.get_field(sco, "_prev_tid") + prev_tid = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing prev_tid context info") + + # prev_prio + field = event.get_field(sco, "_prev_prio") + prev_prio = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing prev_prio context info") + + # prev_state + field = event.get_field(sco, "_prev_state") + prev_state = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing prev_state context info") + + # next_comm + field = event.get_field(sco, "_next_comm") + next_comm = field.get_char_array() + if ctf.field_error(): + print("ERROR: Missing next_comm context info") + + # next_tid + field = event.get_field(sco, "_next_tid") + next_tid = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing next_tid context info") + + # next_prio + field = event.get_field(sco, "_next_prio") + next_prio = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing next_prio context info") + + # Output + print("sched_switch, pid = {}, TS = {}, prev_comm = {},\n\t" + "prev_tid = {}, prev_prio = {}, prev_state = {},\n\t" + "next_comm = {}, next_tid = {}, next_prio = {}".format( + pid, event.get_timestamp(), prev_comm, prev_tid, + prev_prio, prev_state, next_comm, next_tid, next_prio)) + + break # Next event + + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it diff --git a/bindings/python/examples/softirqtimes.py b/bindings/python/examples/softirqtimes.py new file mode 100644 index 0000000..19b2deb --- /dev/null +++ b/bindings/python/examples/softirqtimes.py @@ -0,0 +1,130 @@ +# The script checks the trace for the amount of time +# spent from each softirq_raise to softirq_exit. +# It prints out the min, max (with timestamp), +# average times, the standard deviation and the total count. +# Using the cairoplot module, a .svg graph is also outputted +# showing the taken time in function of the time since the +# beginning of the trace. + +import sys, math +from output_format_modules import cairoplot +from babeltrace import * + +if len(sys.argv) < 2: + raise TypeError("Usage: python softirqtimes.py path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +time_taken = [] +graph_data = [] +max_time = (0.0, 0.0) # (val, ts) + +# tmp template: {(cpu_id, vec):TS raise} +tmp = {} +largest_val = 0 + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +start_time = event.get_timestamp() +while(event is not None): + + event_name = event.get_name() + error = True + appendNext = False + + if event_name == 'softirq_raise' or event_name == 'softirq_exit': + # Recover cpu_id and vec values to make a key to tmp + error = False + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) + field = event.get_field(scope, "cpu_id") + cpu_id = field.get_uint64() + if ctf.field_error(): + print("ERROR: Missing cpu_id info for {}".format( + event.get_name())) + error = True + + scope = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + field = event.get_field(scope, "_vec") + vec = field.get_uint64() + if ctf.field_error(): + print("ERROR: Missing vec info for {}".format( + event.get_name())) + error = True + key = (cpu_id, vec) + + if event_name == 'softirq_raise' and not error: + # Add timestamp to tmp + if key in tmp: + # If key already exists + i = 0 + while True: + # Add index + key = (cpu_id, vec, i) + if key in tmp: + i += 1 + continue + if i > largest_val: + largest_val = i + break + + tmp[key] = event.get_timestamp() + + if event_name == 'softirq_exit' and not error: + # Saving data for output + # Key check + if not (key in tmp): + i = 0 + while i <= largest_val: + key = (key[0], key[1], i) + if key in tmp: + break + i += 1 + + raise_timestamp = tmp[key] + time_data = event.get_timestamp() - tmp.pop(key) + if time_data > max_time[0]: + # max_time = (val, ts) + max_time = (time_data, raise_timestamp) + time_taken.append(time_data) + graph_data.append((raise_timestamp - start_time, time_data)) + + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + + +del ctf_it + +# Standard dev. calc. +try: + mean = sum(time_taken)/float(len(time_taken)) +except ZeroDivisionError: + raise TypeError("empty data") +deviations_squared = [] +for x in time_taken: + deviations_squared.append(math.pow((x - mean), 2)) +try: + stddev = math.sqrt(sum(deviations_squared) / (len(deviations_squared) - 1)) +except ZeroDivisionError: + stddev = '-' + +# Terminal output +print("AVG TIME: {} ns".format(mean)) +print("MIN TIME: {} ns".format(min(time_taken))) +print("MAX TIME: {} ns, TS: {}".format(max_time[0], max_time[1])) +print("STD DEV: {}".format(stddev)) +print("TOTAL COUNT: {}".format(len(time_taken))) + +# Graph output +cairoplot.scatter_plot ( 'softirqtimes.svg', data = graph_data, + width = 5000, height = 4000, border = 20, axis = True, + grid = True, series_colors = ["red"] ) diff --git a/bindings/python/examples/syscalls_by_pid.py b/bindings/python/examples/syscalls_by_pid.py new file mode 100644 index 0000000..f6127ed --- /dev/null +++ b/bindings/python/examples/syscalls_by_pid.py @@ -0,0 +1,61 @@ +# The script checks all syscall in the trace and prints a list +# showing the number of systemcalls executed by each PID +# ordered from greatest to least number of syscalls. +# The trace needs PID context (lttng add-context -k -t pid) + +import sys +from babeltrace import * +from output_format_modules.pprint_table import pprint_table as pprint + +if len(sys.argv) < 2 : + raise TypeError("Usage: python syscalls_by_pid.py path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +data = {} + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +while event is not None: + if event.get_name().find("sys") >= 0: + # Getting scope definition + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) + if sco is None: + print("ERROR: Cannot get definition scope for {}".format( + event.get_name())) + else: + # Getting PID + pid_field = event.get_field(sco, "_pid") + pid = pid_field.get_int64() + + if ctf.field_error(): + print("ERROR: Missing PID info for sched_switch".format( + event.get_name())) + elif pid in data: + data[pid] += 1 + else: + data[pid] = 1 + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it + +# Setting table for output +table = [] +for item in data: + table.append([data[item], item]) # [count, pid] +table.sort(reverse = True) # [big count first, pid] +for i in range(len(table)): + table[i].reverse() # [pid, big count first] +table.insert(0, ["PID", "SYSCALL COUNT"]) +pprint(table) diff --git a/bindings/python/python-complements.c b/bindings/python/python-complements.c new file mode 100644 index 0000000..8c4d811 --- /dev/null +++ b/bindings/python/python-complements.c @@ -0,0 +1,105 @@ +/* python-complements.c + Needed functions for python binding +*/ + +#include "python-complements.h" + +/* FILE functions + ---------------------------------------------------- +*/ + +FILE *_bt_file_open(char *file_path, char *mode) +{ + FILE *fp = stdout; + if (file_path != NULL) + fp = fopen(file_path, mode); + return fp; +} + +void _bt_file_close(FILE *fp) +{ + if (fp != NULL) + fclose(fp); +} + + +/* List-related functions + ---------------------------------------------------- +*/ + +/* ctf-field-list */ +struct definition **_bt_python_field_listcaller( + const struct bt_ctf_event *ctf_event, + const struct definition *scope) +{ + struct definition **list; + unsigned int count; + int ret; + + ret = bt_ctf_get_field_list(ctf_event, scope, + (const struct definition * const **)&list, &count); + + if (ret < 0) /* For python to know an error occured */ + list = NULL; + else /* For python to know the end is reached */ + list[count] = NULL; + + return list; +} + +struct definition *_bt_python_field_one_from_list( + struct definition **list, int index) +{ + return list[index]; +} + +/* event_decl_list */ +struct bt_ctf_event_decl **_bt_python_event_decl_listcaller( + int handle_id, struct bt_context *ctx) +{ + struct bt_ctf_event_decl **list; + unsigned int count; + int ret; + + ret = bt_ctf_get_event_decl_list(handle_id, ctx, + (struct bt_ctf_event_decl * const **)&list, &count); + + if (ret < 0) /* For python to know an error occured */ + list = NULL; + else /* For python to know the end is reached */ + list[count] = NULL; + + return list; +} + +struct bt_ctf_event_decl *_bt_python_decl_one_from_list( + struct bt_ctf_event_decl **list, int index) +{ + return list[index]; +} + +/* decl_fields */ +struct bt_ctf_field_decl **_by_python_field_decl_listcaller( + struct bt_ctf_event_decl *event_decl, + enum bt_ctf_scope scope) +{ + struct bt_ctf_field_decl **list; + unsigned int count; + int ret; + + ret = bt_ctf_get_decl_fields(event_decl, scope, + (const struct bt_ctf_field_decl * const **)&list, &count); + + if (ret < 0) /* For python to know an error occured */ + list = NULL; + else /* For python to know the end is reached */ + list[count] = NULL; + + return list; +} + +struct bt_ctf_field_decl *_bt_python_field_decl_one_from_list( + struct bt_ctf_field_decl **list, int index) +{ + return list[index]; +} diff --git a/bindings/python/python-complements.h b/bindings/python/python-complements.h new file mode 100644 index 0000000..cdd5528 --- /dev/null +++ b/bindings/python/python-complements.h @@ -0,0 +1,36 @@ +/* python-complements.h + Needed functions for python binding +*/ + +#include +#include +#include +#include +#include +#include +#include +#include + +/* File */ +FILE *_bt_file_open(char *file_path, char *mode); +void _bt_file_close(FILE *fp); + +/* ctf-field-list */ +struct definition **_bt_python_field_listcaller( + const struct bt_ctf_event *ctf_event, + const struct definition *scope); +struct definition *_bt_python_field_one_from_list( + struct definition **list, int index); + +/* event_decl_list */ +struct bt_ctf_event_decl **_bt_python_event_decl_listcaller( + int handle_id, struct bt_context *ctx); +struct bt_ctf_event_decl *_bt_python_decl_one_from_list( + struct bt_ctf_event_decl **list, int index); + +/* decl_fields */ +struct bt_ctf_field_decl **_by_python_field_decl_listcaller( + struct bt_ctf_event_decl *event_decl, + enum bt_ctf_scope scope); +struct bt_ctf_field_decl *_bt_python_field_decl_one_from_list( + struct bt_ctf_field_decl **list, int index); diff --git a/bootstrap b/bootstrap index c507425..f6926ca 100755 --- a/bootstrap +++ b/bootstrap @@ -4,7 +4,7 @@ set -x if [ ! -e config ]; then mkdir config fi -aclocal +aclocal -I m4 libtoolize --force --copy autoheader automake --add-missing --copy diff --git a/configure.ac b/configure.ac index d90479d..f9cff9d 100644 --- a/configure.ac +++ b/configure.ac @@ -74,6 +74,41 @@ AC_CHECK_LIB([popt], [poptGetContext], [], [AC_MSG_ERROR([Cannot find popt.])] ) + +# For Python +# SWIG version needed or newer: +swig_version=2.0.0 + +AC_ARG_ENABLE([python], + [AC_HELP_STRING([--disable-python], + [do not compile Python bindings])], + [], [enable_python=yes]) + +AM_CONDITIONAL([USE_PYTHON], [test "x${enable_python:-yes}" = xyes]) + +if test "x${enable_python:-yes}" = xyes; then + AC_MSG_NOTICE([You may configure with --disable-python ]dnl +[if you do not want Python bindings.]) + + AX_PKG_SWIG($swig_version, [], [ AC_MSG_ERROR([SWIG $swig_version or newer is needed]) ]) + AM_PATH_PYTHON + + AC_ARG_VAR([PYTHON_INCLUDE], [Include flags for python, bypassing python-config]) + AC_ARG_VAR([PYTHON_CONFIG], [Path to python-config]) + AS_IF([test -z "$PYTHON_INCLUDE"], [ + AS_IF([test -z "$PYTHON_CONFIG"], [ + AC_PATH_PROGS([PYTHON_CONFIG], + [python$PYTHON_VERSION-config python-config], + [no], + [`dirname $PYTHON`]) + AS_IF([test "$PYTHON_CONFIG" = no], [AC_MSG_ERROR([cannot find python-config for $PYTHON.])]) + ]) + AC_MSG_CHECKING([python include flags]) + PYTHON_INCLUDE=`$PYTHON_CONFIG --includes` + AC_MSG_RESULT([$PYTHON_INCLUDE]) + ]) +fi + pkg_modules="gmodule-2.0 >= 2.0.0" PKG_CHECK_MODULES(GMODULE, [$pkg_modules]) AC_SUBST(PACKAGE_LIBS) @@ -103,6 +138,8 @@ AC_CONFIG_FILES([ lib/Makefile lib/prio_heap/Makefile include/Makefile + bindings/Makefile + bindings/python/Makefile tests/Makefile ]) AC_OUTPUT diff --git a/doc/python-howto.txt b/doc/python-howto.txt new file mode 100644 index 0000000..e2ed751 --- /dev/null +++ b/doc/python-howto.txt @@ -0,0 +1,70 @@ +PYTHON BINDINGS +---------------- + +This is a brief howto for using the Babeltrace Python module. + + +INSTALLATION: + +By default, the Python bindings are installed. +If you do not wish the Python bindings, you can configure with the +--disable-python option during the installation procedure: + + $ ./configure --disable-python + +The Python module is automatically generated using SWIG, therefore the +swig2.0 package on Debian/Ubuntu is requied. + + +USAGE: + +Once installed, the Python module can be used by importing it in Python. +In the Python interpreter: + + >>> import babeltrace + +Then the starting point is to create a context and add a trace to it. + + >>> ctx = babeltrace.Context() + >>> ctx.add_trace("path/to/trace", ) + +Where is a string containing the format name in which the trace +was produced. To print a list of available formats to the standard +output, it is possible to use the print_format_list function. + + >>> out = babeltrace.File(None) # This returns stdout + >>> babeltrace.print_format_list(out) + +When a trace is added to a context, it is opened and ready to read using +an iterator. While creating an iterator, optional starting and ending +position may be specified. So far, only ctf iterator are supported. + + >>> begin_pos = babeltrace.IterPos(babeltrace.SEEK_BEGIN) + >>> iterator = babeltrace.ctf.Iterator(ctx, begin_pos) + +From there, it is possible to read the events. + + >>> event = iterator.read_event() + +It is simple to obtain the timestamp of that event. + + >>> timestamp = event.get_timestamp() + +Let's say that we want to extract the prev_comm context info for a +sched_switch event. To do so, it is needed to set an event scope +with which we can obtain the field wanted. + + >>> if event.get_name == "sched_switch": + ... #prev_comm only for sched_switch events + ... scope = event.get_top_level_scope(babeltrace.ctf.scope.EVENT_FIELDS) + ... field = event.get_field(scope, "_prev_comm") + ... prev_comm = field.get_char_array() + +It is also possible to move on to the next event. + + >>> ret = iterator.next() # Move the iterator + >>> if ret == 0: # No error occured + ... event = iterator.read_event() # Read the next event + +For many usage script examples of the Babeltrace Python module, see the +bindings/python/examples directory. diff --git a/m4/ax_pkg_swig.m4 b/m4/ax_pkg_swig.m4 new file mode 100644 index 0000000..e112f3d --- /dev/null +++ b/m4/ax_pkg_swig.m4 @@ -0,0 +1,135 @@ +# =========================================================================== +# http://www.gnu.org/software/autoconf-archive/ax_pkg_swig.html +# =========================================================================== +# +# SYNOPSIS +# +# AX_PKG_SWIG([major.minor.micro], [action-if-found], [action-if-not-found]) +# +# DESCRIPTION +# +# This macro searches for a SWIG installation on your system. If found, +# then SWIG is AC_SUBST'd; if not found, then $SWIG is empty. If SWIG is +# found, then SWIG_LIB is set to the SWIG library path, and AC_SUBST'd. +# +# You can use the optional first argument to check if the version of the +# available SWIG is greater than or equal to the value of the argument. It +# should have the format: N[.N[.N]] (N is a number between 0 and 999. Only +# the first N is mandatory.) If the version argument is given (e.g. +# 1.3.17), AX_PKG_SWIG checks that the swig package is this version number +# or higher. +# +# As usual, action-if-found is executed if SWIG is found, otherwise +# action-if-not-found is executed. +# +# In configure.in, use as: +# +# AX_PKG_SWIG(1.3.17, [], [ AC_MSG_ERROR([SWIG is required to build..]) ]) +# AX_SWIG_ENABLE_CXX +# AX_SWIG_MULTI_MODULE_SUPPORT +# AX_SWIG_PYTHON +# +# LICENSE +# +# Copyright (c) 2008 Sebastian Huber +# Copyright (c) 2008 Alan W. Irwin +# Copyright (c) 2008 Rafael Laboissiere +# Copyright (c) 2008 Andrew Collier +# Copyright (c) 2011 Murray Cumming +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation; either version 2 of the License, or (at your +# option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program. If not, see . +# +# As a special exception, the respective Autoconf Macro's copyright owner +# gives unlimited permission to copy, distribute and modify the configure +# scripts that are the output of Autoconf when processing the Macro. You +# need not follow the terms of the GNU General Public License when using +# or distributing such scripts, even though portions of the text of the +# Macro appear in them. The GNU General Public License (GPL) does govern +# all other use of the material that constitutes the Autoconf Macro. +# +# This special exception to the GPL applies to versions of the Autoconf +# Macro released by the Autoconf Archive. When you make and distribute a +# modified version of the Autoconf Macro, you may extend this special +# exception to the GPL to apply to your modified version as well. + +#serial 8 + +AC_DEFUN([AX_PKG_SWIG],[ + # Ubuntu has swig 2.0 as /usr/bin/swig2.0 + AC_PATH_PROGS([SWIG],[swig swig2.0]) + if test -z "$SWIG" ; then + m4_ifval([$3],[$3],[:]) + elif test -n "$1" ; then + AC_MSG_CHECKING([SWIG version]) + [swig_version=`$SWIG -version 2>&1 | grep 'SWIG Version' | sed 's/.*\([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\).*/\1/g'`] + AC_MSG_RESULT([$swig_version]) + if test -n "$swig_version" ; then + # Calculate the required version number components + [required=$1] + [required_major=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_major" ; then + [required_major=0] + fi + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] + [required_minor=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_minor" ; then + [required_minor=0] + fi + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] + [required_patch=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_patch" ; then + [required_patch=0] + fi + # Calculate the available version number components + [available=$swig_version] + [available_major=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_major" ; then + [available_major=0] + fi + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] + [available_minor=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_minor" ; then + [available_minor=0] + fi + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] + [available_patch=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_patch" ; then + [available_patch=0] + fi + # Convert the version tuple into a single number for easier comparison. + # Using base 100 should be safe since SWIG internally uses BCD values + # to encode its version number. + required_swig_vernum=`expr $required_major \* 10000 \ + \+ $required_minor \* 100 \+ $required_patch` + available_swig_vernum=`expr $available_major \* 10000 \ + \+ $available_minor \* 100 \+ $available_patch` + + if test $available_swig_vernum -lt $required_swig_vernum; then + AC_MSG_WARN([SWIG version >= $1 is required. You have $swig_version.]) + SWIG='' + m4_ifval([$3],[$3],[]) + else + AC_MSG_CHECKING([for SWIG library]) + SWIG_LIB=`$SWIG -swiglib` + AC_MSG_RESULT([$SWIG_LIB]) + m4_ifval([$2],[$2],[]) + fi + else + AC_MSG_WARN([cannot determine SWIG version]) + SWIG='' + m4_ifval([$3],[$3],[]) + fi + fi + AC_SUBST([SWIG_LIB]) +]) diff --git a/tests/tests-python.py b/tests/tests-python.py new file mode 100644 index 0000000..0bd71c2 --- /dev/null +++ b/tests/tests-python.py @@ -0,0 +1,115 @@ +import unittest +import sys +from babeltrace import * + +class TestBabeltracePythonModule(unittest.TestCase): + + def test_handle_decl(self): + #Context creation, adding trace + ctx = Context() + trace_handle = ctx.add_trace( + "ctf-traces/succeed/lttng-modules-2.0-pre5", + "ctf") + self.assertIsNotNone(trace_handle, "Error adding trace") + + #TraceHandle test + ts_begin = trace_handle.get_timestamp_begin(ctx, CLOCK_REAL) + ts_end = trace_handle.get_timestamp_end(ctx, CLOCK_REAL) + self.assertGreater(ts_end, ts_begin, "Error get_timestamp from trace_handle") + + lst = ctf.get_event_decl_list(trace_handle, ctx) + self.assertIsNotNone(lst, "Error get_event_decl_list") + + name = lst[0].get_name() + self.assertIsNotNone(name) + + fields = lst[0].get_decl_fields(ctf.scope.EVENT_FIELDS) + self.assertIsNotNone(fields, "Error getting FieldDecl list") + + if len(fields) > 0: + self.assertIsNotNone(fields[0].get_name(), + "Error getting name from FieldDecl") + + #Remove trace + ctx.remove_trace(trace_handle) + del ctx + del trace_handle + + + def test_iterator_event(self): + #Context creation, adding trace + ctx = Context() + trace_handle = ctx.add_trace( + "ctf-traces/succeed/lttng-modules-2.0-pre5", + "ctf") + self.assertIsNotNone(trace_handle, "Error adding trace") + + begin_pos = IterPos(SEEK_BEGIN) + it = ctf.Iterator(ctx, begin_pos) + self.assertIsNotNone(it, "Error creating iterator") + + event = it.read_event() + self.assertIsNotNone(event, "Error reading event") + + handle = event.get_handle() + self.assertIsNotNone(handle, "Error getting handle") + + context = event.get_context() + self.assertIsNotNone(context, "Error getting context") + + name = "" + while event is not None and name != "sched_switch": + name = event.get_name() + self.assertIsNotNone(name, "Error getting event name") + + ts = event.get_timestamp() + self.assertGreaterEqual(ts, 0, "Error getting timestamp") + + cycles = event.get_cycles() + self.assertGreaterEqual(cycles, 0, "Error getting cycles") + + if name == "sched_switch": + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) + self.assertIsNotNone(scope, "Error getting scope definition") + + field = event.get_field(scope, "cpu_id") + prev_comm_str = field.get_uint64() + self.assertEqual(ctf.field_error(), 0, "Error getting vec info") + + field_lst = event.get_field_list(scope) + self.assertIsNotNone(field_lst, "Error getting field list") + + fname = field.field_name() + self.assertIsNotNone(fname, "Error getting field name") + + ftype = field.field_type() + self.assertIsNot(ftype, -1, "Error getting field type (unknown)") + + ret = it.next() + self.assertGreaterEqual(ret, 0, "Error moving iterator") + + event = it.read_event() + + pos = it.get_pos() + self.assertIsNotNone(pos._pos, "Error getting iterator position") + + ret = it.set_pos(pos) + self.assertEqual(ret, 0, "Error setting iterator position") + + pos = it.create_time_pos(ts) + self.assertIsNotNone(pos._pos, "Error creating time-based iterator position") + + del it, pos + set_pos = IterPos(SEEK_TIME, ts) + it = ctf.Iterator(ctx, begin_pos, set_pos) + self.assertIsNotNone(it, "Error creating iterator with end_pos") + + + def test_file_class(self): + f = File("test-bitfield.c") + self.assertIsNotNone(f, "Error opening file") + f.close() + + +if __name__ == "__main__": + unittest.main() -- 1.7.9.5 From jdesfossez at efficios.com Fri Aug 10 17:19:53 2012 From: jdesfossez at efficios.com (Julien Desfossez) Date: Fri, 10 Aug 2012 17:19:53 -0400 Subject: [lttng-dev] [LTTNG-TOOLS PATCH] Fix: protect visibility of filter-parser functions Message-ID: <1344633593-5990-1-git-send-email-jdesfossez@efficios.com> From: Mathieu Desnoyers Some functions share the same name between ctf-parser/lexer and filter-parser/lexer. This patch protects the visibility of the functions with the same name to avoid linkage confusions. Signed-off-by: Mathieu Desnoyers Signed-off-by: Julien Desfossez --- src/lib/lttng-ctl/filter-parser.y | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/src/lib/lttng-ctl/filter-parser.y b/src/lib/lttng-ctl/filter-parser.y index ea43352..8be5e43 100644 --- a/src/lib/lttng-ctl/filter-parser.y +++ b/src/lib/lttng-ctl/filter-parser.y @@ -32,13 +32,19 @@ #include "filter-ast.h" #include "filter-parser.h" +__attribute__((visibility("protected"))) int yydebug; int filter_parser_debug = 0; +__attribute__((visibility("protected"))) int yyparse(struct filter_parser_ctx *parser_ctx); +__attribute__((visibility("protected"))) int yylex(union YYSTYPE *yyval, struct filter_parser_ctx *parser_ctx); +__attribute__((visibility("protected"))) int yylex_init_extra(struct filter_parser_ctx *parser_ctx, yyscan_t * ptr_yy_globals); +__attribute__((visibility("protected"))) int yylex_destroy(yyscan_t yyparser_ctx); +__attribute__((visibility("protected"))) void yyrestart(FILE * in_str, yyscan_t parser_ctx); struct gc_string { @@ -55,6 +61,7 @@ static const char *node_type_to_str[] = { [ NODE_UNARY_OP ] = "NODE_UNARY_OP", }; +__attribute__((visibility("protected"))) const char *node_type(struct filter_node *node) { if (node->type < NR_NODE_TYPES) @@ -85,6 +92,7 @@ static struct gc_string *gc_string_alloc(struct filter_parser_ctx *parser_ctx, * gsrc will be garbage collected immediately, and gstr might be. * Should only be used to append characters to a string literal or constant. */ +__attribute__((visibility("protected"))) struct gc_string *gc_string_append(struct filter_parser_ctx *parser_ctx, struct gc_string *gstr, struct gc_string *gsrc) @@ -114,6 +122,7 @@ struct gc_string *gc_string_append(struct filter_parser_ctx *parser_ctx, return gstr; } +__attribute__((visibility("protected"))) void setstring(struct filter_parser_ctx *parser_ctx, YYSTYPE *lvalp, const char *src) { lvalp->gs = gc_string_alloc(parser_ctx, strlen(src) + 1); @@ -175,11 +184,13 @@ static struct filter_node *make_op_node(struct filter_parser_ctx *scanner, return node; } +__attribute__((visibility("protected"))) void yyerror(struct filter_parser_ctx *parser_ctx, const char *str) { fprintf(stderr, "error %s\n", str); } +__attribute__((visibility("protected"))) int yywrap(void) { return 1; -- 1.7.10.4 From david.goulet at polymtl.ca Fri Aug 10 17:22:02 2012 From: david.goulet at polymtl.ca (David Goulet) Date: Fri, 10 Aug 2012 17:22:02 -0400 Subject: [lttng-dev] [LTTNG-TOOLS PATCH] Fix: protect visibility of filter-parser functions In-Reply-To: <1344633593-5990-1-git-send-email-jdesfossez@efficios.com> References: <1344633593-5990-1-git-send-email-jdesfossez@efficios.com> Message-ID: <50257B7A.9060604@polymtl.ca> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Merged! On 10/08/12 05:19 PM, Julien Desfossez wrote: > From: Mathieu Desnoyers > > Some functions share the same name between ctf-parser/lexer and > filter-parser/lexer. > > This patch protects the visibility of the functions with the same name to > avoid linkage confusions. > > Signed-off-by: Mathieu Desnoyers > Signed-off-by: Julien Desfossez --- > src/lib/lttng-ctl/filter-parser.y | 11 +++++++++++ 1 file changed, 11 > insertions(+) > > diff --git a/src/lib/lttng-ctl/filter-parser.y > b/src/lib/lttng-ctl/filter-parser.y index ea43352..8be5e43 100644 --- > a/src/lib/lttng-ctl/filter-parser.y +++ > b/src/lib/lttng-ctl/filter-parser.y @@ -32,13 +32,19 @@ #include > "filter-ast.h" #include "filter-parser.h" > > +__attribute__((visibility("protected"))) int yydebug; int > filter_parser_debug = 0; > > +__attribute__((visibility("protected"))) int yyparse(struct > filter_parser_ctx *parser_ctx); +__attribute__((visibility("protected"))) > int yylex(union YYSTYPE *yyval, struct filter_parser_ctx *parser_ctx); > +__attribute__((visibility("protected"))) int yylex_init_extra(struct > filter_parser_ctx *parser_ctx, yyscan_t * ptr_yy_globals); > +__attribute__((visibility("protected"))) int yylex_destroy(yyscan_t > yyparser_ctx); +__attribute__((visibility("protected"))) void > yyrestart(FILE * in_str, yyscan_t parser_ctx); > > struct gc_string { @@ -55,6 +61,7 @@ static const char *node_type_to_str[] > = { [ NODE_UNARY_OP ] = "NODE_UNARY_OP", }; > > +__attribute__((visibility("protected"))) const char *node_type(struct > filter_node *node) { if (node->type < NR_NODE_TYPES) @@ -85,6 +92,7 @@ > static struct gc_string *gc_string_alloc(struct filter_parser_ctx > *parser_ctx, * gsrc will be garbage collected immediately, and gstr might > be. * Should only be used to append characters to a string literal or > constant. */ +__attribute__((visibility("protected"))) struct gc_string > *gc_string_append(struct filter_parser_ctx *parser_ctx, struct gc_string > *gstr, struct gc_string *gsrc) @@ -114,6 +122,7 @@ struct gc_string > *gc_string_append(struct filter_parser_ctx *parser_ctx, return gstr; } > > +__attribute__((visibility("protected"))) void setstring(struct > filter_parser_ctx *parser_ctx, YYSTYPE *lvalp, const char *src) { lvalp->gs > = gc_string_alloc(parser_ctx, strlen(src) + 1); @@ -175,11 +184,13 @@ > static struct filter_node *make_op_node(struct filter_parser_ctx *scanner, > return node; } > > +__attribute__((visibility("protected"))) void yyerror(struct > filter_parser_ctx *parser_ctx, const char *str) { fprintf(stderr, "error > %s\n", str); } > > +__attribute__((visibility("protected"))) int yywrap(void) { return 1; -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJQJXt6AAoJEELoaioR9I0263gIALoOPhf7ldD7rgtOk+gRcHTM HI/Rr7Fo2tsl8EzyRxpyy+2JZfAOoFAqG17VLAXQ9wozgS3KJ/wgvrRWIhEFvSPv RWlahrg6XHscbrVD5Vf02+L3M0MG6MbaXNYFl2dpHdxKufbmOk3WWW7uxgHAVTpI kWEyMvIeqphxiLqyEF9+LbW0pBQ2Sl/7AjFU6S7ZT5gAGo1EOp3F2LnBXQVBEI9s TJY/rnM/pq0TZAXMRuAY7eALzYYVq6oQ00MAaAgdxk/+ElA9KLd6TxlNOR+orpqD Txp5a1QYTPnavW+6htiq4iQCqDozTy5z1nMoJXuR9Q7UirKjxKfk7DD+baVz7lI= =gdS3 -----END PGP SIGNATURE----- From mathieu.desnoyers at efficios.com Fri Aug 10 17:26:22 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 10 Aug 2012 17:26:22 -0400 Subject: [lttng-dev] [babeltrace PATCH] Babeltrace python module v3 In-Reply-To: <1344632815-6541-1-git-send-email-danny.serres@efficios.com> References: <1344632815-6541-1-git-send-email-danny.serres@efficios.com> Message-ID: <20120810212622.GC364@Krystal> * Danny Serres (danny.serres at efficios.com) wrote: > The Babeltrace Python module can be used to directly control > the Babeltrace API inside Python, using 'import babeltrace'. > > Therefore, it becomes possible to create a Context, add a > trace to it, iterate on it, read events and so on from > within Python. > > SWIG >= 2.0 is used to create the wrapper and its > 'warning md variable unused' bug is patched in Makefile.am > > In the interface file, struct and enum are directly copied > from the include files. All changes to struct bt_iter_pos > and to enums in ctf/events.h and clock-types.h must also > be made in the interface file. Please change the .gitignore: *.m4 to /m4/libtool.m4 /m4/lt~obsolete.m4 /m4/ltoptions.m4 /m4/ltsugar.m4 /m4/ltversion.m4 otherwise git add complains (rightfully). Also, please add a license header at the beginning of each file. Please use a wording similar to: /* * filename.ext * * Babeltrace somescriptname * * Copyright 2012 EfficiOS Inc. * * Author: Danny Serres * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal * in the Software without restriction, including without limitation the rights * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. */ In python, you might need to use: # filename.ext # # Babeltrace somescriptname # # Copyright 2012 EfficiOS Inc. # # Author: Danny Serres # # Permission is hereby granted, free of charge, to any person obtaining a copy # of this software and associated documentation files (the "Software"), to deal # in the Software without restriction, including without limitation the rights # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # copies of the Software, and to permit persons to whom the Software is # furnished to do so, subject to the following conditions: # # The above copyright notice and this permission notice shall be included in # all copies or substantial portions of the Software. The rest looks good, Thanks! Mathieu > > Signed-off-by: Danny Serres > Signed-off-by: Yannick Brosseau > --- > .gitignore | 4 + > Makefile.am | 2 +- > README | 6 + > bindings/Makefile.am | 3 + > bindings/python/Makefile.am | 28 + > bindings/python/babeltrace.i.in | 1079 +++++++++ > bindings/python/examples/babeltrace_and_lttng.py | 107 + > bindings/python/examples/eventcount.py | 66 + > bindings/python/examples/eventcountlist.py | 65 + > bindings/python/examples/events_per_cpu.py | 81 + > bindings/python/examples/example-api-test.py | 59 + > bindings/python/examples/histogram.py | 121 + > .../examples/output_format_modules/cairoplot.py | 2336 ++++++++++++++++++++ > .../examples/output_format_modules/pprint_table.py | 35 + > .../examples/output_format_modules/series.py | 1140 ++++++++++ > bindings/python/examples/sched_switch.py | 110 + > bindings/python/examples/softirqtimes.py | 130 ++ > bindings/python/examples/syscalls_by_pid.py | 61 + > bindings/python/python-complements.c | 105 + > bindings/python/python-complements.h | 36 + > bootstrap | 2 +- > configure.ac | 37 + > doc/python-howto.txt | 70 + > m4/ax_pkg_swig.m4 | 135 ++ > tests/tests-python.py | 115 + > 25 files changed, 5931 insertions(+), 2 deletions(-) > create mode 100644 bindings/Makefile.am > create mode 100644 bindings/python/Makefile.am > create mode 100644 bindings/python/babeltrace.i.in > create mode 100644 bindings/python/examples/babeltrace_and_lttng.py > create mode 100644 bindings/python/examples/eventcount.py > create mode 100644 bindings/python/examples/eventcountlist.py > create mode 100644 bindings/python/examples/events_per_cpu.py > create mode 100644 bindings/python/examples/example-api-test.py > create mode 100644 bindings/python/examples/histogram.py > create mode 100644 bindings/python/examples/output_format_modules/__init__.py > create mode 100755 bindings/python/examples/output_format_modules/cairoplot.py > create mode 100644 bindings/python/examples/output_format_modules/pprint_table.py > create mode 100755 bindings/python/examples/output_format_modules/series.py > create mode 100644 bindings/python/examples/sched_switch.py > create mode 100644 bindings/python/examples/softirqtimes.py > create mode 100644 bindings/python/examples/syscalls_by_pid.py > create mode 100644 bindings/python/python-complements.c > create mode 100644 bindings/python/python-complements.h > create mode 100644 doc/python-howto.txt > create mode 100644 m4/ax_pkg_swig.m4 > create mode 100644 tests/tests-python.py > > diff --git a/.gitignore b/.gitignore > index d6098ac..3fe60e7 100644 > --- a/.gitignore > +++ b/.gitignore > @@ -1,4 +1,5 @@ > /tests/test-bitfield > +*~ > *.o > *.a > *.la > @@ -26,3 +27,6 @@ converter/babeltrace-log > core > formats/ctf/metadata/ctf-parser.output > stamp-h1 > +bindings/python/babeltrace.i > +bindings/python/babeltrace.py > +bindings/python/babeltrace_wrap.c > diff --git a/Makefile.am b/Makefile.am > index 308ee16..6584c5d 100644 > --- a/Makefile.am > +++ b/Makefile.am > @@ -2,7 +2,7 @@ AM_CFLAGS = $(PACKAGE_CFLAGS) -I$(top_srcdir)/include > > ACLOCAL_AMFLAGS = -I m4 > > -SUBDIRS = include types lib formats converter tests doc > +SUBDIRS = include types lib formats converter bindings tests doc > > dist_doc_DATA = ChangeLog LICENSE mit-license.txt gpl-2.0.txt \ > std-ext-lib.txt > diff --git a/README b/README > index 75bf0cf..1687075 100644 > --- a/README > +++ b/README > @@ -25,6 +25,7 @@ BUILDING > make install > ldconfig > > + If you do not want Python bindings, run ./configure --disable-python. > > DEPENDENCIES > ------------ > @@ -44,6 +45,11 @@ To compile Babeltrace, you will need: > libpopt >= 1.13 development libraries > (Debian : libpopt-dev) > (Fedora : popt) > + python headers (optional) > + (Debian/Ubuntu : python-dev) > + swig >= 2.0 (optional) > + (Debian/Ubuntu : swig2.0) > + > > For developers using the git tree: > > diff --git a/bindings/Makefile.am b/bindings/Makefile.am > new file mode 100644 > index 0000000..dcd868d > --- /dev/null > +++ b/bindings/Makefile.am > @@ -0,0 +1,3 @@ > +if USE_PYTHON > +SUBDIRS = python > +endif > diff --git a/bindings/python/Makefile.am b/bindings/python/Makefile.am > new file mode 100644 > index 0000000..579759f > --- /dev/null > +++ b/bindings/python/Makefile.am > @@ -0,0 +1,28 @@ > +babeltrace.i: babeltrace.i.in > + sed "s/BABELTRACE_VERSION_STR/Babeltrace $(PACKAGE_VERSION)/g" babeltrace.i > + > +AM_CFLAGS = -I$(PYTHON_INCLUDE) -I$(top_srcdir)/include/ > + > +EXTRA_DIST = babeltrace.i > +python_PYTHON = babeltrace.py > +pyexec_LTLIBRARIES = _babeltrace.la > + > +MAINTAINERCLEANFILES = babeltrace_wrap.c babeltrace.py > + > +_babeltrace_la_SOURCES = babeltrace_wrap.c python-complements.c > + > +_babeltrace_la_LDFLAGS = -module > + > +_babeltrace_la_CFLAGS = $(GLIB_CFLAGS) $(AM_CFLAGS) > + > +_babeltrace_la_LIBS = $(GLIB_LIBS) > + > +_babeltrace_la_LIBADD = $(top_srcdir)/formats/ctf/libbabeltrace-ctf.la \ > + $(top_srcdir)/formats/ctf-text/libbabeltrace-ctf-text.la > + > +# SWIG 'warning md variable unused' fixed after SWIG build: > +babeltrace_wrap.c: babeltrace.i > + $(SWIG) -python -Wall -I. -I$(top_srcdir)/include babeltrace.i > + sed -i "s/PyObject \*m, \*d, \*md;/PyObject \*m, \*d;\n#if defined(SWIGPYTHON_BUILTIN)\nPyObject *md;\n#endif/g" babeltrace_wrap.c > + sed -i "s/md = d/d/g" babeltrace_wrap.c > + sed -i "s/(void)public_symbol;/(void)public_symbol;\n md = d;/g" babeltrace_wrap.c > diff --git a/bindings/python/babeltrace.i.in b/bindings/python/babeltrace.i.in > new file mode 100644 > index 0000000..49636d1 > --- /dev/null > +++ b/bindings/python/babeltrace.i.in > @@ -0,0 +1,1079 @@ > +/* BABELTRACE PYTHON MODULE interface file */ > + > +%define DOCSTRING > +"BABELTRACE_VERSION_STR > + > +Babeltrace is a trace viewer and converter reading and writing the > +Common Trace Format (CTF). Its main use is to pretty-print CTF > +traces into a human-readable text output. > + > +To use this module, the first step is to create a Context and add a > +trace to it." > +%enddef > + > +%module(docstring=DOCSTRING) babeltrace > + > +%include "typemaps.i" > +%{ > +#define SWIG_FILE_WITH_INIT > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include "python-complements.h" > +%} > + > +typedef unsigned long long uint64_t; > +typedef long long int64_t; > +typedef int bt_intern_str; > + > +/* ================================================================= > + CONTEXT.H, CONTEXT-INTERNAL.H > + ????????????????????????????? > +*/ > + > +%rename("_bt_context_create") bt_context_create(void); > +%rename("_bt_context_add_trace") bt_context_add_trace( > + struct bt_context *ctx, const char *path, const char *format, > + void (*packet_seek)(struct stream_pos *pos, size_t index, int whence), > + struct mmap_stream_list *stream_list, FILE *metadata); > +%rename("_bt_context_remove_trace") bt_context_remove_trace( > + struct bt_context *ctx, int trace_id); > +%rename("_bt_context_get") bt_context_get(struct bt_context *ctx); > +%rename("_bt_context_put") bt_context_put(struct bt_context *ctx); > +%rename("_bt_ctf_event_get_context") bt_ctf_event_get_context( > + const struct bt_ctf_event *event); > + > +struct bt_context *bt_context_create(void); > +int bt_context_add_trace(struct bt_context *ctx, const char *path, const char *format, > + void (*packet_seek)(struct stream_pos *pos, size_t index, int whence), > + struct mmap_stream_list *stream_list, FILE *metadata); > +void bt_context_remove_trace(struct bt_context *ctx, int trace_id); > +void bt_context_get(struct bt_context *ctx); > +void bt_context_put(struct bt_context *ctx); > +struct bt_context *bt_ctf_event_get_context(const struct bt_ctf_event *event); > + > +// class Context to prevent direct access to struct bt_context > +%pythoncode%{ > +class Context: > + """ > + The context represents the object in which a trace_collection is > + open. As long as this structure is allocated, the trace_collection > + is open and the traces it contains can be read and seeked by the > + iterators and callbacks. > + """ > + > + def __init__(self): > + self._c = _bt_context_create() > + > + def __del__(self): > + _bt_context_put(self._c) > + > + def add_trace(self, path, format_str, > + packet_seek=None, stream_list=None, metadata=None): > + """ > + Add a trace by path to the context. > + > + Open a trace. > + > + path is the path to the trace, it is not recursive. > + If "path" is None, stream_list is used instead as a list > + of mmap streams to open for the trace. > + > + format is a string containing the format name in which the trace was > + produced. > + > + packet_seek is not implemented for Python. Should be left None to > + use the default packet_seek handler provided by the trace format. > + > + stream_list is a linked list of streams, it is used to open a trace > + where the trace data is located in memory mapped areas instead of > + trace files, this argument should be None when path is not None. > + > + The metadata parameter acts as a metadata override when not None, > + otherwise the format handles the metadata opening. > + > + Return: the corresponding TraceHandle on success or None on error. > + """ > + if metadata is not None: > + metadata = metadata._file > + > + ret = _bt_context_add_trace(self._c, path, format_str, packet_seek, > + stream_list, metadata) > + if ret < 0: > + return None > + > + th = TraceHandle.__new__(TraceHandle) > + th._id = ret > + return th > + > + def add_traces_recursive(self, path, format_str): > + """ > + Open a trace recursively. > + > + Find each trace present in the subdirectory starting from the given > + path, and add them to the context. > + > + Return a dict of TraceHandle instances (the full path is the key). > + Return None on error. > + """ > + > + import os > + > + trace_handles = {} > + > + noTrace = True > + error = False > + > + for fullpath, dirs, files in os.walk(path): > + if "metadata" in files: > + trace_handle = self.add_trace(fullpath, format_str) > + if trace_handle is None: > + error = True > + continue > + > + trace_handles[fullpath] = trace_handle > + noTrace = False > + > + if noTrace and error: > + return None > + return trace_handles > + > + def remove_trace(self, trace_handle): > + """ > + Remove a trace from the context. > + Effectively closing the trace. > + """ > + try: > + _bt_context_remove_trace(self._c, trace_handle._id) > + except AttributeError: > + raise TypeError("in remove_trace, " > + "argument 2 must be a TraceHandle instance") > +%} > + > + > + > +/* ================================================================= > + FORMAT.H, REGISTRY > + ?????????????????? > +*/ > + > +%rename("lookup_format") bt_lookup_format(bt_intern_str qname); > +%rename("_bt_print_format_list") bt_fprintf_format_list(FILE *fp); > +%rename("register_format") bt_register_format(struct format *format); > + > +extern struct format *bt_lookup_format(bt_intern_str qname); > +extern void bt_fprintf_format_list(FILE *fp); > +extern int bt_register_format(struct format *format); > + > +void format_init(void); > +void format_finalize(void); > + > +%pythoncode %{ > + > +def print_format_list(babeltrace_file): > + """ > + Print a list of available formats to file. > + > + babeltrace_file must be a File instance opened in write mode. > + """ > + try: > + if babeltrace_file._file is not None: > + _bt_print_format_list(babeltrace_file._file) > + except AttributeError: > + raise TypeError("in print_format_list, " > + "argument 1 must be a File instance") > + > +%} > + > + > +/* ================================================================= > + ITERATOR.H, ITERATOR-INTERNAL.H > + ??????????????????????????????? > +*/ > + > +%rename("_bt_iter_create") bt_iter_create(struct bt_context *ctx, > + const struct bt_iter_pos *begin_pos, const struct bt_iter_pos *end_pos); > +%rename("_bt_iter_destroy") bt_iter_destroy(struct bt_iter *iter); > +%rename("_bt_iter_next") bt_iter_next(struct bt_iter *iter); > +%rename("_bt_iter_get_pos") bt_iter_get_pos(struct bt_iter *iter); > +%rename("_bt_iter_free_pos") bt_iter_free_pos(struct bt_iter_pos *pos); > +%rename("_bt_iter_set_pos") bt_iter_set_pos(struct bt_iter *iter, > + const struct bt_iter_pos *pos); > +%rename("_bt_iter_create_time_pos") bt_iter_create_time_pos(struct bt_iter *iter, > + uint64_t timestamp); > + > +struct bt_iter *bt_iter_create(struct bt_context *ctx, > + const struct bt_iter_pos *begin_pos, const struct bt_iter_pos *end_pos); > +void bt_iter_destroy(struct bt_iter *iter); > +int bt_iter_next(struct bt_iter *iter); > +struct bt_iter_pos *bt_iter_get_pos(struct bt_iter *iter); > +void bt_iter_free_pos(struct bt_iter_pos *pos); > +int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *pos); > +struct bt_iter_pos *bt_iter_create_time_pos(struct bt_iter *iter, uint64_t timestamp); > + > +%rename("_bt_iter_pos") bt_iter_pos; > +%rename("SEEK_TIME") BT_SEEK_TIME; > +%rename("SEEK_RESTORE") BT_SEEK_RESTORE; > +%rename("SEEK_CUR") BT_SEEK_CUR; > +%rename("SEEK_BEGIN") BT_SEEK_BEGIN; > +%rename("SEEK_END") BT_SEEK_END; > + > + > +// This struct is taken from iterator.h > +// All changes to the struct must also be made here > +struct bt_iter_pos { > + enum { > + BT_SEEK_TIME, /* uses u.seek_time */ > + BT_SEEK_RESTORE, /* uses u.restore */ > + BT_SEEK_CUR, > + BT_SEEK_BEGIN, > + BT_SEEK_END, > + } type; > + union { > + uint64_t seek_time; > + struct bt_saved_pos *restore; > + } u; > +}; > + > + > +%pythoncode%{ > + > +class IterPos: > + """This class represents the position where to set an iterator.""" > + > + __can_access = False > + > + def __init__(self, seek_type, seek_time = None): > + """ > + seek_type represents the type of seek to use. > + seek_time is the timestamp to seek to when using SEEK_TIME, it > + is expressed in nanoseconds > + Only use SEEK_RESTORE on IterPos obtained from the get_pos function > + in Iter class. > + """ > + > + self._pos = _bt_iter_pos() > + self._pos.type = seek_type > + if seek_time and seek_type == SEEK_TIME: > + self._pos.u.seek_time = seek_time > + self.__can_access = True > + > + def __del__(self): > + if not self.__can_access: > + _bt_iter_free_pos(self._pos) > + > + def _get_type(self): > + if not __can_access: > + raise AttributeError("seek_type is not available") > + return self._pos.type > + > + def _set_type(self, seek_type): > + if not __can_access: > + raise AttributeError("seek_type is not available") > + self._pos.type = seek_type > + > + def _get_time(self): > + if not __can_access: > + raise AttributeError("seek_time is not available") > + > + elif self._pos.type is not SEEK_TIME: > + raise TypeError("seek_type is not SEEK_TIME") > + > + return self._pos.u.seek_time > + > + def _set_time(self, time): > + if not __can_access: > + raise AttributeError("seek_time is not available") > + > + elif self._pos.type is not SEEK_TIME: > + raise TypeError("seek_type is not SEEK_TIME") > + > + self._pos.u.seek_time = time > + > + def _get_pos(self): > + return self._pos > + > + > + seek_type = property(_get_type, _set_type) > + seek_time = property(_get_time, _set_time) > + > + > +class Iterator: > + > + __with_init = False > + > + def __init__(self, context, begin_pos = None, end_pos = None, _no_init = None): > + """ > + Allocate a trace collection iterator. > + > + begin_pos and end_pos are optional parameters to specify the > + position at which the trace collection should be seeked upon > + iterator creation, and the position at which iteration will > + start returning "EOF". > + > + By default, if begin_pos is None, a BT_SEEK_CUR is performed at > + creation. By default, if end_pos is None, a BT_SEEK_END (end of > + trace) is the EOF criterion. > + """ > + if _no_init is None: > + if begin_pos is None: > + bp = None > + else: > + try: > + bp = begin_pos._pos > + except AttributeError: > + raise TypeError("in __init__, " > + "argument 3 must be a IterPos instance") > + > + if end_pos is None: > + ep = None > + else: > + try: > + ep = end_pos._pos > + except AttributeError: > + raise TypeError("in __init__, " > + "argument 4 must be a IterPos instance") > + > + try: > + self._bi = _bt_iter_create(context._c, bp, ep) > + except AttributeError: > + raise TypeError("in __init__, " > + "argument 2 must be a Context instance") > + > + self.__with_init = True > + > + else: > + self._bi = _no_init > + > + def __del__(self): > + if self.__with_init: > + _bt_iter_destroy(self._bi) > + > + def next(self): > + """ > + Move trace collection position to the next event. > + Returns 0 on success, a negative value on error. > + """ > + return _bt_iter_next(self._bi) > + > + def get_pos(self): > + """Return a IterPos class of the current iterator position.""" > + ret = IterPos(0) > + ret.__can_access = False > + ret._pos = _bt_iter_get_pos(self._bi) > + return ret > + > + def set_pos(self, pos): > + """ > + Move the iterator to a given position. > + > + On error, the stream_heap is reinitialized and returned empty. > + Return 0 for success. > + Return EOF if the position requested is after the last event of the > + trace collection. > + Return -EINVAL when called with invalid parameter. > + Return -ENOMEM if the stream_heap could not be properly initialized. > + """ > + try: > + return _bt_iter_set_pos(self._bi, pos._pos) > + except AttributeError: > + raise TypeError("in set_pos, " > + "argument 2 must be a IterPos instance") > + > + def create_time_pos(self, timestamp): > + """ > + Create a position based on time > + This function allocates and returns a new IterPos to be able to > + restore an iterator position based on a timestamp. > + """ > + > + if timestamp < 0: > + raise TypeError("timestamp must be an unsigned int") > + > + ret = IterPos(0) > + ret.__can_access = False > + ret._pos = _bt_iter_create_time_pos(self._bi, timestamp) > + return ret > +%} > + > + > +/* ================================================================= > + CLOCK-TYPE.H > + ???????????? > + *** Enum copied from clock-type.h? > + All changes must also be made here > +*/ > +%rename("CLOCK_CYCLES") BT_CLOCK_CYCLES; > +%rename("CLOCK_REAL") BT_CLOCK_REAL; > + > +enum bt_clock_type { > + BT_CLOCK_CYCLES = 0, > + BT_CLOCK_REAL > +}; > + > +/* ================================================================= > + TRACE-HANDLE.H, TRACE-HANDLE-INTERNAL.H > + ??????????????????????????????????????? > +*/ > + > +%rename("_bt_trace_handle_create") bt_trace_handle_create(struct bt_context *ctx); > +%rename("_bt_trace_handle_destroy") bt_trace_handle_destroy(struct bt_trace_handle *bt); > +struct bt_trace_handle *bt_trace_handle_create(struct bt_context *ctx); > +void bt_trace_handle_destroy(struct bt_trace_handle *bt); > + > +%rename("_bt_trace_handle_get_path") bt_trace_handle_get_path(struct bt_context *ctx, > + int handle_id); > +%rename("_bt_trace_handle_get_timestamp_begin") bt_trace_handle_get_timestamp_begin( > + struct bt_context *ctx, int handle_id, enum bt_clock_type type); > +%rename("_bt_trace_handle_get_timestamp_end") bt_trace_handle_get_timestamp_end( > + struct bt_context *ctx, int handle_id, enum bt_clock_type type); > +%rename("_bt_trace_handle_get_id") bt_trace_handle_get_id(struct bt_trace_handle *th); > +int bt_trace_handle_get_id(struct bt_trace_handle *th); > +const char *bt_trace_handle_get_path(struct bt_context *ctx, int handle_id); > +uint64_t bt_trace_handle_get_timestamp_begin(struct bt_context *ctx, int handle_id, > + enum bt_clock_type type); > +uint64_t bt_trace_handle_get_timestamp_end(struct bt_context *ctx, int handle_id, > + enum bt_clock_type type); > + > +%rename("_bt_ctf_event_get_handle_id") bt_ctf_event_get_handle_id( > + const struct bt_ctf_event *event); > +int bt_ctf_event_get_handle_id(const struct bt_ctf_event *event); > + > + > +%pythoncode%{ > + > +class TraceHandle(object): > + """ > + The TraceHandle allows the user to manipulate a trace file directly. > + It is a unique identifier representing a trace file. > + Do not instantiate. > + """ > + > + def __init__(self): > + raise NotImplementedError("TraceHandle cannot be instantiated") > + > + def __repr__(self): > + return "Babeltrace TraceHandle: trace_id('{}')".format(self._id) > + > + def get_id(self): > + """Return the TraceHandle id.""" > + return self._id > + > + def get_path(self, context): > + """Return the path of a TraceHandle.""" > + try: > + return _bt_trace_handle_get_path(context._c, self._id) > + except AttributeError: > + raise TypeError("in get_path, " > + "argument 2 must be a Context instance") > + > + def get_timestamp_begin(self, context, clock_type): > + """Return the creation time of the buffers of a trace.""" > + try: > + return _bt_trace_handle_get_timestamp_begin( > + context._c, self._id,clock_type) > + except AttributeError: > + raise TypeError("in get_timestamp_begin, " > + "argument 2 must be a Context instance") > + > + def get_timestamp_end(self, context, clock_type): > + """Return the destruction timestamp of the buffers of a trace.""" > + try: > + return _bt_trace_handle_get_timestamp_end( > + context._c, self._id, clock_type) > + except AttributeError: > + raise TypeError("in get_timestamp_end, " > + "argument 2 must be a Context instance") > + > +%} > + > + > + > +// ================================================================= > +// CTF > +// ================================================================= > + > +/* ================================================================= > + ITERATOR.H, EVENTS.H > + ???????????????????? > +*/ > + > +//Iterator > +%rename("_bt_ctf_iter_create") bt_ctf_iter_create(struct bt_context *ctx, > + const struct bt_iter_pos *begin_pos, > + const struct bt_iter_pos *end_pos); > +%rename("_bt_ctf_get_iter") bt_ctf_get_iter(struct bt_ctf_iter *iter); > +%rename("_bt_ctf_iter_destroy") bt_ctf_iter_destroy(struct bt_ctf_iter *iter); > +%rename("_bt_ctf_iter_read_event") bt_ctf_iter_read_event(struct bt_ctf_iter *iter); > + > +struct bt_ctf_iter *bt_ctf_iter_create(struct bt_context *ctx, > + const struct bt_iter_pos *begin_pos, > + const struct bt_iter_pos *end_pos); > +struct bt_iter *bt_ctf_get_iter(struct bt_ctf_iter *iter); > +void bt_ctf_iter_destroy(struct bt_ctf_iter *iter); > +struct bt_ctf_event *bt_ctf_iter_read_event(struct bt_ctf_iter *iter); > + > + > +//Events > + > +%rename("_bt_ctf_get_top_level_scope") bt_ctf_get_top_level_scope(const struct > + bt_ctf_event *event, enum bt_ctf_scope scope); > +%rename("_bt_ctf_event_name") bt_ctf_event_name(const struct bt_ctf_event *ctf_event); > +%rename("_bt_ctf_get_timestamp") bt_ctf_get_timestamp( > + const struct bt_ctf_event *ctf_event); > +%rename("_bt_ctf_get_cycles") bt_ctf_get_cycles( > + const struct bt_ctf_event *ctf_event); > + > +%rename("_bt_ctf_get_field") bt_ctf_get_field(const struct bt_ctf_event *ctf_event, > + const struct definition *scope, const char *field); > +%rename("_bt_ctf_get_index") bt_ctf_get_index(const struct bt_ctf_event *ctf_event, > + const struct definition *field, unsigned int index); > +%rename("_bt_ctf_field_name") bt_ctf_field_name(const struct definition *def); > +%rename("_bt_ctf_field_type") bt_ctf_field_type(const struct definition *def); > +%rename("_bt_ctf_get_int_signedness") bt_ctf_get_int_signedness( > + const struct definition *field); > +%rename("_bt_ctf_get_int_base") bt_ctf_get_int_base(const struct definition *field); > +%rename("_bt_ctf_get_int_byte_order") bt_ctf_get_int_byte_order( > + const struct definition *field); > +%rename("_bt_ctf_get_int_len") bt_ctf_get_int_len(const struct definition *field); > +%rename("_bt_ctf_get_encoding") bt_ctf_get_encoding(const struct definition *field); > +%rename("_bt_ctf_get_array_len") bt_ctf_get_array_len(const struct definition *field); > +%rename("_bt_ctf_get_uint64") bt_ctf_get_uint64(const struct definition *field); > +%rename("_bt_ctf_get_int64") bt_ctf_get_int64(const struct definition *field); > +%rename("_bt_ctf_get_char_array") bt_ctf_get_char_array(const struct definition *field); > +%rename("_bt_ctf_get_string") bt_ctf_get_string(const struct definition *field); > +%rename("_bt_ctf_field_get_error") bt_ctf_field_get_error(void); > +%rename("_bt_ctf_get_decl_event_name") bt_ctf_get_decl_event_name(const struct > + bt_ctf_event_decl *event); > +%rename("_bt_ctf_get_decl_field_name") bt_ctf_get_decl_field_name( > + const struct bt_ctf_field_decl *field); > + > +const struct definition *bt_ctf_get_top_level_scope(const struct bt_ctf_event *ctf_event, > + enum bt_ctf_scope scope); > +const char *bt_ctf_event_name(const struct bt_ctf_event *ctf_event); > +uint64_t bt_ctf_get_timestamp(const struct bt_ctf_event *ctf_event); > +uint64_t bt_ctf_get_cycles(const struct bt_ctf_event *ctf_event); > +const struct definition *bt_ctf_get_field(const struct bt_ctf_event *ctf_event, > + const struct definition *scope, > + const char *field); > +const struct definition *bt_ctf_get_index(const struct bt_ctf_event *ctf_event, > + const struct definition *field, > + unsigned int index); > +const char *bt_ctf_field_name(const struct definition *def); > +enum ctf_type_id bt_ctf_field_type(const struct definition *def); > +int bt_ctf_get_int_signedness(const struct definition *field); > +int bt_ctf_get_int_base(const struct definition *field); > +int bt_ctf_get_int_byte_order(const struct definition *field); > +ssize_t bt_ctf_get_int_len(const struct definition *field); > +enum ctf_string_encoding bt_ctf_get_encoding(const struct definition *field); > +int bt_ctf_get_array_len(const struct definition *field); > +uint64_t bt_ctf_get_uint64(const struct definition *field); > +int64_t bt_ctf_get_int64(const struct definition *field); > +char *bt_ctf_get_char_array(const struct definition *field); > +char *bt_ctf_get_string(const struct definition *field); > +int bt_ctf_field_get_error(void); > +const char *bt_ctf_get_decl_event_name(const struct bt_ctf_event_decl *event); > +const char *bt_ctf_get_decl_field_name(const struct bt_ctf_field_decl *field); > + > +%pythoncode%{ > + > +class ctf: > + > + #enum equivalent, accessible constants > + #These are taken directly from ctf/events.h > + #All changes to enums must also be made here > + class type_id: > + UNKNOWN = 0 > + INTEGER = 1 > + FLOAT = 2 > + ENUM = 3 > + STRING = 4 > + STRUCT = 5 > + UNTAGGED_VARIANT = 6 > + VARIANT = 7 > + ARRAY = 8 > + SEQUENCE = 9 > + NR_CTF_TYPES = 10 > + > + class scope: > + TRACE_PACKET_HEADER = 0 > + STREAM_PACKET_CONTEXT = 1 > + STREAM_EVENT_HEADER = 2 > + STREAM_EVENT_CONTEXT = 3 > + EVENT_CONTEXT = 4 > + EVENT_FIELDS = 5 > + > + class string_encoding: > + NONE = 0 > + UTF8 = 1 > + ASCII = 2 > + UNKNOWN = 3 > + > + class Iterator(Iterator, object): > + """ > + Allocate a CTF trace collection iterator. > + > + begin_pos and end_pos are optional parameters to specify the > + position at which the trace collection should be seeked upon > + iterator creation, and the position at which iteration will > + start returning "EOF". > + > + By default, if begin_pos is None, a SEEK_CUR is performed at > + creation. By default, if end_pos is None, a SEEK_END (end of > + trace) is the EOF criterion. > + > + Only one iterator can be created against a context. If more than one > + iterator is being created for the same context, the second creation > + will return None. The previous iterator must be destroyed before > + creation of the new iterator for this function to succeed. > + """ > + > + def __new__(cls, context, begin_pos = None, end_pos = None): > + # __new__ is used to control the return value > + # as the ctf.Iterator class should return None > + # if bt_ctf_iter_create returns NULL > + > + if begin_pos is None: > + bp = None > + else: > + bp = begin_pos._pos > + if end_pos is None: > + ep = None > + else: > + ep = end_pos._pos > + try: > + it = _bt_ctf_iter_create(context._c, bp, ep) > + except AttributeError: > + raise TypeError("in __init__, " > + "argument 2 must be a Context instance") > + if it is None: > + return None > + > + ret_class = super(ctf.Iterator, cls).__new__(cls) > + ret_class._i = it > + return ret_class > + > + def __init__(self, context, begin_pos = None, end_pos = None): > + Iterator.__init__(self, None, None, None, > + _bt_ctf_get_iter(self._i)) > + > + def __del__(self): > + _bt_ctf_iter_destroy(self._i) > + > + def read_event(self): > + """ > + Read the iterator's current event data. > + Return current event on success, None on end of trace. > + """ > + ret = _bt_ctf_iter_read_event(self._i) > + if ret is None: > + return ret > + ev = ctf.Event.__new__(ctf.Event) > + ev._e = ret > + return ev > + > + > + class Event(object): > + """ > + This class represents an event from the trace. > + It is obtained with read_event() from ctf.Iterator. > + Do not instantiate. > + """ > + > + def __init__(self): > + raise NotImplementedError("ctf.Event cannot be instantiated") > + > + def get_top_level_scope(self, scope): > + """ > + Return a definition of the top-level scope > + Top-level scopes are defined in ctf.scope. > + In order to get a field or a field list, the user needs to pass a > + scope as argument, this scope can be a top-level scope or a scope > + relative to an arbitrary field. This function provides the mapping > + between the scope and the actual definition of top-level scopes. > + On error return None. > + """ > + evDef = ctf.Definition.__new__(ctf.Definition) > + evDef._d = _bt_ctf_get_top_level_scope(self._e, scope) > + if evDef._d is None: > + return None > + return evDef > + > + def get_name(self): > + """Return the name of the event or None on error.""" > + return _bt_ctf_event_name(self._e) > + > + def get_cycles(self): > + """ > + Return the timestamp of the event as written in > + the packet (in cycles) or -1ULL on error. > + """ > + return _bt_ctf_get_cycles(self._e) > + > + def get_timestamp(self): > + """ > + Return the timestamp of the event offsetted with the > + system clock source or -1ULL on error. > + """ > + return _bt_ctf_get_timestamp(self._e) > + > + def get_field(self, scope, field): > + """Return the definition of a specific field.""" > + evDef = ctf.Definition.__new__(ctf.Definition) > + try: > + evDef._d = _bt_ctf_get_field(self._e, scope._d, field) > + except AttributeError: > + raise TypeError("in get_field, argument 2 must be a " > + "Definition (scope) instance") > + return evDef > + > + def get_field_list(self, scope): > + """ > + Return a list of Definitions > + Return None on error. > + """ > + try: > + field_lc = _bt_python_field_listcaller(self._e, scope._d) > + except AttributeError: > + raise TypeError("in get_field_list, argument 2 must be a " > + "Definition (scope) instance") > + > + if field_lc is None: > + return None > + > + def_list = [] > + i = 0 > + while True: > + tmp = ctf.Definition.__new__(ctf.Definition) > + tmp._d = _bt_python_field_one_from_list(field_lc, i) > + > + if tmp._d is None: > + #Last item of list is None, assured in > + #_bt_python_field_listcaller > + break > + > + def_list.append(tmp) > + i += 1 > + return def_list > + > + def get_index(self, field, index): > + """ > + If the field is an array or a sequence, return the element > + at position index, otherwise return None > + """ > + evDef = ctf.Definition.__new__(ctf.Definition) > + try: > + evDef._d = _bt_ctf_get_index(self._e, field._d, index) > + except AttributeError: > + raise TypeError("in get_index, argument 2 must be a " > + "Definition (field) instance") > + > + if evDef._d is None: > + return None > + return evDef > + > + def get_handle(self): > + """ > + Get the TraceHandle associated with an event > + Return None on error > + """ > + ret = _bt_ctf_event_get_handle_id(self._e) > + if ret < 0: > + return None > + > + th = TraceHandle.__new__(TraceHandle) > + th._id = ret > + return th > + > + def get_context(self): > + """ > + Get the context associated with an event. > + Return None on error. > + """ > + ctx = Context() > + ctx._c = _bt_ctf_event_get_context(self._e); > + if ctx._c is None: > + return None > + else: > + return ctx > + > + > + class Definition(object): > + """Definition class. Do not instantiate.""" > + > + def __init__(self): > + raise NotImplementedError("ctf.Definition cannot be instantiated") > + > + def __repr__(self): > + return "Babeltrace Definition: name('{}'), type({})".format( > + self.field_name(), self.field_type()) > + > + def field_name(self): > + """Return the name of a field or None on error.""" > + return _bt_ctf_field_name(self._d) > + > + def field_type(self): > + """Return the type of a field or -1 if unknown.""" > + return _bt_ctf_field_type(self._d) > + > + def get_int_signedness(self): > + """ > + Return the signedness of an integer: > + 0 if unsigned; 1 if signed; -1 on error. > + """ > + return _bt_ctf_get_int_signedness(self._d) > + > + def get_int_base(self): > + """Return the base of an int or a negative value on error.""" > + return _bt_ctf_get_int_base(self._d) > + > + def get_int_byte_order(self): > + """ > + Return the byte order of an int or a negative > + value on error. > + """ > + return _bt_ctf_get_int_byte_order(self._d) > + > + def get_int_len(self): > + """ > + Return the size, in bits, of an int or a negative > + value on error. > + """ > + return _bt_ctf_get_int_len(self._d) > + > + def get_encoding(self): > + """ > + Return the encoding of an int or a string. > + Return a negative value on error. > + """ > + return _bt_ctf_get_encoding(self._d) > + > + def get_array_len(self): > + """ > + Return the len of an array or a negative > + value on error. > + """ > + return _bt_ctf_get_array_len(self._d) > + > + def get_uint64(self): > + """ > + Return the value associated with the field. > + If the field does not exist or is not of the type requested, > + the value returned is undefined. To check if an error occured, > + use the ctf.field_error() function after accessing a field. > + """ > + return _bt_ctf_get_uint64(self._d) > + > + def get_int64(self): > + """ > + Return the value associated with the field. > + If the field does not exist or is not of the type requested, > + the value returned is undefined. To check if an error occured, > + use the ctf.field_error() function after accessing a field. > + """ > + return _bt_ctf_get_int64(self._d) > + > + def get_char_array(self): > + """ > + Return the value associated with the field. > + If the field does not exist or is not of the type requested, > + the value returned is undefined. To check if an error occured, > + use the ctf.field_error() function after accessing a field. > + """ > + return _bt_ctf_get_char_array(self._d) > + > + def get_str(self): > + """ > + Return the value associated with the field. > + If the field does not exist or is not of the type requested, > + the value returned is undefined. To check if an error occured, > + use the ctf.field_error() function after accessing a field. > + """ > + return _bt_ctf_get_string(self._d) > + > + > + class EventDecl(object): > + """Event declaration class. Do not instantiate.""" > + > + def __init__(self): > + raise NotImplementedError("ctf.EventDecl cannot be instantiated") > + > + def __repr__(self): > + return "Babeltrace EventDecl: name {}".format(self.get_name()) > + > + def get_name(self): > + """Return the name of the event or None on error""" > + return _bt_ctf_get_decl_event_name(self._d) > + > + def get_decl_fields(self, scope): > + """ > + Return a list of ctf.FieldDecl > + Return None on error. > + """ > + ptr_list = _by_python_field_decl_listcaller(self._d, scope) > + > + if ptr_list is None: > + return None > + > + decl_list = [] > + i = 0 > + while True: > + tmp = ctf.FieldDecl.__new__(ctf.FieldDecl) > + tmp._d = _bt_python_field_decl_one_from_list( > + ptr_list, i) > + > + if tmp._d is None: > + #Last item of list is None > + break > + > + decl_list.append(tmp) > + i += 1 > + return decl_list > + > + > + class FieldDecl(object): > + """Field declaration class. Do not instantiate.""" > + > + def __init__(self): > + raise NotImplementedError("ctf.FieldDecl cannot be instantiated") > + > + def __repr__(self): > + return "Babeltrace FieldDecl: name {}".format(self.get_name()) > + > + def get_name(self): > + """Return the name of a FieldDecl or None on error""" > + return _bt_ctf_get_decl_field_name(self._d) > + > + > + @staticmethod > + def field_error(): > + """ > + Return the last error code encountered while > + accessing a field and reset the error flag. > + Return 0 if no error, a negative value otherwise. > + """ > + return _bt_ctf_field_get_error() > + > + @staticmethod > + def get_event_decl_list(trace_handle, context): > + """ > + Return a list of ctf.EventDecl > + Return None on error. > + """ > + try: > + handle_id = trace_handle._id > + except AttributeError: > + raise TypeError("in get_event_decl_list, " > + "argument 1 must be a TraceHandle instance") > + try: > + ptr_list = _bt_python_event_decl_listcaller(handle_id, context._c) > + except AttributeError: > + raise TypeError("in get_event_decl_list, " > + "argument 2 must be a Context instance") > + > + if ptr_list is None: > + return None > + > + decl_list = [] > + i = 0 > + while True: > + tmp = ctf.EventDecl.__new__(ctf.EventDecl) > + tmp._d = _bt_python_decl_one_from_list(ptr_list, i) > + > + if tmp._d is None: > + #Last item of list is None > + break > + > + decl_list.append(tmp) > + i += 1 > + return decl_list > + > +%} > + > + > + > +// ================================================================= > +// NEW FUNCTIONS > +// File and list-related > +// python-complements.h > +// ================================================================= > + > +%include python-complements.c > + > +%pythoncode %{ > + > +class File(object): > + """ > + Open a file for babeltrace. > + > + file_path is a string containing the path or None to use the > + standard output in writing mode. > + > + The mode can be 'r', 'w' or 'a' for reading (default), writing or > + appending. The file will be created if it doesn't exist when > + opened for writing or appending; it will be truncated when opened > + for writing. Add a 'b' to the mode for binary files. Add a '+' > + to the mode to allow simultaneous reading and writing. > + """ > + > + def __new__(cls, file_path, mode='r'): > + # __new__ is used to control the return value > + # as the File class should return None > + # if _bt_file_open returns NULL > + > + # Type check > + if file_path is not None and type(file_path) is not str: > + raise TypeError("in method __init__, argument 2 of type 'str'") > + if type(mode) is not str: > + raise TypeError("in method __init__, argument 3 of type 'str'") > + > + # Opening file > + file_ptr = _bt_file_open(file_path, mode) > + if file_ptr is None: > + return None > + > + # Class instantiation > + file_inst = super(File, cls).__new__(cls) > + file_inst._file = file_ptr > + return file_inst > + > + def __init__(self, file_path, mode='r'): > + self._opened = True > + self._use_stdout = False > + > + if file_path is None: > + # use stdout > + file_path = "stdout" > + mode = 'w' > + self._use_stdout = True > + > + self._file_path = file_path > + self._mode = mode > + > + def __del__(self): > + self.close() > + > + def __repr__(self): > + if self._opened: > + stat = 'opened' > + else: > + stat = 'closed' > + return "{} babeltrace File; file_path('{}'), mode('{}')".format( > + stat, self._file_path, self._mode) > + > + def close(self): > + """Close the file. Is also called using del.""" > + if self._opened and not self._use_stdout: > + _bt_file_close(self._file) > + self._opened = False > +%} > diff --git a/bindings/python/examples/babeltrace_and_lttng.py b/bindings/python/examples/babeltrace_and_lttng.py > new file mode 100644 > index 0000000..ef0e35c > --- /dev/null > +++ b/bindings/python/examples/babeltrace_and_lttng.py > @@ -0,0 +1,107 @@ > +# This script uses both lttng-tools and babeltrace > +# python modules. It creates a session, enables > +# events, starts tracing for 2 seconds, stops tracing, > +# destroys the session and outputs the trace in the > +# specified output file. > +# > +# WARNING: will destroy any existing trace having > +# the same name as ses_name > + > + > +# ------------------------------------------------------ > +ses_name = "babeltrace-lttng-test" > +trace_path = "/lttng-traces/babeltrace-lttng-trace/" > +out_file = "babeltrace-lttng-trace-text-output.txt" > +# ------------------------------------------------------ > + > + > +import time > +try: > + import babeltrace, lttng > +except ImportError: > + raise ImportError( "both babeltrace and lttng-tools " > + "python modules must be installed" ) > + > + > +# Errors to raise if something goes wrong > +class LTTngError(Exception): > + pass > +class BabeltraceError(Exception): > + pass > + > + > +# LTTNG-TOOLS > + > +# Making sure session does not already exist > +lttng.destroy(ses_name) > + > +# Creating a new session and handle > +ret = lttng.create(ses_name,trace_path) > +if ret < 0: > + raise LTTngError(lttng.strerror(ret)) > + > +han = None > +han = lttng.Handle(ses_name, lttng.Domain()) > +if han is None: > + raise LTTngError("Handle not created") > + > + > +# Enabling all events > +ret = lttng.enable_event(han, lttng.Event(), None) > +if ret < 0: > + raise LTTngError(lttng.strerror(ret)) > + > + > +# Start, wait, stop > +ret = lttng.start(ses_name) > +if ret < 0: > + raise LTTngError(lttng.strerror(ret)) > +print("Tracing...") > +time.sleep(2) > +print("Stopped.") > +ret = lttng.stop(ses_name) > +if ret < 0: > + raise LTTngError(lttng.strerror(ret)) > + > + > +# Destroying tracing session > +ret = lttng.destroy(ses_name) > +if ret < 0: > + raise LTTngError(lttng.strerror(ret)) > + > + > +# BABELTRACE > + > +# Create context and add trace: > +ctx = babeltrace.Context() > +ret = ctx.add_trace(trace_path + "/kernel", "ctf") > +if ret is None: > + raise BabeltraceError("Error adding trace") > + > +# Iterator setup > +bp = babeltrace.IterPos(babeltrace.SEEK_BEGIN) > +ctf_it = babeltrace.ctf.Iterator(ctx,bp) > + > +# Reading events from trace > +# and outputting timestamps and event names > +# in out_file > +print("Writing trace file...") > +output = open(out_file, "wt") > + > +event = ctf_it.read_event() > +while(event is not None): > + output.write("TS: {}, {} : {}\n".format(event.get_timestamp(), > + event.get_cycles(), event.get_name())) > + > + # Next event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > +# Closing file > +output.close() > + > +# Destroying dynamic elements > +del ctf_it, han > +print("Done.") > diff --git a/bindings/python/examples/eventcount.py b/bindings/python/examples/eventcount.py > new file mode 100644 > index 0000000..2a63b74 > --- /dev/null > +++ b/bindings/python/examples/eventcount.py > @@ -0,0 +1,66 @@ > +# The script prints a count of specified events and > +# their related tid's in a given trace. > +# The trace needs TID context (lttng add-context -k -t tid) > + > +import sys > +from babeltrace import * > +from output_format_modules.pprint_table import pprint_table as pprint > + > +if len(sys.argv) < 3: > + raise TypeError("Usage: python eventcount.py event1 [event2 ...] path/to/trace") > + > +ctx = Context() > +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +counts = {} > + > +# Setting iterator > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx, bp) > + > +# Reading events > +event = ctf_it.read_event() > +while(event is not None): > + for event_type in sys.argv[1:len(sys.argv)-1]: > + if event_type == event.get_name(): > + > + # Getting scope definition > + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) > + if sco is None: > + print("ERROR: Cannot get definition scope for {}".format( > + event.get_name())) > + continue > + > + # Getting TID > + tid_field = event.get_field(sco, "_tid") > + tid = tid_field.get_int64() > + > + if ctf.field_error(): > + print("ERROR: Missing TID info for {}".format( > + event.get_name())) > + continue > + > + tmp = (tid, event.get_name()) > + > + if tmp in counts: > + counts[tmp] += 1 > + else: > + counts[tmp] = 1 > + > + # Next event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > +del ctf_it > + > +# Appending data to table for output > +table = [] > +for item in counts: > + table.append([item[0], item[1], counts[item]]) > +table = sorted(table) > +table.insert(0,["TID", "EVENT", "COUNT"]) > +pprint(table, 2) > diff --git a/bindings/python/examples/eventcountlist.py b/bindings/python/examples/eventcountlist.py > new file mode 100644 > index 0000000..800996c > --- /dev/null > +++ b/bindings/python/examples/eventcountlist.py > @@ -0,0 +1,65 @@ > +# The script prints a count and rate of events. > +# It also outputs a bar graph of count per event, using the cairoplot module. > + > +import sys > +from babeltrace import * > +from output_format_modules import cairoplot > +from output_format_modules.pprint_table import pprint_table as pprint > + > +# Check for path arg: > +if len(sys.argv) < 2: > + raise TypeError("Usage: python eventcountlist.py path/to/trace") > + > +ctx = Context() > +ret = ctx.add_trace(sys.argv[1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +# Events and their assossiated count > +# will be stored as a dict: > +events_count = {} > + > +# Setting iterator: > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx,bp) > + > +prev_event = None > +event = ctf_it.read_event() > + > +start_time = event.get_timestamp() > + > +# Reading events: > +while(event is not None): > + if event.get_name() in events_count: > + events_count[event.get_name()] += 1 > + else: > + events_count[event.get_name()] = 1 > + > + ret = ctf_it.next() > + if ret < 0: > + break > + else: > + prev_event = event > + event = ctf_it.read_event() > + > +if event: > + total_time = event.get_timestamp() - start_time > +else: > + total_time = prev_event.get_timestamp() - start_time > + > +del ctf_it > + > +# Printing encountered events with respective count and rate: > +print("Total time: {} ns".format(total_time)) > +table = [["EVENT", "COUNT", "RATE (Hz)"]] > +for item in sorted(events_count.iterkeys()): > + tmp = [item, events_count[item], > + events_count[item]/(total_time/1000000000.0)] > + table.append(tmp) > +pprint(table) > + > +# Exporting data as bar graph > +cairoplot.vertical_bar_plot ( 'eventcountlist.svg', events_count, 50+85*len(events_count), > + 800, border = 20, display_values = True, grid = True, > + rounded_corners = True, > + x_labels = sorted(events_count.keys()) ) > diff --git a/bindings/python/examples/events_per_cpu.py b/bindings/python/examples/events_per_cpu.py > new file mode 100644 > index 0000000..09cf0e3 > --- /dev/null > +++ b/bindings/python/examples/events_per_cpu.py > @@ -0,0 +1,81 @@ > +# The script opens a trace and prints out CPU statistics > +# for the given trace (event count per CPU, total active > +# time and % of time processing events). > +# It also outputs a .txt file showing each time interval > +# (since the beginning of the trace) in which each CPU > +# was active and the corresponding event. > + > +import sys, multiprocessing > +from output_format_modules.pprint_table import pprint_table as pprint > +from babeltrace import * > + > +if len(sys.argv) < 2: > + raise TypeError("Usage: python events_per_cpu.py path/to/trace") > + > +# Adding trace > +ctx = Context() > +ret = ctx.add_trace(sys.argv[1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +cpu_usage = [] > +nbEvents = 0 > +i = 0 > +while i < multiprocessing.cpu_count(): > + cpu_usage.append([]) > + i += 1 > + > +# Setting iterator > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx, bp) > + > +# Reading events > +event = ctf_it.read_event() > +start_time = event.get_timestamp() > + > +while(event is not None): > + > + event_name = event.get_name() > + ts = event.get_timestamp() > + > + # Getting cpu_id > + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) > + field = event.get_field(scope, "cpu_id") > + cpu_id = field.get_uint64() > + if ctf.field_error(): > + print("ERROR: Missing cpu_id info for {}".format(event.get_name())) > + else: > + cpu_usage[cpu_id].append( (int(ts), event_name) ) > + nbEvents += 1 > + > + # Next Event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > + > +# Outputting > +table = [] > +output = open("events_per_cpu.txt", "wt") > +output.write("(timestamp, event)\n") > + > +for cpu in range(len(cpu_usage)): > + # Setting table > + event_str = str(100.0 * len(cpu_usage[cpu]) / nbEvents) + '000' > + # % is printed with 2 decimals > + table.append([cpu, len(cpu_usage[cpu]), event_str[0:event_str.find('.') + 3] + ' %']) > + > + # Writing to file > + output.write("\n\n\n----------------------\n") > + output.write("CPU {}\n\n".format(cpu)) > + for event in cpu_usage[cpu]: > + output.write(str(event) + '\n') > + > +# Printing table > +table.insert(0, ["CPU ID", "EVENT COUNT", "TRACE EVENT %"]) > +pprint(table) > +print("Total event count: {}".format(nbEvents)) > +print("Total trace time: {} ns".format(ts - start_time)) > + > +output.close() > diff --git a/bindings/python/examples/example-api-test.py b/bindings/python/examples/example-api-test.py > new file mode 100644 > index 0000000..0cfc3ed > --- /dev/null > +++ b/bindings/python/examples/example-api-test.py > @@ -0,0 +1,59 @@ > +# This example uses the babeltrace python module > +# to partially test the api. > + > +import sys > +from babeltrace import * > + > +# Check for path arg: > +if len(sys.argv) < 2: > + raise TypeError("Usage: python example-api-test.py path/to/file") > + > +# Create context and add trace: > +ctx = Context() > +trace_handle = ctx.add_trace(sys.argv[1], "ctf") > +if trace_handle is None: > + raise IOError("Error adding trace") > + > +# Listing events > +lst = ctf.get_event_decl_list(trace_handle, ctx) > +print("--- Event list ---") > +for item in lst: > + print("event : {}".format(item.get_name())) > +print("--- Done ---") > + > +# Iter trace > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx,bp) > +event = ctf_it.read_event() > + > +while(event is not None): > + print("TS: {}, {} : {}".format(event.get_timestamp(), > + event.get_cycles(), event.get_name())) > + > + if event.get_name() == "sched_switch": > + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) > + prev_field = event.get_field(sco, "_prev_comm") > + prev_comm = prev_field.get_char_array() > + > + if ctf.field_error(): > + print("ERROR: Missing prev_comm context info") > + else: > + print("sched_switch prev_comm: {}".format(prev_comm)) > + > + if event.get_name() == "exit_syscall": > + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) > + ret_field = event.get_field(sco, "_ret") > + ret_code = ret_field.get_int64() > + > + if ctf.field_error(): > + print("ERROR: Unable to extract ret") > + else: > + print("exit_syscall ret: {}".format(ret_code)) > + > + ret = ctf_it.next() > + if ret < 0: > + break > + else: > + event = ctf_it.read_event() > + > +del ctf_it > diff --git a/bindings/python/examples/histogram.py b/bindings/python/examples/histogram.py > new file mode 100644 > index 0000000..a24c00d > --- /dev/null > +++ b/bindings/python/examples/histogram.py > @@ -0,0 +1,121 @@ > +# The script checks the number of events in the trace > +# and outputs a table and a .svg histogram for the specified > +# range (microseconds) or the total trace if no range specified. > +# The graph is generated using the cairoplot module. > + > +import sys > +from babeltrace import * > +from output_format_modules import cairoplot > +from output_format_modules.pprint_table import pprint_table as pprint > + > +# ------------------------------------------------ > +# Output settings > + > +# number of intervals: > +nbDiv = 25 # Should not be over 150 > + # for usable graph output > + > +# table output stream (file-like object): > +out = sys.stdout > +# ------------------------------------------------- > + > +if len(sys.argv) < 2 or len(sys.argv) > 4: > + raise TypeError("Usage: python histogram.py [ start_time [end_time] ] path/to/trace") > + > +ctx = Context() > +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +# Check when to start/stop graphing > +sinceBegin = True > +beginTime = 0.0 > +if len(sys.argv) > 2: > + sinceBegin = False > + beginTime = float(sys.argv[1]) > +untilEnd = True > +if len(sys.argv) == 4: > + untilEnd = False > + > +# Setting iterator > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx, bp) > + > +# Reading events > +event = ctf_it.read_event() > +start_time = event.get_timestamp() > +time = 0 > +count = {} > + > +while(event is not None): > + # Microsec. > + time = (event.get_timestamp() - start_time)/1000.0 > + > + # Check if in range > + if not sinceBegin: > + if time < beginTime: > + # Next Event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + continue > + if not untilEnd: > + if time > float(sys.argv[2]): > + break > + > + # Counting events per timestamp: > + if time in count: > + count[time] += 1 > + else: > + count[time] = 1 > + > + # Next Event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > +del ctf_it > + > +# Setting data for output > +interval = (time - beginTime)/nbDiv > +div_begin_time = beginTime > +div_end_time = beginTime + interval > +data = {} > + > +# Prefix for string sorting, considering > +# there should not be over 150 intervals. > +# This would work up to 9999 intervals. > +# If needed, add zeros. > +prefix = 0.0001 > + > +while div_end_time <= time: > + key = str(prefix) + '[' + str(div_begin_time) + ';' + str(div_end_time) + '[' > + for tmp in count: > + if tmp >= div_begin_time and tmp < div_end_time: > + if key in data: > + data[key] += count[tmp] > + else: > + data[key] = count[tmp] > + if not key in data: > + data[key] = 0 > + div_begin_time = div_end_time > + div_end_time += interval > + # Prefix increment > + prefix += 0.001 > + > +table = [] > +x_labels = [] > +for key in sorted(data): > + table.append([key[key.find('['):], data[key]]) > + x_labels.append(key[key.find('['):]) > + > +# Table output > +table.insert(0, ["INTERVAL (us)", "COUNT"]) > +pprint(table, 1, out) > + > +# Graph output > +cairoplot.vertical_bar_plot ( 'histogram.svg', data, 50 + 150*nbDiv, 50*nbDiv, > + border = 20, display_values = True, grid = True, > + x_labels = x_labels, rounded_corners = True ) > diff --git a/bindings/python/examples/output_format_modules/__init__.py b/bindings/python/examples/output_format_modules/__init__.py > new file mode 100644 > index 0000000..e69de29 > diff --git a/bindings/python/examples/output_format_modules/cairoplot.py b/bindings/python/examples/output_format_modules/cairoplot.py > new file mode 100755 > index 0000000..a27113f > --- /dev/null > +++ b/bindings/python/examples/output_format_modules/cairoplot.py > @@ -0,0 +1,2336 @@ > +?#!/usr/bin/env python > +# -*- coding: utf-8 -*- > + > +# CairoPlot.py > +# > +# Copyright (c) 2008 Rodrigo Moreira Ara?jo > +# > +# Author: Rodrigo Moreiro Araujo > +# > +# This program is free software; you can redistribute it and/or > +# modify it under the terms of the GNU Lesser General Public License > +# as published by the Free Software Foundation; either version 2 of > +# the License, or (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU Lesser General Public > +# License along with this program; if not, write to the Free Software > +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 > +# USA > + > +#Contributor: Jo?o S. O. Bueno > + > +#TODO: review BarPlot Code > +#TODO: x_label colision problem on Horizontal Bar Plot > +#TODO: y_label's eat too much space on HBP > + > + > +__version__ = 1.2 > + > +import cairo > +import math > +import random > +from series import Series, Group, Data > + > +HORZ = 0 > +VERT = 1 > +NORM = 2 > + > +COLORS = {"red" : (1.0,0.0,0.0,1.0), "lime" : (0.0,1.0,0.0,1.0), "blue" : (0.0,0.0,1.0,1.0), > + "maroon" : (0.5,0.0,0.0,1.0), "green" : (0.0,0.5,0.0,1.0), "navy" : (0.0,0.0,0.5,1.0), > + "yellow" : (1.0,1.0,0.0,1.0), "magenta" : (1.0,0.0,1.0,1.0), "cyan" : (0.0,1.0,1.0,1.0), > + "orange" : (1.0,0.5,0.0,1.0), "white" : (1.0,1.0,1.0,1.0), "black" : (0.0,0.0,0.0,1.0), > + "gray" : (0.5,0.5,0.5,1.0), "light_gray" : (0.9,0.9,0.9,1.0), > + "transparent" : (0.0,0.0,0.0,0.0)} > + > +THEMES = {"black_red" : [(0.0,0.0,0.0,1.0), (1.0,0.0,0.0,1.0)], > + "red_green_blue" : [(1.0,0.0,0.0,1.0), (0.0,1.0,0.0,1.0), (0.0,0.0,1.0,1.0)], > + "red_orange_yellow" : [(1.0,0.2,0.0,1.0), (1.0,0.7,0.0,1.0), (1.0,1.0,0.0,1.0)], > + "yellow_orange_red" : [(1.0,1.0,0.0,1.0), (1.0,0.7,0.0,1.0), (1.0,0.2,0.0,1.0)], > + "rainbow" : [(1.0,0.0,0.0,1.0), (1.0,0.5,0.0,1.0), (1.0,1.0,0.0,1.0), (0.0,1.0,0.0,1.0), (0.0,0.0,1.0,1.0), (0.3, 0.0, 0.5,1.0), (0.5, 0.0, 1.0, 1.0)]} > + > +def colors_from_theme( theme, series_length, mode = 'solid' ): > + colors = [] > + if theme not in THEMES.keys() : > + raise Exception, "Theme not defined" > + color_steps = THEMES[theme] > + n_colors = len(color_steps) > + if series_length <= n_colors: > + colors = [color + tuple([mode]) for color in color_steps[0:n_colors]] > + else: > + iterations = [(series_length - n_colors)/(n_colors - 1) for i in color_steps[:-1]] > + over_iterations = (series_length - n_colors) % (n_colors - 1) > + for i in range(n_colors - 1): > + if over_iterations <= 0: > + break > + iterations[i] += 1 > + over_iterations -= 1 > + for index,color in enumerate(color_steps[:-1]): > + colors.append(color + tuple([mode])) > + if iterations[index] == 0: > + continue > + next_color = color_steps[index+1] > + color_step = ((next_color[0] - color[0])/(iterations[index] + 1), > + (next_color[1] - color[1])/(iterations[index] + 1), > + (next_color[2] - color[2])/(iterations[index] + 1), > + (next_color[3] - color[3])/(iterations[index] + 1)) > + for i in range( iterations[index] ): > + colors.append((color[0] + color_step[0]*(i+1), > + color[1] + color_step[1]*(i+1), > + color[2] + color_step[2]*(i+1), > + color[3] + color_step[3]*(i+1), > + mode)) > + colors.append(color_steps[-1] + tuple([mode])) > + return colors > + > + > +def other_direction(direction): > + "explicit is better than implicit" > + if direction == HORZ: > + return VERT > + else: > + return HORZ > + > +#Class definition > + > +class Plot(object): > + def __init__(self, > + surface=None, > + data=None, > + width=640, > + height=480, > + background=None, > + border = 0, > + x_labels = None, > + y_labels = None, > + series_colors = None): > + random.seed(2) > + self.create_surface(surface, width, height) > + self.dimensions = {} > + self.dimensions[HORZ] = width > + self.dimensions[VERT] = height > + self.context = cairo.Context(self.surface) > + self.labels={} > + self.labels[HORZ] = x_labels > + self.labels[VERT] = y_labels > + self.load_series(data, x_labels, y_labels, series_colors) > + self.font_size = 10 > + self.set_background (background) > + self.border = border > + self.borders = {} > + self.line_color = (0.5, 0.5, 0.5) > + self.line_width = 0.5 > + self.label_color = (0.0, 0.0, 0.0) > + self.grid_color = (0.8, 0.8, 0.8) > + > + def create_surface(self, surface, width=None, height=None): > + self.filename = None > + if isinstance(surface, cairo.Surface): > + self.surface = surface > + return > + if not type(surface) in (str, unicode): > + raise TypeError("Surface should be either a Cairo surface or a filename, not %s" % surface) > + sufix = surface.rsplit(".")[-1].lower() > + self.filename = surface > + if sufix == "png": > + self.surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, width, height) > + elif sufix == "ps": > + self.surface = cairo.PSSurface(surface, width, height) > + elif sufix == "pdf": > + self.surface = cairo.PSSurface(surface, width, height) > + else: > + if sufix != "svg": > + self.filename += ".svg" > + self.surface = cairo.SVGSurface(self.filename, width, height) > + > + def commit(self): > + try: > + self.context.show_page() > + if self.filename and self.filename.endswith(".png"): > + self.surface.write_to_png(self.filename) > + else: > + self.surface.finish() > + except cairo.Error: > + pass > + > + def load_series (self, data, x_labels=None, y_labels=None, series_colors=None): > + self.series_labels = [] > + self.series = None > + > + #The pretty way > + #if not isinstance(data, Series): > + # # Not an instance of Series > + # self.series = Series(data) > + #else: > + # self.series = data > + # > + #self.series_labels = self.series.get_names() > + > + #TODO: Remove on next version > + # The ugly way, keeping retrocompatibility... > + if callable(data) or type(data) is list and callable(data[0]): # Lambda or List of lambdas > + self.series = data > + self.series_labels = None > + elif isinstance(data, Series): # Instance of Series > + self.series = data > + self.series_labels = data.get_names() > + else: # Anything else > + self.series = Series(data) > + self.series_labels = self.series.get_names() > + > + #TODO: allow user passed series_widths > + self.series_widths = [1.0 for group in self.series] > + > + #TODO: Remove on next version > + self.process_colors( series_colors ) > + > + def process_colors( self, series_colors, length = None, mode = 'solid' ): > + #series_colors might be None, a theme, a string of colors names or a list of color tuples > + if length is None : > + length = len( self.series.to_list() ) > + > + #no colors passed > + if not series_colors: > + #Randomize colors > + self.series_colors = [ [random.random() for i in range(3)] + [1.0, mode] for series in range( length ) ] > + else: > + #Just theme pattern > + if not hasattr( series_colors, "__iter__" ): > + theme = series_colors > + self.series_colors = colors_from_theme( theme.lower(), length ) > + > + #Theme pattern and mode > + elif not hasattr(series_colors, '__delitem__') and not hasattr( series_colors[0], "__iter__" ): > + theme = series_colors[0] > + mode = series_colors[1] > + self.series_colors = colors_from_theme( theme.lower(), length, mode ) > + > + #List > + else: > + self.series_colors = series_colors > + for index, color in enumerate( self.series_colors ): > + #element is a color name > + if not hasattr(color, "__iter__"): > + self.series_colors[index] = COLORS[color.lower()] + tuple([mode]) > + #element is rgb tuple instead of rgba > + elif len( color ) == 3 : > + self.series_colors[index] += (1.0,mode) > + #element has 4 elements, might be rgba tuple or rgb tuple with mode > + elif len( color ) == 4 : > + #last element is mode > + if not hasattr(color[3], "__iter__"): > + self.series_colors[index] += tuple([color[3]]) > + self.series_colors[index][3] = 1.0 > + #last element is alpha > + else: > + self.series_colors[index] += tuple([mode]) > + > + def get_width(self): > + return self.surface.get_width() > + > + def get_height(self): > + return self.surface.get_height() > + > + def set_background(self, background): > + if background is None: > + self.background = (0.0,0.0,0.0,0.0) > + elif type(background) in (cairo.LinearGradient, tuple): > + self.background = background > + elif not hasattr(background,"__iter__"): > + colors = background.split(" ") > + if len(colors) == 1 and colors[0] in COLORS: > + self.background = COLORS[background] > + elif len(colors) > 1: > + self.background = cairo.LinearGradient(self.dimensions[HORZ] / 2, 0, self.dimensions[HORZ] / 2, self.dimensions[VERT]) > + for index,color in enumerate(colors): > + self.background.add_color_stop_rgba(float(index)/(len(colors)-1),*COLORS[color]) > + else: > + raise TypeError ("Background should be either cairo.LinearGradient or a 3/4-tuple, not %s" % type(background)) > + > + def render_background(self): > + if isinstance(self.background, cairo.LinearGradient): > + self.context.set_source(self.background) > + else: > + self.context.set_source_rgba(*self.background) > + self.context.rectangle(0,0, self.dimensions[HORZ], self.dimensions[VERT]) > + self.context.fill() > + > + def render_bounding_box(self): > + self.context.set_source_rgba(*self.line_color) > + self.context.set_line_width(self.line_width) > + self.context.rectangle(self.border, self.border, > + self.dimensions[HORZ] - 2 * self.border, > + self.dimensions[VERT] - 2 * self.border) > + self.context.stroke() > + > + def render(self): > + pass > + > +class ScatterPlot( Plot ): > + def __init__(self, > + surface=None, > + data=None, > + errorx=None, > + errory=None, > + width=640, > + height=480, > + background=None, > + border=0, > + axis = False, > + dash = False, > + discrete = False, > + dots = 0, > + grid = False, > + series_legend = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + z_bounds = None, > + x_title = None, > + y_title = None, > + series_colors = None, > + circle_colors = None ): > + > + self.bounds = {} > + self.bounds[HORZ] = x_bounds > + self.bounds[VERT] = y_bounds > + self.bounds[NORM] = z_bounds > + self.titles = {} > + self.titles[HORZ] = x_title > + self.titles[VERT] = y_title > + self.max_value = {} > + self.axis = axis > + self.discrete = discrete > + self.dots = dots > + self.grid = grid > + self.series_legend = series_legend > + self.variable_radius = False > + self.x_label_angle = math.pi / 2.5 > + self.circle_colors = circle_colors > + > + Plot.__init__(self, surface, data, width, height, background, border, x_labels, y_labels, series_colors) > + > + self.dash = None > + if dash: > + if hasattr(dash, "keys"): > + self.dash = [dash[key] for key in self.series_labels] > + elif max([hasattr(item,'__delitem__') for item in data]) : > + self.dash = dash > + else: > + self.dash = [dash] > + > + self.load_errors(errorx, errory) > + > + def convert_list_to_tuple(self, data): > + #Data must be converted from lists of coordinates to a single > + # list of tuples > + out_data = zip(*data) > + if len(data) == 3: > + self.variable_radius = True > + return out_data > + > + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): > + #TODO: In cairoplot 2.0 keep only the Series instances > + > + # Convert Data and Group to Series > + if isinstance(data, Data) or isinstance(data, Group): > + data = Series(data) > + > + # Series > + if isinstance(data, Series): > + for group in data: > + for item in group: > + if len(item) is 3: > + self.variable_radius = True > + > + #Dictionary with lists > + if hasattr(data, "keys") : > + if hasattr( data.values()[0][0], "__delitem__" ) : > + for key in data.keys() : > + data[key] = self.convert_list_to_tuple(data[key]) > + elif len(data.values()[0][0]) == 3: > + self.variable_radius = True > + #List > + elif hasattr(data[0], "__delitem__") : > + #List of lists > + if hasattr(data[0][0], "__delitem__") : > + for index,value in enumerate(data) : > + data[index] = self.convert_list_to_tuple(value) > + #List > + elif type(data[0][0]) != type((0,0)): > + data = self.convert_list_to_tuple(data) > + #Three dimensional data > + elif len(data[0][0]) == 3: > + self.variable_radius = True > + > + #List with three dimensional tuples > + elif len(data[0]) == 3: > + self.variable_radius = True > + Plot.load_series(self, data, x_labels, y_labels, series_colors) > + self.calc_boundaries() > + self.calc_labels() > + > + def load_errors(self, errorx, errory): > + self.errors = None > + if errorx == None and errory == None: > + return > + self.errors = {} > + self.errors[HORZ] = None > + self.errors[VERT] = None > + #asimetric errors > + if errorx and hasattr(errorx[0], "__delitem__"): > + self.errors[HORZ] = errorx > + #simetric errors > + elif errorx: > + self.errors[HORZ] = [errorx] > + #asimetric errors > + if errory and hasattr(errory[0], "__delitem__"): > + self.errors[VERT] = errory > + #simetric errors > + elif errory: > + self.errors[VERT] = [errory] > + > + def calc_labels(self): > + if not self.labels[HORZ]: > + amplitude = self.bounds[HORZ][1] - self.bounds[HORZ][0] > + if amplitude % 10: #if horizontal labels need floating points > + self.labels[HORZ] = ["%.2lf" % (float(self.bounds[HORZ][0] + (amplitude * i / 10.0))) for i in range(11) ] > + else: > + self.labels[HORZ] = ["%d" % (int(self.bounds[HORZ][0] + (amplitude * i / 10.0))) for i in range(11) ] > + if not self.labels[VERT]: > + amplitude = self.bounds[VERT][1] - self.bounds[VERT][0] > + if amplitude % 10: #if vertical labels need floating points > + self.labels[VERT] = ["%.2lf" % (float(self.bounds[VERT][0] + (amplitude * i / 10.0))) for i in range(11) ] > + else: > + self.labels[VERT] = ["%d" % (int(self.bounds[VERT][0] + (amplitude * i / 10.0))) for i in range(11) ] > + > + def calc_extents(self, direction): > + self.context.set_font_size(self.font_size * 0.8) > + self.max_value[direction] = max(self.context.text_extents(item)[2] for item in self.labels[direction]) > + self.borders[other_direction(direction)] = self.max_value[direction] + self.border + 20 > + > + def calc_boundaries(self): > + #HORZ = 0, VERT = 1, NORM = 2 > + min_data_value = [0,0,0] > + max_data_value = [0,0,0] > + > + for group in self.series: > + if type(group[0].content) in (int, float, long): > + group = [Data((index, item.content)) for index,item in enumerate(group)] > + > + for point in group: > + for index, item in enumerate(point.content): > + if item > max_data_value[index]: > + max_data_value[index] = item > + elif item < min_data_value[index]: > + min_data_value[index] = item > + > + if not self.bounds[HORZ]: > + self.bounds[HORZ] = (min_data_value[HORZ], max_data_value[HORZ]) > + if not self.bounds[VERT]: > + self.bounds[VERT] = (min_data_value[VERT], max_data_value[VERT]) > + if not self.bounds[NORM]: > + self.bounds[NORM] = (min_data_value[NORM], max_data_value[NORM]) > + > + def calc_all_extents(self): > + self.calc_extents(HORZ) > + self.calc_extents(VERT) > + > + self.plot_height = self.dimensions[VERT] - 2 * self.borders[VERT] > + self.plot_width = self.dimensions[HORZ] - 2* self.borders[HORZ] > + > + self.plot_top = self.dimensions[VERT] - self.borders[VERT] > + > + def calc_steps(self): > + #Calculates all the x, y, z and color steps > + series_amplitude = [self.bounds[index][1] - self.bounds[index][0] for index in range(3)] > + > + if series_amplitude[HORZ]: > + self.horizontal_step = float (self.plot_width) / series_amplitude[HORZ] > + else: > + self.horizontal_step = 0.00 > + > + if series_amplitude[VERT]: > + self.vertical_step = float (self.plot_height) / series_amplitude[VERT] > + else: > + self.vertical_step = 0.00 > + > + if series_amplitude[NORM]: > + if self.variable_radius: > + self.z_step = float (self.bounds[NORM][1]) / series_amplitude[NORM] > + if self.circle_colors: > + self.circle_color_step = tuple([float(self.circle_colors[1][i]-self.circle_colors[0][i])/series_amplitude[NORM] for i in range(4)]) > + else: > + self.z_step = 0.00 > + self.circle_color_step = ( 0.0, 0.0, 0.0, 0.0 ) > + > + def get_circle_color(self, value): > + return tuple( [self.circle_colors[0][i] + value*self.circle_color_step[i] for i in range(4)] ) > + > + def render(self): > + self.calc_all_extents() > + self.calc_steps() > + self.render_background() > + self.render_bounding_box() > + if self.axis: > + self.render_axis() > + if self.grid: > + self.render_grid() > + self.render_labels() > + self.render_plot() > + if self.errors: > + self.render_errors() > + if self.series_legend and self.series_labels: > + self.render_legend() > + > + def render_axis(self): > + #Draws both the axis lines and their titles > + cr = self.context > + cr.set_source_rgba(*self.line_color) > + cr.move_to(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) > + cr.line_to(self.borders[HORZ], self.borders[VERT]) > + cr.stroke() > + > + cr.move_to(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) > + cr.line_to(self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) > + cr.stroke() > + > + cr.set_source_rgba(*self.label_color) > + self.context.set_font_size( 1.2 * self.font_size ) > + if self.titles[HORZ]: > + title_width,title_height = cr.text_extents(self.titles[HORZ])[2:4] > + cr.move_to( self.dimensions[HORZ]/2 - title_width/2, self.borders[VERT] - title_height/2 ) > + cr.show_text( self.titles[HORZ] ) > + > + if self.titles[VERT]: > + title_width,title_height = cr.text_extents(self.titles[VERT])[2:4] > + cr.move_to( self.dimensions[HORZ] - self.borders[HORZ] + title_height/2, self.dimensions[VERT]/2 - title_width/2) > + cr.save() > + cr.rotate( math.pi/2 ) > + cr.show_text( self.titles[VERT] ) > + cr.restore() > + > + def render_grid(self): > + cr = self.context > + horizontal_step = float( self.plot_height ) / ( len( self.labels[VERT] ) - 1 ) > + vertical_step = float( self.plot_width ) / ( len( self.labels[HORZ] ) - 1 ) > + > + x = self.borders[HORZ] + vertical_step > + y = self.plot_top - horizontal_step > + > + for label in self.labels[HORZ][:-1]: > + cr.set_source_rgba(*self.grid_color) > + cr.move_to(x, self.dimensions[VERT] - self.borders[VERT]) > + cr.line_to(x, self.borders[VERT]) > + cr.stroke() > + x += vertical_step > + for label in self.labels[VERT][:-1]: > + cr.set_source_rgba(*self.grid_color) > + cr.move_to(self.borders[HORZ], y) > + cr.line_to(self.dimensions[HORZ] - self.borders[HORZ], y) > + cr.stroke() > + y -= horizontal_step > + > + def render_labels(self): > + self.context.set_font_size(self.font_size * 0.8) > + self.render_horz_labels() > + self.render_vert_labels() > + > + def render_horz_labels(self): > + cr = self.context > + step = float( self.plot_width ) / ( len( self.labels[HORZ] ) - 1 ) > + x = self.borders[HORZ] > + y = self.dimensions[VERT] - self.borders[VERT] + 5 > + > + # store rotation matrix from the initial state > + rotation_matrix = cr.get_matrix() > + rotation_matrix.rotate(self.x_label_angle) > + > + cr.set_source_rgba(*self.label_color) > + > + for item in self.labels[HORZ]: > + width = cr.text_extents(item)[2] > + cr.move_to(x, y) > + cr.save() > + cr.set_matrix(rotation_matrix) > + cr.show_text(item) > + cr.restore() > + x += step > + > + def render_vert_labels(self): > + cr = self.context > + step = ( self.plot_height ) / ( len( self.labels[VERT] ) - 1 ) > + y = self.plot_top > + cr.set_source_rgba(*self.label_color) > + for item in self.labels[VERT]: > + width = cr.text_extents(item)[2] > + cr.move_to(self.borders[HORZ] - width - 5,y) > + cr.show_text(item) > + y -= step > + > + def render_legend(self): > + cr = self.context > + cr.set_font_size(self.font_size) > + cr.set_line_width(self.line_width) > + > + widest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[2]) > + tallest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[3]) > + max_width = self.context.text_extents(widest_word)[2] > + max_height = self.context.text_extents(tallest_word)[3] * 1.1 > + > + color_box_height = max_height / 2 > + color_box_width = color_box_height * 2 > + > + #Draw a bounding box > + bounding_box_width = max_width + color_box_width + 15 > + bounding_box_height = (len(self.series_labels)+0.5) * max_height > + cr.set_source_rgba(1,1,1) > + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - bounding_box_width, self.borders[VERT], > + bounding_box_width, bounding_box_height) > + cr.fill() > + > + cr.set_source_rgba(*self.line_color) > + cr.set_line_width(self.line_width) > + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - bounding_box_width, self.borders[VERT], > + bounding_box_width, bounding_box_height) > + cr.stroke() > + > + for idx,key in enumerate(self.series_labels): > + #Draw color box > + cr.set_source_rgba(*self.series_colors[idx][:4]) > + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - max_width - color_box_width - 10, > + self.borders[VERT] + color_box_height + (idx*max_height) , > + color_box_width, color_box_height) > + cr.fill() > + > + cr.set_source_rgba(0, 0, 0) > + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - max_width - color_box_width - 10, > + self.borders[VERT] + color_box_height + (idx*max_height), > + color_box_width, color_box_height) > + cr.stroke() > + > + #Draw series labels > + cr.set_source_rgba(0, 0, 0) > + cr.move_to(self.dimensions[HORZ] - self.borders[HORZ] - max_width - 5, self.borders[VERT] + ((idx+1)*max_height)) > + cr.show_text(key) > + > + def render_errors(self): > + cr = self.context > + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) > + cr.clip() > + radius = self.dots > + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step > + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step > + for index, group in enumerate(self.series): > + cr.set_source_rgba(*self.series_colors[index][:4]) > + for number, data in enumerate(group): > + x = x0 + self.horizontal_step * data.content[0] > + y = self.dimensions[VERT] - y0 - self.vertical_step * data.content[1] > + if self.errors[HORZ]: > + cr.move_to(x, y) > + x1 = x - self.horizontal_step * self.errors[HORZ][0][number] > + cr.line_to(x1, y) > + cr.line_to(x1, y - radius) > + cr.line_to(x1, y + radius) > + cr.stroke() > + if self.errors[HORZ] and len(self.errors[HORZ]) == 2: > + cr.move_to(x, y) > + x1 = x + self.horizontal_step * self.errors[HORZ][1][number] > + cr.line_to(x1, y) > + cr.line_to(x1, y - radius) > + cr.line_to(x1, y + radius) > + cr.stroke() > + if self.errors[VERT]: > + cr.move_to(x, y) > + y1 = y + self.vertical_step * self.errors[VERT][0][number] > + cr.line_to(x, y1) > + cr.line_to(x - radius, y1) > + cr.line_to(x + radius, y1) > + cr.stroke() > + if self.errors[VERT] and len(self.errors[VERT]) == 2: > + cr.move_to(x, y) > + y1 = y - self.vertical_step * self.errors[VERT][1][number] > + cr.line_to(x, y1) > + cr.line_to(x - radius, y1) > + cr.line_to(x + radius, y1) > + cr.stroke() > + > + > + def render_plot(self): > + cr = self.context > + if self.discrete: > + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) > + cr.clip() > + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step > + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step > + radius = self.dots > + for number, group in enumerate (self.series): > + cr.set_source_rgba(*self.series_colors[number][:4]) > + for data in group : > + if self.variable_radius: > + radius = data.content[2]*self.z_step > + if self.circle_colors: > + cr.set_source_rgba( *self.get_circle_color( data.content[2]) ) > + x = x0 + self.horizontal_step*data.content[0] > + y = y0 + self.vertical_step*data.content[1] > + cr.arc(x, self.dimensions[VERT] - y, radius, 0, 2*math.pi) > + cr.fill() > + else: > + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) > + cr.clip() > + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step > + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step > + radius = self.dots > + for number, group in enumerate (self.series): > + last_data = None > + cr.set_source_rgba(*self.series_colors[number][:4]) > + for data in group : > + x = x0 + self.horizontal_step*data.content[0] > + y = y0 + self.vertical_step*data.content[1] > + if self.dots: > + if self.variable_radius: > + radius = data.content[2]*self.z_step > + cr.arc(x, self.dimensions[VERT] - y, radius, 0, 2*math.pi) > + cr.fill() > + if last_data : > + old_x = x0 + self.horizontal_step*last_data.content[0] > + old_y = y0 + self.vertical_step*last_data.content[1] > + cr.move_to( old_x, self.dimensions[VERT] - old_y ) > + cr.line_to( x, self.dimensions[VERT] - y) > + cr.set_line_width(self.series_widths[number]) > + > + #?Display line as dash line > + if self.dash and self.dash[number]: > + s = self.series_widths[number] > + cr.set_dash([s*3, s*3], 0) > + > + cr.stroke() > + cr.set_dash([]) > + last_data = data > + > +class DotLinePlot(ScatterPlot): > + def __init__(self, > + surface=None, > + data=None, > + width=640, > + height=480, > + background=None, > + border=0, > + axis = False, > + dash = False, > + dots = 0, > + grid = False, > + series_legend = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + x_title = None, > + y_title = None, > + series_colors = None): > + > + ScatterPlot.__init__(self, surface, data, None, None, width, height, background, border, > + axis, dash, False, dots, grid, series_legend, x_labels, y_labels, > + x_bounds, y_bounds, None, x_title, y_title, series_colors, None ) > + > + > + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): > + Plot.load_series(self, data, x_labels, y_labels, series_colors) > + for group in self.series : > + for index,data in enumerate(group): > + group[index].content = (index, data.content) > + > + self.calc_boundaries() > + self.calc_labels() > + > +class FunctionPlot(ScatterPlot): > + def __init__(self, > + surface=None, > + data=None, > + width=640, > + height=480, > + background=None, > + border=0, > + axis = False, > + discrete = False, > + dots = 0, > + grid = False, > + series_legend = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + x_title = None, > + y_title = None, > + series_colors = None, > + step = 1): > + > + self.function = data > + self.step = step > + self.discrete = discrete > + > + data, x_bounds = self.load_series_from_function( self.function, x_bounds ) > + > + ScatterPlot.__init__(self, surface, data, None, None, width, height, background, border, > + axis, False, discrete, dots, grid, series_legend, x_labels, y_labels, > + x_bounds, y_bounds, None, x_title, y_title, series_colors, None ) > + > + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): > + Plot.load_series(self, data, x_labels, y_labels, series_colors) > + > + if len(self.series[0][0]) is 1: > + for group_id, group in enumerate(self.series) : > + for index,data in enumerate(group): > + group[index].content = (self.bounds[HORZ][0] + self.step*index, data.content) > + > + self.calc_boundaries() > + self.calc_labels() > + > + def load_series_from_function( self, function, x_bounds ): > + #TODO: Add the possibility for the user to define multiple functions with different discretization parameters > + > + #This function converts a function, a list of functions or a dictionary > + #of functions into its corresponding array of data > + series = Series() > + > + if isinstance(function, Group) or isinstance(function, Data): > + function = Series(function) > + > + # If is instance of Series > + if isinstance(function, Series): > + # Overwrite any bounds passed by the function > + x_bounds = (function.range[0],function.range[-1]) > + > + #if no bounds are provided > + if x_bounds == None: > + x_bounds = (0,10) > + > + > + #TODO: Finish the dict translation > + if hasattr(function, "keys"): #dictionary: > + for key in function.keys(): > + group = Group(name=key) > + #data[ key ] = [] > + i = x_bounds[0] > + while i <= x_bounds[1] : > + group.add_data(function[ key ](i)) > + #data[ key ].append( function[ key ](i) ) > + i += self.step > + series.add_group(group) > + > + elif hasattr(function, "__delitem__"): #list of functions > + for index,f in enumerate( function ) : > + group = Group() > + #data.append( [] ) > + i = x_bounds[0] > + while i <= x_bounds[1] : > + group.add_data(f(i)) > + #data[ index ].append( f(i) ) > + i += self.step > + series.add_group(group) > + > + elif isinstance(function, Series): # instance of Series > + series = function > + > + else: #function > + group = Group() > + i = x_bounds[0] > + while i <= x_bounds[1] : > + group.add_data(function(i)) > + i += self.step > + series.add_group(group) > + > + > + return series, x_bounds > + > + def calc_labels(self): > + if not self.labels[HORZ]: > + self.labels[HORZ] = [] > + i = self.bounds[HORZ][0] > + while i<=self.bounds[HORZ][1]: > + self.labels[HORZ].append(str(i)) > + i += float(self.bounds[HORZ][1] - self.bounds[HORZ][0])/10 > + ScatterPlot.calc_labels(self) > + > + def render_plot(self): > + if not self.discrete: > + ScatterPlot.render_plot(self) > + else: > + last = None > + cr = self.context > + for number, group in enumerate (self.series): > + cr.set_source_rgba(*self.series_colors[number][:4]) > + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step > + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step > + for data in group: > + x = x0 + self.horizontal_step * data.content[0] > + y = y0 + self.vertical_step * data.content[1] > + cr.move_to(x, self.dimensions[VERT] - y) > + cr.line_to(x, self.plot_top) > + cr.set_line_width(self.series_widths[number]) > + cr.stroke() > + if self.dots: > + cr.new_path() > + cr.arc(x, self.dimensions[VERT] - y, 3, 0, 2.1 * math.pi) > + cr.close_path() > + cr.fill() > + > +class BarPlot(Plot): > + def __init__(self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + border = 0, > + display_values = False, > + grid = False, > + rounded_corners = False, > + stack = False, > + three_dimension = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + series_colors = None, > + main_dir = None): > + > + self.bounds = {} > + self.bounds[HORZ] = x_bounds > + self.bounds[VERT] = y_bounds > + self.display_values = display_values > + self.grid = grid > + self.rounded_corners = rounded_corners > + self.stack = stack > + self.three_dimension = three_dimension > + self.x_label_angle = math.pi / 2.5 > + self.main_dir = main_dir > + self.max_value = {} > + self.plot_dimensions = {} > + self.steps = {} > + self.value_label_color = (0.5,0.5,0.5,1.0) > + > + Plot.__init__(self, surface, data, width, height, background, border, x_labels, y_labels, series_colors) > + > + def load_series(self, data, x_labels = None, y_labels = None, series_colors = None): > + Plot.load_series(self, data, x_labels, y_labels, series_colors) > + self.calc_boundaries() > + > + def process_colors(self, series_colors): > + #Data for a BarPlot might be a List or a List of Lists. > + #On the first case, colors must be generated for all bars, > + #On the second, colors must be generated for each of the inner lists. > + > + #TODO: Didn't get it... > + #if hasattr(self.data[0], '__getitem__'): > + # length = max(len(series) for series in self.data) > + #else: > + # length = len( self.data ) > + > + length = max(len(group) for group in self.series) > + > + Plot.process_colors( self, series_colors, length, 'linear') > + > + def calc_boundaries(self): > + if not self.bounds[self.main_dir]: > + if self.stack: > + max_data_value = max(sum(group.to_list()) for group in self.series) > + else: > + max_data_value = max(max(group.to_list()) for group in self.series) > + self.bounds[self.main_dir] = (0, max_data_value) > + if not self.bounds[other_direction(self.main_dir)]: > + self.bounds[other_direction(self.main_dir)] = (0, len(self.series)) > + > + def calc_extents(self, direction): > + self.max_value[direction] = 0 > + if self.labels[direction]: > + widest_word = max(self.labels[direction], key = lambda item: self.context.text_extents(item)[2]) > + self.max_value[direction] = self.context.text_extents(widest_word)[3 - direction] > + self.borders[other_direction(direction)] = (2-direction)*self.max_value[direction] + self.border + direction*(5) > + else: > + self.borders[other_direction(direction)] = self.border > + > + def calc_horz_extents(self): > + self.calc_extents(HORZ) > + > + def calc_vert_extents(self): > + self.calc_extents(VERT) > + > + def calc_all_extents(self): > + self.calc_horz_extents() > + self.calc_vert_extents() > + other_dir = other_direction(self.main_dir) > + self.value_label = 0 > + if self.display_values: > + if self.stack: > + self.value_label = self.context.text_extents(str(max(sum(group.to_list()) for group in self.series)))[2 + self.main_dir] > + else: > + self.value_label = self.context.text_extents(str(max(max(group.to_list()) for group in self.series)))[2 + self.main_dir] > + if self.labels[self.main_dir]: > + self.plot_dimensions[self.main_dir] = self.dimensions[self.main_dir] - 2*self.borders[self.main_dir] - self.value_label > + else: > + self.plot_dimensions[self.main_dir] = self.dimensions[self.main_dir] - self.borders[self.main_dir] - 1.2*self.border - self.value_label > + self.plot_dimensions[other_dir] = self.dimensions[other_dir] - self.borders[other_dir] - self.border > + self.plot_top = self.dimensions[VERT] - self.borders[VERT] > + > + def calc_steps(self): > + other_dir = other_direction(self.main_dir) > + self.series_amplitude = self.bounds[self.main_dir][1] - self.bounds[self.main_dir][0] > + if self.series_amplitude: > + self.steps[self.main_dir] = float(self.plot_dimensions[self.main_dir])/self.series_amplitude > + else: > + self.steps[self.main_dir] = 0.00 > + series_length = len(self.series) > + self.steps[other_dir] = float(self.plot_dimensions[other_dir])/(series_length + 0.1*(series_length + 1)) > + self.space = 0.1*self.steps[other_dir] > + > + def render(self): > + self.calc_all_extents() > + self.calc_steps() > + self.render_background() > + self.render_bounding_box() > + if self.grid: > + self.render_grid() > + if self.three_dimension: > + self.render_ground() > + if self.display_values: > + self.render_values() > + self.render_labels() > + self.render_plot() > + if self.series_labels: > + self.render_legend() > + > + def draw_3d_rectangle_front(self, x0, y0, x1, y1, shift): > + self.context.rectangle(x0-shift, y0+shift, x1-x0, y1-y0) > + > + def draw_3d_rectangle_side(self, x0, y0, x1, y1, shift): > + self.context.move_to(x1-shift,y0+shift) > + self.context.line_to(x1, y0) > + self.context.line_to(x1, y1) > + self.context.line_to(x1-shift, y1+shift) > + self.context.line_to(x1-shift, y0+shift) > + self.context.close_path() > + > + def draw_3d_rectangle_top(self, x0, y0, x1, y1, shift): > + self.context.move_to(x0-shift,y0+shift) > + self.context.line_to(x0, y0) > + self.context.line_to(x1, y0) > + self.context.line_to(x1-shift, y0+shift) > + self.context.line_to(x0-shift, y0+shift) > + self.context.close_path() > + > + def draw_round_rectangle(self, x0, y0, x1, y1): > + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) > + self.context.line_to(x1-5, y0) > + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) > + self.context.line_to(x1, y1-5) > + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) > + self.context.line_to(x0+5, y1) > + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) > + self.context.line_to(x0, y0+5) > + self.context.close_path() > + > + def render_ground(self): > + self.draw_3d_rectangle_front(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], > + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) > + self.context.fill() > + > + self.draw_3d_rectangle_side (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], > + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) > + self.context.fill() > + > + self.draw_3d_rectangle_top (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], > + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) > + self.context.fill() > + > + def render_labels(self): > + self.context.set_font_size(self.font_size * 0.8) > + if self.labels[HORZ]: > + self.render_horz_labels() > + if self.labels[VERT]: > + self.render_vert_labels() > + > + def render_legend(self): > + cr = self.context > + cr.set_font_size(self.font_size) > + cr.set_line_width(self.line_width) > + > + widest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[2]) > + tallest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[3]) > + max_width = self.context.text_extents(widest_word)[2] > + max_height = self.context.text_extents(tallest_word)[3] * 1.1 + 5 > + > + color_box_height = max_height / 2 > + color_box_width = color_box_height * 2 > + > + #Draw a bounding box > + bounding_box_width = max_width + color_box_width + 15 > + bounding_box_height = (len(self.series_labels)+0.5) * max_height > + cr.set_source_rgba(1,1,1) > + cr.rectangle(self.dimensions[HORZ] - self.border - bounding_box_width, self.border, > + bounding_box_width, bounding_box_height) > + cr.fill() > + > + cr.set_source_rgba(*self.line_color) > + cr.set_line_width(self.line_width) > + cr.rectangle(self.dimensions[HORZ] - self.border - bounding_box_width, self.border, > + bounding_box_width, bounding_box_height) > + cr.stroke() > + > + for idx,key in enumerate(self.series_labels): > + #Draw color box > + cr.set_source_rgba(*self.series_colors[idx][:4]) > + cr.rectangle(self.dimensions[HORZ] - self.border - max_width - color_box_width - 10, > + self.border + color_box_height + (idx*max_height) , > + color_box_width, color_box_height) > + cr.fill() > + > + cr.set_source_rgba(0, 0, 0) > + cr.rectangle(self.dimensions[HORZ] - self.border - max_width - color_box_width - 10, > + self.border + color_box_height + (idx*max_height), > + color_box_width, color_box_height) > + cr.stroke() > + > + #Draw series labels > + cr.set_source_rgba(0, 0, 0) > + cr.move_to(self.dimensions[HORZ] - self.border - max_width - 5, self.border + ((idx+1)*max_height)) > + cr.show_text(key) > + > + > +class HorizontalBarPlot(BarPlot): > + def __init__(self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + border = 0, > + display_values = False, > + grid = False, > + rounded_corners = False, > + stack = False, > + three_dimension = False, > + series_labels = None, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + series_colors = None): > + > + BarPlot.__init__(self, surface, data, width, height, background, border, > + display_values, grid, rounded_corners, stack, three_dimension, > + x_labels, y_labels, x_bounds, y_bounds, series_colors, HORZ) > + self.series_labels = series_labels > + > + def calc_vert_extents(self): > + self.calc_extents(VERT) > + if self.labels[HORZ] and not self.labels[VERT]: > + self.borders[HORZ] += 10 > + > + def draw_rectangle_bottom(self, x0, y0, x1, y1): > + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) > + self.context.line_to(x0, y0+5) > + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) > + self.context.line_to(x1, y0) > + self.context.line_to(x1, y1) > + self.context.line_to(x0+5, y1) > + self.context.close_path() > + > + def draw_rectangle_top(self, x0, y0, x1, y1): > + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) > + self.context.line_to(x1, y1-5) > + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) > + self.context.line_to(x0, y1) > + self.context.line_to(x0, y0) > + self.context.line_to(x1, y0) > + self.context.close_path() > + > + def draw_rectangle(self, index, length, x0, y0, x1, y1): > + if length == 1: > + BarPlot.draw_rectangle(self, x0, y0, x1, y1) > + elif index == 0: > + self.draw_rectangle_bottom(x0, y0, x1, y1) > + elif index == length-1: > + self.draw_rectangle_top(x0, y0, x1, y1) > + else: > + self.context.rectangle(x0, y0, x1-x0, y1-y0) > + > + #TODO: Review BarPlot.render_grid code > + def render_grid(self): > + self.context.set_source_rgba(0.8, 0.8, 0.8) > + if self.labels[HORZ]: > + self.context.set_font_size(self.font_size * 0.8) > + step = (self.dimensions[HORZ] - 2*self.borders[HORZ] - self.value_label)/(len(self.labels[HORZ])-1) > + x = self.borders[HORZ] > + next_x = 0 > + for item in self.labels[HORZ]: > + width = self.context.text_extents(item)[2] > + if x - width/2 > next_x and x - width/2 > self.border: > + self.context.move_to(x, self.border) > + self.context.line_to(x, self.dimensions[VERT] - self.borders[VERT]) > + self.context.stroke() > + next_x = x + width/2 > + x += step > + else: > + lines = 11 > + horizontal_step = float(self.plot_dimensions[HORZ])/(lines-1) > + x = self.borders[HORZ] > + for y in xrange(0, lines): > + self.context.move_to(x, self.border) > + self.context.line_to(x, self.dimensions[VERT] - self.borders[VERT]) > + self.context.stroke() > + x += horizontal_step > + > + def render_horz_labels(self): > + step = (self.dimensions[HORZ] - 2*self.borders[HORZ])/(len(self.labels[HORZ])-1) > + x = self.borders[HORZ] > + next_x = 0 > + > + for item in self.labels[HORZ]: > + self.context.set_source_rgba(*self.label_color) > + width = self.context.text_extents(item)[2] > + if x - width/2 > next_x and x - width/2 > self.border: > + self.context.move_to(x - width/2, self.dimensions[VERT] - self.borders[VERT] + self.max_value[HORZ] + 3) > + self.context.show_text(item) > + next_x = x + width/2 > + x += step > + > + def render_vert_labels(self): > + series_length = len(self.labels[VERT]) > + step = (self.plot_dimensions[VERT] - (series_length + 1)*self.space)/(len(self.labels[VERT])) > + y = self.border + step/2 + self.space > + > + for item in self.labels[VERT]: > + self.context.set_source_rgba(*self.label_color) > + width, height = self.context.text_extents(item)[2:4] > + self.context.move_to(self.borders[HORZ] - width - 5, y + height/2) > + self.context.show_text(item) > + y += step + self.space > + self.labels[VERT].reverse() > + > + def render_values(self): > + self.context.set_source_rgba(*self.value_label_color) > + self.context.set_font_size(self.font_size * 0.8) > + if self.stack: > + for i,group in enumerate(self.series): > + value = sum(group.to_list()) > + height = self.context.text_extents(str(value))[3] > + x = self.borders[HORZ] + value*self.steps[HORZ] + 2 > + y = self.borders[VERT] + (i+0.5)*self.steps[VERT] + (i+1)*self.space + height/2 > + self.context.move_to(x, y) > + self.context.show_text(str(value)) > + else: > + for i,group in enumerate(self.series): > + inner_step = self.steps[VERT]/len(group) > + y0 = self.border + i*self.steps[VERT] + (i+1)*self.space > + for number,data in enumerate(group): > + height = self.context.text_extents(str(data.content))[3] > + self.context.move_to(self.borders[HORZ] + data.content*self.steps[HORZ] + 2, y0 + 0.5*inner_step + height/2, ) > + self.context.show_text(str(data.content)) > + y0 += inner_step > + > + def render_plot(self): > + if self.stack: > + for i,group in enumerate(self.series): > + x0 = self.borders[HORZ] > + y0 = self.borders[VERT] + i*self.steps[VERT] + (i+1)*self.space > + for number,data in enumerate(group): > + if self.series_colors[number][4] in ('radial','linear') : > + linear = cairo.LinearGradient( data.content*self.steps[HORZ]/2, y0, data.content*self.steps[HORZ]/2, y0 + self.steps[VERT] ) > + color = self.series_colors[number] > + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) > + linear.add_color_stop_rgba(1.0, *color[:4]) > + self.context.set_source(linear) > + elif self.series_colors[number][4] == 'solid': > + self.context.set_source_rgba(*self.series_colors[number][:4]) > + if self.rounded_corners: > + self.draw_rectangle(number, len(group), x0, y0, x0+data.content*self.steps[HORZ], y0+self.steps[VERT]) > + self.context.fill() > + else: > + self.context.rectangle(x0, y0, data.content*self.steps[HORZ], self.steps[VERT]) > + self.context.fill() > + x0 += data.content*self.steps[HORZ] > + else: > + for i,group in enumerate(self.series): > + inner_step = self.steps[VERT]/len(group) > + x0 = self.borders[HORZ] > + y0 = self.border + i*self.steps[VERT] + (i+1)*self.space > + for number,data in enumerate(group): > + linear = cairo.LinearGradient(data.content*self.steps[HORZ]/2, y0, data.content*self.steps[HORZ]/2, y0 + inner_step) > + color = self.series_colors[number] > + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) > + linear.add_color_stop_rgba(1.0, *color[:4]) > + self.context.set_source(linear) > + if self.rounded_corners and data.content != 0: > + BarPlot.draw_round_rectangle(self,x0, y0, x0 + data.content*self.steps[HORZ], y0 + inner_step) > + self.context.fill() > + else: > + self.context.rectangle(x0, y0, data.content*self.steps[HORZ], inner_step) > + self.context.fill() > + y0 += inner_step > + > +class VerticalBarPlot(BarPlot): > + def __init__(self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + border = 0, > + display_values = False, > + grid = False, > + rounded_corners = False, > + stack = False, > + three_dimension = False, > + series_labels = None, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + series_colors = None): > + > + BarPlot.__init__(self, surface, data, width, height, background, border, > + display_values, grid, rounded_corners, stack, three_dimension, > + x_labels, y_labels, x_bounds, y_bounds, series_colors, VERT) > + self.series_labels = series_labels > + > + def calc_vert_extents(self): > + self.calc_extents(VERT) > + if self.labels[VERT] and not self.labels[HORZ]: > + self.borders[VERT] += 10 > + > + def draw_rectangle_bottom(self, x0, y0, x1, y1): > + self.context.move_to(x1,y1) > + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) > + self.context.line_to(x0+5, y1) > + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) > + self.context.line_to(x0, y0) > + self.context.line_to(x1, y0) > + self.context.line_to(x1, y1) > + self.context.close_path() > + > + def draw_rectangle_top(self, x0, y0, x1, y1): > + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) > + self.context.line_to(x1-5, y0) > + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) > + self.context.line_to(x1, y1) > + self.context.line_to(x0, y1) > + self.context.line_to(x0, y0) > + self.context.close_path() > + > + def draw_rectangle(self, index, length, x0, y0, x1, y1): > + if length == 1: > + BarPlot.draw_rectangle(self, x0, y0, x1, y1) > + elif index == 0: > + self.draw_rectangle_bottom(x0, y0, x1, y1) > + elif index == length-1: > + self.draw_rectangle_top(x0, y0, x1, y1) > + else: > + self.context.rectangle(x0, y0, x1-x0, y1-y0) > + > + def render_grid(self): > + self.context.set_source_rgba(0.8, 0.8, 0.8) > + if self.labels[VERT]: > + lines = len(self.labels[VERT]) > + vertical_step = float(self.plot_dimensions[self.main_dir])/(lines-1) > + y = self.borders[VERT] + self.value_label > + else: > + lines = 11 > + vertical_step = float(self.plot_dimensions[self.main_dir])/(lines-1) > + y = 1.2*self.border + self.value_label > + for x in xrange(0, lines): > + self.context.move_to(self.borders[HORZ], y) > + self.context.line_to(self.dimensions[HORZ] - self.border, y) > + self.context.stroke() > + y += vertical_step > + > + def render_ground(self): > + self.draw_3d_rectangle_front(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], > + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) > + self.context.fill() > + > + self.draw_3d_rectangle_side (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], > + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) > + self.context.fill() > + > + self.draw_3d_rectangle_top (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], > + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) > + self.context.fill() > + > + def render_horz_labels(self): > + series_length = len(self.labels[HORZ]) > + step = float (self.plot_dimensions[HORZ] - (series_length + 1)*self.space)/len(self.labels[HORZ]) > + x = self.borders[HORZ] + step/2 + self.space > + next_x = 0 > + > + for item in self.labels[HORZ]: > + self.context.set_source_rgba(*self.label_color) > + width = self.context.text_extents(item)[2] > + if x - width/2 > next_x and x - width/2 > self.borders[HORZ]: > + self.context.move_to(x - width/2, self.dimensions[VERT] - self.borders[VERT] + self.max_value[HORZ] + 3) > + self.context.show_text(item) > + next_x = x + width/2 > + x += step + self.space > + > + def render_vert_labels(self): > + self.context.set_source_rgba(*self.label_color) > + y = self.borders[VERT] + self.value_label > + step = (self.dimensions[VERT] - 2*self.borders[VERT] - self.value_label)/(len(self.labels[VERT]) - 1) > + self.labels[VERT].reverse() > + for item in self.labels[VERT]: > + width, height = self.context.text_extents(item)[2:4] > + self.context.move_to(self.borders[HORZ] - width - 5, y + height/2) > + self.context.show_text(item) > + y += step > + self.labels[VERT].reverse() > + > + def render_values(self): > + self.context.set_source_rgba(*self.value_label_color) > + self.context.set_font_size(self.font_size * 0.8) > + if self.stack: > + for i,group in enumerate(self.series): > + value = sum(group.to_list()) > + width = self.context.text_extents(str(value))[2] > + x = self.borders[HORZ] + (i+0.5)*self.steps[HORZ] + (i+1)*self.space - width/2 > + y = value*self.steps[VERT] + 2 > + self.context.move_to(x, self.plot_top-y) > + self.context.show_text(str(value)) > + else: > + for i,group in enumerate(self.series): > + inner_step = self.steps[HORZ]/len(group) > + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space > + for number,data in enumerate(group): > + width = self.context.text_extents(str(data.content))[2] > + self.context.move_to(x0 + 0.5*inner_step - width/2, self.plot_top - data.content*self.steps[VERT] - 2) > + self.context.show_text(str(data.content)) > + x0 += inner_step > + > + def render_plot(self): > + if self.stack: > + for i,group in enumerate(self.series): > + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space > + y0 = 0 > + for number,data in enumerate(group): > + if self.series_colors[number][4] in ('linear','radial'): > + linear = cairo.LinearGradient( x0, data.content*self.steps[VERT]/2, x0 + self.steps[HORZ], data.content*self.steps[VERT]/2 ) > + color = self.series_colors[number] > + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) > + linear.add_color_stop_rgba(1.0, *color[:4]) > + self.context.set_source(linear) > + elif self.series_colors[number][4] == 'solid': > + self.context.set_source_rgba(*self.series_colors[number][:4]) > + if self.rounded_corners: > + self.draw_rectangle(number, len(group), x0, self.plot_top - y0 - data.content*self.steps[VERT], x0 + self.steps[HORZ], self.plot_top - y0) > + self.context.fill() > + else: > + self.context.rectangle(x0, self.plot_top - y0 - data.content*self.steps[VERT], self.steps[HORZ], data.content*self.steps[VERT]) > + self.context.fill() > + y0 += data.content*self.steps[VERT] > + else: > + for i,group in enumerate(self.series): > + inner_step = self.steps[HORZ]/len(group) > + y0 = self.borders[VERT] > + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space > + for number,data in enumerate(group): > + if self.series_colors[number][4] == 'linear': > + linear = cairo.LinearGradient( x0, data.content*self.steps[VERT]/2, x0 + inner_step, data.content*self.steps[VERT]/2 ) > + color = self.series_colors[number] > + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) > + linear.add_color_stop_rgba(1.0, *color[:4]) > + self.context.set_source(linear) > + elif self.series_colors[number][4] == 'solid': > + self.context.set_source_rgba(*self.series_colors[number][:4]) > + if self.rounded_corners and data.content != 0: > + BarPlot.draw_round_rectangle(self, x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top) > + self.context.fill() > + elif self.three_dimension: > + self.draw_3d_rectangle_front(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) > + self.context.fill() > + self.draw_3d_rectangle_side(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) > + self.context.fill() > + self.draw_3d_rectangle_top(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) > + self.context.fill() > + else: > + self.context.rectangle(x0, self.plot_top - data.content*self.steps[VERT], inner_step, data.content*self.steps[VERT]) > + self.context.fill() > + > + x0 += inner_step > + > +class StreamChart(VerticalBarPlot): > + def __init__(self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + border = 0, > + grid = False, > + series_legend = None, > + x_labels = None, > + x_bounds = None, > + y_bounds = None, > + series_colors = None): > + > + VerticalBarPlot.__init__(self, surface, data, width, height, background, border, > + False, grid, False, True, False, > + None, x_labels, None, x_bounds, y_bounds, series_colors) > + > + def calc_steps(self): > + other_dir = other_direction(self.main_dir) > + self.series_amplitude = self.bounds[self.main_dir][1] - self.bounds[self.main_dir][0] > + if self.series_amplitude: > + self.steps[self.main_dir] = float(self.plot_dimensions[self.main_dir])/self.series_amplitude > + else: > + self.steps[self.main_dir] = 0.00 > + series_length = len(self.data) > + self.steps[other_dir] = float(self.plot_dimensions[other_dir])/series_length > + > + def render_legend(self): > + pass > + > + def ground(self, index): > + sum_values = sum(self.data[index]) > + return -0.5*sum_values > + > + def calc_angles(self): > + middle = self.plot_top - self.plot_dimensions[VERT]/2.0 > + self.angles = [tuple([0.0 for x in range(len(self.data)+1)])] > + for x_index in range(1, len(self.data)-1): > + t = [] > + x0 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] > + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] > + y0 = middle - self.ground(x_index-1)*self.steps[VERT] > + y2 = middle - self.ground(x_index+1)*self.steps[VERT] > + t.append(math.atan(float(y0-y2)/(x0-x2))) > + for data_index in range(len(self.data[x_index])): > + x0 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] > + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] > + y0 = middle - self.ground(x_index-1)*self.steps[VERT] - self.data[x_index-1][data_index]*self.steps[VERT] > + y2 = middle - self.ground(x_index+1)*self.steps[VERT] - self.data[x_index+1][data_index]*self.steps[VERT] > + > + for i in range(0,data_index): > + y0 -= self.data[x_index-1][i]*self.steps[VERT] > + y2 -= self.data[x_index+1][i]*self.steps[VERT] > + > + if data_index == len(self.data[0])-1 and False: > + self.context.set_source_rgba(0.0,0.0,0.0,0.3) > + self.context.move_to(x0,y0) > + self.context.line_to(x2,y2) > + self.context.stroke() > + self.context.arc(x0,y0,2,0,2*math.pi) > + self.context.fill() > + t.append(math.atan(float(y0-y2)/(x0-x2))) > + self.angles.append(tuple(t)) > + self.angles.append(tuple([0.0 for x in range(len(self.data)+1)])) > + > + def render_plot(self): > + self.calc_angles() > + middle = self.plot_top - self.plot_dimensions[VERT]/2.0 > + p = 0.4*self.steps[HORZ] > + for data_index in range(len(self.data[0])-1,-1,-1): > + self.context.set_source_rgba(*self.series_colors[data_index][:4]) > + > + #draw the upper line > + for x_index in range(len(self.data)-1) : > + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] > + y1 = middle - self.ground(x_index)*self.steps[VERT] - self.data[x_index][data_index]*self.steps[VERT] > + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] > + y2 = middle - self.ground(x_index + 1)*self.steps[VERT] - self.data[x_index + 1][data_index]*self.steps[VERT] > + > + for i in range(0,data_index): > + y1 -= self.data[x_index][i]*self.steps[VERT] > + y2 -= self.data[x_index+1][i]*self.steps[VERT] > + > + if x_index == 0: > + self.context.move_to(x1,y1) > + > + ang1 = self.angles[x_index][data_index+1] > + ang2 = self.angles[x_index+1][data_index+1] + math.pi > + self.context.curve_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1), > + x2+p*math.cos(ang2),y2+p*math.sin(ang2), > + x2,y2) > + > + for x_index in range(len(self.data)-1,0,-1) : > + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] > + y1 = middle - self.ground(x_index)*self.steps[VERT] > + x2 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] > + y2 = middle - self.ground(x_index - 1)*self.steps[VERT] > + > + for i in range(0,data_index): > + y1 -= self.data[x_index][i]*self.steps[VERT] > + y2 -= self.data[x_index-1][i]*self.steps[VERT] > + > + if x_index == len(self.data)-1: > + self.context.line_to(x1,y1+2) > + > + #revert angles by pi degrees to take the turn back > + ang1 = self.angles[x_index][data_index] + math.pi > + ang2 = self.angles[x_index-1][data_index] > + self.context.curve_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1), > + x2+p*math.cos(ang2),y2+p*math.sin(ang2), > + x2,y2+2) > + > + self.context.close_path() > + self.context.fill() > + > + if False: > + self.context.move_to(self.borders[HORZ] + 0.5*self.steps[HORZ], middle) > + for x_index in range(len(self.data)-1) : > + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] > + y1 = middle - self.ground(x_index)*self.steps[VERT] - self.data[x_index][data_index]*self.steps[VERT] > + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] > + y2 = middle - self.ground(x_index + 1)*self.steps[VERT] - self.data[x_index + 1][data_index]*self.steps[VERT] > + > + for i in range(0,data_index): > + y1 -= self.data[x_index][i]*self.steps[VERT] > + y2 -= self.data[x_index+1][i]*self.steps[VERT] > + > + ang1 = self.angles[x_index][data_index+1] > + ang2 = self.angles[x_index+1][data_index+1] + math.pi > + self.context.set_source_rgba(1.0,0.0,0.0) > + self.context.arc(x1+p*math.cos(ang1),y1+p*math.sin(ang1),2,0,2*math.pi) > + self.context.fill() > + self.context.set_source_rgba(0.0,0.0,0.0) > + self.context.arc(x2+p*math.cos(ang2),y2+p*math.sin(ang2),2,0,2*math.pi) > + self.context.fill() > + '''self.context.set_source_rgba(0.0,0.0,0.0,0.3) > + self.context.arc(x2,y2,2,0,2*math.pi) > + self.context.fill()''' > + self.context.move_to(x1,y1) > + self.context.line_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1)) > + self.context.stroke() > + self.context.move_to(x2,y2) > + self.context.line_to(x2+p*math.cos(ang2),y2+p*math.sin(ang2)) > + self.context.stroke() > + if False: > + for x_index in range(len(self.data)-1,0,-1) : > + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] > + y1 = middle - self.ground(x_index)*self.steps[VERT] > + x2 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] > + y2 = middle - self.ground(x_index - 1)*self.steps[VERT] > + > + for i in range(0,data_index): > + y1 -= self.data[x_index][i]*self.steps[VERT] > + y2 -= self.data[x_index-1][i]*self.steps[VERT] > + > + #revert angles by pi degrees to take the turn back > + ang1 = self.angles[x_index][data_index] + math.pi > + ang2 = self.angles[x_index-1][data_index] > + self.context.set_source_rgba(0.0,1.0,0.0) > + self.context.arc(x1+p*math.cos(ang1),y1+p*math.sin(ang1),2,0,2*math.pi) > + self.context.fill() > + self.context.set_source_rgba(0.0,0.0,1.0) > + self.context.arc(x2+p*math.cos(ang2),y2+p*math.sin(ang2),2,0,2*math.pi) > + self.context.fill() > + '''self.context.set_source_rgba(0.0,0.0,0.0,0.3) > + self.context.arc(x2,y2,2,0,2*math.pi) > + self.context.fill()''' > + self.context.move_to(x1,y1) > + self.context.line_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1)) > + self.context.stroke() > + self.context.move_to(x2,y2) > + self.context.line_to(x2+p*math.cos(ang2),y2+p*math.sin(ang2)) > + self.context.stroke() > + #break > + > + #self.context.arc(self.dimensions[HORZ]/2, self.dimensions[VERT]/2,50,0,3*math.pi/2) > + #self.context.fill() > + > + > +class PiePlot(Plot): > + #TODO: Check the old cairoplot, graphs aren't matching > + def __init__ (self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + gradient = False, > + shadow = False, > + colors = None): > + > + Plot.__init__( self, surface, data, width, height, background, series_colors = colors ) > + self.center = (self.dimensions[HORZ]/2, self.dimensions[VERT]/2) > + self.total = sum( self.series.to_list() ) > + self.radius = min(self.dimensions[HORZ]/3,self.dimensions[VERT]/3) > + self.gradient = gradient > + self.shadow = shadow > + > + def sort_function(x,y): > + return x.content - y.content > + > + def load_series(self, data, x_labels=None, y_labels=None, series_colors=None): > + Plot.load_series(self, data, x_labels, y_labels, series_colors) > + # Already done inside series > + #self.data = sorted(self.data) > + > + def draw_piece(self, angle, next_angle): > + self.context.move_to(self.center[0],self.center[1]) > + self.context.line_to(self.center[0] + self.radius*math.cos(angle), self.center[1] + self.radius*math.sin(angle)) > + self.context.arc(self.center[0], self.center[1], self.radius, angle, next_angle) > + self.context.line_to(self.center[0], self.center[1]) > + self.context.close_path() > + > + def render(self): > + self.render_background() > + self.render_bounding_box() > + if self.shadow: > + self.render_shadow() > + self.render_plot() > + self.render_series_labels() > + > + def render_shadow(self): > + horizontal_shift = 3 > + vertical_shift = 3 > + self.context.set_source_rgba(0, 0, 0, 0.5) > + self.context.arc(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.radius, 0, 2*math.pi) > + self.context.fill() > + > + def render_series_labels(self): > + angle = 0 > + next_angle = 0 > + x0,y0 = self.center > + cr = self.context > + for number,key in enumerate(self.series_labels): > + # self.data[number] should be just a number > + data = sum(self.series[number].to_list()) > + > + next_angle = angle + 2.0*math.pi*data/self.total > + cr.set_source_rgba(*self.series_colors[number][:4]) > + w = cr.text_extents(key)[2] > + if (angle + next_angle)/2 < math.pi/2 or (angle + next_angle)/2 > 3*math.pi/2: > + cr.move_to(x0 + (self.radius+10)*math.cos((angle+next_angle)/2), y0 + (self.radius+10)*math.sin((angle+next_angle)/2) ) > + else: > + cr.move_to(x0 + (self.radius+10)*math.cos((angle+next_angle)/2) - w, y0 + (self.radius+10)*math.sin((angle+next_angle)/2) ) > + cr.show_text(key) > + angle = next_angle > + > + def render_plot(self): > + angle = 0 > + next_angle = 0 > + x0,y0 = self.center > + cr = self.context > + for number,group in enumerate(self.series): > + # Group should be just a number > + data = sum(group.to_list()) > + next_angle = angle + 2.0*math.pi*data/self.total > + if self.gradient or self.series_colors[number][4] in ('linear','radial'): > + gradient_color = cairo.RadialGradient(self.center[0], self.center[1], 0, self.center[0], self.center[1], self.radius) > + gradient_color.add_color_stop_rgba(0.3, *self.series_colors[number][:4]) > + gradient_color.add_color_stop_rgba(1, self.series_colors[number][0]*0.7, > + self.series_colors[number][1]*0.7, > + self.series_colors[number][2]*0.7, > + self.series_colors[number][3]) > + cr.set_source(gradient_color) > + else: > + cr.set_source_rgba(*self.series_colors[number][:4]) > + > + self.draw_piece(angle, next_angle) > + cr.fill() > + > + cr.set_source_rgba(1.0, 1.0, 1.0) > + self.draw_piece(angle, next_angle) > + cr.stroke() > + > + angle = next_angle > + > +class DonutPlot(PiePlot): > + def __init__ (self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + gradient = False, > + shadow = False, > + colors = None, > + inner_radius=-1): > + > + Plot.__init__( self, surface, data, width, height, background, series_colors = colors ) > + > + self.center = ( self.dimensions[HORZ]/2, self.dimensions[VERT]/2 ) > + self.total = sum( self.series.to_list() ) > + self.radius = min( self.dimensions[HORZ]/3,self.dimensions[VERT]/3 ) > + self.inner_radius = inner_radius*self.radius > + > + if inner_radius == -1: > + self.inner_radius = self.radius/3 > + > + self.gradient = gradient > + self.shadow = shadow > + > + def draw_piece(self, angle, next_angle): > + self.context.move_to(self.center[0] + (self.inner_radius)*math.cos(angle), self.center[1] + (self.inner_radius)*math.sin(angle)) > + self.context.line_to(self.center[0] + self.radius*math.cos(angle), self.center[1] + self.radius*math.sin(angle)) > + self.context.arc(self.center[0], self.center[1], self.radius, angle, next_angle) > + self.context.line_to(self.center[0] + (self.inner_radius)*math.cos(next_angle), self.center[1] + (self.inner_radius)*math.sin(next_angle)) > + self.context.arc_negative(self.center[0], self.center[1], self.inner_radius, next_angle, angle) > + self.context.close_path() > + > + def render_shadow(self): > + horizontal_shift = 3 > + vertical_shift = 3 > + self.context.set_source_rgba(0, 0, 0, 0.5) > + self.context.arc(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.inner_radius, 0, 2*math.pi) > + self.context.arc_negative(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.radius, 0, -2*math.pi) > + self.context.fill() > + > +class GanttChart (Plot) : > + def __init__(self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + x_labels = None, > + y_labels = None, > + colors = None): > + self.bounds = {} > + self.max_value = {} > + Plot.__init__(self, surface, data, width, height, x_labels = x_labels, y_labels = y_labels, series_colors = colors) > + > + def load_series(self, data, x_labels=None, y_labels=None, series_colors=None): > + Plot.load_series(self, data, x_labels, y_labels, series_colors) > + self.calc_boundaries() > + > + def calc_boundaries(self): > + self.bounds[HORZ] = (0,len(self.series)) > + end_pos = max(self.series.to_list()) > + > + #for group in self.series: > + # if hasattr(item, "__delitem__"): > + # for sub_item in item: > + # end_pos = max(sub_item) > + # else: > + # end_pos = max(item) > + self.bounds[VERT] = (0,end_pos) > + > + def calc_extents(self, direction): > + self.max_value[direction] = 0 > + if self.labels[direction]: > + self.max_value[direction] = max(self.context.text_extents(item)[2] for item in self.labels[direction]) > + else: > + self.max_value[direction] = self.context.text_extents( str(self.bounds[direction][1] + 1) )[2] > + > + def calc_horz_extents(self): > + self.calc_extents(HORZ) > + self.borders[HORZ] = 100 + self.max_value[HORZ] > + > + def calc_vert_extents(self): > + self.calc_extents(VERT) > + self.borders[VERT] = self.dimensions[VERT]/(self.bounds[HORZ][1] + 1) > + > + def calc_steps(self): > + self.horizontal_step = (self.dimensions[HORZ] - self.borders[HORZ])/(len(self.labels[VERT])) > + self.vertical_step = self.borders[VERT] > + > + def render(self): > + self.calc_horz_extents() > + self.calc_vert_extents() > + self.calc_steps() > + self.render_background() > + > + self.render_labels() > + self.render_grid() > + self.render_plot() > + > + def render_background(self): > + cr = self.context > + cr.set_source_rgba(255,255,255) > + cr.rectangle(0,0,self.dimensions[HORZ], self.dimensions[VERT]) > + cr.fill() > + for number,group in enumerate(self.series): > + linear = cairo.LinearGradient(self.dimensions[HORZ]/2, self.borders[VERT] + number*self.vertical_step, > + self.dimensions[HORZ]/2, self.borders[VERT] + (number+1)*self.vertical_step) > + linear.add_color_stop_rgba(0,1.0,1.0,1.0,1.0) > + linear.add_color_stop_rgba(1.0,0.9,0.9,0.9,1.0) > + cr.set_source(linear) > + cr.rectangle(0,self.borders[VERT] + number*self.vertical_step,self.dimensions[HORZ],self.vertical_step) > + cr.fill() > + > + def render_grid(self): > + cr = self.context > + cr.set_source_rgba(0.7, 0.7, 0.7) > + cr.set_dash((1,0,0,0,0,0,1)) > + cr.set_line_width(0.5) > + for number,label in enumerate(self.labels[VERT]): > + h = cr.text_extents(label)[3] > + cr.move_to(self.borders[HORZ] + number*self.horizontal_step, self.vertical_step/2 + h) > + cr.line_to(self.borders[HORZ] + number*self.horizontal_step, self.dimensions[VERT]) > + cr.stroke() > + > + def render_labels(self): > + self.context.set_font_size(0.02 * self.dimensions[HORZ]) > + > + self.render_horz_labels() > + self.render_vert_labels() > + > + def render_horz_labels(self): > + cr = self.context > + labels = self.labels[HORZ] > + if not labels: > + labels = [str(i) for i in range(1, self.bounds[HORZ][1] + 1) ] > + for number,label in enumerate(labels): > + if label != None: > + cr.set_source_rgba(0.5, 0.5, 0.5) > + w,h = cr.text_extents(label)[2], cr.text_extents(label)[3] > + cr.move_to(40,self.borders[VERT] + number*self.vertical_step + self.vertical_step/2 + h/2) > + cr.show_text(label) > + > + def render_vert_labels(self): > + cr = self.context > + labels = self.labels[VERT] > + if not labels: > + labels = [str(i) for i in range(1, self.bounds[VERT][1] + 1) ] > + for number,label in enumerate(labels): > + w,h = cr.text_extents(label)[2], cr.text_extents(label)[3] > + cr.move_to(self.borders[HORZ] + number*self.horizontal_step - w/2, self.vertical_step/2) > + cr.show_text(label) > + > + def render_rectangle(self, x0, y0, x1, y1, color): > + self.draw_shadow(x0, y0, x1, y1) > + self.draw_rectangle(x0, y0, x1, y1, color) > + > + def draw_rectangular_shadow(self, gradient, x0, y0, w, h): > + self.context.set_source(gradient) > + self.context.rectangle(x0,y0,w,h) > + self.context.fill() > + > + def draw_circular_shadow(self, x, y, radius, ang_start, ang_end, mult, shadow): > + gradient = cairo.RadialGradient(x, y, 0, x, y, 2*radius) > + gradient.add_color_stop_rgba(0, 0, 0, 0, shadow) > + gradient.add_color_stop_rgba(1, 0, 0, 0, 0) > + self.context.set_source(gradient) > + self.context.move_to(x,y) > + self.context.line_to(x + mult[0]*radius,y + mult[1]*radius) > + self.context.arc(x, y, 8, ang_start, ang_end) > + self.context.line_to(x,y) > + self.context.close_path() > + self.context.fill() > + > + def draw_rectangle(self, x0, y0, x1, y1, color): > + cr = self.context > + middle = (x0+x1)/2 > + linear = cairo.LinearGradient(middle,y0,middle,y1) > + linear.add_color_stop_rgba(0,3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) > + linear.add_color_stop_rgba(1,*color[:4]) > + cr.set_source(linear) > + > + cr.arc(x0+5, y0+5, 5, 0, 2*math.pi) > + cr.arc(x1-5, y0+5, 5, 0, 2*math.pi) > + cr.arc(x0+5, y1-5, 5, 0, 2*math.pi) > + cr.arc(x1-5, y1-5, 5, 0, 2*math.pi) > + cr.rectangle(x0+5,y0,x1-x0-10,y1-y0) > + cr.rectangle(x0,y0+5,x1-x0,y1-y0-10) > + cr.fill() > + > + def draw_shadow(self, x0, y0, x1, y1): > + shadow = 0.4 > + h_mid = (x0+x1)/2 > + v_mid = (y0+y1)/2 > + h_linear_1 = cairo.LinearGradient(h_mid,y0-4,h_mid,y0+4) > + h_linear_2 = cairo.LinearGradient(h_mid,y1-4,h_mid,y1+4) > + v_linear_1 = cairo.LinearGradient(x0-4,v_mid,x0+4,v_mid) > + v_linear_2 = cairo.LinearGradient(x1-4,v_mid,x1+4,v_mid) > + > + h_linear_1.add_color_stop_rgba( 0, 0, 0, 0, 0) > + h_linear_1.add_color_stop_rgba( 1, 0, 0, 0, shadow) > + h_linear_2.add_color_stop_rgba( 0, 0, 0, 0, shadow) > + h_linear_2.add_color_stop_rgba( 1, 0, 0, 0, 0) > + v_linear_1.add_color_stop_rgba( 0, 0, 0, 0, 0) > + v_linear_1.add_color_stop_rgba( 1, 0, 0, 0, shadow) > + v_linear_2.add_color_stop_rgba( 0, 0, 0, 0, shadow) > + v_linear_2.add_color_stop_rgba( 1, 0, 0, 0, 0) > + > + self.draw_rectangular_shadow(h_linear_1,x0+4,y0-4,x1-x0-8,8) > + self.draw_rectangular_shadow(h_linear_2,x0+4,y1-4,x1-x0-8,8) > + self.draw_rectangular_shadow(v_linear_1,x0-4,y0+4,8,y1-y0-8) > + self.draw_rectangular_shadow(v_linear_2,x1-4,y0+4,8,y1-y0-8) > + > + self.draw_circular_shadow(x0+4, y0+4, 4, math.pi, 3*math.pi/2, (-1,0), shadow) > + self.draw_circular_shadow(x1-4, y0+4, 4, 3*math.pi/2, 2*math.pi, (0,-1), shadow) > + self.draw_circular_shadow(x0+4, y1-4, 4, math.pi/2, math.pi, (0,1), shadow) > + self.draw_circular_shadow(x1-4, y1-4, 4, 0, math.pi/2, (1,0), shadow) > + > + def render_plot(self): > + for index,group in enumerate(self.series): > + for data in group: > + self.render_rectangle(self.borders[HORZ] + data.content[0]*self.horizontal_step, > + self.borders[VERT] + index*self.vertical_step + self.vertical_step/4.0, > + self.borders[HORZ] + data.content[1]*self.horizontal_step, > + self.borders[VERT] + index*self.vertical_step + 3.0*self.vertical_step/4.0, > + self.series_colors[index]) > + > +# Function definition > + > +def scatter_plot(name, > + data = None, > + errorx = None, > + errory = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + border = 0, > + axis = False, > + dash = False, > + discrete = False, > + dots = False, > + grid = False, > + series_legend = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + z_bounds = None, > + x_title = None, > + y_title = None, > + series_colors = None, > + circle_colors = None): > + > + ''' > + - Function to plot scatter data. > + > + - Parameters > + > + data - The values to be ploted might be passed in a two basic: > + list of points: [(0,0), (0,1), (0,2)] or [(0,0,1), (0,1,4), (0,2,1)] > + lists of coordinates: [ [0,0,0] , [0,1,2] ] or [ [0,0,0] , [0,1,2] , [1,4,1] ] > + Notice that these kinds of that can be grouped in order to form more complex data > + using lists of lists or dictionaries; > + series_colors - Define color values for each of the series > + circle_colors - Define a lower and an upper bound for the circle colors for variable radius > + (3 dimensions) series > + ''' > + > + plot = ScatterPlot( name, data, errorx, errory, width, height, background, border, > + axis, dash, discrete, dots, grid, series_legend, x_labels, y_labels, > + x_bounds, y_bounds, z_bounds, x_title, y_title, series_colors, circle_colors ) > + plot.render() > + plot.commit() > + > +def dot_line_plot(name, > + data, > + width, > + height, > + background = "white light_gray", > + border = 0, > + axis = False, > + dash = False, > + dots = False, > + grid = False, > + series_legend = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + x_title = None, > + y_title = None, > + series_colors = None): > + ''' > + - Function to plot graphics using dots and lines. > + > + dot_line_plot (name, data, width, height, background = "white light_gray", border = 0, axis = False, grid = False, x_labels = None, y_labels = None, x_bounds = None, y_bounds = None) > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; > + data - The list, list of lists or dictionary holding the data to be plotted; > + width, height - Dimensions of the output image; > + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. > + If left None, a gray to white gradient will be generated; > + border - Distance in pixels of a square border into which the graphics will be drawn; > + axis - Whether or not the axis are to be drawn; > + dash - Boolean or a list or a dictionary of booleans indicating whether or not the associated series should be drawn in dashed mode; > + dots - Whether or not dots should be drawn on each point; > + grid - Whether or not the gris is to be drawn; > + series_legend - Whether or not the legend is to be drawn; > + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; > + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; > + x_title - Whether or not to plot a title over the x axis. > + y_title - Whether or not to plot a title over the y axis. > + > + - Examples of use > + > + data = [0, 1, 3, 8, 9, 0, 10, 10, 2, 1] > + CairoPlot.dot_line_plot('teste', data, 400, 300) > + > + data = { "john" : [10, 10, 10, 10, 30], "mary" : [0, 0, 3, 5, 15], "philip" : [13, 32, 11, 25, 2] } > + x_labels = ["jan/2008", "feb/2008", "mar/2008", "apr/2008", "may/2008" ] > + CairoPlot.dot_line_plot( 'test', data, 400, 300, axis = True, grid = True, > + series_legend = True, x_labels = x_labels ) > + ''' > + plot = DotLinePlot( name, data, width, height, background, border, > + axis, dash, dots, grid, series_legend, x_labels, y_labels, > + x_bounds, y_bounds, x_title, y_title, series_colors ) > + plot.render() > + plot.commit() > + > +def function_plot(name, > + data, > + width, > + height, > + background = "white light_gray", > + border = 0, > + axis = True, > + dots = False, > + discrete = False, > + grid = False, > + series_legend = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + x_title = None, > + y_title = None, > + series_colors = None, > + step = 1): > + > + ''' > + - Function to plot functions. > + > + function_plot(name, data, width, height, background = "white light_gray", border = 0, axis = True, grid = False, dots = False, x_labels = None, y_labels = None, x_bounds = None, y_bounds = None, step = 1, discrete = False) > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; > + data - The list, list of lists or dictionary holding the data to be plotted; > + width, height - Dimensions of the output image; > + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. > + If left None, a gray to white gradient will be generated; > + border - Distance in pixels of a square border into which the graphics will be drawn; > + axis - Whether or not the axis are to be drawn; > + grid - Whether or not the gris is to be drawn; > + dots - Whether or not dots should be shown at each point; > + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; > + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; > + step - the horizontal distance from one point to the other. The smaller, the smoother the curve will be; > + discrete - whether or not the function should be plotted in discrete format. > + > + - Example of use > + > + data = lambda x : x**2 > + CairoPlot.function_plot('function4', data, 400, 300, grid = True, x_bounds=(-10,10), step = 0.1) > + ''' > + > + plot = FunctionPlot( name, data, width, height, background, border, > + axis, discrete, dots, grid, series_legend, x_labels, y_labels, > + x_bounds, y_bounds, x_title, y_title, series_colors, step ) > + plot.render() > + plot.commit() > + > +def pie_plot( name, data, width, height, background = "white light_gray", gradient = False, shadow = False, colors = None ): > + > + ''' > + - Function to plot pie graphics. > + > + pie_plot(name, data, width, height, background = "white light_gray", gradient = False, colors = None) > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; > + data - The list, list of lists or dictionary holding the data to be plotted; > + width, height - Dimensions of the output image; > + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. > + If left None, a gray to white gradient will be generated; > + gradient - Whether or not the pie color will be painted with a gradient; > + shadow - Whether or not there will be a shadow behind the pie; > + colors - List of slices colors. > + > + - Example of use > + > + teste_data = {"john" : 123, "mary" : 489, "philip" : 890 , "suzy" : 235} > + CairoPlot.pie_plot("pie_teste", teste_data, 500, 500) > + ''' > + > + plot = PiePlot( name, data, width, height, background, gradient, shadow, colors ) > + plot.render() > + plot.commit() > + > +def donut_plot(name, data, width, height, background = "white light_gray", gradient = False, shadow = False, colors = None, inner_radius = -1): > + > + ''' > + - Function to plot donut graphics. > + > + donut_plot(name, data, width, height, background = "white light_gray", gradient = False, inner_radius = -1) > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; > + data - The list, list of lists or dictionary holding the data to be plotted; > + width, height - Dimensions of the output image; > + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. > + If left None, a gray to white gradient will be generated; > + shadow - Whether or not there will be a shadow behind the donut; > + gradient - Whether or not the donut color will be painted with a gradient; > + colors - List of slices colors; > + inner_radius - The radius of the donut's inner circle. > + > + - Example of use > + > + teste_data = {"john" : 123, "mary" : 489, "philip" : 890 , "suzy" : 235} > + CairoPlot.donut_plot("donut_teste", teste_data, 500, 500) > + ''' > + > + plot = DonutPlot(name, data, width, height, background, gradient, shadow, colors, inner_radius) > + plot.render() > + plot.commit() > + > +def gantt_chart(name, pieces, width, height, x_labels, y_labels, colors): > + > + ''' > + - Function to generate Gantt Charts. > + > + gantt_chart(name, pieces, width, height, x_labels, y_labels, colors): > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; > + pieces - A list defining the spaces to be drawn. The user must pass, for each line, the index of its start and the index of its end. If a line must have two or more spaces, they must be passed inside a list; > + width, height - Dimensions of the output image; > + x_labels - A list of names for each of the vertical lines; > + y_labels - A list of names for each of the horizontal spaces; > + colors - List containing the colors expected for each of the horizontal spaces > + > + - Example of use > + > + pieces = [ (0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,8)] > + x_labels = [ 'teste01', 'teste02', 'teste03', 'teste04'] > + y_labels = [ '0001', '0002', '0003', '0004', '0005', '0006', '0007', '0008', '0009', '0010' ] > + colors = [ (1.0, 0.0, 0.0), (1.0, 0.7, 0.0), (1.0, 1.0, 0.0), (0.0, 1.0, 0.0) ] > + CairoPlot.gantt_chart('gantt_teste', pieces, 600, 300, x_labels, y_labels, colors) > + ''' > + > + plot = GanttChart(name, pieces, width, height, x_labels, y_labels, colors) > + plot.render() > + plot.commit() > + > +def vertical_bar_plot(name, > + data, > + width, > + height, > + background = "white light_gray", > + border = 0, > + display_values = False, > + grid = False, > + rounded_corners = False, > + stack = False, > + three_dimension = False, > + series_labels = None, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + colors = None): > + #TODO: Fix docstring for vertical_bar_plot > + ''' > + - Function to generate vertical Bar Plot Charts. > + > + bar_plot(name, data, width, height, background, border, grid, rounded_corners, three_dimension, > + x_labels, y_labels, x_bounds, y_bounds, colors): > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtime; > + data - The list, list of lists or dictionary holding the data to be plotted; > + width, height - Dimensions of the output image; > + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. > + If left None, a gray to white gradient will be generated; > + border - Distance in pixels of a square border into which the graphics will be drawn; > + grid - Whether or not the gris is to be drawn; > + rounded_corners - Whether or not the bars should have rounded corners; > + three_dimension - Whether or not the bars should be drawn in pseudo 3D; > + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; > + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; > + colors - List containing the colors expected for each of the bars. > + > + - Example of use > + > + data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] > + CairoPlot.vertical_bar_plot ('bar2', data, 400, 300, border = 20, grid = True, rounded_corners = False) > + ''' > + > + plot = VerticalBarPlot(name, data, width, height, background, border, > + display_values, grid, rounded_corners, stack, three_dimension, > + series_labels, x_labels, y_labels, x_bounds, y_bounds, colors) > + plot.render() > + plot.commit() > + > +def horizontal_bar_plot(name, > + data, > + width, > + height, > + background = "white light_gray", > + border = 0, > + display_values = False, > + grid = False, > + rounded_corners = False, > + stack = False, > + three_dimension = False, > + series_labels = None, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + colors = None): > + > + #TODO: Fix docstring for horizontal_bar_plot > + ''' > + - Function to generate Horizontal Bar Plot Charts. > + > + bar_plot(name, data, width, height, background, border, grid, rounded_corners, three_dimension, > + x_labels, y_labels, x_bounds, y_bounds, colors): > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtime; > + data - The list, list of lists or dictionary holding the data to be plotted; > + width, height - Dimensions of the output image; > + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. > + If left None, a gray to white gradient will be generated; > + border - Distance in pixels of a square border into which the graphics will be drawn; > + grid - Whether or not the gris is to be drawn; > + rounded_corners - Whether or not the bars should have rounded corners; > + three_dimension - Whether or not the bars should be drawn in pseudo 3D; > + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; > + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; > + colors - List containing the colors expected for each of the bars. > + > + - Example of use > + > + data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] > + CairoPlot.bar_plot ('bar2', data, 400, 300, border = 20, grid = True, rounded_corners = False) > + ''' > + > + plot = HorizontalBarPlot(name, data, width, height, background, border, > + display_values, grid, rounded_corners, stack, three_dimension, > + series_labels, x_labels, y_labels, x_bounds, y_bounds, colors) > + plot.render() > + plot.commit() > + > +def stream_chart(name, > + data, > + width, > + height, > + background = "white light_gray", > + border = 0, > + grid = False, > + series_legend = None, > + x_labels = None, > + x_bounds = None, > + y_bounds = None, > + colors = None): > + > + #TODO: Fix docstring for horizontal_bar_plot > + plot = StreamChart(name, data, width, height, background, border, > + grid, series_legend, x_labels, x_bounds, y_bounds, colors) > + plot.render() > + plot.commit() > + > + > +if __name__ == "__main__": > + import tests > + import seriestests > diff --git a/bindings/python/examples/output_format_modules/pprint_table.py b/bindings/python/examples/output_format_modules/pprint_table.py > new file mode 100644 > index 0000000..1cd8620 > --- /dev/null > +++ b/bindings/python/examples/output_format_modules/pprint_table.py > @@ -0,0 +1,35 @@ > +# This module is used to pretty-print a table > +# Adapted from > +# http://ginstrom.com/scribbles/2007/09/04/pretty-printing-a-table-in-python/ > + > +import sys > + > +def get_max_width(table, index): > + """Get the maximum width of the given column index""" > + > + return max([len(str(row[index])) for row in table]) > + > + > +def pprint_table(table, nbLeft=1, out=sys.stdout): > + """ > + Prints out a table of data, padded for alignment > + @param table: The table to print. A list of lists. > + Each row must have the same number of columns. > + @param nbLeft: The number of columns aligned left > + @param out: Output stream (file-like object) > + """ > + > + col_paddings = [] > + > + for i in range(len(table[0])): > + col_paddings.append(get_max_width(table, i)) > + > + for row in table: > + # left cols > + for i in range(nbLeft): > + print >> out, str(row[i]).ljust(col_paddings[i] + 1), > + # rest of the cols > + for i in range(nbLeft, len(row)): > + col = str(row[i]).rjust(col_paddings[i] + 2) > + print >> out, col, > + print >> out > diff --git a/bindings/python/examples/output_format_modules/series.py b/bindings/python/examples/output_format_modules/series.py > new file mode 100755 > index 0000000..8e8b236 > --- /dev/null > +++ b/bindings/python/examples/output_format_modules/series.py > @@ -0,0 +1,1140 @@ > +#!/usr/bin/env python > +# -*- coding: utf-8 -*- > + > +# Serie.py > +# > +# Copyright (c) 2008 Magnun Leno da Silva > +# > +# Author: Magnun Leno da Silva > +# > +# This program is free software; you can redistribute it and/or > +# modify it under the terms of the GNU Lesser General Public License > +# as published by the Free Software Foundation; either version 2 of > +# the License, or (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU Lesser General Public > +# License along with this program; if not, write to the Free Software > +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 > +# USA > + > +# Contributor: Rodrigo Moreiro Araujo > + > +#import cairoplot > +import doctest > + > +NUMTYPES = (int, float, long) > +LISTTYPES = (list, tuple) > +STRTYPES = (str, unicode) > +FILLING_TYPES = ['linear', 'solid', 'gradient'] > +DEFAULT_COLOR_FILLING = 'solid' > +#TODO: Define default color list > +DEFAULT_COLOR_LIST = None > + > +class Data(object): > + ''' > + Class that models the main data structure. > + It can hold: > + - a number type (int, float or long) > + - a tuple, witch represents a point and can have 2 or 3 items (x,y,z) > + - if a list is passed it will be converted to a tuple. > + > + obs: In case a tuple is passed it will convert to tuple > + ''' > + def __init__(self, data=None, name=None, parent=None): > + ''' > + Starts main atributes from the Data class > + @name - Name for each point; > + @content - The real data, can be an int, float, long or tuple, which > + represents a point (x,y) or (x,y,z); > + @parent - A pointer that give the data access to it's parent. > + > + Usage: > + >>> d = Data(name='empty'); print d > + empty: () > + >>> d = Data((1,1),'point a'); print d > + point a: (1, 1) > + >>> d = Data((1,2,3),'point b'); print d > + point b: (1, 2, 3) > + >>> d = Data([2,3],'point c'); print d > + point c: (2, 3) > + >>> d = Data(12, 'simple value'); print d > + simple value: 12 > + ''' > + # Initial values > + self.__content = None > + self.__name = None > + > + # Setting passed values > + self.parent = parent > + self.name = name > + self.content = data > + > + # Name property > + @apply > + def name(): > + doc = ''' > + Name is a read/write property that controls the input of name. > + - If passed an invalid value it cleans the name with None > + > + Usage: > + >>> d = Data(13); d.name = 'name_test'; print d > + name_test: 13 > + >>> d.name = 11; print d > + 13 > + >>> d.name = 'other_name'; print d > + other_name: 13 > + >>> d.name = None; print d > + 13 > + >>> d.name = 'last_name'; print d > + last_name: 13 > + >>> d.name = ''; print d > + 13 > + ''' > + def fget(self): > + ''' > + returns the name as a string > + ''' > + return self.__name > + > + def fset(self, name): > + ''' > + Sets the name of the Data > + ''' > + if type(name) in STRTYPES and len(name) > 0: > + self.__name = name > + else: > + self.__name = None > + > + > + > + return property(**locals()) > + > + # Content property > + @apply > + def content(): > + doc = ''' > + Content is a read/write property that validate the data passed > + and return it. > + > + Usage: > + >>> d = Data(); d.content = 13; d.content > + 13 > + >>> d = Data(); d.content = (1,2); d.content > + (1, 2) > + >>> d = Data(); d.content = (1,2,3); d.content > + (1, 2, 3) > + >>> d = Data(); d.content = [1,2,3]; d.content > + (1, 2, 3) > + >>> d = Data(); d.content = [1.5,.2,3.3]; d.content > + (1.5, 0.20000000000000001, 3.2999999999999998) > + ''' > + def fget(self): > + ''' > + Return the content of Data > + ''' > + return self.__content > + > + def fset(self, data): > + ''' > + Ensures that data is a valid tuple/list or a number (int, float > + or long) > + ''' > + # Type: None > + if data is None: > + self.__content = None > + return > + > + # Type: Int or Float > + elif type(data) in NUMTYPES: > + self.__content = data > + > + # Type: List or Tuple > + elif type(data) in LISTTYPES: > + # Ensures the correct size > + if len(data) not in (2, 3): > + raise TypeError, "Data (as list/tuple) must have 2 or 3 items" > + return > + > + # Ensures that all items in list/tuple is a number > + isnum = lambda x : type(x) not in NUMTYPES > + > + if max(map(isnum, data)): > + # An item in data isn't an int or a float > + raise TypeError, "All content of data must be a number (int or float)" > + > + # Convert the tuple to list > + if type(data) is list: > + data = tuple(data) > + > + # Append a copy and sets the type > + self.__content = data[:] > + > + # Unknown type! > + else: > + self.__content = None > + raise TypeError, "Data must be an int, float or a tuple with two or three items" > + return > + > + return property(**locals()) > + > + > + def clear(self): > + ''' > + Clear the all Data (content, name and parent) > + ''' > + self.content = None > + self.name = None > + self.parent = None > + > + def copy(self): > + ''' > + Returns a copy of the Data structure > + ''' > + # The copy > + new_data = Data() > + if self.content is not None: > + # If content is a point > + if type(self.content) is tuple: > + new_data.__content = self.content[:] > + > + # If content is a number > + else: > + new_data.__content = self.content > + > + # If it has a name > + if self.name is not None: > + new_data.__name = self.name > + > + return new_data > + > + def __str__(self): > + ''' > + Return a string representation of the Data structure > + ''' > + if self.name is None: > + if self.content is None: > + return '' > + return str(self.content) > + else: > + if self.content is None: > + return self.name+": ()" > + return self.name+": "+str(self.content) > + > + def __len__(self): > + ''' > + Return the length of the Data. > + - If it's a number return 1; > + - If it's a list return it's length; > + - If its None return 0. > + ''' > + if self.content is None: > + return 0 > + elif type(self.content) in NUMTYPES: > + return 1 > + return len(self.content) > + > + > + > + > +class Group(object): > + ''' > + Class that models a group of data. Every value (int, float, long, tuple > + or list) passed is converted to a list of Data. > + It can receive: > + - A single number (int, float, long); > + - A list of numbers; > + - A tuple of numbers; > + - An instance of Data; > + - A list of Data; > + > + Obs: If a tuple with 2 or 3 items is passed it is converted to a point. > + If a tuple with only 1 item is passed it's converted to a number; > + If a tuple with more than 2 items is passed it's converted to a > + list of numbers > + ''' > + def __init__(self, group=None, name=None, parent=None): > + ''' > + Starts main atributes in Group instance. > + @data_list - a list of data which forms the group; > + @range - a range that represent the x axis of possible functions; > + @name - name of the data group; > + @parent - the Serie parent of this group. > + > + Usage: > + >>> g = Group(13, 'simple number'); print g > + simple number ['13'] > + >>> g = Group((1,2), 'simple point'); print g > + simple point ['(1, 2)'] > + >>> g = Group([1,2,3,4], 'list of numbers'); print g > + list of numbers ['1', '2', '3', '4'] > + >>> g = Group((1,2,3,4),'int in tuple'); print g > + int in tuple ['1', '2', '3', '4'] > + >>> g = Group([(1,2),(2,3),(3,4)], 'list of points'); print g > + list of points ['(1, 2)', '(2, 3)', '(3, 4)'] > + >>> g = Group([[1,2,3],[1,2,3]], '2D coordinate lists'); print g > + 2D coordinated lists ['(1, 1)', '(2, 2)', '(3, 3)'] > + >>> g = Group([[1,2],[1,2],[1,2]], '3D coordinate lists'); print g > + 3D coordinated lists ['(1, 1, 1)', '(2, 2, 2)'] > + ''' > + # Initial values > + self.__data_list = [] > + self.__range = [] > + self.__name = None > + > + > + self.parent = parent > + self.name = name > + self.data_list = group > + > + # Name property > + @apply > + def name(): > + doc = ''' > + Name is a read/write property that controls the input of name. > + - If passed an invalid value it cleans the name with None > + > + Usage: > + >>> g = Group(13); g.name = 'name_test'; print g > + name_test ['13'] > + >>> g.name = 11; print g > + ['13'] > + >>> g.name = 'other_name'; print g > + other_name ['13'] > + >>> g.name = None; print g > + ['13'] > + >>> g.name = 'last_name'; print g > + last_name ['13'] > + >>> g.name = ''; print g > + ['13'] > + ''' > + def fget(self): > + ''' > + Returns the name as a string > + ''' > + return self.__name > + > + def fset(self, name): > + ''' > + Sets the name of the Group > + ''' > + if type(name) in STRTYPES and len(name) > 0: > + self.__name = name > + else: > + self.__name = None > + > + return property(**locals()) > + > + # data_list property > + @apply > + def data_list(): > + doc = ''' > + The data_list is a read/write property that can be a list of > + numbers, a list of points or a list of 2 or 3 coordinate lists. This > + property uses mainly the self.add_data method. > + > + Usage: > + >>> g = Group(); g.data_list = 13; print g > + ['13'] > + >>> g.data_list = (1,2); print g > + ['(1, 2)'] > + >>> g.data_list = Data((1,2),'point a'); print g > + ['point a: (1, 2)'] > + >>> g.data_list = [1,2,3]; print g > + ['1', '2', '3'] > + >>> g.data_list = (1,2,3,4); print g > + ['1', '2', '3', '4'] > + >>> g.data_list = [(1,2),(2,3),(3,4)]; print g > + ['(1, 2)', '(2, 3)', '(3, 4)'] > + >>> g.data_list = [[1,2],[1,2]]; print g > + ['(1, 1)', '(2, 2)'] > + >>> g.data_list = [[1,2],[1,2],[1,2]]; print g > + ['(1, 1, 1)', '(2, 2, 2)'] > + >>> g.range = (10); g.data_list = lambda x:x**2; print g > + ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)'] > + ''' > + def fget(self): > + ''' > + Returns the value of data_list > + ''' > + return self.__data_list > + > + def fset(self, group): > + ''' > + Ensures that group is valid. > + ''' > + # None > + if group is None: > + self.__data_list = [] > + > + # Int/float/long or Instance of Data > + elif type(group) in NUMTYPES or isinstance(group, Data): > + # Clean data_list > + self.__data_list = [] > + self.add_data(group) > + > + # One point > + elif type(group) is tuple and len(group) in (2,3): > + self.__data_list = [] > + self.add_data(group) > + > + # list of items > + elif type(group) in LISTTYPES and type(group[0]) is not list: > + # Clean data_list > + self.__data_list = [] > + for item in group: > + # try to append and catch an exception > + self.add_data(item) > + > + # function lambda > + elif callable(group): > + # Explicit is better than implicit > + function = group > + # Has range > + if len(self.range) is not 0: > + # Clean data_list > + self.__data_list = [] > + # Generate values for the lambda function > + for x in self.range: > + #self.add_data((x,round(group(x),2))) > + self.add_data((x,function(x))) > + > + # Only have range in parent > + elif self.parent is not None and len(self.parent.range) is not 0: > + # Copy parent range > + self.__range = self.parent.range[:] > + # Clean data_list > + self.__data_list = [] > + # Generate values for the lambda function > + for x in self.range: > + #self.add_data((x,round(group(x),2))) > + self.add_data((x,function(x))) > + > + # Don't have range anywhere > + else: > + # x_data don't exist > + raise Exception, "Data argument is valid but to use function type please set x_range first" > + > + # Coordinate Lists > + elif type(group) in LISTTYPES and type(group[0]) is list: > + # Clean data_list > + self.__data_list = [] > + data = [] > + if len(group) == 3: > + data = zip(group[0], group[1], group[2]) > + elif len(group) == 2: > + data = zip(group[0], group[1]) > + else: > + raise TypeError, "Only one list of coordinates was received." > + > + for item in data: > + self.add_data(item) > + > + else: > + raise TypeError, "Group type not supported" > + > + return property(**locals()) > + > + @apply > + def range(): > + doc = ''' > + The range is a read/write property that generates a range of values > + for the x axis of the functions. When passed a tuple it almost works > + like the built-in range funtion: > + - 1 item, represent the end of the range started from 0; > + - 2 items, represents the start and the end, respectively; > + - 3 items, the last one represents the step; > + > + When passed a list the range function understands as a valid range. > + > + Usage: > + >>> g = Group(); g.range = 10; print g.range > + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0] > + >>> g = Group(); g.range = (5); print g.range > + [0.0, 1.0, 2.0, 3.0, 4.0] > + >>> g = Group(); g.range = (1,7); print g.range > + [1.0, 2.0, 3.0, 4.0, 5.0, 6.0] > + >>> g = Group(); g.range = (0,10,2); print g.range > + [0.0, 2.0, 4.0, 6.0, 8.0] > + >>> > + >>> g = Group(); g.range = [0]; print g.range > + [0.0] > + >>> g = Group(); g.range = [0,10,20]; print g.range > + [0.0, 10.0, 20.0] > + ''' > + def fget(self): > + ''' > + Returns the range > + ''' > + return self.__range > + > + def fset(self, x_range): > + ''' > + Controls the input of a valid type and generate the range > + ''' > + # if passed a simple number convert to tuple > + if type(x_range) in NUMTYPES: > + x_range = (x_range,) > + > + # A list, just convert to float > + if type(x_range) is list and len(x_range) > 0: > + # Convert all to float > + x_range = map(float, x_range) > + # Prevents repeated values and convert back to list > + self.__range = list(set(x_range[:])) > + # Sort the list to ascending order > + self.__range.sort() > + > + # A tuple, must check the lengths and generate the values > + elif type(x_range) is tuple and len(x_range) in (1,2,3): > + # Convert all to float > + x_range = map(float, x_range) > + > + # Inital values > + start = 0.0 > + step = 1.0 > + end = 0.0 > + > + # Only the end and it can't be less or iqual to 0 > + if len(x_range) is 1 and x_range > 0: > + end = x_range[0] > + > + # The start and the end but the start must be less then the end > + elif len(x_range) is 2 and x_range[0] < x_range[1]: > + start = x_range[0] > + end = x_range[1] > + > + # All 3, but the start must be less then the end > + elif x_range[0] <= x_range[1]: > + start = x_range[0] > + end = x_range[1] > + step = x_range[2] > + > + # Starts the range > + self.__range = [] > + # Generate the range > + # Can't use the range function because it doesn't support float values > + while start < end: > + self.__range.append(start) > + start += step > + > + # Incorrect type > + else: > + raise Exception, "x_range must be a list with one or more items or a tuple with 2 or 3 items" > + > + return property(**locals()) > + > + def add_data(self, data, name=None): > + ''' > + Append a new data to the data_list. > + - If data is an instance of Data, append it > + - If it's an int, float, tuple or list create an instance of Data and append it > + > + Usage: > + >>> g = Group() > + >>> g.add_data(12); print g > + ['12'] > + >>> g.add_data(7,'other'); print g > + ['12', 'other: 7'] > + >>> > + >>> g = Group() > + >>> g.add_data((1,1),'a'); print g > + ['a: (1, 1)'] > + >>> g.add_data((2,2),'b'); print g > + ['a: (1, 1)', 'b: (2, 2)'] > + >>> > + >>> g.add_data(Data((1,2),'c')); print g > + ['a: (1, 1)', 'b: (2, 2)', 'c: (1, 2)'] > + ''' > + if not isinstance(data, Data): > + # Try to convert > + data = Data(data,name,self) > + > + if data.content is not None: > + self.__data_list.append(data.copy()) > + self.__data_list[-1].parent = self > + > + > + def to_list(self): > + ''' > + Returns the group as a list of numbers (int, float or long) or a > + list of tuples (points 2D or 3D). > + > + Usage: > + >>> g = Group([1,2,3,4],'g1'); g.to_list() > + [1, 2, 3, 4] > + >>> g = Group([(1,2),(2,3),(3,4)],'g2'); g.to_list() > + [(1, 2), (2, 3), (3, 4)] > + >>> g = Group([(1,2,3),(3,4,5)],'g2'); g.to_list() > + [(1, 2, 3), (3, 4, 5)] > + ''' > + return [data.content for data in self] > + > + def copy(self): > + ''' > + Returns a copy of this group > + ''' > + new_group = Group() > + new_group.__name = self.__name > + if self.__range is not None: > + new_group.__range = self.__range[:] > + for data in self: > + new_group.add_data(data.copy()) > + return new_group > + > + def get_names(self): > + ''' > + Return a list with the names of all data in this group > + ''' > + names = [] > + for data in self: > + if data.name is None: > + names.append('Data '+str(data.index()+1)) > + else: > + names.append(data.name) > + return names > + > + > + def __str__ (self): > + ''' > + Returns a string representing the Group > + ''' > + ret = "" > + if self.name is not None: > + ret += self.name + " " > + if len(self) > 0: > + list_str = [str(item) for item in self] > + ret += str(list_str) > + else: > + ret += "[]" > + return ret > + > + def __getitem__(self, key): > + ''' > + Makes a Group iterable, based in the data_list property > + ''' > + return self.data_list[key] > + > + def __len__(self): > + ''' > + Returns the length of the Group, based in the data_list property > + ''' > + return len(self.data_list) > + > + > +class Colors(object): > + ''' > + Class that models the colors its labels (names) and its properties, RGB > + and filling type. > + > + It can receive: > + - A list where each item is a list with 3 or 4 items. The > + first 3 items represent the RGB values and the last argument > + defines the filling type. The list will be converted to a dict > + and each color will receve a name based in its position in the > + list. > + - A dictionary where each key will be the color name and its item > + can be a list with 3 or 4 items. The first 3 items represent > + the RGB colors and the last argument defines the filling type. > + ''' > + def __init__(self, color_list=None): > + ''' > + Start the color_list property > + @ color_list - the list or dict contaning the colors properties. > + ''' > + self.__color_list = None > + > + self.color_list = color_list > + > + @apply > + def color_list(): > + doc = ''' > + >>> c = Colors([[1,1,1],[2,2,2,'linear'],[3,3,3,'gradient']]) > + >>> print c.color_list > + {'Color 2': [2, 2, 2, 'linear'], 'Color 3': [3, 3, 3, 'gradient'], 'Color 1': [1, 1, 1, 'solid']} > + >>> c.color_list = [[1,1,1],(2,2,2,'solid'),(3,3,3,'linear')] > + >>> print c.color_list > + {'Color 2': [2, 2, 2, 'solid'], 'Color 3': [3, 3, 3, 'linear'], 'Color 1': [1, 1, 1, 'solid']} > + >>> c.color_list = {'a':[1,1,1],'b':(2,2,2,'solid'),'c':(3,3,3,'linear'), 'd':(4,4,4)} > + >>> print c.color_list > + {'a': [1, 1, 1, 'solid'], 'c': [3, 3, 3, 'linear'], 'b': [2, 2, 2, 'solid'], 'd': [4, 4, 4, 'solid']} > + ''' > + def fget(self): > + ''' > + Return the color list > + ''' > + return self.__color_list > + > + def fset(self, color_list): > + ''' > + Format the color list to a dictionary > + ''' > + if color_list is None: > + self.__color_list = None > + return > + > + if type(color_list) in LISTTYPES and type(color_list[0]) in LISTTYPES: > + old_color_list = color_list[:] > + color_list = {} > + for index, color in enumerate(old_color_list): > + if len(color) is 3 and max(map(type, color)) in NUMTYPES: > + color_list['Color '+str(index+1)] = list(color)+[DEFAULT_COLOR_FILLING] > + elif len(color) is 4 and max(map(type, color[:-1])) in NUMTYPES and color[-1] in FILLING_TYPES: > + color_list['Color '+str(index+1)] = list(color) > + else: > + raise TypeError, "Unsuported color format" > + elif type(color_list) is not dict: > + raise TypeError, "Unsuported color format" > + > + for name, color in color_list.items(): > + if len(color) is 3: > + if max(map(type, color)) in NUMTYPES: > + color_list[name] = list(color)+[DEFAULT_COLOR_FILLING] > + else: > + raise TypeError, "Unsuported color format" > + elif len(color) is 4: > + if max(map(type, color[:-1])) in NUMTYPES and color[-1] in FILLING_TYPES: > + color_list[name] = list(color) > + else: > + raise TypeError, "Unsuported color format" > + self.__color_list = color_list.copy() > + > + return property(**locals()) > + > + > +class Series(object): > + ''' > + Class that models a Series (group of groups). Every value (int, float, > + long, tuple or list) passed is converted to a list of Group or Data. > + It can receive: > + - a single number or point, will be converted to a Group of one Data; > + - a list of numbers, will be converted to a group of numbers; > + - a list of tuples, will converted to a single Group of points; > + - a list of lists of numbers, each 'sublist' will be converted to a > + group of numbers; > + - a list of lists of tuples, each 'sublist' will be converted to a > + group of points; > + - a list of lists of lists, the content of the 'sublist' will be > + processed as coordinated lists and the result will be converted to > + a group of points; > + - a Dictionary where each item can be the same of the list: number, > + point, list of numbers, list of points or list of lists (coordinated > + lists); > + - an instance of Data; > + - an instance of group. > + ''' > + def __init__(self, series=None, name=None, property=[], colors=None): > + ''' > + Starts main atributes in Group instance. > + @series - a list, dict of data of which the series is composed; > + @name - name of the series; > + @property - a list/dict of properties to be used in the plots of > + this Series > + > + Usage: > + >>> print Series([1,2,3,4]) > + ["Group 1 ['1', '2', '3', '4']"] > + >>> print Series([[1,2,3],[4,5,6]]) > + ["Group 1 ['1', '2', '3']", "Group 2 ['4', '5', '6']"] > + >>> print Series((1,2)) > + ["Group 1 ['(1, 2)']"] > + >>> print Series([(1,2),(2,3)]) > + ["Group 1 ['(1, 2)', '(2, 3)']"] > + >>> print Series([[(1,2),(2,3)],[(4,5),(5,6)]]) > + ["Group 1 ['(1, 2)', '(2, 3)']", "Group 2 ['(4, 5)', '(5, 6)']"] > + >>> print Series([[[1,2,3],[1,2,3],[1,2,3]]]) > + ["Group 1 ['(1, 1, 1)', '(2, 2, 2)', '(3, 3, 3)']"] > + >>> print Series({'g1':[1,2,3], 'g2':[4,5,6]}) > + ["g1 ['1', '2', '3']", "g2 ['4', '5', '6']"] > + >>> print Series({'g1':[(1,2),(2,3)], 'g2':[(4,5),(5,6)]}) > + ["g1 ['(1, 2)', '(2, 3)']", "g2 ['(4, 5)', '(5, 6)']"] > + >>> print Series({'g1':[[1,2],[1,2]], 'g2':[[4,5],[4,5]]}) > + ["g1 ['(1, 1)', '(2, 2)']", "g2 ['(4, 4)', '(5, 5)']"] > + >>> print Series(Data(1,'d1')) > + ["Group 1 ['d1: 1']"] > + >>> print Series(Group([(1,2),(2,3)],'g1')) > + ["g1 ['(1, 2)', '(2, 3)']"] > + ''' > + # Intial values > + self.__group_list = [] > + self.__name = None > + self.__range = None > + > + # TODO: Implement colors with filling > + self.__colors = None > + > + self.name = name > + self.group_list = series > + self.colors = colors > + > + # Name property > + @apply > + def name(): > + doc = ''' > + Name is a read/write property that controls the input of name. > + - If passed an invalid value it cleans the name with None > + > + Usage: > + >>> s = Series(13); s.name = 'name_test'; print s > + name_test ["Group 1 ['13']"] > + >>> s.name = 11; print s > + ["Group 1 ['13']"] > + >>> s.name = 'other_name'; print s > + other_name ["Group 1 ['13']"] > + >>> s.name = None; print s > + ["Group 1 ['13']"] > + >>> s.name = 'last_name'; print s > + last_name ["Group 1 ['13']"] > + >>> s.name = ''; print s > + ["Group 1 ['13']"] > + ''' > + def fget(self): > + ''' > + Returns the name as a string > + ''' > + return self.__name > + > + def fset(self, name): > + ''' > + Sets the name of the Group > + ''' > + if type(name) in STRTYPES and len(name) > 0: > + self.__name = name > + else: > + self.__name = None > + > + return property(**locals()) > + > + > + > + # Colors property > + @apply > + def colors(): > + doc = ''' > + >>> s = Series() > + >>> s.colors = [[1,1,1],[2,2,2,'linear'],[3,3,3,'gradient']] > + >>> print s.colors > + {'Color 2': [2, 2, 2, 'linear'], 'Color 3': [3, 3, 3, 'gradient'], 'Color 1': [1, 1, 1, 'solid']} > + >>> s.colors = [[1,1,1],(2,2,2,'solid'),(3,3,3,'linear')] > + >>> print s.colors > + {'Color 2': [2, 2, 2, 'solid'], 'Color 3': [3, 3, 3, 'linear'], 'Color 1': [1, 1, 1, 'solid']} > + >>> s.colors = {'a':[1,1,1],'b':(2,2,2,'solid'),'c':(3,3,3,'linear'), 'd':(4,4,4)} > + >>> print s.colors > + {'a': [1, 1, 1, 'solid'], 'c': [3, 3, 3, 'linear'], 'b': [2, 2, 2, 'solid'], 'd': [4, 4, 4, 'solid']} > + ''' > + def fget(self): > + ''' > + Return the color list > + ''' > + return self.__colors.color_list > + > + def fset(self, colors): > + ''' > + Format the color list to a dictionary > + ''' > + self.__colors = Colors(colors) > + > + return property(**locals()) > + > + @apply > + def range(): > + doc = ''' > + The range is a read/write property that generates a range of values > + for the x axis of the functions. When passed a tuple it almost works > + like the built-in range funtion: > + - 1 item, represent the end of the range started from 0; > + - 2 items, represents the start and the end, respectively; > + - 3 items, the last one represents the step; > + > + When passed a list the range function understands as a valid range. > + > + Usage: > + >>> s = Series(); s.range = 10; print s.range > + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0] > + >>> s = Series(); s.range = (5); print s.range > + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0] > + >>> s = Series(); s.range = (1,7); print s.range > + [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0] > + >>> s = Series(); s.range = (0,10,2); print s.range > + [0.0, 2.0, 4.0, 6.0, 8.0, 10.0] > + >>> > + >>> s = Series(); s.range = [0]; print s.range > + [0.0] > + >>> s = Series(); s.range = [0,10,20]; print s.range > + [0.0, 10.0, 20.0] > + ''' > + def fget(self): > + ''' > + Returns the range > + ''' > + return self.__range > + > + def fset(self, x_range): > + ''' > + Controls the input of a valid type and generate the range > + ''' > + # if passed a simple number convert to tuple > + if type(x_range) in NUMTYPES: > + x_range = (x_range,) > + > + # A list, just convert to float > + if type(x_range) is list and len(x_range) > 0: > + # Convert all to float > + x_range = map(float, x_range) > + # Prevents repeated values and convert back to list > + self.__range = list(set(x_range[:])) > + # Sort the list to ascending order > + self.__range.sort() > + > + # A tuple, must check the lengths and generate the values > + elif type(x_range) is tuple and len(x_range) in (1,2,3): > + # Convert all to float > + x_range = map(float, x_range) > + > + # Inital values > + start = 0.0 > + step = 1.0 > + end = 0.0 > + > + # Only the end and it can't be less or iqual to 0 > + if len(x_range) is 1 and x_range > 0: > + end = x_range[0] > + > + # The start and the end but the start must be lesser then the end > + elif len(x_range) is 2 and x_range[0] < x_range[1]: > + start = x_range[0] > + end = x_range[1] > + > + # All 3, but the start must be lesser then the end > + elif x_range[0] < x_range[1]: > + start = x_range[0] > + end = x_range[1] > + step = x_range[2] > + > + # Starts the range > + self.__range = [] > + # Generate the range > + # Cnat use the range function becouse it don't suport float values > + while start <= end: > + self.__range.append(start) > + start += step > + > + # Incorrect type > + else: > + raise Exception, "x_range must be a list with one or more item or a tuple with 2 or 3 items" > + > + return property(**locals()) > + > + @apply > + def group_list(): > + doc = ''' > + The group_list is a read/write property used to pre-process the list > + of Groups. > + It can be: > + - a single number, point or lambda, will be converted to a single > + Group of one Data; > + - a list of numbers, will be converted to a group of numbers; > + - a list of tuples, will converted to a single Group of points; > + - a list of lists of numbers, each 'sublist' will be converted to > + a group of numbers; > + - a list of lists of tuples, each 'sublist' will be converted to a > + group of points; > + - a list of lists of lists, the content of the 'sublist' will be > + processed as coordinated lists and the result will be converted > + to a group of points; > + - a list of lambdas, each lambda represents a Group; > + - a Dictionary where each item can be the same of the list: number, > + point, list of numbers, list of points, list of lists > + (coordinated lists) or lambdas > + - an instance of Data; > + - an instance of group. > + > + Usage: > + >>> s = Series() > + >>> s.group_list = [1,2,3,4]; print s > + ["Group 1 ['1', '2', '3', '4']"] > + >>> s.group_list = [[1,2,3],[4,5,6]]; print s > + ["Group 1 ['1', '2', '3']", "Group 2 ['4', '5', '6']"] > + >>> s.group_list = (1,2); print s > + ["Group 1 ['(1, 2)']"] > + >>> s.group_list = [(1,2),(2,3)]; print s > + ["Group 1 ['(1, 2)', '(2, 3)']"] > + >>> s.group_list = [[(1,2),(2,3)],[(4,5),(5,6)]]; print s > + ["Group 1 ['(1, 2)', '(2, 3)']", "Group 2 ['(4, 5)', '(5, 6)']"] > + >>> s.group_list = [[[1,2,3],[1,2,3],[1,2,3]]]; print s > + ["Group 1 ['(1, 1, 1)', '(2, 2, 2)', '(3, 3, 3)']"] > + >>> s.group_list = [(0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,9)]; print s > + ["Group 1 ['(0.5, 5.5)']", "Group 2 ['(0, 4)', '(6, 8)']", "Group 3 ['(5.5, 7)']", "Group 4 ['(7, 9)']"] > + >>> s.group_list = {'g1':[1,2,3], 'g2':[4,5,6]}; print s > + ["g1 ['1', '2', '3']", "g2 ['4', '5', '6']"] > + >>> s.group_list = {'g1':[(1,2),(2,3)], 'g2':[(4,5),(5,6)]}; print s > + ["g1 ['(1, 2)', '(2, 3)']", "g2 ['(4, 5)', '(5, 6)']"] > + >>> s.group_list = {'g1':[[1,2],[1,2]], 'g2':[[4,5],[4,5]]}; print s > + ["g1 ['(1, 1)', '(2, 2)']", "g2 ['(4, 4)', '(5, 5)']"] > + >>> s.range = 10 > + >>> s.group_list = lambda x:x*2 > + >>> s.group_list = [lambda x:x*2, lambda x:x**2, lambda x:x**3]; print s > + ["Group 1 ['(0.0, 0.0)', '(1.0, 2.0)', '(2.0, 4.0)', '(3.0, 6.0)', '(4.0, 8.0)', '(5.0, 10.0)', '(6.0, 12.0)', '(7.0, 14.0)', '(8.0, 16.0)', '(9.0, 18.0)', '(10.0, 20.0)']", "Group 2 ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)', '(10.0, 100.0)']", "Group 3 ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 8.0)', '(3.0, 27.0)', '(4.0, 64.0)', '(5.0, 125.0)', '(6.0, 216.0)', '(7.0, 343.0)', '(8.0, 512.0)', '(9.0, 729.0)', '(10.0, 1000.0)']"] > + >>> s.group_list = {'linear':lambda x:x*2, 'square':lambda x:x**2, 'cubic':lambda x:x**3}; print s > + ["cubic ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 8.0)', '(3.0, 27.0)', '(4.0, 64.0)', '(5.0, 125.0)', '(6.0, 216.0)', '(7.0, 343.0)', '(8.0, 512.0)', '(9.0, 729.0)', '(10.0, 1000.0)']", "linear ['(0.0, 0.0)', '(1.0, 2.0)', '(2.0, 4.0)', '(3.0, 6.0)', '(4.0, 8.0)', '(5.0, 10.0)', '(6.0, 12.0)', '(7.0, 14.0)', '(8.0, 16.0)', '(9.0, 18.0)', '(10.0, 20.0)']", "square ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)', '(10.0, 100.0)']"] > + >>> s.group_list = Data(1,'d1'); print s > + ["Group 1 ['d1: 1']"] > + >>> s.group_list = Group([(1,2),(2,3)],'g1'); print s > + ["g1 ['(1, 2)', '(2, 3)']"] > + ''' > + def fget(self): > + ''' > + Return the group list. > + ''' > + return self.__group_list > + > + def fset(self, series): > + ''' > + Controls the input of a valid group list. > + ''' > + #TODO: Add support to the following strem of data: [ (0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,9)] > + > + # Type: None > + if series is None: > + self.__group_list = [] > + > + # List or Tuple > + elif type(series) in LISTTYPES: > + self.__group_list = [] > + > + is_function = lambda x: callable(x) > + # Groups > + if list in map(type, series) or max(map(is_function, series)): > + for group in series: > + self.add_group(group) > + > + # single group > + else: > + self.add_group(series) > + > + #old code > + ## List of numbers > + #if type(series[0]) in NUMTYPES or type(series[0]) is tuple: > + # print series > + # self.add_group(series) > + # > + ## List of anything else > + #else: > + # for group in series: > + # self.add_group(group) > + > + # Dict representing series of groups > + elif type(series) is dict: > + self.__group_list = [] > + names = series.keys() > + names.sort() > + for name in names: > + self.add_group(Group(series[name],name,self)) > + > + # A single lambda > + elif callable(series): > + self.__group_list = [] > + self.add_group(series) > + > + # Int/float, instance of Group or Data > + elif type(series) in NUMTYPES or isinstance(series, Group) or isinstance(series, Data): > + self.__group_list = [] > + self.add_group(series) > + > + # Default > + else: > + raise TypeError, "Serie type not supported" > + > + return property(**locals()) > + > + def add_group(self, group, name=None): > + ''' > + Append a new group in group_list > + ''' > + if not isinstance(group, Group): > + #Try to convert > + group = Group(group, name, self) > + > + if len(group.data_list) is not 0: > + # Auto naming groups > + if group.name is None: > + group.name = "Group "+str(len(self.__group_list)+1) > + > + self.__group_list.append(group) > + self.__group_list[-1].parent = self > + > + def copy(self): > + ''' > + Returns a copy of the Series > + ''' > + new_series = Series() > + new_series.__name = self.__name > + if self.__range is not None: > + new_series.__range = self.__range[:] > + #Add color property in the copy method > + #self.__colors = None > + > + for group in self: > + new_series.add_group(group.copy()) > + > + return new_series > + > + def get_names(self): > + ''' > + Returns a list of the names of all groups in the Serie > + ''' > + names = [] > + for group in self: > + if group.name is None: > + names.append('Group '+str(group.index()+1)) > + else: > + names.append(group.name) > + > + return names > + > + def to_list(self): > + ''' > + Returns a list with the content of all groups and data > + ''' > + big_list = [] > + for group in self: > + for data in group: > + if type(data.content) in NUMTYPES: > + big_list.append(data.content) > + else: > + big_list = big_list + list(data.content) > + return big_list > + > + def __getitem__(self, key): > + ''' > + Makes the Series iterable, based in the group_list property > + ''' > + return self.__group_list[key] > + > + def __str__(self): > + ''' > + Returns a string that represents the Series > + ''' > + ret = "" > + if self.name is not None: > + ret += self.name + " " > + if len(self) > 0: > + list_str = [str(item) for item in self] > + ret += str(list_str) > + else: > + ret += "[]" > + return ret > + > + def __len__(self): > + ''' > + Returns the length of the Series, based in the group_lsit property > + ''' > + return len(self.group_list) > + > + > +if __name__ == '__main__': > + doctest.testmod() > diff --git a/bindings/python/examples/sched_switch.py b/bindings/python/examples/sched_switch.py > new file mode 100644 > index 0000000..f252ab5 > --- /dev/null > +++ b/bindings/python/examples/sched_switch.py > @@ -0,0 +1,110 @@ > +# The script takes one optional argument (pid) > +# The script will read events based on pid and > +# print the scheduler switches happening with the process. > +# If no arguments are passed, it displays all the scheduler switches. > +# This can be used to understand which tasks schedule out the current > +# process being traced, and when it gets scheduled in again. > +# The trace needs PID context (lttng add-context -k -t pid) > + > +import sys > +from babeltrace import * > + > +if len(sys.argv) < 2 or len(sys.argv) > 3: > + raise TypeError("Usage: python sched_switch.py [pid] path/to/trace") > +elif len(sys.argv) == 3: > + usePID = True > +else: > + usePID = False > + > + > +ctx = Context() > +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +# Setting iterator > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx, bp) > + > +# Reading events > +event = ctf_it.read_event() > +while event is not None: > + while True: > + if event.get_name() == "sched_switch": > + # Getting scope definition > + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) > + if sco is None: > + print("ERROR: Cannot get definition scope for sched_switch") > + break # Next event > + > + # Getting PID > + pid_field = event.get_field(sco, "_pid") > + pid = pid_field.get_int64() > + > + if ctf.field_error(): > + print("ERROR: Missing PID info for sched_switch") > + break # Next event > + > + if usePID and (pid != long(sys.argv[1])): > + break # Next event > + > + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) > + > + # prev_comm > + field = event.get_field(sco, "_prev_comm") > + prev_comm = field.get_char_array() > + if ctf.field_error(): > + print("ERROR: Missing prev_comm context info") > + > + # prev_tid > + field = event.get_field(sco, "_prev_tid") > + prev_tid = field.get_int64() > + if ctf.field_error(): > + print("ERROR: Missing prev_tid context info") > + > + # prev_prio > + field = event.get_field(sco, "_prev_prio") > + prev_prio = field.get_int64() > + if ctf.field_error(): > + print("ERROR: Missing prev_prio context info") > + > + # prev_state > + field = event.get_field(sco, "_prev_state") > + prev_state = field.get_int64() > + if ctf.field_error(): > + print("ERROR: Missing prev_state context info") > + > + # next_comm > + field = event.get_field(sco, "_next_comm") > + next_comm = field.get_char_array() > + if ctf.field_error(): > + print("ERROR: Missing next_comm context info") > + > + # next_tid > + field = event.get_field(sco, "_next_tid") > + next_tid = field.get_int64() > + if ctf.field_error(): > + print("ERROR: Missing next_tid context info") > + > + # next_prio > + field = event.get_field(sco, "_next_prio") > + next_prio = field.get_int64() > + if ctf.field_error(): > + print("ERROR: Missing next_prio context info") > + > + # Output > + print("sched_switch, pid = {}, TS = {}, prev_comm = {},\n\t" > + "prev_tid = {}, prev_prio = {}, prev_state = {},\n\t" > + "next_comm = {}, next_tid = {}, next_prio = {}".format( > + pid, event.get_timestamp(), prev_comm, prev_tid, > + prev_prio, prev_state, next_comm, next_tid, next_prio)) > + > + break # Next event > + > + # Next event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > +del ctf_it > diff --git a/bindings/python/examples/softirqtimes.py b/bindings/python/examples/softirqtimes.py > new file mode 100644 > index 0000000..19b2deb > --- /dev/null > +++ b/bindings/python/examples/softirqtimes.py > @@ -0,0 +1,130 @@ > +# The script checks the trace for the amount of time > +# spent from each softirq_raise to softirq_exit. > +# It prints out the min, max (with timestamp), > +# average times, the standard deviation and the total count. > +# Using the cairoplot module, a .svg graph is also outputted > +# showing the taken time in function of the time since the > +# beginning of the trace. > + > +import sys, math > +from output_format_modules import cairoplot > +from babeltrace import * > + > +if len(sys.argv) < 2: > + raise TypeError("Usage: python softirqtimes.py path/to/trace") > + > +ctx = Context() > +ret = ctx.add_trace(sys.argv[1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +time_taken = [] > +graph_data = [] > +max_time = (0.0, 0.0) # (val, ts) > + > +# tmp template: {(cpu_id, vec):TS raise} > +tmp = {} > +largest_val = 0 > + > +# Setting iterator > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx, bp) > + > +# Reading events > +event = ctf_it.read_event() > +start_time = event.get_timestamp() > +while(event is not None): > + > + event_name = event.get_name() > + error = True > + appendNext = False > + > + if event_name == 'softirq_raise' or event_name == 'softirq_exit': > + # Recover cpu_id and vec values to make a key to tmp > + error = False > + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) > + field = event.get_field(scope, "cpu_id") > + cpu_id = field.get_uint64() > + if ctf.field_error(): > + print("ERROR: Missing cpu_id info for {}".format( > + event.get_name())) > + error = True > + > + scope = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) > + field = event.get_field(scope, "_vec") > + vec = field.get_uint64() > + if ctf.field_error(): > + print("ERROR: Missing vec info for {}".format( > + event.get_name())) > + error = True > + key = (cpu_id, vec) > + > + if event_name == 'softirq_raise' and not error: > + # Add timestamp to tmp > + if key in tmp: > + # If key already exists > + i = 0 > + while True: > + # Add index > + key = (cpu_id, vec, i) > + if key in tmp: > + i += 1 > + continue > + if i > largest_val: > + largest_val = i > + break > + > + tmp[key] = event.get_timestamp() > + > + if event_name == 'softirq_exit' and not error: > + # Saving data for output > + # Key check > + if not (key in tmp): > + i = 0 > + while i <= largest_val: > + key = (key[0], key[1], i) > + if key in tmp: > + break > + i += 1 > + > + raise_timestamp = tmp[key] > + time_data = event.get_timestamp() - tmp.pop(key) > + if time_data > max_time[0]: > + # max_time = (val, ts) > + max_time = (time_data, raise_timestamp) > + time_taken.append(time_data) > + graph_data.append((raise_timestamp - start_time, time_data)) > + > + # Next Event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > + > +del ctf_it > + > +# Standard dev. calc. > +try: > + mean = sum(time_taken)/float(len(time_taken)) > +except ZeroDivisionError: > + raise TypeError("empty data") > +deviations_squared = [] > +for x in time_taken: > + deviations_squared.append(math.pow((x - mean), 2)) > +try: > + stddev = math.sqrt(sum(deviations_squared) / (len(deviations_squared) - 1)) > +except ZeroDivisionError: > + stddev = '-' > + > +# Terminal output > +print("AVG TIME: {} ns".format(mean)) > +print("MIN TIME: {} ns".format(min(time_taken))) > +print("MAX TIME: {} ns, TS: {}".format(max_time[0], max_time[1])) > +print("STD DEV: {}".format(stddev)) > +print("TOTAL COUNT: {}".format(len(time_taken))) > + > +# Graph output > +cairoplot.scatter_plot ( 'softirqtimes.svg', data = graph_data, > + width = 5000, height = 4000, border = 20, axis = True, > + grid = True, series_colors = ["red"] ) > diff --git a/bindings/python/examples/syscalls_by_pid.py b/bindings/python/examples/syscalls_by_pid.py > new file mode 100644 > index 0000000..f6127ed > --- /dev/null > +++ b/bindings/python/examples/syscalls_by_pid.py > @@ -0,0 +1,61 @@ > +# The script checks all syscall in the trace and prints a list > +# showing the number of systemcalls executed by each PID > +# ordered from greatest to least number of syscalls. > +# The trace needs PID context (lttng add-context -k -t pid) > + > +import sys > +from babeltrace import * > +from output_format_modules.pprint_table import pprint_table as pprint > + > +if len(sys.argv) < 2 : > + raise TypeError("Usage: python syscalls_by_pid.py path/to/trace") > + > +ctx = Context() > +ret = ctx.add_trace(sys.argv[1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +data = {} > + > +# Setting iterator > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx, bp) > + > +# Reading events > +event = ctf_it.read_event() > +while event is not None: > + if event.get_name().find("sys") >= 0: > + # Getting scope definition > + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) > + if sco is None: > + print("ERROR: Cannot get definition scope for {}".format( > + event.get_name())) > + else: > + # Getting PID > + pid_field = event.get_field(sco, "_pid") > + pid = pid_field.get_int64() > + > + if ctf.field_error(): > + print("ERROR: Missing PID info for sched_switch".format( > + event.get_name())) > + elif pid in data: > + data[pid] += 1 > + else: > + data[pid] = 1 > + # Next event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > +del ctf_it > + > +# Setting table for output > +table = [] > +for item in data: > + table.append([data[item], item]) # [count, pid] > +table.sort(reverse = True) # [big count first, pid] > +for i in range(len(table)): > + table[i].reverse() # [pid, big count first] > +table.insert(0, ["PID", "SYSCALL COUNT"]) > +pprint(table) > diff --git a/bindings/python/python-complements.c b/bindings/python/python-complements.c > new file mode 100644 > index 0000000..8c4d811 > --- /dev/null > +++ b/bindings/python/python-complements.c > @@ -0,0 +1,105 @@ > +/* python-complements.c > + Needed functions for python binding > +*/ > + > +#include "python-complements.h" > + > +/* FILE functions > + ---------------------------------------------------- > +*/ > + > +FILE *_bt_file_open(char *file_path, char *mode) > +{ > + FILE *fp = stdout; > + if (file_path != NULL) > + fp = fopen(file_path, mode); > + return fp; > +} > + > +void _bt_file_close(FILE *fp) > +{ > + if (fp != NULL) > + fclose(fp); > +} > + > + > +/* List-related functions > + ---------------------------------------------------- > +*/ > + > +/* ctf-field-list */ > +struct definition **_bt_python_field_listcaller( > + const struct bt_ctf_event *ctf_event, > + const struct definition *scope) > +{ > + struct definition **list; > + unsigned int count; > + int ret; > + > + ret = bt_ctf_get_field_list(ctf_event, scope, > + (const struct definition * const **)&list, &count); > + > + if (ret < 0) /* For python to know an error occured */ > + list = NULL; > + else /* For python to know the end is reached */ > + list[count] = NULL; > + > + return list; > +} > + > +struct definition *_bt_python_field_one_from_list( > + struct definition **list, int index) > +{ > + return list[index]; > +} > + > +/* event_decl_list */ > +struct bt_ctf_event_decl **_bt_python_event_decl_listcaller( > + int handle_id, struct bt_context *ctx) > +{ > + struct bt_ctf_event_decl **list; > + unsigned int count; > + int ret; > + > + ret = bt_ctf_get_event_decl_list(handle_id, ctx, > + (struct bt_ctf_event_decl * const **)&list, &count); > + > + if (ret < 0) /* For python to know an error occured */ > + list = NULL; > + else /* For python to know the end is reached */ > + list[count] = NULL; > + > + return list; > +} > + > +struct bt_ctf_event_decl *_bt_python_decl_one_from_list( > + struct bt_ctf_event_decl **list, int index) > +{ > + return list[index]; > +} > + > +/* decl_fields */ > +struct bt_ctf_field_decl **_by_python_field_decl_listcaller( > + struct bt_ctf_event_decl *event_decl, > + enum bt_ctf_scope scope) > +{ > + struct bt_ctf_field_decl **list; > + unsigned int count; > + int ret; > + > + ret = bt_ctf_get_decl_fields(event_decl, scope, > + (const struct bt_ctf_field_decl * const **)&list, &count); > + > + if (ret < 0) /* For python to know an error occured */ > + list = NULL; > + else /* For python to know the end is reached */ > + list[count] = NULL; > + > + return list; > +} > + > +struct bt_ctf_field_decl *_bt_python_field_decl_one_from_list( > + struct bt_ctf_field_decl **list, int index) > +{ > + return list[index]; > +} > diff --git a/bindings/python/python-complements.h b/bindings/python/python-complements.h > new file mode 100644 > index 0000000..cdd5528 > --- /dev/null > +++ b/bindings/python/python-complements.h > @@ -0,0 +1,36 @@ > +/* python-complements.h > + Needed functions for python binding > +*/ > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +/* File */ > +FILE *_bt_file_open(char *file_path, char *mode); > +void _bt_file_close(FILE *fp); > + > +/* ctf-field-list */ > +struct definition **_bt_python_field_listcaller( > + const struct bt_ctf_event *ctf_event, > + const struct definition *scope); > +struct definition *_bt_python_field_one_from_list( > + struct definition **list, int index); > + > +/* event_decl_list */ > +struct bt_ctf_event_decl **_bt_python_event_decl_listcaller( > + int handle_id, struct bt_context *ctx); > +struct bt_ctf_event_decl *_bt_python_decl_one_from_list( > + struct bt_ctf_event_decl **list, int index); > + > +/* decl_fields */ > +struct bt_ctf_field_decl **_by_python_field_decl_listcaller( > + struct bt_ctf_event_decl *event_decl, > + enum bt_ctf_scope scope); > +struct bt_ctf_field_decl *_bt_python_field_decl_one_from_list( > + struct bt_ctf_field_decl **list, int index); > diff --git a/bootstrap b/bootstrap > index c507425..f6926ca 100755 > --- a/bootstrap > +++ b/bootstrap > @@ -4,7 +4,7 @@ set -x > if [ ! -e config ]; then > mkdir config > fi > -aclocal > +aclocal -I m4 > libtoolize --force --copy > autoheader > automake --add-missing --copy > diff --git a/configure.ac b/configure.ac > index d90479d..f9cff9d 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -74,6 +74,41 @@ AC_CHECK_LIB([popt], [poptGetContext], [], > [AC_MSG_ERROR([Cannot find popt.])] > ) > > + > +# For Python > +# SWIG version needed or newer: > +swig_version=2.0.0 > + > +AC_ARG_ENABLE([python], > + [AC_HELP_STRING([--disable-python], > + [do not compile Python bindings])], > + [], [enable_python=yes]) > + > +AM_CONDITIONAL([USE_PYTHON], [test "x${enable_python:-yes}" = xyes]) > + > +if test "x${enable_python:-yes}" = xyes; then > + AC_MSG_NOTICE([You may configure with --disable-python ]dnl > +[if you do not want Python bindings.]) > + > + AX_PKG_SWIG($swig_version, [], [ AC_MSG_ERROR([SWIG $swig_version or newer is needed]) ]) > + AM_PATH_PYTHON > + > + AC_ARG_VAR([PYTHON_INCLUDE], [Include flags for python, bypassing python-config]) > + AC_ARG_VAR([PYTHON_CONFIG], [Path to python-config]) > + AS_IF([test -z "$PYTHON_INCLUDE"], [ > + AS_IF([test -z "$PYTHON_CONFIG"], [ > + AC_PATH_PROGS([PYTHON_CONFIG], > + [python$PYTHON_VERSION-config python-config], > + [no], > + [`dirname $PYTHON`]) > + AS_IF([test "$PYTHON_CONFIG" = no], [AC_MSG_ERROR([cannot find python-config for $PYTHON.])]) > + ]) > + AC_MSG_CHECKING([python include flags]) > + PYTHON_INCLUDE=`$PYTHON_CONFIG --includes` > + AC_MSG_RESULT([$PYTHON_INCLUDE]) > + ]) > +fi > + > pkg_modules="gmodule-2.0 >= 2.0.0" > PKG_CHECK_MODULES(GMODULE, [$pkg_modules]) > AC_SUBST(PACKAGE_LIBS) > @@ -103,6 +138,8 @@ AC_CONFIG_FILES([ > lib/Makefile > lib/prio_heap/Makefile > include/Makefile > + bindings/Makefile > + bindings/python/Makefile > tests/Makefile > ]) > AC_OUTPUT > diff --git a/doc/python-howto.txt b/doc/python-howto.txt > new file mode 100644 > index 0000000..e2ed751 > --- /dev/null > +++ b/doc/python-howto.txt > @@ -0,0 +1,70 @@ > +PYTHON BINDINGS > +---------------- > + > +This is a brief howto for using the Babeltrace Python module. > + > + > +INSTALLATION: > + > +By default, the Python bindings are installed. > +If you do not wish the Python bindings, you can configure with the > +--disable-python option during the installation procedure: > + > + $ ./configure --disable-python > + > +The Python module is automatically generated using SWIG, therefore the > +swig2.0 package on Debian/Ubuntu is requied. > + > + > +USAGE: > + > +Once installed, the Python module can be used by importing it in Python. > +In the Python interpreter: > + > + >>> import babeltrace > + > +Then the starting point is to create a context and add a trace to it. > + > + >>> ctx = babeltrace.Context() > + >>> ctx.add_trace("path/to/trace", ) > + > +Where is a string containing the format name in which the trace > +was produced. To print a list of available formats to the standard > +output, it is possible to use the print_format_list function. > + > + >>> out = babeltrace.File(None) # This returns stdout > + >>> babeltrace.print_format_list(out) > + > +When a trace is added to a context, it is opened and ready to read using > +an iterator. While creating an iterator, optional starting and ending > +position may be specified. So far, only ctf iterator are supported. > + > + >>> begin_pos = babeltrace.IterPos(babeltrace.SEEK_BEGIN) > + >>> iterator = babeltrace.ctf.Iterator(ctx, begin_pos) > + > +From there, it is possible to read the events. > + > + >>> event = iterator.read_event() > + > +It is simple to obtain the timestamp of that event. > + > + >>> timestamp = event.get_timestamp() > + > +Let's say that we want to extract the prev_comm context info for a > +sched_switch event. To do so, it is needed to set an event scope > +with which we can obtain the field wanted. > + > + >>> if event.get_name == "sched_switch": > + ... #prev_comm only for sched_switch events > + ... scope = event.get_top_level_scope(babeltrace.ctf.scope.EVENT_FIELDS) > + ... field = event.get_field(scope, "_prev_comm") > + ... prev_comm = field.get_char_array() > + > +It is also possible to move on to the next event. > + > + >>> ret = iterator.next() # Move the iterator > + >>> if ret == 0: # No error occured > + ... event = iterator.read_event() # Read the next event > + > +For many usage script examples of the Babeltrace Python module, see the > +bindings/python/examples directory. > diff --git a/m4/ax_pkg_swig.m4 b/m4/ax_pkg_swig.m4 > new file mode 100644 > index 0000000..e112f3d > --- /dev/null > +++ b/m4/ax_pkg_swig.m4 > @@ -0,0 +1,135 @@ > +# =========================================================================== > +# http://www.gnu.org/software/autoconf-archive/ax_pkg_swig.html > +# =========================================================================== > +# > +# SYNOPSIS > +# > +# AX_PKG_SWIG([major.minor.micro], [action-if-found], [action-if-not-found]) > +# > +# DESCRIPTION > +# > +# This macro searches for a SWIG installation on your system. If found, > +# then SWIG is AC_SUBST'd; if not found, then $SWIG is empty. If SWIG is > +# found, then SWIG_LIB is set to the SWIG library path, and AC_SUBST'd. > +# > +# You can use the optional first argument to check if the version of the > +# available SWIG is greater than or equal to the value of the argument. It > +# should have the format: N[.N[.N]] (N is a number between 0 and 999. Only > +# the first N is mandatory.) If the version argument is given (e.g. > +# 1.3.17), AX_PKG_SWIG checks that the swig package is this version number > +# or higher. > +# > +# As usual, action-if-found is executed if SWIG is found, otherwise > +# action-if-not-found is executed. > +# > +# In configure.in, use as: > +# > +# AX_PKG_SWIG(1.3.17, [], [ AC_MSG_ERROR([SWIG is required to build..]) ]) > +# AX_SWIG_ENABLE_CXX > +# AX_SWIG_MULTI_MODULE_SUPPORT > +# AX_SWIG_PYTHON > +# > +# LICENSE > +# > +# Copyright (c) 2008 Sebastian Huber > +# Copyright (c) 2008 Alan W. Irwin > +# Copyright (c) 2008 Rafael Laboissiere > +# Copyright (c) 2008 Andrew Collier > +# Copyright (c) 2011 Murray Cumming > +# > +# This program is free software; you can redistribute it and/or modify it > +# under the terms of the GNU General Public License as published by the > +# Free Software Foundation; either version 2 of the License, or (at your > +# option) any later version. > +# > +# This program is distributed in the hope that it will be useful, but > +# WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General > +# Public License for more details. > +# > +# You should have received a copy of the GNU General Public License along > +# with this program. If not, see . > +# > +# As a special exception, the respective Autoconf Macro's copyright owner > +# gives unlimited permission to copy, distribute and modify the configure > +# scripts that are the output of Autoconf when processing the Macro. You > +# need not follow the terms of the GNU General Public License when using > +# or distributing such scripts, even though portions of the text of the > +# Macro appear in them. The GNU General Public License (GPL) does govern > +# all other use of the material that constitutes the Autoconf Macro. > +# > +# This special exception to the GPL applies to versions of the Autoconf > +# Macro released by the Autoconf Archive. When you make and distribute a > +# modified version of the Autoconf Macro, you may extend this special > +# exception to the GPL to apply to your modified version as well. > + > +#serial 8 > + > +AC_DEFUN([AX_PKG_SWIG],[ > + # Ubuntu has swig 2.0 as /usr/bin/swig2.0 > + AC_PATH_PROGS([SWIG],[swig swig2.0]) > + if test -z "$SWIG" ; then > + m4_ifval([$3],[$3],[:]) > + elif test -n "$1" ; then > + AC_MSG_CHECKING([SWIG version]) > + [swig_version=`$SWIG -version 2>&1 | grep 'SWIG Version' | sed 's/.*\([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\).*/\1/g'`] > + AC_MSG_RESULT([$swig_version]) > + if test -n "$swig_version" ; then > + # Calculate the required version number components > + [required=$1] > + [required_major=`echo $required | sed 's/[^0-9].*//'`] > + if test -z "$required_major" ; then > + [required_major=0] > + fi > + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] > + [required_minor=`echo $required | sed 's/[^0-9].*//'`] > + if test -z "$required_minor" ; then > + [required_minor=0] > + fi > + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] > + [required_patch=`echo $required | sed 's/[^0-9].*//'`] > + if test -z "$required_patch" ; then > + [required_patch=0] > + fi > + # Calculate the available version number components > + [available=$swig_version] > + [available_major=`echo $available | sed 's/[^0-9].*//'`] > + if test -z "$available_major" ; then > + [available_major=0] > + fi > + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] > + [available_minor=`echo $available | sed 's/[^0-9].*//'`] > + if test -z "$available_minor" ; then > + [available_minor=0] > + fi > + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] > + [available_patch=`echo $available | sed 's/[^0-9].*//'`] > + if test -z "$available_patch" ; then > + [available_patch=0] > + fi > + # Convert the version tuple into a single number for easier comparison. > + # Using base 100 should be safe since SWIG internally uses BCD values > + # to encode its version number. > + required_swig_vernum=`expr $required_major \* 10000 \ > + \+ $required_minor \* 100 \+ $required_patch` > + available_swig_vernum=`expr $available_major \* 10000 \ > + \+ $available_minor \* 100 \+ $available_patch` > + > + if test $available_swig_vernum -lt $required_swig_vernum; then > + AC_MSG_WARN([SWIG version >= $1 is required. You have $swig_version.]) > + SWIG='' > + m4_ifval([$3],[$3],[]) > + else > + AC_MSG_CHECKING([for SWIG library]) > + SWIG_LIB=`$SWIG -swiglib` > + AC_MSG_RESULT([$SWIG_LIB]) > + m4_ifval([$2],[$2],[]) > + fi > + else > + AC_MSG_WARN([cannot determine SWIG version]) > + SWIG='' > + m4_ifval([$3],[$3],[]) > + fi > + fi > + AC_SUBST([SWIG_LIB]) > +]) > diff --git a/tests/tests-python.py b/tests/tests-python.py > new file mode 100644 > index 0000000..0bd71c2 > --- /dev/null > +++ b/tests/tests-python.py > @@ -0,0 +1,115 @@ > +import unittest > +import sys > +from babeltrace import * > + > +class TestBabeltracePythonModule(unittest.TestCase): > + > + def test_handle_decl(self): > + #Context creation, adding trace > + ctx = Context() > + trace_handle = ctx.add_trace( > + "ctf-traces/succeed/lttng-modules-2.0-pre5", > + "ctf") > + self.assertIsNotNone(trace_handle, "Error adding trace") > + > + #TraceHandle test > + ts_begin = trace_handle.get_timestamp_begin(ctx, CLOCK_REAL) > + ts_end = trace_handle.get_timestamp_end(ctx, CLOCK_REAL) > + self.assertGreater(ts_end, ts_begin, "Error get_timestamp from trace_handle") > + > + lst = ctf.get_event_decl_list(trace_handle, ctx) > + self.assertIsNotNone(lst, "Error get_event_decl_list") > + > + name = lst[0].get_name() > + self.assertIsNotNone(name) > + > + fields = lst[0].get_decl_fields(ctf.scope.EVENT_FIELDS) > + self.assertIsNotNone(fields, "Error getting FieldDecl list") > + > + if len(fields) > 0: > + self.assertIsNotNone(fields[0].get_name(), > + "Error getting name from FieldDecl") > + > + #Remove trace > + ctx.remove_trace(trace_handle) > + del ctx > + del trace_handle > + > + > + def test_iterator_event(self): > + #Context creation, adding trace > + ctx = Context() > + trace_handle = ctx.add_trace( > + "ctf-traces/succeed/lttng-modules-2.0-pre5", > + "ctf") > + self.assertIsNotNone(trace_handle, "Error adding trace") > + > + begin_pos = IterPos(SEEK_BEGIN) > + it = ctf.Iterator(ctx, begin_pos) > + self.assertIsNotNone(it, "Error creating iterator") > + > + event = it.read_event() > + self.assertIsNotNone(event, "Error reading event") > + > + handle = event.get_handle() > + self.assertIsNotNone(handle, "Error getting handle") > + > + context = event.get_context() > + self.assertIsNotNone(context, "Error getting context") > + > + name = "" > + while event is not None and name != "sched_switch": > + name = event.get_name() > + self.assertIsNotNone(name, "Error getting event name") > + > + ts = event.get_timestamp() > + self.assertGreaterEqual(ts, 0, "Error getting timestamp") > + > + cycles = event.get_cycles() > + self.assertGreaterEqual(cycles, 0, "Error getting cycles") > + > + if name == "sched_switch": > + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) > + self.assertIsNotNone(scope, "Error getting scope definition") > + > + field = event.get_field(scope, "cpu_id") > + prev_comm_str = field.get_uint64() > + self.assertEqual(ctf.field_error(), 0, "Error getting vec info") > + > + field_lst = event.get_field_list(scope) > + self.assertIsNotNone(field_lst, "Error getting field list") > + > + fname = field.field_name() > + self.assertIsNotNone(fname, "Error getting field name") > + > + ftype = field.field_type() > + self.assertIsNot(ftype, -1, "Error getting field type (unknown)") > + > + ret = it.next() > + self.assertGreaterEqual(ret, 0, "Error moving iterator") > + > + event = it.read_event() > + > + pos = it.get_pos() > + self.assertIsNotNone(pos._pos, "Error getting iterator position") > + > + ret = it.set_pos(pos) > + self.assertEqual(ret, 0, "Error setting iterator position") > + > + pos = it.create_time_pos(ts) > + self.assertIsNotNone(pos._pos, "Error creating time-based iterator position") > + > + del it, pos > + set_pos = IterPos(SEEK_TIME, ts) > + it = ctf.Iterator(ctx, begin_pos, set_pos) > + self.assertIsNotNone(it, "Error creating iterator with end_pos") > + > + > + def test_file_class(self): > + f = File("test-bitfield.c") > + self.assertIsNotNone(f, "Error opening file") > + f.close() > + > + > +if __name__ == "__main__": > + unittest.main() > -- > 1.7.9.5 > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From jdesfossez at efficios.com Fri Aug 10 17:32:35 2012 From: jdesfossez at efficios.com (Julien Desfossez) Date: Fri, 10 Aug 2012 17:32:35 -0400 Subject: [lttng-dev] [BABELTRACE PATCH] Fix: safety checks for opening mmap traces Message-ID: <1344634355-10923-1-git-send-email-jdesfossez@efficios.com> When adding a mmap trace in the context, we don't have any tracefile and index, some safety checks were missing in the path handling and timestamp manipulation. Signed-off-by: Julien Desfossez --- formats/ctf/ctf.c | 9 +++++++++ lib/context.c | 6 ++++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/formats/ctf/ctf.c b/formats/ctf/ctf.c index f33fa33..69917f7 100644 --- a/formats/ctf/ctf.c +++ b/formats/ctf/ctf.c @@ -167,6 +167,9 @@ uint64_t ctf_timestamp_begin(struct trace_descriptor *descriptor, if (!stream_pos->packet_real_index) goto error; + if (stream_pos->packet_real_index->len <= 0) + continue; + if (type == BT_CLOCK_REAL) { index = &g_array_index(stream_pos->packet_real_index, struct packet_index, @@ -225,6 +228,9 @@ uint64_t ctf_timestamp_end(struct trace_descriptor *descriptor, if (!stream_pos->packet_real_index) goto error; + if (stream_pos->packet_real_index->len <= 0) + continue; + if (type == BT_CLOCK_REAL) { index = &g_array_index(stream_pos->packet_real_index, struct packet_index, @@ -1845,6 +1851,9 @@ int ctf_convert_index_timestamp(struct trace_descriptor *tdp) stream_pos->packet_real_index = g_array_new(FALSE, TRUE, sizeof(struct packet_index)); + if (!stream_pos->packet_cycles_index) + continue; + for (k = 0; k < stream_pos->packet_cycles_index->len; k++) { struct packet_index *index; struct packet_index new_index; diff --git a/lib/context.c b/lib/context.c index 373944c..e2c1739 100644 --- a/lib/context.c +++ b/lib/context.c @@ -104,8 +104,10 @@ int bt_context_add_trace(struct bt_context *ctx, const char *path, } handle->format = fmt; handle->td = td; - strncpy(handle->path, path, PATH_MAX); - handle->path[PATH_MAX - 1] = '\0'; + if (path) { + strncpy(handle->path, path, PATH_MAX); + handle->path[PATH_MAX - 1] = '\0'; + } if (fmt->set_handle) fmt->set_handle(td, handle); -- 1.7.10.4 From mathieu.desnoyers at efficios.com Fri Aug 10 17:34:24 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 10 Aug 2012 17:34:24 -0400 Subject: [lttng-dev] [babeltrace PATCH] Remove trace-collection.h from include_headers In-Reply-To: <1344537779-19779-1-git-send-email-danny.serres@efficios.com> References: <1344537779-19779-1-git-send-email-danny.serres@efficios.com> Message-ID: <20120810213424.GA601@Krystal> merged, thanks! Mathieu * Danny Serres (danny.serres at efficios.com) wrote: > Signed-off-by: Danny Serres > --- > include/Makefile.am | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/include/Makefile.am b/include/Makefile.am > index f7488da..79247a0 100644 > --- a/include/Makefile.am > +++ b/include/Makefile.am > @@ -3,7 +3,6 @@ babeltraceinclude_HEADERS = \ > babeltrace/format.h \ > babeltrace/context.h \ > babeltrace/iterator.h \ > - babeltrace/trace-collection.h \ > babeltrace/trace-handle.h \ > babeltrace/list.h \ > babeltrace/clock-types.h > @@ -21,6 +20,7 @@ noinst_HEADERS = \ > babeltrace/context-internal.h \ > babeltrace/iterator-internal.h \ > babeltrace/prio_heap.h \ > + babeltrace/trace-collection.h \ > babeltrace/types.h \ > babeltrace/ctf-ir/metadata.h \ > babeltrace/ctf/events-internal.h \ > -- > 1.7.9.5 > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Fri Aug 10 17:37:18 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 10 Aug 2012 17:37:18 -0400 Subject: [lttng-dev] [BABELTRACE PATCH] Fix: safety checks for opening mmap traces In-Reply-To: <1344634355-10923-1-git-send-email-jdesfossez@efficios.com> References: <1344634355-10923-1-git-send-email-jdesfossez@efficios.com> Message-ID: <20120810213717.GB601@Krystal> * Julien Desfossez (jdesfossez at efficios.com) wrote: > When adding a mmap trace in the context, we don't have any tracefile and > index, some safety checks were missing in the path handling and > timestamp manipulation. > > Signed-off-by: Julien Desfossez merged, thanks! Mathieu > --- > formats/ctf/ctf.c | 9 +++++++++ > lib/context.c | 6 ++++-- > 2 files changed, 13 insertions(+), 2 deletions(-) > > diff --git a/formats/ctf/ctf.c b/formats/ctf/ctf.c > index f33fa33..69917f7 100644 > --- a/formats/ctf/ctf.c > +++ b/formats/ctf/ctf.c > @@ -167,6 +167,9 @@ uint64_t ctf_timestamp_begin(struct trace_descriptor *descriptor, > if (!stream_pos->packet_real_index) > goto error; > > + if (stream_pos->packet_real_index->len <= 0) > + continue; > + > if (type == BT_CLOCK_REAL) { > index = &g_array_index(stream_pos->packet_real_index, > struct packet_index, > @@ -225,6 +228,9 @@ uint64_t ctf_timestamp_end(struct trace_descriptor *descriptor, > if (!stream_pos->packet_real_index) > goto error; > > + if (stream_pos->packet_real_index->len <= 0) > + continue; > + > if (type == BT_CLOCK_REAL) { > index = &g_array_index(stream_pos->packet_real_index, > struct packet_index, > @@ -1845,6 +1851,9 @@ int ctf_convert_index_timestamp(struct trace_descriptor *tdp) > stream_pos->packet_real_index = g_array_new(FALSE, TRUE, > sizeof(struct packet_index)); > > + if (!stream_pos->packet_cycles_index) > + continue; > + > for (k = 0; k < stream_pos->packet_cycles_index->len; k++) { > struct packet_index *index; > struct packet_index new_index; > diff --git a/lib/context.c b/lib/context.c > index 373944c..e2c1739 100644 > --- a/lib/context.c > +++ b/lib/context.c > @@ -104,8 +104,10 @@ int bt_context_add_trace(struct bt_context *ctx, const char *path, > } > handle->format = fmt; > handle->td = td; > - strncpy(handle->path, path, PATH_MAX); > - handle->path[PATH_MAX - 1] = '\0'; > + if (path) { > + strncpy(handle->path, path, PATH_MAX); > + handle->path[PATH_MAX - 1] = '\0'; > + } > > if (fmt->set_handle) > fmt->set_handle(td, handle); > -- > 1.7.10.4 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From danny.serres at efficios.com Fri Aug 10 17:46:11 2012 From: danny.serres at efficios.com (Danny Serres) Date: Fri, 10 Aug 2012 17:46:11 -0400 Subject: [lttng-dev] =?utf-8?q?=5Bbabeltrace_PATCH=5D_Babeltrace_python_mo?= =?utf-8?q?dule_v4?= Message-ID: <1344635171-7654-1-git-send-email-danny.serres@efficios.com> The Babeltrace Python module can be used to directly control the Babeltrace API inside Python, using 'import babeltrace'. Therefore, it becomes possible to create a Context, add a trace to it, iterate on it, read events and so on from within Python. SWIG >= 2.0 is used to create the wrapper and its 'warning md variable unused' bug is patched in Makefile.am In the interface file, struct and enum are directly copied from the include files. All changes to struct bt_iter_pos and to enums in ctf/events.h and clock-types.h must also be made in the interface file. Signed-off-by: Danny Serres Signed-off-by: Yannick Brosseau --- .gitignore | 10 +- Makefile.am | 2 +- README | 6 + bindings/Makefile.am | 3 + bindings/python/Makefile.am | 28 + bindings/python/babeltrace.i.in | 1098 +++++++++ bindings/python/examples/babeltrace_and_lttng.py | 126 ++ bindings/python/examples/eventcount.py | 84 + bindings/python/examples/eventcountlist.py | 83 + bindings/python/examples/events_per_cpu.py | 99 + bindings/python/examples/example-api-test.py | 77 + bindings/python/examples/histogram.py | 139 ++ .../examples/output_format_modules/cairoplot.py | 2336 ++++++++++++++++++++ .../examples/output_format_modules/pprint_table.py | 37 + .../examples/output_format_modules/series.py | 1140 ++++++++++ bindings/python/examples/sched_switch.py | 128 ++ bindings/python/examples/softirqtimes.py | 153 ++ bindings/python/examples/syscalls_by_pid.py | 84 + bindings/python/python-complements.c | 121 + bindings/python/python-complements.h | 52 + bootstrap | 2 +- configure.ac | 37 + doc/python-howto.txt | 70 + m4/ax_pkg_swig.m4 | 135 ++ tests/tests-python.py | 115 + 25 files changed, 6162 insertions(+), 3 deletions(-) create mode 100644 bindings/Makefile.am create mode 100644 bindings/python/Makefile.am create mode 100644 bindings/python/babeltrace.i.in create mode 100644 bindings/python/examples/babeltrace_and_lttng.py create mode 100644 bindings/python/examples/eventcount.py create mode 100644 bindings/python/examples/eventcountlist.py create mode 100644 bindings/python/examples/events_per_cpu.py create mode 100644 bindings/python/examples/example-api-test.py create mode 100644 bindings/python/examples/histogram.py create mode 100644 bindings/python/examples/output_format_modules/__init__.py create mode 100755 bindings/python/examples/output_format_modules/cairoplot.py create mode 100644 bindings/python/examples/output_format_modules/pprint_table.py create mode 100755 bindings/python/examples/output_format_modules/series.py create mode 100644 bindings/python/examples/sched_switch.py create mode 100644 bindings/python/examples/softirqtimes.py create mode 100644 bindings/python/examples/syscalls_by_pid.py create mode 100644 bindings/python/python-complements.c create mode 100644 bindings/python/python-complements.h create mode 100644 doc/python-howto.txt create mode 100644 m4/ax_pkg_swig.m4 create mode 100644 tests/tests-python.py diff --git a/.gitignore b/.gitignore index d6098ac..d7bce4d 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ /tests/test-bitfield +*~ *.o *.a *.la @@ -15,7 +16,11 @@ ctf-parser-test /config.h.in /config.status *.log -*.m4 +/m4/libtool.m4 +/m4/lt~obsolete.m4 +/m4/ltoptions.m4 +/m4/ltsugar.m4 +/m4/ltversion.m4 libtool /configure Makefile @@ -26,3 +31,6 @@ converter/babeltrace-log core formats/ctf/metadata/ctf-parser.output stamp-h1 +bindings/python/babeltrace.i +bindings/python/babeltrace.py +bindings/python/babeltrace_wrap.c diff --git a/Makefile.am b/Makefile.am index 308ee16..6584c5d 100644 --- a/Makefile.am +++ b/Makefile.am @@ -2,7 +2,7 @@ AM_CFLAGS = $(PACKAGE_CFLAGS) -I$(top_srcdir)/include ACLOCAL_AMFLAGS = -I m4 -SUBDIRS = include types lib formats converter tests doc +SUBDIRS = include types lib formats converter bindings tests doc dist_doc_DATA = ChangeLog LICENSE mit-license.txt gpl-2.0.txt \ std-ext-lib.txt diff --git a/README b/README index 75bf0cf..1687075 100644 --- a/README +++ b/README @@ -25,6 +25,7 @@ BUILDING make install ldconfig + If you do not want Python bindings, run ./configure --disable-python. DEPENDENCIES ------------ @@ -44,6 +45,11 @@ To compile Babeltrace, you will need: libpopt >= 1.13 development libraries (Debian : libpopt-dev) (Fedora : popt) + python headers (optional) + (Debian/Ubuntu : python-dev) + swig >= 2.0 (optional) + (Debian/Ubuntu : swig2.0) + For developers using the git tree: diff --git a/bindings/Makefile.am b/bindings/Makefile.am new file mode 100644 index 0000000..dcd868d --- /dev/null +++ b/bindings/Makefile.am @@ -0,0 +1,3 @@ +if USE_PYTHON +SUBDIRS = python +endif diff --git a/bindings/python/Makefile.am b/bindings/python/Makefile.am new file mode 100644 index 0000000..579759f --- /dev/null +++ b/bindings/python/Makefile.am @@ -0,0 +1,28 @@ +babeltrace.i: babeltrace.i.in + sed "s/BABELTRACE_VERSION_STR/Babeltrace $(PACKAGE_VERSION)/g" babeltrace.i + +AM_CFLAGS = -I$(PYTHON_INCLUDE) -I$(top_srcdir)/include/ + +EXTRA_DIST = babeltrace.i +python_PYTHON = babeltrace.py +pyexec_LTLIBRARIES = _babeltrace.la + +MAINTAINERCLEANFILES = babeltrace_wrap.c babeltrace.py + +_babeltrace_la_SOURCES = babeltrace_wrap.c python-complements.c + +_babeltrace_la_LDFLAGS = -module + +_babeltrace_la_CFLAGS = $(GLIB_CFLAGS) $(AM_CFLAGS) + +_babeltrace_la_LIBS = $(GLIB_LIBS) + +_babeltrace_la_LIBADD = $(top_srcdir)/formats/ctf/libbabeltrace-ctf.la \ + $(top_srcdir)/formats/ctf-text/libbabeltrace-ctf-text.la + +# SWIG 'warning md variable unused' fixed after SWIG build: +babeltrace_wrap.c: babeltrace.i + $(SWIG) -python -Wall -I. -I$(top_srcdir)/include babeltrace.i + sed -i "s/PyObject \*m, \*d, \*md;/PyObject \*m, \*d;\n#if defined(SWIGPYTHON_BUILTIN)\nPyObject *md;\n#endif/g" babeltrace_wrap.c + sed -i "s/md = d/d/g" babeltrace_wrap.c + sed -i "s/(void)public_symbol;/(void)public_symbol;\n md = d;/g" babeltrace_wrap.c diff --git a/bindings/python/babeltrace.i.in b/bindings/python/babeltrace.i.in new file mode 100644 index 0000000..c8e4923 --- /dev/null +++ b/bindings/python/babeltrace.i.in @@ -0,0 +1,1098 @@ +/* + * babeltrace.i.in + * + * Babeltrace Python Module interface file + * + * Copyright 2012 EfficiOS Inc. + * + * Author: Danny Serres + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + */ + + +%define DOCSTRING +"BABELTRACE_VERSION_STR + +Babeltrace is a trace viewer and converter reading and writing the +Common Trace Format (CTF). Its main use is to pretty-print CTF +traces into a human-readable text output. + +To use this module, the first step is to create a Context and add a +trace to it." +%enddef + +%module(docstring=DOCSTRING) babeltrace + +%include "typemaps.i" +%{ +#define SWIG_FILE_WITH_INIT +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "python-complements.h" +%} + +typedef unsigned long long uint64_t; +typedef long long int64_t; +typedef int bt_intern_str; + +/* ================================================================= + CONTEXT.H, CONTEXT-INTERNAL.H + ????????????????????????????? +*/ + +%rename("_bt_context_create") bt_context_create(void); +%rename("_bt_context_add_trace") bt_context_add_trace( + struct bt_context *ctx, const char *path, const char *format, + void (*packet_seek)(struct stream_pos *pos, size_t index, int whence), + struct mmap_stream_list *stream_list, FILE *metadata); +%rename("_bt_context_remove_trace") bt_context_remove_trace( + struct bt_context *ctx, int trace_id); +%rename("_bt_context_get") bt_context_get(struct bt_context *ctx); +%rename("_bt_context_put") bt_context_put(struct bt_context *ctx); +%rename("_bt_ctf_event_get_context") bt_ctf_event_get_context( + const struct bt_ctf_event *event); + +struct bt_context *bt_context_create(void); +int bt_context_add_trace(struct bt_context *ctx, const char *path, const char *format, + void (*packet_seek)(struct stream_pos *pos, size_t index, int whence), + struct mmap_stream_list *stream_list, FILE *metadata); +void bt_context_remove_trace(struct bt_context *ctx, int trace_id); +void bt_context_get(struct bt_context *ctx); +void bt_context_put(struct bt_context *ctx); +struct bt_context *bt_ctf_event_get_context(const struct bt_ctf_event *event); + +// class Context to prevent direct access to struct bt_context +%pythoncode%{ +class Context: + """ + The context represents the object in which a trace_collection is + open. As long as this structure is allocated, the trace_collection + is open and the traces it contains can be read and seeked by the + iterators and callbacks. + """ + + def __init__(self): + self._c = _bt_context_create() + + def __del__(self): + _bt_context_put(self._c) + + def add_trace(self, path, format_str, + packet_seek=None, stream_list=None, metadata=None): + """ + Add a trace by path to the context. + + Open a trace. + + path is the path to the trace, it is not recursive. + If "path" is None, stream_list is used instead as a list + of mmap streams to open for the trace. + + format is a string containing the format name in which the trace was + produced. + + packet_seek is not implemented for Python. Should be left None to + use the default packet_seek handler provided by the trace format. + + stream_list is a linked list of streams, it is used to open a trace + where the trace data is located in memory mapped areas instead of + trace files, this argument should be None when path is not None. + + The metadata parameter acts as a metadata override when not None, + otherwise the format handles the metadata opening. + + Return: the corresponding TraceHandle on success or None on error. + """ + if metadata is not None: + metadata = metadata._file + + ret = _bt_context_add_trace(self._c, path, format_str, packet_seek, + stream_list, metadata) + if ret < 0: + return None + + th = TraceHandle.__new__(TraceHandle) + th._id = ret + return th + + def add_traces_recursive(self, path, format_str): + """ + Open a trace recursively. + + Find each trace present in the subdirectory starting from the given + path, and add them to the context. + + Return a dict of TraceHandle instances (the full path is the key). + Return None on error. + """ + + import os + + trace_handles = {} + + noTrace = True + error = False + + for fullpath, dirs, files in os.walk(path): + if "metadata" in files: + trace_handle = self.add_trace(fullpath, format_str) + if trace_handle is None: + error = True + continue + + trace_handles[fullpath] = trace_handle + noTrace = False + + if noTrace and error: + return None + return trace_handles + + def remove_trace(self, trace_handle): + """ + Remove a trace from the context. + Effectively closing the trace. + """ + try: + _bt_context_remove_trace(self._c, trace_handle._id) + except AttributeError: + raise TypeError("in remove_trace, " + "argument 2 must be a TraceHandle instance") +%} + + + +/* ================================================================= + FORMAT.H, REGISTRY + ?????????????????? +*/ + +%rename("lookup_format") bt_lookup_format(bt_intern_str qname); +%rename("_bt_print_format_list") bt_fprintf_format_list(FILE *fp); +%rename("register_format") bt_register_format(struct format *format); + +extern struct format *bt_lookup_format(bt_intern_str qname); +extern void bt_fprintf_format_list(FILE *fp); +extern int bt_register_format(struct format *format); + +void format_init(void); +void format_finalize(void); + +%pythoncode %{ + +def print_format_list(babeltrace_file): + """ + Print a list of available formats to file. + + babeltrace_file must be a File instance opened in write mode. + """ + try: + if babeltrace_file._file is not None: + _bt_print_format_list(babeltrace_file._file) + except AttributeError: + raise TypeError("in print_format_list, " + "argument 1 must be a File instance") + +%} + + +/* ================================================================= + ITERATOR.H, ITERATOR-INTERNAL.H + ??????????????????????????????? +*/ + +%rename("_bt_iter_create") bt_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, const struct bt_iter_pos *end_pos); +%rename("_bt_iter_destroy") bt_iter_destroy(struct bt_iter *iter); +%rename("_bt_iter_next") bt_iter_next(struct bt_iter *iter); +%rename("_bt_iter_get_pos") bt_iter_get_pos(struct bt_iter *iter); +%rename("_bt_iter_free_pos") bt_iter_free_pos(struct bt_iter_pos *pos); +%rename("_bt_iter_set_pos") bt_iter_set_pos(struct bt_iter *iter, + const struct bt_iter_pos *pos); +%rename("_bt_iter_create_time_pos") bt_iter_create_time_pos(struct bt_iter *iter, + uint64_t timestamp); + +struct bt_iter *bt_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, const struct bt_iter_pos *end_pos); +void bt_iter_destroy(struct bt_iter *iter); +int bt_iter_next(struct bt_iter *iter); +struct bt_iter_pos *bt_iter_get_pos(struct bt_iter *iter); +void bt_iter_free_pos(struct bt_iter_pos *pos); +int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *pos); +struct bt_iter_pos *bt_iter_create_time_pos(struct bt_iter *iter, uint64_t timestamp); + +%rename("_bt_iter_pos") bt_iter_pos; +%rename("SEEK_TIME") BT_SEEK_TIME; +%rename("SEEK_RESTORE") BT_SEEK_RESTORE; +%rename("SEEK_CUR") BT_SEEK_CUR; +%rename("SEEK_BEGIN") BT_SEEK_BEGIN; +%rename("SEEK_END") BT_SEEK_END; + + +// This struct is taken from iterator.h +// All changes to the struct must also be made here +struct bt_iter_pos { + enum { + BT_SEEK_TIME, /* uses u.seek_time */ + BT_SEEK_RESTORE, /* uses u.restore */ + BT_SEEK_CUR, + BT_SEEK_BEGIN, + BT_SEEK_END, + } type; + union { + uint64_t seek_time; + struct bt_saved_pos *restore; + } u; +}; + + +%pythoncode%{ + +class IterPos: + """This class represents the position where to set an iterator.""" + + __can_access = False + + def __init__(self, seek_type, seek_time = None): + """ + seek_type represents the type of seek to use. + seek_time is the timestamp to seek to when using SEEK_TIME, it + is expressed in nanoseconds + Only use SEEK_RESTORE on IterPos obtained from the get_pos function + in Iter class. + """ + + self._pos = _bt_iter_pos() + self._pos.type = seek_type + if seek_time and seek_type == SEEK_TIME: + self._pos.u.seek_time = seek_time + self.__can_access = True + + def __del__(self): + if not self.__can_access: + _bt_iter_free_pos(self._pos) + + def _get_type(self): + if not __can_access: + raise AttributeError("seek_type is not available") + return self._pos.type + + def _set_type(self, seek_type): + if not __can_access: + raise AttributeError("seek_type is not available") + self._pos.type = seek_type + + def _get_time(self): + if not __can_access: + raise AttributeError("seek_time is not available") + + elif self._pos.type is not SEEK_TIME: + raise TypeError("seek_type is not SEEK_TIME") + + return self._pos.u.seek_time + + def _set_time(self, time): + if not __can_access: + raise AttributeError("seek_time is not available") + + elif self._pos.type is not SEEK_TIME: + raise TypeError("seek_type is not SEEK_TIME") + + self._pos.u.seek_time = time + + def _get_pos(self): + return self._pos + + + seek_type = property(_get_type, _set_type) + seek_time = property(_get_time, _set_time) + + +class Iterator: + + __with_init = False + + def __init__(self, context, begin_pos = None, end_pos = None, _no_init = None): + """ + Allocate a trace collection iterator. + + begin_pos and end_pos are optional parameters to specify the + position at which the trace collection should be seeked upon + iterator creation, and the position at which iteration will + start returning "EOF". + + By default, if begin_pos is None, a BT_SEEK_CUR is performed at + creation. By default, if end_pos is None, a BT_SEEK_END (end of + trace) is the EOF criterion. + """ + if _no_init is None: + if begin_pos is None: + bp = None + else: + try: + bp = begin_pos._pos + except AttributeError: + raise TypeError("in __init__, " + "argument 3 must be a IterPos instance") + + if end_pos is None: + ep = None + else: + try: + ep = end_pos._pos + except AttributeError: + raise TypeError("in __init__, " + "argument 4 must be a IterPos instance") + + try: + self._bi = _bt_iter_create(context._c, bp, ep) + except AttributeError: + raise TypeError("in __init__, " + "argument 2 must be a Context instance") + + self.__with_init = True + + else: + self._bi = _no_init + + def __del__(self): + if self.__with_init: + _bt_iter_destroy(self._bi) + + def next(self): + """ + Move trace collection position to the next event. + Returns 0 on success, a negative value on error. + """ + return _bt_iter_next(self._bi) + + def get_pos(self): + """Return a IterPos class of the current iterator position.""" + ret = IterPos(0) + ret.__can_access = False + ret._pos = _bt_iter_get_pos(self._bi) + return ret + + def set_pos(self, pos): + """ + Move the iterator to a given position. + + On error, the stream_heap is reinitialized and returned empty. + Return 0 for success. + Return EOF if the position requested is after the last event of the + trace collection. + Return -EINVAL when called with invalid parameter. + Return -ENOMEM if the stream_heap could not be properly initialized. + """ + try: + return _bt_iter_set_pos(self._bi, pos._pos) + except AttributeError: + raise TypeError("in set_pos, " + "argument 2 must be a IterPos instance") + + def create_time_pos(self, timestamp): + """ + Create a position based on time + This function allocates and returns a new IterPos to be able to + restore an iterator position based on a timestamp. + """ + + if timestamp < 0: + raise TypeError("timestamp must be an unsigned int") + + ret = IterPos(0) + ret.__can_access = False + ret._pos = _bt_iter_create_time_pos(self._bi, timestamp) + return ret +%} + + +/* ================================================================= + CLOCK-TYPE.H + ???????????? + *** Enum copied from clock-type.h? + All changes must also be made here +*/ +%rename("CLOCK_CYCLES") BT_CLOCK_CYCLES; +%rename("CLOCK_REAL") BT_CLOCK_REAL; + +enum bt_clock_type { + BT_CLOCK_CYCLES = 0, + BT_CLOCK_REAL +}; + +/* ================================================================= + TRACE-HANDLE.H, TRACE-HANDLE-INTERNAL.H + ??????????????????????????????????????? +*/ + +%rename("_bt_trace_handle_create") bt_trace_handle_create(struct bt_context *ctx); +%rename("_bt_trace_handle_destroy") bt_trace_handle_destroy(struct bt_trace_handle *bt); +struct bt_trace_handle *bt_trace_handle_create(struct bt_context *ctx); +void bt_trace_handle_destroy(struct bt_trace_handle *bt); + +%rename("_bt_trace_handle_get_path") bt_trace_handle_get_path(struct bt_context *ctx, + int handle_id); +%rename("_bt_trace_handle_get_timestamp_begin") bt_trace_handle_get_timestamp_begin( + struct bt_context *ctx, int handle_id, enum bt_clock_type type); +%rename("_bt_trace_handle_get_timestamp_end") bt_trace_handle_get_timestamp_end( + struct bt_context *ctx, int handle_id, enum bt_clock_type type); +%rename("_bt_trace_handle_get_id") bt_trace_handle_get_id(struct bt_trace_handle *th); +int bt_trace_handle_get_id(struct bt_trace_handle *th); +const char *bt_trace_handle_get_path(struct bt_context *ctx, int handle_id); +uint64_t bt_trace_handle_get_timestamp_begin(struct bt_context *ctx, int handle_id, + enum bt_clock_type type); +uint64_t bt_trace_handle_get_timestamp_end(struct bt_context *ctx, int handle_id, + enum bt_clock_type type); + +%rename("_bt_ctf_event_get_handle_id") bt_ctf_event_get_handle_id( + const struct bt_ctf_event *event); +int bt_ctf_event_get_handle_id(const struct bt_ctf_event *event); + + +%pythoncode%{ + +class TraceHandle(object): + """ + The TraceHandle allows the user to manipulate a trace file directly. + It is a unique identifier representing a trace file. + Do not instantiate. + """ + + def __init__(self): + raise NotImplementedError("TraceHandle cannot be instantiated") + + def __repr__(self): + return "Babeltrace TraceHandle: trace_id('{}')".format(self._id) + + def get_id(self): + """Return the TraceHandle id.""" + return self._id + + def get_path(self, context): + """Return the path of a TraceHandle.""" + try: + return _bt_trace_handle_get_path(context._c, self._id) + except AttributeError: + raise TypeError("in get_path, " + "argument 2 must be a Context instance") + + def get_timestamp_begin(self, context, clock_type): + """Return the creation time of the buffers of a trace.""" + try: + return _bt_trace_handle_get_timestamp_begin( + context._c, self._id,clock_type) + except AttributeError: + raise TypeError("in get_timestamp_begin, " + "argument 2 must be a Context instance") + + def get_timestamp_end(self, context, clock_type): + """Return the destruction timestamp of the buffers of a trace.""" + try: + return _bt_trace_handle_get_timestamp_end( + context._c, self._id, clock_type) + except AttributeError: + raise TypeError("in get_timestamp_end, " + "argument 2 must be a Context instance") + +%} + + + +// ================================================================= +// CTF +// ================================================================= + +/* ================================================================= + ITERATOR.H, EVENTS.H + ???????????????????? +*/ + +//Iterator +%rename("_bt_ctf_iter_create") bt_ctf_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, + const struct bt_iter_pos *end_pos); +%rename("_bt_ctf_get_iter") bt_ctf_get_iter(struct bt_ctf_iter *iter); +%rename("_bt_ctf_iter_destroy") bt_ctf_iter_destroy(struct bt_ctf_iter *iter); +%rename("_bt_ctf_iter_read_event") bt_ctf_iter_read_event(struct bt_ctf_iter *iter); + +struct bt_ctf_iter *bt_ctf_iter_create(struct bt_context *ctx, + const struct bt_iter_pos *begin_pos, + const struct bt_iter_pos *end_pos); +struct bt_iter *bt_ctf_get_iter(struct bt_ctf_iter *iter); +void bt_ctf_iter_destroy(struct bt_ctf_iter *iter); +struct bt_ctf_event *bt_ctf_iter_read_event(struct bt_ctf_iter *iter); + + +//Events + +%rename("_bt_ctf_get_top_level_scope") bt_ctf_get_top_level_scope(const struct + bt_ctf_event *event, enum bt_ctf_scope scope); +%rename("_bt_ctf_event_name") bt_ctf_event_name(const struct bt_ctf_event *ctf_event); +%rename("_bt_ctf_get_timestamp") bt_ctf_get_timestamp( + const struct bt_ctf_event *ctf_event); +%rename("_bt_ctf_get_cycles") bt_ctf_get_cycles( + const struct bt_ctf_event *ctf_event); + +%rename("_bt_ctf_get_field") bt_ctf_get_field(const struct bt_ctf_event *ctf_event, + const struct definition *scope, const char *field); +%rename("_bt_ctf_get_index") bt_ctf_get_index(const struct bt_ctf_event *ctf_event, + const struct definition *field, unsigned int index); +%rename("_bt_ctf_field_name") bt_ctf_field_name(const struct definition *def); +%rename("_bt_ctf_field_type") bt_ctf_field_type(const struct definition *def); +%rename("_bt_ctf_get_int_signedness") bt_ctf_get_int_signedness( + const struct definition *field); +%rename("_bt_ctf_get_int_base") bt_ctf_get_int_base(const struct definition *field); +%rename("_bt_ctf_get_int_byte_order") bt_ctf_get_int_byte_order( + const struct definition *field); +%rename("_bt_ctf_get_int_len") bt_ctf_get_int_len(const struct definition *field); +%rename("_bt_ctf_get_encoding") bt_ctf_get_encoding(const struct definition *field); +%rename("_bt_ctf_get_array_len") bt_ctf_get_array_len(const struct definition *field); +%rename("_bt_ctf_get_uint64") bt_ctf_get_uint64(const struct definition *field); +%rename("_bt_ctf_get_int64") bt_ctf_get_int64(const struct definition *field); +%rename("_bt_ctf_get_char_array") bt_ctf_get_char_array(const struct definition *field); +%rename("_bt_ctf_get_string") bt_ctf_get_string(const struct definition *field); +%rename("_bt_ctf_field_get_error") bt_ctf_field_get_error(void); +%rename("_bt_ctf_get_decl_event_name") bt_ctf_get_decl_event_name(const struct + bt_ctf_event_decl *event); +%rename("_bt_ctf_get_decl_field_name") bt_ctf_get_decl_field_name( + const struct bt_ctf_field_decl *field); + +const struct definition *bt_ctf_get_top_level_scope(const struct bt_ctf_event *ctf_event, + enum bt_ctf_scope scope); +const char *bt_ctf_event_name(const struct bt_ctf_event *ctf_event); +uint64_t bt_ctf_get_timestamp(const struct bt_ctf_event *ctf_event); +uint64_t bt_ctf_get_cycles(const struct bt_ctf_event *ctf_event); +const struct definition *bt_ctf_get_field(const struct bt_ctf_event *ctf_event, + const struct definition *scope, + const char *field); +const struct definition *bt_ctf_get_index(const struct bt_ctf_event *ctf_event, + const struct definition *field, + unsigned int index); +const char *bt_ctf_field_name(const struct definition *def); +enum ctf_type_id bt_ctf_field_type(const struct definition *def); +int bt_ctf_get_int_signedness(const struct definition *field); +int bt_ctf_get_int_base(const struct definition *field); +int bt_ctf_get_int_byte_order(const struct definition *field); +ssize_t bt_ctf_get_int_len(const struct definition *field); +enum ctf_string_encoding bt_ctf_get_encoding(const struct definition *field); +int bt_ctf_get_array_len(const struct definition *field); +uint64_t bt_ctf_get_uint64(const struct definition *field); +int64_t bt_ctf_get_int64(const struct definition *field); +char *bt_ctf_get_char_array(const struct definition *field); +char *bt_ctf_get_string(const struct definition *field); +int bt_ctf_field_get_error(void); +const char *bt_ctf_get_decl_event_name(const struct bt_ctf_event_decl *event); +const char *bt_ctf_get_decl_field_name(const struct bt_ctf_field_decl *field); + +%pythoncode%{ + +class ctf: + + #enum equivalent, accessible constants + #These are taken directly from ctf/events.h + #All changes to enums must also be made here + class type_id: + UNKNOWN = 0 + INTEGER = 1 + FLOAT = 2 + ENUM = 3 + STRING = 4 + STRUCT = 5 + UNTAGGED_VARIANT = 6 + VARIANT = 7 + ARRAY = 8 + SEQUENCE = 9 + NR_CTF_TYPES = 10 + + class scope: + TRACE_PACKET_HEADER = 0 + STREAM_PACKET_CONTEXT = 1 + STREAM_EVENT_HEADER = 2 + STREAM_EVENT_CONTEXT = 3 + EVENT_CONTEXT = 4 + EVENT_FIELDS = 5 + + class string_encoding: + NONE = 0 + UTF8 = 1 + ASCII = 2 + UNKNOWN = 3 + + class Iterator(Iterator, object): + """ + Allocate a CTF trace collection iterator. + + begin_pos and end_pos are optional parameters to specify the + position at which the trace collection should be seeked upon + iterator creation, and the position at which iteration will + start returning "EOF". + + By default, if begin_pos is None, a SEEK_CUR is performed at + creation. By default, if end_pos is None, a SEEK_END (end of + trace) is the EOF criterion. + + Only one iterator can be created against a context. If more than one + iterator is being created for the same context, the second creation + will return None. The previous iterator must be destroyed before + creation of the new iterator for this function to succeed. + """ + + def __new__(cls, context, begin_pos = None, end_pos = None): + # __new__ is used to control the return value + # as the ctf.Iterator class should return None + # if bt_ctf_iter_create returns NULL + + if begin_pos is None: + bp = None + else: + bp = begin_pos._pos + if end_pos is None: + ep = None + else: + ep = end_pos._pos + try: + it = _bt_ctf_iter_create(context._c, bp, ep) + except AttributeError: + raise TypeError("in __init__, " + "argument 2 must be a Context instance") + if it is None: + return None + + ret_class = super(ctf.Iterator, cls).__new__(cls) + ret_class._i = it + return ret_class + + def __init__(self, context, begin_pos = None, end_pos = None): + Iterator.__init__(self, None, None, None, + _bt_ctf_get_iter(self._i)) + + def __del__(self): + _bt_ctf_iter_destroy(self._i) + + def read_event(self): + """ + Read the iterator's current event data. + Return current event on success, None on end of trace. + """ + ret = _bt_ctf_iter_read_event(self._i) + if ret is None: + return ret + ev = ctf.Event.__new__(ctf.Event) + ev._e = ret + return ev + + + class Event(object): + """ + This class represents an event from the trace. + It is obtained with read_event() from ctf.Iterator. + Do not instantiate. + """ + + def __init__(self): + raise NotImplementedError("ctf.Event cannot be instantiated") + + def get_top_level_scope(self, scope): + """ + Return a definition of the top-level scope + Top-level scopes are defined in ctf.scope. + In order to get a field or a field list, the user needs to pass a + scope as argument, this scope can be a top-level scope or a scope + relative to an arbitrary field. This function provides the mapping + between the scope and the actual definition of top-level scopes. + On error return None. + """ + evDef = ctf.Definition.__new__(ctf.Definition) + evDef._d = _bt_ctf_get_top_level_scope(self._e, scope) + if evDef._d is None: + return None + return evDef + + def get_name(self): + """Return the name of the event or None on error.""" + return _bt_ctf_event_name(self._e) + + def get_cycles(self): + """ + Return the timestamp of the event as written in + the packet (in cycles) or -1ULL on error. + """ + return _bt_ctf_get_cycles(self._e) + + def get_timestamp(self): + """ + Return the timestamp of the event offsetted with the + system clock source or -1ULL on error. + """ + return _bt_ctf_get_timestamp(self._e) + + def get_field(self, scope, field): + """Return the definition of a specific field.""" + evDef = ctf.Definition.__new__(ctf.Definition) + try: + evDef._d = _bt_ctf_get_field(self._e, scope._d, field) + except AttributeError: + raise TypeError("in get_field, argument 2 must be a " + "Definition (scope) instance") + return evDef + + def get_field_list(self, scope): + """ + Return a list of Definitions + Return None on error. + """ + try: + field_lc = _bt_python_field_listcaller(self._e, scope._d) + except AttributeError: + raise TypeError("in get_field_list, argument 2 must be a " + "Definition (scope) instance") + + if field_lc is None: + return None + + def_list = [] + i = 0 + while True: + tmp = ctf.Definition.__new__(ctf.Definition) + tmp._d = _bt_python_field_one_from_list(field_lc, i) + + if tmp._d is None: + #Last item of list is None, assured in + #_bt_python_field_listcaller + break + + def_list.append(tmp) + i += 1 + return def_list + + def get_index(self, field, index): + """ + If the field is an array or a sequence, return the element + at position index, otherwise return None + """ + evDef = ctf.Definition.__new__(ctf.Definition) + try: + evDef._d = _bt_ctf_get_index(self._e, field._d, index) + except AttributeError: + raise TypeError("in get_index, argument 2 must be a " + "Definition (field) instance") + + if evDef._d is None: + return None + return evDef + + def get_handle(self): + """ + Get the TraceHandle associated with an event + Return None on error + """ + ret = _bt_ctf_event_get_handle_id(self._e) + if ret < 0: + return None + + th = TraceHandle.__new__(TraceHandle) + th._id = ret + return th + + def get_context(self): + """ + Get the context associated with an event. + Return None on error. + """ + ctx = Context() + ctx._c = _bt_ctf_event_get_context(self._e); + if ctx._c is None: + return None + else: + return ctx + + + class Definition(object): + """Definition class. Do not instantiate.""" + + def __init__(self): + raise NotImplementedError("ctf.Definition cannot be instantiated") + + def __repr__(self): + return "Babeltrace Definition: name('{}'), type({})".format( + self.field_name(), self.field_type()) + + def field_name(self): + """Return the name of a field or None on error.""" + return _bt_ctf_field_name(self._d) + + def field_type(self): + """Return the type of a field or -1 if unknown.""" + return _bt_ctf_field_type(self._d) + + def get_int_signedness(self): + """ + Return the signedness of an integer: + 0 if unsigned; 1 if signed; -1 on error. + """ + return _bt_ctf_get_int_signedness(self._d) + + def get_int_base(self): + """Return the base of an int or a negative value on error.""" + return _bt_ctf_get_int_base(self._d) + + def get_int_byte_order(self): + """ + Return the byte order of an int or a negative + value on error. + """ + return _bt_ctf_get_int_byte_order(self._d) + + def get_int_len(self): + """ + Return the size, in bits, of an int or a negative + value on error. + """ + return _bt_ctf_get_int_len(self._d) + + def get_encoding(self): + """ + Return the encoding of an int or a string. + Return a negative value on error. + """ + return _bt_ctf_get_encoding(self._d) + + def get_array_len(self): + """ + Return the len of an array or a negative + value on error. + """ + return _bt_ctf_get_array_len(self._d) + + def get_uint64(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_uint64(self._d) + + def get_int64(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_int64(self._d) + + def get_char_array(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_char_array(self._d) + + def get_str(self): + """ + Return the value associated with the field. + If the field does not exist or is not of the type requested, + the value returned is undefined. To check if an error occured, + use the ctf.field_error() function after accessing a field. + """ + return _bt_ctf_get_string(self._d) + + + class EventDecl(object): + """Event declaration class. Do not instantiate.""" + + def __init__(self): + raise NotImplementedError("ctf.EventDecl cannot be instantiated") + + def __repr__(self): + return "Babeltrace EventDecl: name {}".format(self.get_name()) + + def get_name(self): + """Return the name of the event or None on error""" + return _bt_ctf_get_decl_event_name(self._d) + + def get_decl_fields(self, scope): + """ + Return a list of ctf.FieldDecl + Return None on error. + """ + ptr_list = _by_python_field_decl_listcaller(self._d, scope) + + if ptr_list is None: + return None + + decl_list = [] + i = 0 + while True: + tmp = ctf.FieldDecl.__new__(ctf.FieldDecl) + tmp._d = _bt_python_field_decl_one_from_list( + ptr_list, i) + + if tmp._d is None: + #Last item of list is None + break + + decl_list.append(tmp) + i += 1 + return decl_list + + + class FieldDecl(object): + """Field declaration class. Do not instantiate.""" + + def __init__(self): + raise NotImplementedError("ctf.FieldDecl cannot be instantiated") + + def __repr__(self): + return "Babeltrace FieldDecl: name {}".format(self.get_name()) + + def get_name(self): + """Return the name of a FieldDecl or None on error""" + return _bt_ctf_get_decl_field_name(self._d) + + + @staticmethod + def field_error(): + """ + Return the last error code encountered while + accessing a field and reset the error flag. + Return 0 if no error, a negative value otherwise. + """ + return _bt_ctf_field_get_error() + + @staticmethod + def get_event_decl_list(trace_handle, context): + """ + Return a list of ctf.EventDecl + Return None on error. + """ + try: + handle_id = trace_handle._id + except AttributeError: + raise TypeError("in get_event_decl_list, " + "argument 1 must be a TraceHandle instance") + try: + ptr_list = _bt_python_event_decl_listcaller(handle_id, context._c) + except AttributeError: + raise TypeError("in get_event_decl_list, " + "argument 2 must be a Context instance") + + if ptr_list is None: + return None + + decl_list = [] + i = 0 + while True: + tmp = ctf.EventDecl.__new__(ctf.EventDecl) + tmp._d = _bt_python_decl_one_from_list(ptr_list, i) + + if tmp._d is None: + #Last item of list is None + break + + decl_list.append(tmp) + i += 1 + return decl_list + +%} + + + +// ================================================================= +// NEW FUNCTIONS +// File and list-related +// python-complements.h +// ================================================================= + +%include python-complements.c + +%pythoncode %{ + +class File(object): + """ + Open a file for babeltrace. + + file_path is a string containing the path or None to use the + standard output in writing mode. + + The mode can be 'r', 'w' or 'a' for reading (default), writing or + appending. The file will be created if it doesn't exist when + opened for writing or appending; it will be truncated when opened + for writing. Add a 'b' to the mode for binary files. Add a '+' + to the mode to allow simultaneous reading and writing. + """ + + def __new__(cls, file_path, mode='r'): + # __new__ is used to control the return value + # as the File class should return None + # if _bt_file_open returns NULL + + # Type check + if file_path is not None and type(file_path) is not str: + raise TypeError("in method __init__, argument 2 of type 'str'") + if type(mode) is not str: + raise TypeError("in method __init__, argument 3 of type 'str'") + + # Opening file + file_ptr = _bt_file_open(file_path, mode) + if file_ptr is None: + return None + + # Class instantiation + file_inst = super(File, cls).__new__(cls) + file_inst._file = file_ptr + return file_inst + + def __init__(self, file_path, mode='r'): + self._opened = True + self._use_stdout = False + + if file_path is None: + # use stdout + file_path = "stdout" + mode = 'w' + self._use_stdout = True + + self._file_path = file_path + self._mode = mode + + def __del__(self): + self.close() + + def __repr__(self): + if self._opened: + stat = 'opened' + else: + stat = 'closed' + return "{} babeltrace File; file_path('{}'), mode('{}')".format( + stat, self._file_path, self._mode) + + def close(self): + """Close the file. Is also called using del.""" + if self._opened and not self._use_stdout: + _bt_file_close(self._file) + self._opened = False +%} diff --git a/bindings/python/examples/babeltrace_and_lttng.py b/bindings/python/examples/babeltrace_and_lttng.py new file mode 100644 index 0000000..cb44796 --- /dev/null +++ b/bindings/python/examples/babeltrace_and_lttng.py @@ -0,0 +1,126 @@ +# babeltrace_and_lttng.py +# +# Babeltrace and LTTng example script +# +# Copyright 2012 EfficiOS Inc. +# +# Author: Danny Serres +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. + + +# This script uses both lttng-tools and babeltrace +# python modules. It creates a session, enables +# events, starts tracing for 2 seconds, stops tracing, +# destroys the session and outputs the trace in the +# specified output file. +# +# WARNING: will destroy any existing trace having +# the same name as ses_name + + +# ------------------------------------------------------ +ses_name = "babeltrace-lttng-test" +trace_path = "/lttng-traces/babeltrace-lttng-trace/" +out_file = "babeltrace-lttng-trace-text-output.txt" +# ------------------------------------------------------ + + +import time +try: + import babeltrace, lttng +except ImportError: + raise ImportError( "both babeltrace and lttng-tools " + "python modules must be installed" ) + + +# Errors to raise if something goes wrong +class LTTngError(Exception): + pass +class BabeltraceError(Exception): + pass + + +# LTTNG-TOOLS + +# Making sure session does not already exist +lttng.destroy(ses_name) + +# Creating a new session and handle +ret = lttng.create(ses_name,trace_path) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + +han = None +han = lttng.Handle(ses_name, lttng.Domain()) +if han is None: + raise LTTngError("Handle not created") + + +# Enabling all events +ret = lttng.enable_event(han, lttng.Event(), None) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + + +# Start, wait, stop +ret = lttng.start(ses_name) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) +print("Tracing...") +time.sleep(2) +print("Stopped.") +ret = lttng.stop(ses_name) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + + +# Destroying tracing session +ret = lttng.destroy(ses_name) +if ret < 0: + raise LTTngError(lttng.strerror(ret)) + + +# BABELTRACE + +# Create context and add trace: +ctx = babeltrace.Context() +ret = ctx.add_trace(trace_path + "/kernel", "ctf") +if ret is None: + raise BabeltraceError("Error adding trace") + +# Iterator setup +bp = babeltrace.IterPos(babeltrace.SEEK_BEGIN) +ctf_it = babeltrace.ctf.Iterator(ctx,bp) + +# Reading events from trace +# and outputting timestamps and event names +# in out_file +print("Writing trace file...") +output = open(out_file, "wt") + +event = ctf_it.read_event() +while(event is not None): + output.write("TS: {}, {} : {}\n".format(event.get_timestamp(), + event.get_cycles(), event.get_name())) + + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +# Closing file +output.close() + +# Destroying dynamic elements +del ctf_it, han +print("Done.") diff --git a/bindings/python/examples/eventcount.py b/bindings/python/examples/eventcount.py new file mode 100644 index 0000000..5e96a43 --- /dev/null +++ b/bindings/python/examples/eventcount.py @@ -0,0 +1,84 @@ +# eventcount.py +# +# Babeltrace event count example script +# +# Copyright 2012 EfficiOS Inc. +# +# Author: Danny Serres +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. + +# The script prints a count of specified events and +# their related tid's in a given trace. +# The trace needs TID context (lttng add-context -k -t tid) + +import sys +from babeltrace import * +from output_format_modules.pprint_table import pprint_table as pprint + +if len(sys.argv) < 3: + raise TypeError("Usage: python eventcount.py event1 [event2 ...] path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +counts = {} + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +while(event is not None): + for event_type in sys.argv[1:len(sys.argv)-1]: + if event_type == event.get_name(): + + # Getting scope definition + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) + if sco is None: + print("ERROR: Cannot get definition scope for {}".format( + event.get_name())) + continue + + # Getting TID + tid_field = event.get_field(sco, "_tid") + tid = tid_field.get_int64() + + if ctf.field_error(): + print("ERROR: Missing TID info for {}".format( + event.get_name())) + continue + + tmp = (tid, event.get_name()) + + if tmp in counts: + counts[tmp] += 1 + else: + counts[tmp] = 1 + + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it + +# Appending data to table for output +table = [] +for item in counts: + table.append([item[0], item[1], counts[item]]) +table = sorted(table) +table.insert(0,["TID", "EVENT", "COUNT"]) +pprint(table, 2) diff --git a/bindings/python/examples/eventcountlist.py b/bindings/python/examples/eventcountlist.py new file mode 100644 index 0000000..945a960 --- /dev/null +++ b/bindings/python/examples/eventcountlist.py @@ -0,0 +1,83 @@ +# eventcountlist.py +# +# Babeltrace event count list example script +# +# Copyright 2012 EfficiOS Inc. +# +# Author: Danny Serres +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. + +# The script prints a count and rate of events. +# It also outputs a bar graph of count per event, using the cairoplot module. + +import sys +from babeltrace import * +from output_format_modules import cairoplot +from output_format_modules.pprint_table import pprint_table as pprint + +# Check for path arg: +if len(sys.argv) < 2: + raise TypeError("Usage: python eventcountlist.py path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +# Events and their assossiated count +# will be stored as a dict: +events_count = {} + +# Setting iterator: +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx,bp) + +prev_event = None +event = ctf_it.read_event() + +start_time = event.get_timestamp() + +# Reading events: +while(event is not None): + if event.get_name() in events_count: + events_count[event.get_name()] += 1 + else: + events_count[event.get_name()] = 1 + + ret = ctf_it.next() + if ret < 0: + break + else: + prev_event = event + event = ctf_it.read_event() + +if event: + total_time = event.get_timestamp() - start_time +else: + total_time = prev_event.get_timestamp() - start_time + +del ctf_it + +# Printing encountered events with respective count and rate: +print("Total time: {} ns".format(total_time)) +table = [["EVENT", "COUNT", "RATE (Hz)"]] +for item in sorted(events_count.iterkeys()): + tmp = [item, events_count[item], + events_count[item]/(total_time/1000000000.0)] + table.append(tmp) +pprint(table) + +# Exporting data as bar graph +cairoplot.vertical_bar_plot ( 'eventcountlist.svg', events_count, 50+85*len(events_count), + 800, border = 20, display_values = True, grid = True, + rounded_corners = True, + x_labels = sorted(events_count.keys()) ) diff --git a/bindings/python/examples/events_per_cpu.py b/bindings/python/examples/events_per_cpu.py new file mode 100644 index 0000000..be497ec --- /dev/null +++ b/bindings/python/examples/events_per_cpu.py @@ -0,0 +1,99 @@ +# events_per_cpu.py +# +# Babeltrace events per cpu example script +# +# Copyright 2012 EfficiOS Inc. +# +# Author: Danny Serres +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. + +# The script opens a trace and prints out CPU statistics +# for the given trace (event count per CPU, total active +# time and % of time processing events). +# It also outputs a .txt file showing each time interval +# (since the beginning of the trace) in which each CPU +# was active and the corresponding event. + +import sys, multiprocessing +from output_format_modules.pprint_table import pprint_table as pprint +from babeltrace import * + +if len(sys.argv) < 2: + raise TypeError("Usage: python events_per_cpu.py path/to/trace") + +# Adding trace +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +cpu_usage = [] +nbEvents = 0 +i = 0 +while i < multiprocessing.cpu_count(): + cpu_usage.append([]) + i += 1 + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +start_time = event.get_timestamp() + +while(event is not None): + + event_name = event.get_name() + ts = event.get_timestamp() + + # Getting cpu_id + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) + field = event.get_field(scope, "cpu_id") + cpu_id = field.get_uint64() + if ctf.field_error(): + print("ERROR: Missing cpu_id info for {}".format(event.get_name())) + else: + cpu_usage[cpu_id].append( (int(ts), event_name) ) + nbEvents += 1 + + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + + +# Outputting +table = [] +output = open("events_per_cpu.txt", "wt") +output.write("(timestamp, event)\n") + +for cpu in range(len(cpu_usage)): + # Setting table + event_str = str(100.0 * len(cpu_usage[cpu]) / nbEvents) + '000' + # % is printed with 2 decimals + table.append([cpu, len(cpu_usage[cpu]), event_str[0:event_str.find('.') + 3] + ' %']) + + # Writing to file + output.write("\n\n\n----------------------\n") + output.write("CPU {}\n\n".format(cpu)) + for event in cpu_usage[cpu]: + output.write(str(event) + '\n') + +# Printing table +table.insert(0, ["CPU ID", "EVENT COUNT", "TRACE EVENT %"]) +pprint(table) +print("Total event count: {}".format(nbEvents)) +print("Total trace time: {} ns".format(ts - start_time)) + +output.close() diff --git a/bindings/python/examples/example-api-test.py b/bindings/python/examples/example-api-test.py new file mode 100644 index 0000000..104f2d5 --- /dev/null +++ b/bindings/python/examples/example-api-test.py @@ -0,0 +1,77 @@ +# example_api_test.py +# +# Babeltrace example script based on the Babeltrace API test script +# +# Copyright 2012 EfficiOS Inc. +# +# Author: Danny Serres +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. + +# This example uses the babeltrace python module +# to partially test the api. + +import sys +from babeltrace import * + +# Check for path arg: +if len(sys.argv) < 2: + raise TypeError("Usage: python example-api-test.py path/to/file") + +# Create context and add trace: +ctx = Context() +trace_handle = ctx.add_trace(sys.argv[1], "ctf") +if trace_handle is None: + raise IOError("Error adding trace") + +# Listing events +lst = ctf.get_event_decl_list(trace_handle, ctx) +print("--- Event list ---") +for item in lst: + print("event : {}".format(item.get_name())) +print("--- Done ---") + +# Iter trace +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx,bp) +event = ctf_it.read_event() + +while(event is not None): + print("TS: {}, {} : {}".format(event.get_timestamp(), + event.get_cycles(), event.get_name())) + + if event.get_name() == "sched_switch": + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + prev_field = event.get_field(sco, "_prev_comm") + prev_comm = prev_field.get_char_array() + + if ctf.field_error(): + print("ERROR: Missing prev_comm context info") + else: + print("sched_switch prev_comm: {}".format(prev_comm)) + + if event.get_name() == "exit_syscall": + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + ret_field = event.get_field(sco, "_ret") + ret_code = ret_field.get_int64() + + if ctf.field_error(): + print("ERROR: Unable to extract ret") + else: + print("exit_syscall ret: {}".format(ret_code)) + + ret = ctf_it.next() + if ret < 0: + break + else: + event = ctf_it.read_event() + +del ctf_it diff --git a/bindings/python/examples/histogram.py b/bindings/python/examples/histogram.py new file mode 100644 index 0000000..44616a6 --- /dev/null +++ b/bindings/python/examples/histogram.py @@ -0,0 +1,139 @@ +# histogram.py +# +# Babeltrace histogram example script +# +# Copyright 2012 EfficiOS Inc. +# +# Author: Danny Serres +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. + +# The script checks the number of events in the trace +# and outputs a table and a .svg histogram for the specified +# range (microseconds) or the total trace if no range specified. +# The graph is generated using the cairoplot module. + +import sys +from babeltrace import * +from output_format_modules import cairoplot +from output_format_modules.pprint_table import pprint_table as pprint + +# ------------------------------------------------ +# Output settings + +# number of intervals: +nbDiv = 25 # Should not be over 150 + # for usable graph output + +# table output stream (file-like object): +out = sys.stdout +# ------------------------------------------------- + +if len(sys.argv) < 2 or len(sys.argv) > 4: + raise TypeError("Usage: python histogram.py [ start_time [end_time] ] path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +# Check when to start/stop graphing +sinceBegin = True +beginTime = 0.0 +if len(sys.argv) > 2: + sinceBegin = False + beginTime = float(sys.argv[1]) +untilEnd = True +if len(sys.argv) == 4: + untilEnd = False + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +start_time = event.get_timestamp() +time = 0 +count = {} + +while(event is not None): + # Microsec. + time = (event.get_timestamp() - start_time)/1000.0 + + # Check if in range + if not sinceBegin: + if time < beginTime: + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + continue + if not untilEnd: + if time > float(sys.argv[2]): + break + + # Counting events per timestamp: + if time in count: + count[time] += 1 + else: + count[time] = 1 + + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it + +# Setting data for output +interval = (time - beginTime)/nbDiv +div_begin_time = beginTime +div_end_time = beginTime + interval +data = {} + +# Prefix for string sorting, considering +# there should not be over 150 intervals. +# This would work up to 9999 intervals. +# If needed, add zeros. +prefix = 0.0001 + +while div_end_time <= time: + key = str(prefix) + '[' + str(div_begin_time) + ';' + str(div_end_time) + '[' + for tmp in count: + if tmp >= div_begin_time and tmp < div_end_time: + if key in data: + data[key] += count[tmp] + else: + data[key] = count[tmp] + if not key in data: + data[key] = 0 + div_begin_time = div_end_time + div_end_time += interval + # Prefix increment + prefix += 0.001 + +table = [] +x_labels = [] +for key in sorted(data): + table.append([key[key.find('['):], data[key]]) + x_labels.append(key[key.find('['):]) + +# Table output +table.insert(0, ["INTERVAL (us)", "COUNT"]) +pprint(table, 1, out) + +# Graph output +cairoplot.vertical_bar_plot ( 'histogram.svg', data, 50 + 150*nbDiv, 50*nbDiv, + border = 20, display_values = True, grid = True, + x_labels = x_labels, rounded_corners = True ) diff --git a/bindings/python/examples/output_format_modules/__init__.py b/bindings/python/examples/output_format_modules/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/bindings/python/examples/output_format_modules/cairoplot.py b/bindings/python/examples/output_format_modules/cairoplot.py new file mode 100755 index 0000000..a27113f --- /dev/null +++ b/bindings/python/examples/output_format_modules/cairoplot.py @@ -0,0 +1,2336 @@ +?#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# CairoPlot.py +# +# Copyright (c) 2008 Rodrigo Moreira Ara?jo +# +# Author: Rodrigo Moreiro Araujo +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public License +# as published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 +# USA + +#Contributor: Jo?o S. O. Bueno + +#TODO: review BarPlot Code +#TODO: x_label colision problem on Horizontal Bar Plot +#TODO: y_label's eat too much space on HBP + + +__version__ = 1.2 + +import cairo +import math +import random +from series import Series, Group, Data + +HORZ = 0 +VERT = 1 +NORM = 2 + +COLORS = {"red" : (1.0,0.0,0.0,1.0), "lime" : (0.0,1.0,0.0,1.0), "blue" : (0.0,0.0,1.0,1.0), + "maroon" : (0.5,0.0,0.0,1.0), "green" : (0.0,0.5,0.0,1.0), "navy" : (0.0,0.0,0.5,1.0), + "yellow" : (1.0,1.0,0.0,1.0), "magenta" : (1.0,0.0,1.0,1.0), "cyan" : (0.0,1.0,1.0,1.0), + "orange" : (1.0,0.5,0.0,1.0), "white" : (1.0,1.0,1.0,1.0), "black" : (0.0,0.0,0.0,1.0), + "gray" : (0.5,0.5,0.5,1.0), "light_gray" : (0.9,0.9,0.9,1.0), + "transparent" : (0.0,0.0,0.0,0.0)} + +THEMES = {"black_red" : [(0.0,0.0,0.0,1.0), (1.0,0.0,0.0,1.0)], + "red_green_blue" : [(1.0,0.0,0.0,1.0), (0.0,1.0,0.0,1.0), (0.0,0.0,1.0,1.0)], + "red_orange_yellow" : [(1.0,0.2,0.0,1.0), (1.0,0.7,0.0,1.0), (1.0,1.0,0.0,1.0)], + "yellow_orange_red" : [(1.0,1.0,0.0,1.0), (1.0,0.7,0.0,1.0), (1.0,0.2,0.0,1.0)], + "rainbow" : [(1.0,0.0,0.0,1.0), (1.0,0.5,0.0,1.0), (1.0,1.0,0.0,1.0), (0.0,1.0,0.0,1.0), (0.0,0.0,1.0,1.0), (0.3, 0.0, 0.5,1.0), (0.5, 0.0, 1.0, 1.0)]} + +def colors_from_theme( theme, series_length, mode = 'solid' ): + colors = [] + if theme not in THEMES.keys() : + raise Exception, "Theme not defined" + color_steps = THEMES[theme] + n_colors = len(color_steps) + if series_length <= n_colors: + colors = [color + tuple([mode]) for color in color_steps[0:n_colors]] + else: + iterations = [(series_length - n_colors)/(n_colors - 1) for i in color_steps[:-1]] + over_iterations = (series_length - n_colors) % (n_colors - 1) + for i in range(n_colors - 1): + if over_iterations <= 0: + break + iterations[i] += 1 + over_iterations -= 1 + for index,color in enumerate(color_steps[:-1]): + colors.append(color + tuple([mode])) + if iterations[index] == 0: + continue + next_color = color_steps[index+1] + color_step = ((next_color[0] - color[0])/(iterations[index] + 1), + (next_color[1] - color[1])/(iterations[index] + 1), + (next_color[2] - color[2])/(iterations[index] + 1), + (next_color[3] - color[3])/(iterations[index] + 1)) + for i in range( iterations[index] ): + colors.append((color[0] + color_step[0]*(i+1), + color[1] + color_step[1]*(i+1), + color[2] + color_step[2]*(i+1), + color[3] + color_step[3]*(i+1), + mode)) + colors.append(color_steps[-1] + tuple([mode])) + return colors + + +def other_direction(direction): + "explicit is better than implicit" + if direction == HORZ: + return VERT + else: + return HORZ + +#Class definition + +class Plot(object): + def __init__(self, + surface=None, + data=None, + width=640, + height=480, + background=None, + border = 0, + x_labels = None, + y_labels = None, + series_colors = None): + random.seed(2) + self.create_surface(surface, width, height) + self.dimensions = {} + self.dimensions[HORZ] = width + self.dimensions[VERT] = height + self.context = cairo.Context(self.surface) + self.labels={} + self.labels[HORZ] = x_labels + self.labels[VERT] = y_labels + self.load_series(data, x_labels, y_labels, series_colors) + self.font_size = 10 + self.set_background (background) + self.border = border + self.borders = {} + self.line_color = (0.5, 0.5, 0.5) + self.line_width = 0.5 + self.label_color = (0.0, 0.0, 0.0) + self.grid_color = (0.8, 0.8, 0.8) + + def create_surface(self, surface, width=None, height=None): + self.filename = None + if isinstance(surface, cairo.Surface): + self.surface = surface + return + if not type(surface) in (str, unicode): + raise TypeError("Surface should be either a Cairo surface or a filename, not %s" % surface) + sufix = surface.rsplit(".")[-1].lower() + self.filename = surface + if sufix == "png": + self.surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, width, height) + elif sufix == "ps": + self.surface = cairo.PSSurface(surface, width, height) + elif sufix == "pdf": + self.surface = cairo.PSSurface(surface, width, height) + else: + if sufix != "svg": + self.filename += ".svg" + self.surface = cairo.SVGSurface(self.filename, width, height) + + def commit(self): + try: + self.context.show_page() + if self.filename and self.filename.endswith(".png"): + self.surface.write_to_png(self.filename) + else: + self.surface.finish() + except cairo.Error: + pass + + def load_series (self, data, x_labels=None, y_labels=None, series_colors=None): + self.series_labels = [] + self.series = None + + #The pretty way + #if not isinstance(data, Series): + # # Not an instance of Series + # self.series = Series(data) + #else: + # self.series = data + # + #self.series_labels = self.series.get_names() + + #TODO: Remove on next version + # The ugly way, keeping retrocompatibility... + if callable(data) or type(data) is list and callable(data[0]): # Lambda or List of lambdas + self.series = data + self.series_labels = None + elif isinstance(data, Series): # Instance of Series + self.series = data + self.series_labels = data.get_names() + else: # Anything else + self.series = Series(data) + self.series_labels = self.series.get_names() + + #TODO: allow user passed series_widths + self.series_widths = [1.0 for group in self.series] + + #TODO: Remove on next version + self.process_colors( series_colors ) + + def process_colors( self, series_colors, length = None, mode = 'solid' ): + #series_colors might be None, a theme, a string of colors names or a list of color tuples + if length is None : + length = len( self.series.to_list() ) + + #no colors passed + if not series_colors: + #Randomize colors + self.series_colors = [ [random.random() for i in range(3)] + [1.0, mode] for series in range( length ) ] + else: + #Just theme pattern + if not hasattr( series_colors, "__iter__" ): + theme = series_colors + self.series_colors = colors_from_theme( theme.lower(), length ) + + #Theme pattern and mode + elif not hasattr(series_colors, '__delitem__') and not hasattr( series_colors[0], "__iter__" ): + theme = series_colors[0] + mode = series_colors[1] + self.series_colors = colors_from_theme( theme.lower(), length, mode ) + + #List + else: + self.series_colors = series_colors + for index, color in enumerate( self.series_colors ): + #element is a color name + if not hasattr(color, "__iter__"): + self.series_colors[index] = COLORS[color.lower()] + tuple([mode]) + #element is rgb tuple instead of rgba + elif len( color ) == 3 : + self.series_colors[index] += (1.0,mode) + #element has 4 elements, might be rgba tuple or rgb tuple with mode + elif len( color ) == 4 : + #last element is mode + if not hasattr(color[3], "__iter__"): + self.series_colors[index] += tuple([color[3]]) + self.series_colors[index][3] = 1.0 + #last element is alpha + else: + self.series_colors[index] += tuple([mode]) + + def get_width(self): + return self.surface.get_width() + + def get_height(self): + return self.surface.get_height() + + def set_background(self, background): + if background is None: + self.background = (0.0,0.0,0.0,0.0) + elif type(background) in (cairo.LinearGradient, tuple): + self.background = background + elif not hasattr(background,"__iter__"): + colors = background.split(" ") + if len(colors) == 1 and colors[0] in COLORS: + self.background = COLORS[background] + elif len(colors) > 1: + self.background = cairo.LinearGradient(self.dimensions[HORZ] / 2, 0, self.dimensions[HORZ] / 2, self.dimensions[VERT]) + for index,color in enumerate(colors): + self.background.add_color_stop_rgba(float(index)/(len(colors)-1),*COLORS[color]) + else: + raise TypeError ("Background should be either cairo.LinearGradient or a 3/4-tuple, not %s" % type(background)) + + def render_background(self): + if isinstance(self.background, cairo.LinearGradient): + self.context.set_source(self.background) + else: + self.context.set_source_rgba(*self.background) + self.context.rectangle(0,0, self.dimensions[HORZ], self.dimensions[VERT]) + self.context.fill() + + def render_bounding_box(self): + self.context.set_source_rgba(*self.line_color) + self.context.set_line_width(self.line_width) + self.context.rectangle(self.border, self.border, + self.dimensions[HORZ] - 2 * self.border, + self.dimensions[VERT] - 2 * self.border) + self.context.stroke() + + def render(self): + pass + +class ScatterPlot( Plot ): + def __init__(self, + surface=None, + data=None, + errorx=None, + errory=None, + width=640, + height=480, + background=None, + border=0, + axis = False, + dash = False, + discrete = False, + dots = 0, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + z_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + circle_colors = None ): + + self.bounds = {} + self.bounds[HORZ] = x_bounds + self.bounds[VERT] = y_bounds + self.bounds[NORM] = z_bounds + self.titles = {} + self.titles[HORZ] = x_title + self.titles[VERT] = y_title + self.max_value = {} + self.axis = axis + self.discrete = discrete + self.dots = dots + self.grid = grid + self.series_legend = series_legend + self.variable_radius = False + self.x_label_angle = math.pi / 2.5 + self.circle_colors = circle_colors + + Plot.__init__(self, surface, data, width, height, background, border, x_labels, y_labels, series_colors) + + self.dash = None + if dash: + if hasattr(dash, "keys"): + self.dash = [dash[key] for key in self.series_labels] + elif max([hasattr(item,'__delitem__') for item in data]) : + self.dash = dash + else: + self.dash = [dash] + + self.load_errors(errorx, errory) + + def convert_list_to_tuple(self, data): + #Data must be converted from lists of coordinates to a single + # list of tuples + out_data = zip(*data) + if len(data) == 3: + self.variable_radius = True + return out_data + + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): + #TODO: In cairoplot 2.0 keep only the Series instances + + # Convert Data and Group to Series + if isinstance(data, Data) or isinstance(data, Group): + data = Series(data) + + # Series + if isinstance(data, Series): + for group in data: + for item in group: + if len(item) is 3: + self.variable_radius = True + + #Dictionary with lists + if hasattr(data, "keys") : + if hasattr( data.values()[0][0], "__delitem__" ) : + for key in data.keys() : + data[key] = self.convert_list_to_tuple(data[key]) + elif len(data.values()[0][0]) == 3: + self.variable_radius = True + #List + elif hasattr(data[0], "__delitem__") : + #List of lists + if hasattr(data[0][0], "__delitem__") : + for index,value in enumerate(data) : + data[index] = self.convert_list_to_tuple(value) + #List + elif type(data[0][0]) != type((0,0)): + data = self.convert_list_to_tuple(data) + #Three dimensional data + elif len(data[0][0]) == 3: + self.variable_radius = True + + #List with three dimensional tuples + elif len(data[0]) == 3: + self.variable_radius = True + Plot.load_series(self, data, x_labels, y_labels, series_colors) + self.calc_boundaries() + self.calc_labels() + + def load_errors(self, errorx, errory): + self.errors = None + if errorx == None and errory == None: + return + self.errors = {} + self.errors[HORZ] = None + self.errors[VERT] = None + #asimetric errors + if errorx and hasattr(errorx[0], "__delitem__"): + self.errors[HORZ] = errorx + #simetric errors + elif errorx: + self.errors[HORZ] = [errorx] + #asimetric errors + if errory and hasattr(errory[0], "__delitem__"): + self.errors[VERT] = errory + #simetric errors + elif errory: + self.errors[VERT] = [errory] + + def calc_labels(self): + if not self.labels[HORZ]: + amplitude = self.bounds[HORZ][1] - self.bounds[HORZ][0] + if amplitude % 10: #if horizontal labels need floating points + self.labels[HORZ] = ["%.2lf" % (float(self.bounds[HORZ][0] + (amplitude * i / 10.0))) for i in range(11) ] + else: + self.labels[HORZ] = ["%d" % (int(self.bounds[HORZ][0] + (amplitude * i / 10.0))) for i in range(11) ] + if not self.labels[VERT]: + amplitude = self.bounds[VERT][1] - self.bounds[VERT][0] + if amplitude % 10: #if vertical labels need floating points + self.labels[VERT] = ["%.2lf" % (float(self.bounds[VERT][0] + (amplitude * i / 10.0))) for i in range(11) ] + else: + self.labels[VERT] = ["%d" % (int(self.bounds[VERT][0] + (amplitude * i / 10.0))) for i in range(11) ] + + def calc_extents(self, direction): + self.context.set_font_size(self.font_size * 0.8) + self.max_value[direction] = max(self.context.text_extents(item)[2] for item in self.labels[direction]) + self.borders[other_direction(direction)] = self.max_value[direction] + self.border + 20 + + def calc_boundaries(self): + #HORZ = 0, VERT = 1, NORM = 2 + min_data_value = [0,0,0] + max_data_value = [0,0,0] + + for group in self.series: + if type(group[0].content) in (int, float, long): + group = [Data((index, item.content)) for index,item in enumerate(group)] + + for point in group: + for index, item in enumerate(point.content): + if item > max_data_value[index]: + max_data_value[index] = item + elif item < min_data_value[index]: + min_data_value[index] = item + + if not self.bounds[HORZ]: + self.bounds[HORZ] = (min_data_value[HORZ], max_data_value[HORZ]) + if not self.bounds[VERT]: + self.bounds[VERT] = (min_data_value[VERT], max_data_value[VERT]) + if not self.bounds[NORM]: + self.bounds[NORM] = (min_data_value[NORM], max_data_value[NORM]) + + def calc_all_extents(self): + self.calc_extents(HORZ) + self.calc_extents(VERT) + + self.plot_height = self.dimensions[VERT] - 2 * self.borders[VERT] + self.plot_width = self.dimensions[HORZ] - 2* self.borders[HORZ] + + self.plot_top = self.dimensions[VERT] - self.borders[VERT] + + def calc_steps(self): + #Calculates all the x, y, z and color steps + series_amplitude = [self.bounds[index][1] - self.bounds[index][0] for index in range(3)] + + if series_amplitude[HORZ]: + self.horizontal_step = float (self.plot_width) / series_amplitude[HORZ] + else: + self.horizontal_step = 0.00 + + if series_amplitude[VERT]: + self.vertical_step = float (self.plot_height) / series_amplitude[VERT] + else: + self.vertical_step = 0.00 + + if series_amplitude[NORM]: + if self.variable_radius: + self.z_step = float (self.bounds[NORM][1]) / series_amplitude[NORM] + if self.circle_colors: + self.circle_color_step = tuple([float(self.circle_colors[1][i]-self.circle_colors[0][i])/series_amplitude[NORM] for i in range(4)]) + else: + self.z_step = 0.00 + self.circle_color_step = ( 0.0, 0.0, 0.0, 0.0 ) + + def get_circle_color(self, value): + return tuple( [self.circle_colors[0][i] + value*self.circle_color_step[i] for i in range(4)] ) + + def render(self): + self.calc_all_extents() + self.calc_steps() + self.render_background() + self.render_bounding_box() + if self.axis: + self.render_axis() + if self.grid: + self.render_grid() + self.render_labels() + self.render_plot() + if self.errors: + self.render_errors() + if self.series_legend and self.series_labels: + self.render_legend() + + def render_axis(self): + #Draws both the axis lines and their titles + cr = self.context + cr.set_source_rgba(*self.line_color) + cr.move_to(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) + cr.line_to(self.borders[HORZ], self.borders[VERT]) + cr.stroke() + + cr.move_to(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) + cr.line_to(self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) + cr.stroke() + + cr.set_source_rgba(*self.label_color) + self.context.set_font_size( 1.2 * self.font_size ) + if self.titles[HORZ]: + title_width,title_height = cr.text_extents(self.titles[HORZ])[2:4] + cr.move_to( self.dimensions[HORZ]/2 - title_width/2, self.borders[VERT] - title_height/2 ) + cr.show_text( self.titles[HORZ] ) + + if self.titles[VERT]: + title_width,title_height = cr.text_extents(self.titles[VERT])[2:4] + cr.move_to( self.dimensions[HORZ] - self.borders[HORZ] + title_height/2, self.dimensions[VERT]/2 - title_width/2) + cr.save() + cr.rotate( math.pi/2 ) + cr.show_text( self.titles[VERT] ) + cr.restore() + + def render_grid(self): + cr = self.context + horizontal_step = float( self.plot_height ) / ( len( self.labels[VERT] ) - 1 ) + vertical_step = float( self.plot_width ) / ( len( self.labels[HORZ] ) - 1 ) + + x = self.borders[HORZ] + vertical_step + y = self.plot_top - horizontal_step + + for label in self.labels[HORZ][:-1]: + cr.set_source_rgba(*self.grid_color) + cr.move_to(x, self.dimensions[VERT] - self.borders[VERT]) + cr.line_to(x, self.borders[VERT]) + cr.stroke() + x += vertical_step + for label in self.labels[VERT][:-1]: + cr.set_source_rgba(*self.grid_color) + cr.move_to(self.borders[HORZ], y) + cr.line_to(self.dimensions[HORZ] - self.borders[HORZ], y) + cr.stroke() + y -= horizontal_step + + def render_labels(self): + self.context.set_font_size(self.font_size * 0.8) + self.render_horz_labels() + self.render_vert_labels() + + def render_horz_labels(self): + cr = self.context + step = float( self.plot_width ) / ( len( self.labels[HORZ] ) - 1 ) + x = self.borders[HORZ] + y = self.dimensions[VERT] - self.borders[VERT] + 5 + + # store rotation matrix from the initial state + rotation_matrix = cr.get_matrix() + rotation_matrix.rotate(self.x_label_angle) + + cr.set_source_rgba(*self.label_color) + + for item in self.labels[HORZ]: + width = cr.text_extents(item)[2] + cr.move_to(x, y) + cr.save() + cr.set_matrix(rotation_matrix) + cr.show_text(item) + cr.restore() + x += step + + def render_vert_labels(self): + cr = self.context + step = ( self.plot_height ) / ( len( self.labels[VERT] ) - 1 ) + y = self.plot_top + cr.set_source_rgba(*self.label_color) + for item in self.labels[VERT]: + width = cr.text_extents(item)[2] + cr.move_to(self.borders[HORZ] - width - 5,y) + cr.show_text(item) + y -= step + + def render_legend(self): + cr = self.context + cr.set_font_size(self.font_size) + cr.set_line_width(self.line_width) + + widest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[2]) + tallest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[3]) + max_width = self.context.text_extents(widest_word)[2] + max_height = self.context.text_extents(tallest_word)[3] * 1.1 + + color_box_height = max_height / 2 + color_box_width = color_box_height * 2 + + #Draw a bounding box + bounding_box_width = max_width + color_box_width + 15 + bounding_box_height = (len(self.series_labels)+0.5) * max_height + cr.set_source_rgba(1,1,1) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - bounding_box_width, self.borders[VERT], + bounding_box_width, bounding_box_height) + cr.fill() + + cr.set_source_rgba(*self.line_color) + cr.set_line_width(self.line_width) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - bounding_box_width, self.borders[VERT], + bounding_box_width, bounding_box_height) + cr.stroke() + + for idx,key in enumerate(self.series_labels): + #Draw color box + cr.set_source_rgba(*self.series_colors[idx][:4]) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - max_width - color_box_width - 10, + self.borders[VERT] + color_box_height + (idx*max_height) , + color_box_width, color_box_height) + cr.fill() + + cr.set_source_rgba(0, 0, 0) + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - max_width - color_box_width - 10, + self.borders[VERT] + color_box_height + (idx*max_height), + color_box_width, color_box_height) + cr.stroke() + + #Draw series labels + cr.set_source_rgba(0, 0, 0) + cr.move_to(self.dimensions[HORZ] - self.borders[HORZ] - max_width - 5, self.borders[VERT] + ((idx+1)*max_height)) + cr.show_text(key) + + def render_errors(self): + cr = self.context + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) + cr.clip() + radius = self.dots + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + for index, group in enumerate(self.series): + cr.set_source_rgba(*self.series_colors[index][:4]) + for number, data in enumerate(group): + x = x0 + self.horizontal_step * data.content[0] + y = self.dimensions[VERT] - y0 - self.vertical_step * data.content[1] + if self.errors[HORZ]: + cr.move_to(x, y) + x1 = x - self.horizontal_step * self.errors[HORZ][0][number] + cr.line_to(x1, y) + cr.line_to(x1, y - radius) + cr.line_to(x1, y + radius) + cr.stroke() + if self.errors[HORZ] and len(self.errors[HORZ]) == 2: + cr.move_to(x, y) + x1 = x + self.horizontal_step * self.errors[HORZ][1][number] + cr.line_to(x1, y) + cr.line_to(x1, y - radius) + cr.line_to(x1, y + radius) + cr.stroke() + if self.errors[VERT]: + cr.move_to(x, y) + y1 = y + self.vertical_step * self.errors[VERT][0][number] + cr.line_to(x, y1) + cr.line_to(x - radius, y1) + cr.line_to(x + radius, y1) + cr.stroke() + if self.errors[VERT] and len(self.errors[VERT]) == 2: + cr.move_to(x, y) + y1 = y - self.vertical_step * self.errors[VERT][1][number] + cr.line_to(x, y1) + cr.line_to(x - radius, y1) + cr.line_to(x + radius, y1) + cr.stroke() + + + def render_plot(self): + cr = self.context + if self.discrete: + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) + cr.clip() + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + radius = self.dots + for number, group in enumerate (self.series): + cr.set_source_rgba(*self.series_colors[number][:4]) + for data in group : + if self.variable_radius: + radius = data.content[2]*self.z_step + if self.circle_colors: + cr.set_source_rgba( *self.get_circle_color( data.content[2]) ) + x = x0 + self.horizontal_step*data.content[0] + y = y0 + self.vertical_step*data.content[1] + cr.arc(x, self.dimensions[VERT] - y, radius, 0, 2*math.pi) + cr.fill() + else: + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) + cr.clip() + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + radius = self.dots + for number, group in enumerate (self.series): + last_data = None + cr.set_source_rgba(*self.series_colors[number][:4]) + for data in group : + x = x0 + self.horizontal_step*data.content[0] + y = y0 + self.vertical_step*data.content[1] + if self.dots: + if self.variable_radius: + radius = data.content[2]*self.z_step + cr.arc(x, self.dimensions[VERT] - y, radius, 0, 2*math.pi) + cr.fill() + if last_data : + old_x = x0 + self.horizontal_step*last_data.content[0] + old_y = y0 + self.vertical_step*last_data.content[1] + cr.move_to( old_x, self.dimensions[VERT] - old_y ) + cr.line_to( x, self.dimensions[VERT] - y) + cr.set_line_width(self.series_widths[number]) + + #?Display line as dash line + if self.dash and self.dash[number]: + s = self.series_widths[number] + cr.set_dash([s*3, s*3], 0) + + cr.stroke() + cr.set_dash([]) + last_data = data + +class DotLinePlot(ScatterPlot): + def __init__(self, + surface=None, + data=None, + width=640, + height=480, + background=None, + border=0, + axis = False, + dash = False, + dots = 0, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None): + + ScatterPlot.__init__(self, surface, data, None, None, width, height, background, border, + axis, dash, False, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, None, x_title, y_title, series_colors, None ) + + + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + for group in self.series : + for index,data in enumerate(group): + group[index].content = (index, data.content) + + self.calc_boundaries() + self.calc_labels() + +class FunctionPlot(ScatterPlot): + def __init__(self, + surface=None, + data=None, + width=640, + height=480, + background=None, + border=0, + axis = False, + discrete = False, + dots = 0, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + step = 1): + + self.function = data + self.step = step + self.discrete = discrete + + data, x_bounds = self.load_series_from_function( self.function, x_bounds ) + + ScatterPlot.__init__(self, surface, data, None, None, width, height, background, border, + axis, False, discrete, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, None, x_title, y_title, series_colors, None ) + + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + + if len(self.series[0][0]) is 1: + for group_id, group in enumerate(self.series) : + for index,data in enumerate(group): + group[index].content = (self.bounds[HORZ][0] + self.step*index, data.content) + + self.calc_boundaries() + self.calc_labels() + + def load_series_from_function( self, function, x_bounds ): + #TODO: Add the possibility for the user to define multiple functions with different discretization parameters + + #This function converts a function, a list of functions or a dictionary + #of functions into its corresponding array of data + series = Series() + + if isinstance(function, Group) or isinstance(function, Data): + function = Series(function) + + # If is instance of Series + if isinstance(function, Series): + # Overwrite any bounds passed by the function + x_bounds = (function.range[0],function.range[-1]) + + #if no bounds are provided + if x_bounds == None: + x_bounds = (0,10) + + + #TODO: Finish the dict translation + if hasattr(function, "keys"): #dictionary: + for key in function.keys(): + group = Group(name=key) + #data[ key ] = [] + i = x_bounds[0] + while i <= x_bounds[1] : + group.add_data(function[ key ](i)) + #data[ key ].append( function[ key ](i) ) + i += self.step + series.add_group(group) + + elif hasattr(function, "__delitem__"): #list of functions + for index,f in enumerate( function ) : + group = Group() + #data.append( [] ) + i = x_bounds[0] + while i <= x_bounds[1] : + group.add_data(f(i)) + #data[ index ].append( f(i) ) + i += self.step + series.add_group(group) + + elif isinstance(function, Series): # instance of Series + series = function + + else: #function + group = Group() + i = x_bounds[0] + while i <= x_bounds[1] : + group.add_data(function(i)) + i += self.step + series.add_group(group) + + + return series, x_bounds + + def calc_labels(self): + if not self.labels[HORZ]: + self.labels[HORZ] = [] + i = self.bounds[HORZ][0] + while i<=self.bounds[HORZ][1]: + self.labels[HORZ].append(str(i)) + i += float(self.bounds[HORZ][1] - self.bounds[HORZ][0])/10 + ScatterPlot.calc_labels(self) + + def render_plot(self): + if not self.discrete: + ScatterPlot.render_plot(self) + else: + last = None + cr = self.context + for number, group in enumerate (self.series): + cr.set_source_rgba(*self.series_colors[number][:4]) + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step + for data in group: + x = x0 + self.horizontal_step * data.content[0] + y = y0 + self.vertical_step * data.content[1] + cr.move_to(x, self.dimensions[VERT] - y) + cr.line_to(x, self.plot_top) + cr.set_line_width(self.series_widths[number]) + cr.stroke() + if self.dots: + cr.new_path() + cr.arc(x, self.dimensions[VERT] - y, 3, 0, 2.1 * math.pi) + cr.close_path() + cr.fill() + +class BarPlot(Plot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None, + main_dir = None): + + self.bounds = {} + self.bounds[HORZ] = x_bounds + self.bounds[VERT] = y_bounds + self.display_values = display_values + self.grid = grid + self.rounded_corners = rounded_corners + self.stack = stack + self.three_dimension = three_dimension + self.x_label_angle = math.pi / 2.5 + self.main_dir = main_dir + self.max_value = {} + self.plot_dimensions = {} + self.steps = {} + self.value_label_color = (0.5,0.5,0.5,1.0) + + Plot.__init__(self, surface, data, width, height, background, border, x_labels, y_labels, series_colors) + + def load_series(self, data, x_labels = None, y_labels = None, series_colors = None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + self.calc_boundaries() + + def process_colors(self, series_colors): + #Data for a BarPlot might be a List or a List of Lists. + #On the first case, colors must be generated for all bars, + #On the second, colors must be generated for each of the inner lists. + + #TODO: Didn't get it... + #if hasattr(self.data[0], '__getitem__'): + # length = max(len(series) for series in self.data) + #else: + # length = len( self.data ) + + length = max(len(group) for group in self.series) + + Plot.process_colors( self, series_colors, length, 'linear') + + def calc_boundaries(self): + if not self.bounds[self.main_dir]: + if self.stack: + max_data_value = max(sum(group.to_list()) for group in self.series) + else: + max_data_value = max(max(group.to_list()) for group in self.series) + self.bounds[self.main_dir] = (0, max_data_value) + if not self.bounds[other_direction(self.main_dir)]: + self.bounds[other_direction(self.main_dir)] = (0, len(self.series)) + + def calc_extents(self, direction): + self.max_value[direction] = 0 + if self.labels[direction]: + widest_word = max(self.labels[direction], key = lambda item: self.context.text_extents(item)[2]) + self.max_value[direction] = self.context.text_extents(widest_word)[3 - direction] + self.borders[other_direction(direction)] = (2-direction)*self.max_value[direction] + self.border + direction*(5) + else: + self.borders[other_direction(direction)] = self.border + + def calc_horz_extents(self): + self.calc_extents(HORZ) + + def calc_vert_extents(self): + self.calc_extents(VERT) + + def calc_all_extents(self): + self.calc_horz_extents() + self.calc_vert_extents() + other_dir = other_direction(self.main_dir) + self.value_label = 0 + if self.display_values: + if self.stack: + self.value_label = self.context.text_extents(str(max(sum(group.to_list()) for group in self.series)))[2 + self.main_dir] + else: + self.value_label = self.context.text_extents(str(max(max(group.to_list()) for group in self.series)))[2 + self.main_dir] + if self.labels[self.main_dir]: + self.plot_dimensions[self.main_dir] = self.dimensions[self.main_dir] - 2*self.borders[self.main_dir] - self.value_label + else: + self.plot_dimensions[self.main_dir] = self.dimensions[self.main_dir] - self.borders[self.main_dir] - 1.2*self.border - self.value_label + self.plot_dimensions[other_dir] = self.dimensions[other_dir] - self.borders[other_dir] - self.border + self.plot_top = self.dimensions[VERT] - self.borders[VERT] + + def calc_steps(self): + other_dir = other_direction(self.main_dir) + self.series_amplitude = self.bounds[self.main_dir][1] - self.bounds[self.main_dir][0] + if self.series_amplitude: + self.steps[self.main_dir] = float(self.plot_dimensions[self.main_dir])/self.series_amplitude + else: + self.steps[self.main_dir] = 0.00 + series_length = len(self.series) + self.steps[other_dir] = float(self.plot_dimensions[other_dir])/(series_length + 0.1*(series_length + 1)) + self.space = 0.1*self.steps[other_dir] + + def render(self): + self.calc_all_extents() + self.calc_steps() + self.render_background() + self.render_bounding_box() + if self.grid: + self.render_grid() + if self.three_dimension: + self.render_ground() + if self.display_values: + self.render_values() + self.render_labels() + self.render_plot() + if self.series_labels: + self.render_legend() + + def draw_3d_rectangle_front(self, x0, y0, x1, y1, shift): + self.context.rectangle(x0-shift, y0+shift, x1-x0, y1-y0) + + def draw_3d_rectangle_side(self, x0, y0, x1, y1, shift): + self.context.move_to(x1-shift,y0+shift) + self.context.line_to(x1, y0) + self.context.line_to(x1, y1) + self.context.line_to(x1-shift, y1+shift) + self.context.line_to(x1-shift, y0+shift) + self.context.close_path() + + def draw_3d_rectangle_top(self, x0, y0, x1, y1, shift): + self.context.move_to(x0-shift,y0+shift) + self.context.line_to(x0, y0) + self.context.line_to(x1, y0) + self.context.line_to(x1-shift, y0+shift) + self.context.line_to(x0-shift, y0+shift) + self.context.close_path() + + def draw_round_rectangle(self, x0, y0, x1, y1): + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) + self.context.line_to(x1-5, y0) + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) + self.context.line_to(x1, y1-5) + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) + self.context.line_to(x0+5, y1) + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) + self.context.line_to(x0, y0+5) + self.context.close_path() + + def render_ground(self): + self.draw_3d_rectangle_front(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_side (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_top (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + def render_labels(self): + self.context.set_font_size(self.font_size * 0.8) + if self.labels[HORZ]: + self.render_horz_labels() + if self.labels[VERT]: + self.render_vert_labels() + + def render_legend(self): + cr = self.context + cr.set_font_size(self.font_size) + cr.set_line_width(self.line_width) + + widest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[2]) + tallest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[3]) + max_width = self.context.text_extents(widest_word)[2] + max_height = self.context.text_extents(tallest_word)[3] * 1.1 + 5 + + color_box_height = max_height / 2 + color_box_width = color_box_height * 2 + + #Draw a bounding box + bounding_box_width = max_width + color_box_width + 15 + bounding_box_height = (len(self.series_labels)+0.5) * max_height + cr.set_source_rgba(1,1,1) + cr.rectangle(self.dimensions[HORZ] - self.border - bounding_box_width, self.border, + bounding_box_width, bounding_box_height) + cr.fill() + + cr.set_source_rgba(*self.line_color) + cr.set_line_width(self.line_width) + cr.rectangle(self.dimensions[HORZ] - self.border - bounding_box_width, self.border, + bounding_box_width, bounding_box_height) + cr.stroke() + + for idx,key in enumerate(self.series_labels): + #Draw color box + cr.set_source_rgba(*self.series_colors[idx][:4]) + cr.rectangle(self.dimensions[HORZ] - self.border - max_width - color_box_width - 10, + self.border + color_box_height + (idx*max_height) , + color_box_width, color_box_height) + cr.fill() + + cr.set_source_rgba(0, 0, 0) + cr.rectangle(self.dimensions[HORZ] - self.border - max_width - color_box_width - 10, + self.border + color_box_height + (idx*max_height), + color_box_width, color_box_height) + cr.stroke() + + #Draw series labels + cr.set_source_rgba(0, 0, 0) + cr.move_to(self.dimensions[HORZ] - self.border - max_width - 5, self.border + ((idx+1)*max_height)) + cr.show_text(key) + + +class HorizontalBarPlot(BarPlot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None): + + BarPlot.__init__(self, surface, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, series_colors, HORZ) + self.series_labels = series_labels + + def calc_vert_extents(self): + self.calc_extents(VERT) + if self.labels[HORZ] and not self.labels[VERT]: + self.borders[HORZ] += 10 + + def draw_rectangle_bottom(self, x0, y0, x1, y1): + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) + self.context.line_to(x0, y0+5) + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) + self.context.line_to(x1, y0) + self.context.line_to(x1, y1) + self.context.line_to(x0+5, y1) + self.context.close_path() + + def draw_rectangle_top(self, x0, y0, x1, y1): + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) + self.context.line_to(x1, y1-5) + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) + self.context.line_to(x0, y1) + self.context.line_to(x0, y0) + self.context.line_to(x1, y0) + self.context.close_path() + + def draw_rectangle(self, index, length, x0, y0, x1, y1): + if length == 1: + BarPlot.draw_rectangle(self, x0, y0, x1, y1) + elif index == 0: + self.draw_rectangle_bottom(x0, y0, x1, y1) + elif index == length-1: + self.draw_rectangle_top(x0, y0, x1, y1) + else: + self.context.rectangle(x0, y0, x1-x0, y1-y0) + + #TODO: Review BarPlot.render_grid code + def render_grid(self): + self.context.set_source_rgba(0.8, 0.8, 0.8) + if self.labels[HORZ]: + self.context.set_font_size(self.font_size * 0.8) + step = (self.dimensions[HORZ] - 2*self.borders[HORZ] - self.value_label)/(len(self.labels[HORZ])-1) + x = self.borders[HORZ] + next_x = 0 + for item in self.labels[HORZ]: + width = self.context.text_extents(item)[2] + if x - width/2 > next_x and x - width/2 > self.border: + self.context.move_to(x, self.border) + self.context.line_to(x, self.dimensions[VERT] - self.borders[VERT]) + self.context.stroke() + next_x = x + width/2 + x += step + else: + lines = 11 + horizontal_step = float(self.plot_dimensions[HORZ])/(lines-1) + x = self.borders[HORZ] + for y in xrange(0, lines): + self.context.move_to(x, self.border) + self.context.line_to(x, self.dimensions[VERT] - self.borders[VERT]) + self.context.stroke() + x += horizontal_step + + def render_horz_labels(self): + step = (self.dimensions[HORZ] - 2*self.borders[HORZ])/(len(self.labels[HORZ])-1) + x = self.borders[HORZ] + next_x = 0 + + for item in self.labels[HORZ]: + self.context.set_source_rgba(*self.label_color) + width = self.context.text_extents(item)[2] + if x - width/2 > next_x and x - width/2 > self.border: + self.context.move_to(x - width/2, self.dimensions[VERT] - self.borders[VERT] + self.max_value[HORZ] + 3) + self.context.show_text(item) + next_x = x + width/2 + x += step + + def render_vert_labels(self): + series_length = len(self.labels[VERT]) + step = (self.plot_dimensions[VERT] - (series_length + 1)*self.space)/(len(self.labels[VERT])) + y = self.border + step/2 + self.space + + for item in self.labels[VERT]: + self.context.set_source_rgba(*self.label_color) + width, height = self.context.text_extents(item)[2:4] + self.context.move_to(self.borders[HORZ] - width - 5, y + height/2) + self.context.show_text(item) + y += step + self.space + self.labels[VERT].reverse() + + def render_values(self): + self.context.set_source_rgba(*self.value_label_color) + self.context.set_font_size(self.font_size * 0.8) + if self.stack: + for i,group in enumerate(self.series): + value = sum(group.to_list()) + height = self.context.text_extents(str(value))[3] + x = self.borders[HORZ] + value*self.steps[HORZ] + 2 + y = self.borders[VERT] + (i+0.5)*self.steps[VERT] + (i+1)*self.space + height/2 + self.context.move_to(x, y) + self.context.show_text(str(value)) + else: + for i,group in enumerate(self.series): + inner_step = self.steps[VERT]/len(group) + y0 = self.border + i*self.steps[VERT] + (i+1)*self.space + for number,data in enumerate(group): + height = self.context.text_extents(str(data.content))[3] + self.context.move_to(self.borders[HORZ] + data.content*self.steps[HORZ] + 2, y0 + 0.5*inner_step + height/2, ) + self.context.show_text(str(data.content)) + y0 += inner_step + + def render_plot(self): + if self.stack: + for i,group in enumerate(self.series): + x0 = self.borders[HORZ] + y0 = self.borders[VERT] + i*self.steps[VERT] + (i+1)*self.space + for number,data in enumerate(group): + if self.series_colors[number][4] in ('radial','linear') : + linear = cairo.LinearGradient( data.content*self.steps[HORZ]/2, y0, data.content*self.steps[HORZ]/2, y0 + self.steps[VERT] ) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + elif self.series_colors[number][4] == 'solid': + self.context.set_source_rgba(*self.series_colors[number][:4]) + if self.rounded_corners: + self.draw_rectangle(number, len(group), x0, y0, x0+data.content*self.steps[HORZ], y0+self.steps[VERT]) + self.context.fill() + else: + self.context.rectangle(x0, y0, data.content*self.steps[HORZ], self.steps[VERT]) + self.context.fill() + x0 += data.content*self.steps[HORZ] + else: + for i,group in enumerate(self.series): + inner_step = self.steps[VERT]/len(group) + x0 = self.borders[HORZ] + y0 = self.border + i*self.steps[VERT] + (i+1)*self.space + for number,data in enumerate(group): + linear = cairo.LinearGradient(data.content*self.steps[HORZ]/2, y0, data.content*self.steps[HORZ]/2, y0 + inner_step) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + if self.rounded_corners and data.content != 0: + BarPlot.draw_round_rectangle(self,x0, y0, x0 + data.content*self.steps[HORZ], y0 + inner_step) + self.context.fill() + else: + self.context.rectangle(x0, y0, data.content*self.steps[HORZ], inner_step) + self.context.fill() + y0 += inner_step + +class VerticalBarPlot(BarPlot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None): + + BarPlot.__init__(self, surface, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, series_colors, VERT) + self.series_labels = series_labels + + def calc_vert_extents(self): + self.calc_extents(VERT) + if self.labels[VERT] and not self.labels[HORZ]: + self.borders[VERT] += 10 + + def draw_rectangle_bottom(self, x0, y0, x1, y1): + self.context.move_to(x1,y1) + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) + self.context.line_to(x0+5, y1) + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) + self.context.line_to(x0, y0) + self.context.line_to(x1, y0) + self.context.line_to(x1, y1) + self.context.close_path() + + def draw_rectangle_top(self, x0, y0, x1, y1): + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) + self.context.line_to(x1-5, y0) + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) + self.context.line_to(x1, y1) + self.context.line_to(x0, y1) + self.context.line_to(x0, y0) + self.context.close_path() + + def draw_rectangle(self, index, length, x0, y0, x1, y1): + if length == 1: + BarPlot.draw_rectangle(self, x0, y0, x1, y1) + elif index == 0: + self.draw_rectangle_bottom(x0, y0, x1, y1) + elif index == length-1: + self.draw_rectangle_top(x0, y0, x1, y1) + else: + self.context.rectangle(x0, y0, x1-x0, y1-y0) + + def render_grid(self): + self.context.set_source_rgba(0.8, 0.8, 0.8) + if self.labels[VERT]: + lines = len(self.labels[VERT]) + vertical_step = float(self.plot_dimensions[self.main_dir])/(lines-1) + y = self.borders[VERT] + self.value_label + else: + lines = 11 + vertical_step = float(self.plot_dimensions[self.main_dir])/(lines-1) + y = 1.2*self.border + self.value_label + for x in xrange(0, lines): + self.context.move_to(self.borders[HORZ], y) + self.context.line_to(self.dimensions[HORZ] - self.border, y) + self.context.stroke() + y += vertical_step + + def render_ground(self): + self.draw_3d_rectangle_front(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_side (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + self.draw_3d_rectangle_top (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) + self.context.fill() + + def render_horz_labels(self): + series_length = len(self.labels[HORZ]) + step = float (self.plot_dimensions[HORZ] - (series_length + 1)*self.space)/len(self.labels[HORZ]) + x = self.borders[HORZ] + step/2 + self.space + next_x = 0 + + for item in self.labels[HORZ]: + self.context.set_source_rgba(*self.label_color) + width = self.context.text_extents(item)[2] + if x - width/2 > next_x and x - width/2 > self.borders[HORZ]: + self.context.move_to(x - width/2, self.dimensions[VERT] - self.borders[VERT] + self.max_value[HORZ] + 3) + self.context.show_text(item) + next_x = x + width/2 + x += step + self.space + + def render_vert_labels(self): + self.context.set_source_rgba(*self.label_color) + y = self.borders[VERT] + self.value_label + step = (self.dimensions[VERT] - 2*self.borders[VERT] - self.value_label)/(len(self.labels[VERT]) - 1) + self.labels[VERT].reverse() + for item in self.labels[VERT]: + width, height = self.context.text_extents(item)[2:4] + self.context.move_to(self.borders[HORZ] - width - 5, y + height/2) + self.context.show_text(item) + y += step + self.labels[VERT].reverse() + + def render_values(self): + self.context.set_source_rgba(*self.value_label_color) + self.context.set_font_size(self.font_size * 0.8) + if self.stack: + for i,group in enumerate(self.series): + value = sum(group.to_list()) + width = self.context.text_extents(str(value))[2] + x = self.borders[HORZ] + (i+0.5)*self.steps[HORZ] + (i+1)*self.space - width/2 + y = value*self.steps[VERT] + 2 + self.context.move_to(x, self.plot_top-y) + self.context.show_text(str(value)) + else: + for i,group in enumerate(self.series): + inner_step = self.steps[HORZ]/len(group) + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space + for number,data in enumerate(group): + width = self.context.text_extents(str(data.content))[2] + self.context.move_to(x0 + 0.5*inner_step - width/2, self.plot_top - data.content*self.steps[VERT] - 2) + self.context.show_text(str(data.content)) + x0 += inner_step + + def render_plot(self): + if self.stack: + for i,group in enumerate(self.series): + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space + y0 = 0 + for number,data in enumerate(group): + if self.series_colors[number][4] in ('linear','radial'): + linear = cairo.LinearGradient( x0, data.content*self.steps[VERT]/2, x0 + self.steps[HORZ], data.content*self.steps[VERT]/2 ) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + elif self.series_colors[number][4] == 'solid': + self.context.set_source_rgba(*self.series_colors[number][:4]) + if self.rounded_corners: + self.draw_rectangle(number, len(group), x0, self.plot_top - y0 - data.content*self.steps[VERT], x0 + self.steps[HORZ], self.plot_top - y0) + self.context.fill() + else: + self.context.rectangle(x0, self.plot_top - y0 - data.content*self.steps[VERT], self.steps[HORZ], data.content*self.steps[VERT]) + self.context.fill() + y0 += data.content*self.steps[VERT] + else: + for i,group in enumerate(self.series): + inner_step = self.steps[HORZ]/len(group) + y0 = self.borders[VERT] + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space + for number,data in enumerate(group): + if self.series_colors[number][4] == 'linear': + linear = cairo.LinearGradient( x0, data.content*self.steps[VERT]/2, x0 + inner_step, data.content*self.steps[VERT]/2 ) + color = self.series_colors[number] + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1.0, *color[:4]) + self.context.set_source(linear) + elif self.series_colors[number][4] == 'solid': + self.context.set_source_rgba(*self.series_colors[number][:4]) + if self.rounded_corners and data.content != 0: + BarPlot.draw_round_rectangle(self, x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top) + self.context.fill() + elif self.three_dimension: + self.draw_3d_rectangle_front(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) + self.context.fill() + self.draw_3d_rectangle_side(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) + self.context.fill() + self.draw_3d_rectangle_top(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) + self.context.fill() + else: + self.context.rectangle(x0, self.plot_top - data.content*self.steps[VERT], inner_step, data.content*self.steps[VERT]) + self.context.fill() + + x0 += inner_step + +class StreamChart(VerticalBarPlot): + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + grid = False, + series_legend = None, + x_labels = None, + x_bounds = None, + y_bounds = None, + series_colors = None): + + VerticalBarPlot.__init__(self, surface, data, width, height, background, border, + False, grid, False, True, False, + None, x_labels, None, x_bounds, y_bounds, series_colors) + + def calc_steps(self): + other_dir = other_direction(self.main_dir) + self.series_amplitude = self.bounds[self.main_dir][1] - self.bounds[self.main_dir][0] + if self.series_amplitude: + self.steps[self.main_dir] = float(self.plot_dimensions[self.main_dir])/self.series_amplitude + else: + self.steps[self.main_dir] = 0.00 + series_length = len(self.data) + self.steps[other_dir] = float(self.plot_dimensions[other_dir])/series_length + + def render_legend(self): + pass + + def ground(self, index): + sum_values = sum(self.data[index]) + return -0.5*sum_values + + def calc_angles(self): + middle = self.plot_top - self.plot_dimensions[VERT]/2.0 + self.angles = [tuple([0.0 for x in range(len(self.data)+1)])] + for x_index in range(1, len(self.data)-1): + t = [] + x0 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y0 = middle - self.ground(x_index-1)*self.steps[VERT] + y2 = middle - self.ground(x_index+1)*self.steps[VERT] + t.append(math.atan(float(y0-y2)/(x0-x2))) + for data_index in range(len(self.data[x_index])): + x0 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y0 = middle - self.ground(x_index-1)*self.steps[VERT] - self.data[x_index-1][data_index]*self.steps[VERT] + y2 = middle - self.ground(x_index+1)*self.steps[VERT] - self.data[x_index+1][data_index]*self.steps[VERT] + + for i in range(0,data_index): + y0 -= self.data[x_index-1][i]*self.steps[VERT] + y2 -= self.data[x_index+1][i]*self.steps[VERT] + + if data_index == len(self.data[0])-1 and False: + self.context.set_source_rgba(0.0,0.0,0.0,0.3) + self.context.move_to(x0,y0) + self.context.line_to(x2,y2) + self.context.stroke() + self.context.arc(x0,y0,2,0,2*math.pi) + self.context.fill() + t.append(math.atan(float(y0-y2)/(x0-x2))) + self.angles.append(tuple(t)) + self.angles.append(tuple([0.0 for x in range(len(self.data)+1)])) + + def render_plot(self): + self.calc_angles() + middle = self.plot_top - self.plot_dimensions[VERT]/2.0 + p = 0.4*self.steps[HORZ] + for data_index in range(len(self.data[0])-1,-1,-1): + self.context.set_source_rgba(*self.series_colors[data_index][:4]) + + #draw the upper line + for x_index in range(len(self.data)-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] - self.data[x_index][data_index]*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index + 1)*self.steps[VERT] - self.data[x_index + 1][data_index]*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index+1][i]*self.steps[VERT] + + if x_index == 0: + self.context.move_to(x1,y1) + + ang1 = self.angles[x_index][data_index+1] + ang2 = self.angles[x_index+1][data_index+1] + math.pi + self.context.curve_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1), + x2+p*math.cos(ang2),y2+p*math.sin(ang2), + x2,y2) + + for x_index in range(len(self.data)-1,0,-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index - 1)*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index-1][i]*self.steps[VERT] + + if x_index == len(self.data)-1: + self.context.line_to(x1,y1+2) + + #revert angles by pi degrees to take the turn back + ang1 = self.angles[x_index][data_index] + math.pi + ang2 = self.angles[x_index-1][data_index] + self.context.curve_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1), + x2+p*math.cos(ang2),y2+p*math.sin(ang2), + x2,y2+2) + + self.context.close_path() + self.context.fill() + + if False: + self.context.move_to(self.borders[HORZ] + 0.5*self.steps[HORZ], middle) + for x_index in range(len(self.data)-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] - self.data[x_index][data_index]*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index + 1)*self.steps[VERT] - self.data[x_index + 1][data_index]*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index+1][i]*self.steps[VERT] + + ang1 = self.angles[x_index][data_index+1] + ang2 = self.angles[x_index+1][data_index+1] + math.pi + self.context.set_source_rgba(1.0,0.0,0.0) + self.context.arc(x1+p*math.cos(ang1),y1+p*math.sin(ang1),2,0,2*math.pi) + self.context.fill() + self.context.set_source_rgba(0.0,0.0,0.0) + self.context.arc(x2+p*math.cos(ang2),y2+p*math.sin(ang2),2,0,2*math.pi) + self.context.fill() + '''self.context.set_source_rgba(0.0,0.0,0.0,0.3) + self.context.arc(x2,y2,2,0,2*math.pi) + self.context.fill()''' + self.context.move_to(x1,y1) + self.context.line_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1)) + self.context.stroke() + self.context.move_to(x2,y2) + self.context.line_to(x2+p*math.cos(ang2),y2+p*math.sin(ang2)) + self.context.stroke() + if False: + for x_index in range(len(self.data)-1,0,-1) : + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] + y1 = middle - self.ground(x_index)*self.steps[VERT] + x2 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] + y2 = middle - self.ground(x_index - 1)*self.steps[VERT] + + for i in range(0,data_index): + y1 -= self.data[x_index][i]*self.steps[VERT] + y2 -= self.data[x_index-1][i]*self.steps[VERT] + + #revert angles by pi degrees to take the turn back + ang1 = self.angles[x_index][data_index] + math.pi + ang2 = self.angles[x_index-1][data_index] + self.context.set_source_rgba(0.0,1.0,0.0) + self.context.arc(x1+p*math.cos(ang1),y1+p*math.sin(ang1),2,0,2*math.pi) + self.context.fill() + self.context.set_source_rgba(0.0,0.0,1.0) + self.context.arc(x2+p*math.cos(ang2),y2+p*math.sin(ang2),2,0,2*math.pi) + self.context.fill() + '''self.context.set_source_rgba(0.0,0.0,0.0,0.3) + self.context.arc(x2,y2,2,0,2*math.pi) + self.context.fill()''' + self.context.move_to(x1,y1) + self.context.line_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1)) + self.context.stroke() + self.context.move_to(x2,y2) + self.context.line_to(x2+p*math.cos(ang2),y2+p*math.sin(ang2)) + self.context.stroke() + #break + + #self.context.arc(self.dimensions[HORZ]/2, self.dimensions[VERT]/2,50,0,3*math.pi/2) + #self.context.fill() + + +class PiePlot(Plot): + #TODO: Check the old cairoplot, graphs aren't matching + def __init__ (self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + gradient = False, + shadow = False, + colors = None): + + Plot.__init__( self, surface, data, width, height, background, series_colors = colors ) + self.center = (self.dimensions[HORZ]/2, self.dimensions[VERT]/2) + self.total = sum( self.series.to_list() ) + self.radius = min(self.dimensions[HORZ]/3,self.dimensions[VERT]/3) + self.gradient = gradient + self.shadow = shadow + + def sort_function(x,y): + return x.content - y.content + + def load_series(self, data, x_labels=None, y_labels=None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + # Already done inside series + #self.data = sorted(self.data) + + def draw_piece(self, angle, next_angle): + self.context.move_to(self.center[0],self.center[1]) + self.context.line_to(self.center[0] + self.radius*math.cos(angle), self.center[1] + self.radius*math.sin(angle)) + self.context.arc(self.center[0], self.center[1], self.radius, angle, next_angle) + self.context.line_to(self.center[0], self.center[1]) + self.context.close_path() + + def render(self): + self.render_background() + self.render_bounding_box() + if self.shadow: + self.render_shadow() + self.render_plot() + self.render_series_labels() + + def render_shadow(self): + horizontal_shift = 3 + vertical_shift = 3 + self.context.set_source_rgba(0, 0, 0, 0.5) + self.context.arc(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.radius, 0, 2*math.pi) + self.context.fill() + + def render_series_labels(self): + angle = 0 + next_angle = 0 + x0,y0 = self.center + cr = self.context + for number,key in enumerate(self.series_labels): + # self.data[number] should be just a number + data = sum(self.series[number].to_list()) + + next_angle = angle + 2.0*math.pi*data/self.total + cr.set_source_rgba(*self.series_colors[number][:4]) + w = cr.text_extents(key)[2] + if (angle + next_angle)/2 < math.pi/2 or (angle + next_angle)/2 > 3*math.pi/2: + cr.move_to(x0 + (self.radius+10)*math.cos((angle+next_angle)/2), y0 + (self.radius+10)*math.sin((angle+next_angle)/2) ) + else: + cr.move_to(x0 + (self.radius+10)*math.cos((angle+next_angle)/2) - w, y0 + (self.radius+10)*math.sin((angle+next_angle)/2) ) + cr.show_text(key) + angle = next_angle + + def render_plot(self): + angle = 0 + next_angle = 0 + x0,y0 = self.center + cr = self.context + for number,group in enumerate(self.series): + # Group should be just a number + data = sum(group.to_list()) + next_angle = angle + 2.0*math.pi*data/self.total + if self.gradient or self.series_colors[number][4] in ('linear','radial'): + gradient_color = cairo.RadialGradient(self.center[0], self.center[1], 0, self.center[0], self.center[1], self.radius) + gradient_color.add_color_stop_rgba(0.3, *self.series_colors[number][:4]) + gradient_color.add_color_stop_rgba(1, self.series_colors[number][0]*0.7, + self.series_colors[number][1]*0.7, + self.series_colors[number][2]*0.7, + self.series_colors[number][3]) + cr.set_source(gradient_color) + else: + cr.set_source_rgba(*self.series_colors[number][:4]) + + self.draw_piece(angle, next_angle) + cr.fill() + + cr.set_source_rgba(1.0, 1.0, 1.0) + self.draw_piece(angle, next_angle) + cr.stroke() + + angle = next_angle + +class DonutPlot(PiePlot): + def __init__ (self, + surface = None, + data = None, + width = 640, + height = 480, + background = "white light_gray", + gradient = False, + shadow = False, + colors = None, + inner_radius=-1): + + Plot.__init__( self, surface, data, width, height, background, series_colors = colors ) + + self.center = ( self.dimensions[HORZ]/2, self.dimensions[VERT]/2 ) + self.total = sum( self.series.to_list() ) + self.radius = min( self.dimensions[HORZ]/3,self.dimensions[VERT]/3 ) + self.inner_radius = inner_radius*self.radius + + if inner_radius == -1: + self.inner_radius = self.radius/3 + + self.gradient = gradient + self.shadow = shadow + + def draw_piece(self, angle, next_angle): + self.context.move_to(self.center[0] + (self.inner_radius)*math.cos(angle), self.center[1] + (self.inner_radius)*math.sin(angle)) + self.context.line_to(self.center[0] + self.radius*math.cos(angle), self.center[1] + self.radius*math.sin(angle)) + self.context.arc(self.center[0], self.center[1], self.radius, angle, next_angle) + self.context.line_to(self.center[0] + (self.inner_radius)*math.cos(next_angle), self.center[1] + (self.inner_radius)*math.sin(next_angle)) + self.context.arc_negative(self.center[0], self.center[1], self.inner_radius, next_angle, angle) + self.context.close_path() + + def render_shadow(self): + horizontal_shift = 3 + vertical_shift = 3 + self.context.set_source_rgba(0, 0, 0, 0.5) + self.context.arc(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.inner_radius, 0, 2*math.pi) + self.context.arc_negative(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.radius, 0, -2*math.pi) + self.context.fill() + +class GanttChart (Plot) : + def __init__(self, + surface = None, + data = None, + width = 640, + height = 480, + x_labels = None, + y_labels = None, + colors = None): + self.bounds = {} + self.max_value = {} + Plot.__init__(self, surface, data, width, height, x_labels = x_labels, y_labels = y_labels, series_colors = colors) + + def load_series(self, data, x_labels=None, y_labels=None, series_colors=None): + Plot.load_series(self, data, x_labels, y_labels, series_colors) + self.calc_boundaries() + + def calc_boundaries(self): + self.bounds[HORZ] = (0,len(self.series)) + end_pos = max(self.series.to_list()) + + #for group in self.series: + # if hasattr(item, "__delitem__"): + # for sub_item in item: + # end_pos = max(sub_item) + # else: + # end_pos = max(item) + self.bounds[VERT] = (0,end_pos) + + def calc_extents(self, direction): + self.max_value[direction] = 0 + if self.labels[direction]: + self.max_value[direction] = max(self.context.text_extents(item)[2] for item in self.labels[direction]) + else: + self.max_value[direction] = self.context.text_extents( str(self.bounds[direction][1] + 1) )[2] + + def calc_horz_extents(self): + self.calc_extents(HORZ) + self.borders[HORZ] = 100 + self.max_value[HORZ] + + def calc_vert_extents(self): + self.calc_extents(VERT) + self.borders[VERT] = self.dimensions[VERT]/(self.bounds[HORZ][1] + 1) + + def calc_steps(self): + self.horizontal_step = (self.dimensions[HORZ] - self.borders[HORZ])/(len(self.labels[VERT])) + self.vertical_step = self.borders[VERT] + + def render(self): + self.calc_horz_extents() + self.calc_vert_extents() + self.calc_steps() + self.render_background() + + self.render_labels() + self.render_grid() + self.render_plot() + + def render_background(self): + cr = self.context + cr.set_source_rgba(255,255,255) + cr.rectangle(0,0,self.dimensions[HORZ], self.dimensions[VERT]) + cr.fill() + for number,group in enumerate(self.series): + linear = cairo.LinearGradient(self.dimensions[HORZ]/2, self.borders[VERT] + number*self.vertical_step, + self.dimensions[HORZ]/2, self.borders[VERT] + (number+1)*self.vertical_step) + linear.add_color_stop_rgba(0,1.0,1.0,1.0,1.0) + linear.add_color_stop_rgba(1.0,0.9,0.9,0.9,1.0) + cr.set_source(linear) + cr.rectangle(0,self.borders[VERT] + number*self.vertical_step,self.dimensions[HORZ],self.vertical_step) + cr.fill() + + def render_grid(self): + cr = self.context + cr.set_source_rgba(0.7, 0.7, 0.7) + cr.set_dash((1,0,0,0,0,0,1)) + cr.set_line_width(0.5) + for number,label in enumerate(self.labels[VERT]): + h = cr.text_extents(label)[3] + cr.move_to(self.borders[HORZ] + number*self.horizontal_step, self.vertical_step/2 + h) + cr.line_to(self.borders[HORZ] + number*self.horizontal_step, self.dimensions[VERT]) + cr.stroke() + + def render_labels(self): + self.context.set_font_size(0.02 * self.dimensions[HORZ]) + + self.render_horz_labels() + self.render_vert_labels() + + def render_horz_labels(self): + cr = self.context + labels = self.labels[HORZ] + if not labels: + labels = [str(i) for i in range(1, self.bounds[HORZ][1] + 1) ] + for number,label in enumerate(labels): + if label != None: + cr.set_source_rgba(0.5, 0.5, 0.5) + w,h = cr.text_extents(label)[2], cr.text_extents(label)[3] + cr.move_to(40,self.borders[VERT] + number*self.vertical_step + self.vertical_step/2 + h/2) + cr.show_text(label) + + def render_vert_labels(self): + cr = self.context + labels = self.labels[VERT] + if not labels: + labels = [str(i) for i in range(1, self.bounds[VERT][1] + 1) ] + for number,label in enumerate(labels): + w,h = cr.text_extents(label)[2], cr.text_extents(label)[3] + cr.move_to(self.borders[HORZ] + number*self.horizontal_step - w/2, self.vertical_step/2) + cr.show_text(label) + + def render_rectangle(self, x0, y0, x1, y1, color): + self.draw_shadow(x0, y0, x1, y1) + self.draw_rectangle(x0, y0, x1, y1, color) + + def draw_rectangular_shadow(self, gradient, x0, y0, w, h): + self.context.set_source(gradient) + self.context.rectangle(x0,y0,w,h) + self.context.fill() + + def draw_circular_shadow(self, x, y, radius, ang_start, ang_end, mult, shadow): + gradient = cairo.RadialGradient(x, y, 0, x, y, 2*radius) + gradient.add_color_stop_rgba(0, 0, 0, 0, shadow) + gradient.add_color_stop_rgba(1, 0, 0, 0, 0) + self.context.set_source(gradient) + self.context.move_to(x,y) + self.context.line_to(x + mult[0]*radius,y + mult[1]*radius) + self.context.arc(x, y, 8, ang_start, ang_end) + self.context.line_to(x,y) + self.context.close_path() + self.context.fill() + + def draw_rectangle(self, x0, y0, x1, y1, color): + cr = self.context + middle = (x0+x1)/2 + linear = cairo.LinearGradient(middle,y0,middle,y1) + linear.add_color_stop_rgba(0,3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) + linear.add_color_stop_rgba(1,*color[:4]) + cr.set_source(linear) + + cr.arc(x0+5, y0+5, 5, 0, 2*math.pi) + cr.arc(x1-5, y0+5, 5, 0, 2*math.pi) + cr.arc(x0+5, y1-5, 5, 0, 2*math.pi) + cr.arc(x1-5, y1-5, 5, 0, 2*math.pi) + cr.rectangle(x0+5,y0,x1-x0-10,y1-y0) + cr.rectangle(x0,y0+5,x1-x0,y1-y0-10) + cr.fill() + + def draw_shadow(self, x0, y0, x1, y1): + shadow = 0.4 + h_mid = (x0+x1)/2 + v_mid = (y0+y1)/2 + h_linear_1 = cairo.LinearGradient(h_mid,y0-4,h_mid,y0+4) + h_linear_2 = cairo.LinearGradient(h_mid,y1-4,h_mid,y1+4) + v_linear_1 = cairo.LinearGradient(x0-4,v_mid,x0+4,v_mid) + v_linear_2 = cairo.LinearGradient(x1-4,v_mid,x1+4,v_mid) + + h_linear_1.add_color_stop_rgba( 0, 0, 0, 0, 0) + h_linear_1.add_color_stop_rgba( 1, 0, 0, 0, shadow) + h_linear_2.add_color_stop_rgba( 0, 0, 0, 0, shadow) + h_linear_2.add_color_stop_rgba( 1, 0, 0, 0, 0) + v_linear_1.add_color_stop_rgba( 0, 0, 0, 0, 0) + v_linear_1.add_color_stop_rgba( 1, 0, 0, 0, shadow) + v_linear_2.add_color_stop_rgba( 0, 0, 0, 0, shadow) + v_linear_2.add_color_stop_rgba( 1, 0, 0, 0, 0) + + self.draw_rectangular_shadow(h_linear_1,x0+4,y0-4,x1-x0-8,8) + self.draw_rectangular_shadow(h_linear_2,x0+4,y1-4,x1-x0-8,8) + self.draw_rectangular_shadow(v_linear_1,x0-4,y0+4,8,y1-y0-8) + self.draw_rectangular_shadow(v_linear_2,x1-4,y0+4,8,y1-y0-8) + + self.draw_circular_shadow(x0+4, y0+4, 4, math.pi, 3*math.pi/2, (-1,0), shadow) + self.draw_circular_shadow(x1-4, y0+4, 4, 3*math.pi/2, 2*math.pi, (0,-1), shadow) + self.draw_circular_shadow(x0+4, y1-4, 4, math.pi/2, math.pi, (0,1), shadow) + self.draw_circular_shadow(x1-4, y1-4, 4, 0, math.pi/2, (1,0), shadow) + + def render_plot(self): + for index,group in enumerate(self.series): + for data in group: + self.render_rectangle(self.borders[HORZ] + data.content[0]*self.horizontal_step, + self.borders[VERT] + index*self.vertical_step + self.vertical_step/4.0, + self.borders[HORZ] + data.content[1]*self.horizontal_step, + self.borders[VERT] + index*self.vertical_step + 3.0*self.vertical_step/4.0, + self.series_colors[index]) + +# Function definition + +def scatter_plot(name, + data = None, + errorx = None, + errory = None, + width = 640, + height = 480, + background = "white light_gray", + border = 0, + axis = False, + dash = False, + discrete = False, + dots = False, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + z_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + circle_colors = None): + + ''' + - Function to plot scatter data. + + - Parameters + + data - The values to be ploted might be passed in a two basic: + list of points: [(0,0), (0,1), (0,2)] or [(0,0,1), (0,1,4), (0,2,1)] + lists of coordinates: [ [0,0,0] , [0,1,2] ] or [ [0,0,0] , [0,1,2] , [1,4,1] ] + Notice that these kinds of that can be grouped in order to form more complex data + using lists of lists or dictionaries; + series_colors - Define color values for each of the series + circle_colors - Define a lower and an upper bound for the circle colors for variable radius + (3 dimensions) series + ''' + + plot = ScatterPlot( name, data, errorx, errory, width, height, background, border, + axis, dash, discrete, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, z_bounds, x_title, y_title, series_colors, circle_colors ) + plot.render() + plot.commit() + +def dot_line_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + axis = False, + dash = False, + dots = False, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None): + ''' + - Function to plot graphics using dots and lines. + + dot_line_plot (name, data, width, height, background = "white light_gray", border = 0, axis = False, grid = False, x_labels = None, y_labels = None, x_bounds = None, y_bounds = None) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + axis - Whether or not the axis are to be drawn; + dash - Boolean or a list or a dictionary of booleans indicating whether or not the associated series should be drawn in dashed mode; + dots - Whether or not dots should be drawn on each point; + grid - Whether or not the gris is to be drawn; + series_legend - Whether or not the legend is to be drawn; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + x_title - Whether or not to plot a title over the x axis. + y_title - Whether or not to plot a title over the y axis. + + - Examples of use + + data = [0, 1, 3, 8, 9, 0, 10, 10, 2, 1] + CairoPlot.dot_line_plot('teste', data, 400, 300) + + data = { "john" : [10, 10, 10, 10, 30], "mary" : [0, 0, 3, 5, 15], "philip" : [13, 32, 11, 25, 2] } + x_labels = ["jan/2008", "feb/2008", "mar/2008", "apr/2008", "may/2008" ] + CairoPlot.dot_line_plot( 'test', data, 400, 300, axis = True, grid = True, + series_legend = True, x_labels = x_labels ) + ''' + plot = DotLinePlot( name, data, width, height, background, border, + axis, dash, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, x_title, y_title, series_colors ) + plot.render() + plot.commit() + +def function_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + axis = True, + dots = False, + discrete = False, + grid = False, + series_legend = False, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + x_title = None, + y_title = None, + series_colors = None, + step = 1): + + ''' + - Function to plot functions. + + function_plot(name, data, width, height, background = "white light_gray", border = 0, axis = True, grid = False, dots = False, x_labels = None, y_labels = None, x_bounds = None, y_bounds = None, step = 1, discrete = False) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + axis - Whether or not the axis are to be drawn; + grid - Whether or not the gris is to be drawn; + dots - Whether or not dots should be shown at each point; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + step - the horizontal distance from one point to the other. The smaller, the smoother the curve will be; + discrete - whether or not the function should be plotted in discrete format. + + - Example of use + + data = lambda x : x**2 + CairoPlot.function_plot('function4', data, 400, 300, grid = True, x_bounds=(-10,10), step = 0.1) + ''' + + plot = FunctionPlot( name, data, width, height, background, border, + axis, discrete, dots, grid, series_legend, x_labels, y_labels, + x_bounds, y_bounds, x_title, y_title, series_colors, step ) + plot.render() + plot.commit() + +def pie_plot( name, data, width, height, background = "white light_gray", gradient = False, shadow = False, colors = None ): + + ''' + - Function to plot pie graphics. + + pie_plot(name, data, width, height, background = "white light_gray", gradient = False, colors = None) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + gradient - Whether or not the pie color will be painted with a gradient; + shadow - Whether or not there will be a shadow behind the pie; + colors - List of slices colors. + + - Example of use + + teste_data = {"john" : 123, "mary" : 489, "philip" : 890 , "suzy" : 235} + CairoPlot.pie_plot("pie_teste", teste_data, 500, 500) + ''' + + plot = PiePlot( name, data, width, height, background, gradient, shadow, colors ) + plot.render() + plot.commit() + +def donut_plot(name, data, width, height, background = "white light_gray", gradient = False, shadow = False, colors = None, inner_radius = -1): + + ''' + - Function to plot donut graphics. + + donut_plot(name, data, width, height, background = "white light_gray", gradient = False, inner_radius = -1) + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + shadow - Whether or not there will be a shadow behind the donut; + gradient - Whether or not the donut color will be painted with a gradient; + colors - List of slices colors; + inner_radius - The radius of the donut's inner circle. + + - Example of use + + teste_data = {"john" : 123, "mary" : 489, "philip" : 890 , "suzy" : 235} + CairoPlot.donut_plot("donut_teste", teste_data, 500, 500) + ''' + + plot = DonutPlot(name, data, width, height, background, gradient, shadow, colors, inner_radius) + plot.render() + plot.commit() + +def gantt_chart(name, pieces, width, height, x_labels, y_labels, colors): + + ''' + - Function to generate Gantt Charts. + + gantt_chart(name, pieces, width, height, x_labels, y_labels, colors): + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; + pieces - A list defining the spaces to be drawn. The user must pass, for each line, the index of its start and the index of its end. If a line must have two or more spaces, they must be passed inside a list; + width, height - Dimensions of the output image; + x_labels - A list of names for each of the vertical lines; + y_labels - A list of names for each of the horizontal spaces; + colors - List containing the colors expected for each of the horizontal spaces + + - Example of use + + pieces = [ (0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,8)] + x_labels = [ 'teste01', 'teste02', 'teste03', 'teste04'] + y_labels = [ '0001', '0002', '0003', '0004', '0005', '0006', '0007', '0008', '0009', '0010' ] + colors = [ (1.0, 0.0, 0.0), (1.0, 0.7, 0.0), (1.0, 1.0, 0.0), (0.0, 1.0, 0.0) ] + CairoPlot.gantt_chart('gantt_teste', pieces, 600, 300, x_labels, y_labels, colors) + ''' + + plot = GanttChart(name, pieces, width, height, x_labels, y_labels, colors) + plot.render() + plot.commit() + +def vertical_bar_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + colors = None): + #TODO: Fix docstring for vertical_bar_plot + ''' + - Function to generate vertical Bar Plot Charts. + + bar_plot(name, data, width, height, background, border, grid, rounded_corners, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, colors): + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtime; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + grid - Whether or not the gris is to be drawn; + rounded_corners - Whether or not the bars should have rounded corners; + three_dimension - Whether or not the bars should be drawn in pseudo 3D; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + colors - List containing the colors expected for each of the bars. + + - Example of use + + data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] + CairoPlot.vertical_bar_plot ('bar2', data, 400, 300, border = 20, grid = True, rounded_corners = False) + ''' + + plot = VerticalBarPlot(name, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + series_labels, x_labels, y_labels, x_bounds, y_bounds, colors) + plot.render() + plot.commit() + +def horizontal_bar_plot(name, + data, + width, + height, + background = "white light_gray", + border = 0, + display_values = False, + grid = False, + rounded_corners = False, + stack = False, + three_dimension = False, + series_labels = None, + x_labels = None, + y_labels = None, + x_bounds = None, + y_bounds = None, + colors = None): + + #TODO: Fix docstring for horizontal_bar_plot + ''' + - Function to generate Horizontal Bar Plot Charts. + + bar_plot(name, data, width, height, background, border, grid, rounded_corners, three_dimension, + x_labels, y_labels, x_bounds, y_bounds, colors): + + - Parameters + + name - Name of the desired output file, no need to input the .svg as it will be added at runtime; + data - The list, list of lists or dictionary holding the data to be plotted; + width, height - Dimensions of the output image; + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. + If left None, a gray to white gradient will be generated; + border - Distance in pixels of a square border into which the graphics will be drawn; + grid - Whether or not the gris is to be drawn; + rounded_corners - Whether or not the bars should have rounded corners; + three_dimension - Whether or not the bars should be drawn in pseudo 3D; + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; + colors - List containing the colors expected for each of the bars. + + - Example of use + + data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] + CairoPlot.bar_plot ('bar2', data, 400, 300, border = 20, grid = True, rounded_corners = False) + ''' + + plot = HorizontalBarPlot(name, data, width, height, background, border, + display_values, grid, rounded_corners, stack, three_dimension, + series_labels, x_labels, y_labels, x_bounds, y_bounds, colors) + plot.render() + plot.commit() + +def stream_chart(name, + data, + width, + height, + background = "white light_gray", + border = 0, + grid = False, + series_legend = None, + x_labels = None, + x_bounds = None, + y_bounds = None, + colors = None): + + #TODO: Fix docstring for horizontal_bar_plot + plot = StreamChart(name, data, width, height, background, border, + grid, series_legend, x_labels, x_bounds, y_bounds, colors) + plot.render() + plot.commit() + + +if __name__ == "__main__": + import tests + import seriestests diff --git a/bindings/python/examples/output_format_modules/pprint_table.py b/bindings/python/examples/output_format_modules/pprint_table.py new file mode 100644 index 0000000..a7e8255 --- /dev/null +++ b/bindings/python/examples/output_format_modules/pprint_table.py @@ -0,0 +1,37 @@ +# pprint_table.py +# +# This module is used to pretty-print a table +# Adapted from +# http://ginstrom.com/scribbles/2007/09/04/pretty-printing-a-table-in-python/ + +import sys + +def get_max_width(table, index): + """Get the maximum width of the given column index""" + + return max([len(str(row[index])) for row in table]) + + +def pprint_table(table, nbLeft=1, out=sys.stdout): + """ + Prints out a table of data, padded for alignment + @param table: The table to print. A list of lists. + Each row must have the same number of columns. + @param nbLeft: The number of columns aligned left + @param out: Output stream (file-like object) + """ + + col_paddings = [] + + for i in range(len(table[0])): + col_paddings.append(get_max_width(table, i)) + + for row in table: + # left cols + for i in range(nbLeft): + print >> out, str(row[i]).ljust(col_paddings[i] + 1), + # rest of the cols + for i in range(nbLeft, len(row)): + col = str(row[i]).rjust(col_paddings[i] + 2) + print >> out, col, + print >> out diff --git a/bindings/python/examples/output_format_modules/series.py b/bindings/python/examples/output_format_modules/series.py new file mode 100755 index 0000000..8e8b236 --- /dev/null +++ b/bindings/python/examples/output_format_modules/series.py @@ -0,0 +1,1140 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# Serie.py +# +# Copyright (c) 2008 Magnun Leno da Silva +# +# Author: Magnun Leno da Silva +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public License +# as published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 +# USA + +# Contributor: Rodrigo Moreiro Araujo + +#import cairoplot +import doctest + +NUMTYPES = (int, float, long) +LISTTYPES = (list, tuple) +STRTYPES = (str, unicode) +FILLING_TYPES = ['linear', 'solid', 'gradient'] +DEFAULT_COLOR_FILLING = 'solid' +#TODO: Define default color list +DEFAULT_COLOR_LIST = None + +class Data(object): + ''' + Class that models the main data structure. + It can hold: + - a number type (int, float or long) + - a tuple, witch represents a point and can have 2 or 3 items (x,y,z) + - if a list is passed it will be converted to a tuple. + + obs: In case a tuple is passed it will convert to tuple + ''' + def __init__(self, data=None, name=None, parent=None): + ''' + Starts main atributes from the Data class + @name - Name for each point; + @content - The real data, can be an int, float, long or tuple, which + represents a point (x,y) or (x,y,z); + @parent - A pointer that give the data access to it's parent. + + Usage: + >>> d = Data(name='empty'); print d + empty: () + >>> d = Data((1,1),'point a'); print d + point a: (1, 1) + >>> d = Data((1,2,3),'point b'); print d + point b: (1, 2, 3) + >>> d = Data([2,3],'point c'); print d + point c: (2, 3) + >>> d = Data(12, 'simple value'); print d + simple value: 12 + ''' + # Initial values + self.__content = None + self.__name = None + + # Setting passed values + self.parent = parent + self.name = name + self.content = data + + # Name property + @apply + def name(): + doc = ''' + Name is a read/write property that controls the input of name. + - If passed an invalid value it cleans the name with None + + Usage: + >>> d = Data(13); d.name = 'name_test'; print d + name_test: 13 + >>> d.name = 11; print d + 13 + >>> d.name = 'other_name'; print d + other_name: 13 + >>> d.name = None; print d + 13 + >>> d.name = 'last_name'; print d + last_name: 13 + >>> d.name = ''; print d + 13 + ''' + def fget(self): + ''' + returns the name as a string + ''' + return self.__name + + def fset(self, name): + ''' + Sets the name of the Data + ''' + if type(name) in STRTYPES and len(name) > 0: + self.__name = name + else: + self.__name = None + + + + return property(**locals()) + + # Content property + @apply + def content(): + doc = ''' + Content is a read/write property that validate the data passed + and return it. + + Usage: + >>> d = Data(); d.content = 13; d.content + 13 + >>> d = Data(); d.content = (1,2); d.content + (1, 2) + >>> d = Data(); d.content = (1,2,3); d.content + (1, 2, 3) + >>> d = Data(); d.content = [1,2,3]; d.content + (1, 2, 3) + >>> d = Data(); d.content = [1.5,.2,3.3]; d.content + (1.5, 0.20000000000000001, 3.2999999999999998) + ''' + def fget(self): + ''' + Return the content of Data + ''' + return self.__content + + def fset(self, data): + ''' + Ensures that data is a valid tuple/list or a number (int, float + or long) + ''' + # Type: None + if data is None: + self.__content = None + return + + # Type: Int or Float + elif type(data) in NUMTYPES: + self.__content = data + + # Type: List or Tuple + elif type(data) in LISTTYPES: + # Ensures the correct size + if len(data) not in (2, 3): + raise TypeError, "Data (as list/tuple) must have 2 or 3 items" + return + + # Ensures that all items in list/tuple is a number + isnum = lambda x : type(x) not in NUMTYPES + + if max(map(isnum, data)): + # An item in data isn't an int or a float + raise TypeError, "All content of data must be a number (int or float)" + + # Convert the tuple to list + if type(data) is list: + data = tuple(data) + + # Append a copy and sets the type + self.__content = data[:] + + # Unknown type! + else: + self.__content = None + raise TypeError, "Data must be an int, float or a tuple with two or three items" + return + + return property(**locals()) + + + def clear(self): + ''' + Clear the all Data (content, name and parent) + ''' + self.content = None + self.name = None + self.parent = None + + def copy(self): + ''' + Returns a copy of the Data structure + ''' + # The copy + new_data = Data() + if self.content is not None: + # If content is a point + if type(self.content) is tuple: + new_data.__content = self.content[:] + + # If content is a number + else: + new_data.__content = self.content + + # If it has a name + if self.name is not None: + new_data.__name = self.name + + return new_data + + def __str__(self): + ''' + Return a string representation of the Data structure + ''' + if self.name is None: + if self.content is None: + return '' + return str(self.content) + else: + if self.content is None: + return self.name+": ()" + return self.name+": "+str(self.content) + + def __len__(self): + ''' + Return the length of the Data. + - If it's a number return 1; + - If it's a list return it's length; + - If its None return 0. + ''' + if self.content is None: + return 0 + elif type(self.content) in NUMTYPES: + return 1 + return len(self.content) + + + + +class Group(object): + ''' + Class that models a group of data. Every value (int, float, long, tuple + or list) passed is converted to a list of Data. + It can receive: + - A single number (int, float, long); + - A list of numbers; + - A tuple of numbers; + - An instance of Data; + - A list of Data; + + Obs: If a tuple with 2 or 3 items is passed it is converted to a point. + If a tuple with only 1 item is passed it's converted to a number; + If a tuple with more than 2 items is passed it's converted to a + list of numbers + ''' + def __init__(self, group=None, name=None, parent=None): + ''' + Starts main atributes in Group instance. + @data_list - a list of data which forms the group; + @range - a range that represent the x axis of possible functions; + @name - name of the data group; + @parent - the Serie parent of this group. + + Usage: + >>> g = Group(13, 'simple number'); print g + simple number ['13'] + >>> g = Group((1,2), 'simple point'); print g + simple point ['(1, 2)'] + >>> g = Group([1,2,3,4], 'list of numbers'); print g + list of numbers ['1', '2', '3', '4'] + >>> g = Group((1,2,3,4),'int in tuple'); print g + int in tuple ['1', '2', '3', '4'] + >>> g = Group([(1,2),(2,3),(3,4)], 'list of points'); print g + list of points ['(1, 2)', '(2, 3)', '(3, 4)'] + >>> g = Group([[1,2,3],[1,2,3]], '2D coordinate lists'); print g + 2D coordinated lists ['(1, 1)', '(2, 2)', '(3, 3)'] + >>> g = Group([[1,2],[1,2],[1,2]], '3D coordinate lists'); print g + 3D coordinated lists ['(1, 1, 1)', '(2, 2, 2)'] + ''' + # Initial values + self.__data_list = [] + self.__range = [] + self.__name = None + + + self.parent = parent + self.name = name + self.data_list = group + + # Name property + @apply + def name(): + doc = ''' + Name is a read/write property that controls the input of name. + - If passed an invalid value it cleans the name with None + + Usage: + >>> g = Group(13); g.name = 'name_test'; print g + name_test ['13'] + >>> g.name = 11; print g + ['13'] + >>> g.name = 'other_name'; print g + other_name ['13'] + >>> g.name = None; print g + ['13'] + >>> g.name = 'last_name'; print g + last_name ['13'] + >>> g.name = ''; print g + ['13'] + ''' + def fget(self): + ''' + Returns the name as a string + ''' + return self.__name + + def fset(self, name): + ''' + Sets the name of the Group + ''' + if type(name) in STRTYPES and len(name) > 0: + self.__name = name + else: + self.__name = None + + return property(**locals()) + + # data_list property + @apply + def data_list(): + doc = ''' + The data_list is a read/write property that can be a list of + numbers, a list of points or a list of 2 or 3 coordinate lists. This + property uses mainly the self.add_data method. + + Usage: + >>> g = Group(); g.data_list = 13; print g + ['13'] + >>> g.data_list = (1,2); print g + ['(1, 2)'] + >>> g.data_list = Data((1,2),'point a'); print g + ['point a: (1, 2)'] + >>> g.data_list = [1,2,3]; print g + ['1', '2', '3'] + >>> g.data_list = (1,2,3,4); print g + ['1', '2', '3', '4'] + >>> g.data_list = [(1,2),(2,3),(3,4)]; print g + ['(1, 2)', '(2, 3)', '(3, 4)'] + >>> g.data_list = [[1,2],[1,2]]; print g + ['(1, 1)', '(2, 2)'] + >>> g.data_list = [[1,2],[1,2],[1,2]]; print g + ['(1, 1, 1)', '(2, 2, 2)'] + >>> g.range = (10); g.data_list = lambda x:x**2; print g + ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)'] + ''' + def fget(self): + ''' + Returns the value of data_list + ''' + return self.__data_list + + def fset(self, group): + ''' + Ensures that group is valid. + ''' + # None + if group is None: + self.__data_list = [] + + # Int/float/long or Instance of Data + elif type(group) in NUMTYPES or isinstance(group, Data): + # Clean data_list + self.__data_list = [] + self.add_data(group) + + # One point + elif type(group) is tuple and len(group) in (2,3): + self.__data_list = [] + self.add_data(group) + + # list of items + elif type(group) in LISTTYPES and type(group[0]) is not list: + # Clean data_list + self.__data_list = [] + for item in group: + # try to append and catch an exception + self.add_data(item) + + # function lambda + elif callable(group): + # Explicit is better than implicit + function = group + # Has range + if len(self.range) is not 0: + # Clean data_list + self.__data_list = [] + # Generate values for the lambda function + for x in self.range: + #self.add_data((x,round(group(x),2))) + self.add_data((x,function(x))) + + # Only have range in parent + elif self.parent is not None and len(self.parent.range) is not 0: + # Copy parent range + self.__range = self.parent.range[:] + # Clean data_list + self.__data_list = [] + # Generate values for the lambda function + for x in self.range: + #self.add_data((x,round(group(x),2))) + self.add_data((x,function(x))) + + # Don't have range anywhere + else: + # x_data don't exist + raise Exception, "Data argument is valid but to use function type please set x_range first" + + # Coordinate Lists + elif type(group) in LISTTYPES and type(group[0]) is list: + # Clean data_list + self.__data_list = [] + data = [] + if len(group) == 3: + data = zip(group[0], group[1], group[2]) + elif len(group) == 2: + data = zip(group[0], group[1]) + else: + raise TypeError, "Only one list of coordinates was received." + + for item in data: + self.add_data(item) + + else: + raise TypeError, "Group type not supported" + + return property(**locals()) + + @apply + def range(): + doc = ''' + The range is a read/write property that generates a range of values + for the x axis of the functions. When passed a tuple it almost works + like the built-in range funtion: + - 1 item, represent the end of the range started from 0; + - 2 items, represents the start and the end, respectively; + - 3 items, the last one represents the step; + + When passed a list the range function understands as a valid range. + + Usage: + >>> g = Group(); g.range = 10; print g.range + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0] + >>> g = Group(); g.range = (5); print g.range + [0.0, 1.0, 2.0, 3.0, 4.0] + >>> g = Group(); g.range = (1,7); print g.range + [1.0, 2.0, 3.0, 4.0, 5.0, 6.0] + >>> g = Group(); g.range = (0,10,2); print g.range + [0.0, 2.0, 4.0, 6.0, 8.0] + >>> + >>> g = Group(); g.range = [0]; print g.range + [0.0] + >>> g = Group(); g.range = [0,10,20]; print g.range + [0.0, 10.0, 20.0] + ''' + def fget(self): + ''' + Returns the range + ''' + return self.__range + + def fset(self, x_range): + ''' + Controls the input of a valid type and generate the range + ''' + # if passed a simple number convert to tuple + if type(x_range) in NUMTYPES: + x_range = (x_range,) + + # A list, just convert to float + if type(x_range) is list and len(x_range) > 0: + # Convert all to float + x_range = map(float, x_range) + # Prevents repeated values and convert back to list + self.__range = list(set(x_range[:])) + # Sort the list to ascending order + self.__range.sort() + + # A tuple, must check the lengths and generate the values + elif type(x_range) is tuple and len(x_range) in (1,2,3): + # Convert all to float + x_range = map(float, x_range) + + # Inital values + start = 0.0 + step = 1.0 + end = 0.0 + + # Only the end and it can't be less or iqual to 0 + if len(x_range) is 1 and x_range > 0: + end = x_range[0] + + # The start and the end but the start must be less then the end + elif len(x_range) is 2 and x_range[0] < x_range[1]: + start = x_range[0] + end = x_range[1] + + # All 3, but the start must be less then the end + elif x_range[0] <= x_range[1]: + start = x_range[0] + end = x_range[1] + step = x_range[2] + + # Starts the range + self.__range = [] + # Generate the range + # Can't use the range function because it doesn't support float values + while start < end: + self.__range.append(start) + start += step + + # Incorrect type + else: + raise Exception, "x_range must be a list with one or more items or a tuple with 2 or 3 items" + + return property(**locals()) + + def add_data(self, data, name=None): + ''' + Append a new data to the data_list. + - If data is an instance of Data, append it + - If it's an int, float, tuple or list create an instance of Data and append it + + Usage: + >>> g = Group() + >>> g.add_data(12); print g + ['12'] + >>> g.add_data(7,'other'); print g + ['12', 'other: 7'] + >>> + >>> g = Group() + >>> g.add_data((1,1),'a'); print g + ['a: (1, 1)'] + >>> g.add_data((2,2),'b'); print g + ['a: (1, 1)', 'b: (2, 2)'] + >>> + >>> g.add_data(Data((1,2),'c')); print g + ['a: (1, 1)', 'b: (2, 2)', 'c: (1, 2)'] + ''' + if not isinstance(data, Data): + # Try to convert + data = Data(data,name,self) + + if data.content is not None: + self.__data_list.append(data.copy()) + self.__data_list[-1].parent = self + + + def to_list(self): + ''' + Returns the group as a list of numbers (int, float or long) or a + list of tuples (points 2D or 3D). + + Usage: + >>> g = Group([1,2,3,4],'g1'); g.to_list() + [1, 2, 3, 4] + >>> g = Group([(1,2),(2,3),(3,4)],'g2'); g.to_list() + [(1, 2), (2, 3), (3, 4)] + >>> g = Group([(1,2,3),(3,4,5)],'g2'); g.to_list() + [(1, 2, 3), (3, 4, 5)] + ''' + return [data.content for data in self] + + def copy(self): + ''' + Returns a copy of this group + ''' + new_group = Group() + new_group.__name = self.__name + if self.__range is not None: + new_group.__range = self.__range[:] + for data in self: + new_group.add_data(data.copy()) + return new_group + + def get_names(self): + ''' + Return a list with the names of all data in this group + ''' + names = [] + for data in self: + if data.name is None: + names.append('Data '+str(data.index()+1)) + else: + names.append(data.name) + return names + + + def __str__ (self): + ''' + Returns a string representing the Group + ''' + ret = "" + if self.name is not None: + ret += self.name + " " + if len(self) > 0: + list_str = [str(item) for item in self] + ret += str(list_str) + else: + ret += "[]" + return ret + + def __getitem__(self, key): + ''' + Makes a Group iterable, based in the data_list property + ''' + return self.data_list[key] + + def __len__(self): + ''' + Returns the length of the Group, based in the data_list property + ''' + return len(self.data_list) + + +class Colors(object): + ''' + Class that models the colors its labels (names) and its properties, RGB + and filling type. + + It can receive: + - A list where each item is a list with 3 or 4 items. The + first 3 items represent the RGB values and the last argument + defines the filling type. The list will be converted to a dict + and each color will receve a name based in its position in the + list. + - A dictionary where each key will be the color name and its item + can be a list with 3 or 4 items. The first 3 items represent + the RGB colors and the last argument defines the filling type. + ''' + def __init__(self, color_list=None): + ''' + Start the color_list property + @ color_list - the list or dict contaning the colors properties. + ''' + self.__color_list = None + + self.color_list = color_list + + @apply + def color_list(): + doc = ''' + >>> c = Colors([[1,1,1],[2,2,2,'linear'],[3,3,3,'gradient']]) + >>> print c.color_list + {'Color 2': [2, 2, 2, 'linear'], 'Color 3': [3, 3, 3, 'gradient'], 'Color 1': [1, 1, 1, 'solid']} + >>> c.color_list = [[1,1,1],(2,2,2,'solid'),(3,3,3,'linear')] + >>> print c.color_list + {'Color 2': [2, 2, 2, 'solid'], 'Color 3': [3, 3, 3, 'linear'], 'Color 1': [1, 1, 1, 'solid']} + >>> c.color_list = {'a':[1,1,1],'b':(2,2,2,'solid'),'c':(3,3,3,'linear'), 'd':(4,4,4)} + >>> print c.color_list + {'a': [1, 1, 1, 'solid'], 'c': [3, 3, 3, 'linear'], 'b': [2, 2, 2, 'solid'], 'd': [4, 4, 4, 'solid']} + ''' + def fget(self): + ''' + Return the color list + ''' + return self.__color_list + + def fset(self, color_list): + ''' + Format the color list to a dictionary + ''' + if color_list is None: + self.__color_list = None + return + + if type(color_list) in LISTTYPES and type(color_list[0]) in LISTTYPES: + old_color_list = color_list[:] + color_list = {} + for index, color in enumerate(old_color_list): + if len(color) is 3 and max(map(type, color)) in NUMTYPES: + color_list['Color '+str(index+1)] = list(color)+[DEFAULT_COLOR_FILLING] + elif len(color) is 4 and max(map(type, color[:-1])) in NUMTYPES and color[-1] in FILLING_TYPES: + color_list['Color '+str(index+1)] = list(color) + else: + raise TypeError, "Unsuported color format" + elif type(color_list) is not dict: + raise TypeError, "Unsuported color format" + + for name, color in color_list.items(): + if len(color) is 3: + if max(map(type, color)) in NUMTYPES: + color_list[name] = list(color)+[DEFAULT_COLOR_FILLING] + else: + raise TypeError, "Unsuported color format" + elif len(color) is 4: + if max(map(type, color[:-1])) in NUMTYPES and color[-1] in FILLING_TYPES: + color_list[name] = list(color) + else: + raise TypeError, "Unsuported color format" + self.__color_list = color_list.copy() + + return property(**locals()) + + +class Series(object): + ''' + Class that models a Series (group of groups). Every value (int, float, + long, tuple or list) passed is converted to a list of Group or Data. + It can receive: + - a single number or point, will be converted to a Group of one Data; + - a list of numbers, will be converted to a group of numbers; + - a list of tuples, will converted to a single Group of points; + - a list of lists of numbers, each 'sublist' will be converted to a + group of numbers; + - a list of lists of tuples, each 'sublist' will be converted to a + group of points; + - a list of lists of lists, the content of the 'sublist' will be + processed as coordinated lists and the result will be converted to + a group of points; + - a Dictionary where each item can be the same of the list: number, + point, list of numbers, list of points or list of lists (coordinated + lists); + - an instance of Data; + - an instance of group. + ''' + def __init__(self, series=None, name=None, property=[], colors=None): + ''' + Starts main atributes in Group instance. + @series - a list, dict of data of which the series is composed; + @name - name of the series; + @property - a list/dict of properties to be used in the plots of + this Series + + Usage: + >>> print Series([1,2,3,4]) + ["Group 1 ['1', '2', '3', '4']"] + >>> print Series([[1,2,3],[4,5,6]]) + ["Group 1 ['1', '2', '3']", "Group 2 ['4', '5', '6']"] + >>> print Series((1,2)) + ["Group 1 ['(1, 2)']"] + >>> print Series([(1,2),(2,3)]) + ["Group 1 ['(1, 2)', '(2, 3)']"] + >>> print Series([[(1,2),(2,3)],[(4,5),(5,6)]]) + ["Group 1 ['(1, 2)', '(2, 3)']", "Group 2 ['(4, 5)', '(5, 6)']"] + >>> print Series([[[1,2,3],[1,2,3],[1,2,3]]]) + ["Group 1 ['(1, 1, 1)', '(2, 2, 2)', '(3, 3, 3)']"] + >>> print Series({'g1':[1,2,3], 'g2':[4,5,6]}) + ["g1 ['1', '2', '3']", "g2 ['4', '5', '6']"] + >>> print Series({'g1':[(1,2),(2,3)], 'g2':[(4,5),(5,6)]}) + ["g1 ['(1, 2)', '(2, 3)']", "g2 ['(4, 5)', '(5, 6)']"] + >>> print Series({'g1':[[1,2],[1,2]], 'g2':[[4,5],[4,5]]}) + ["g1 ['(1, 1)', '(2, 2)']", "g2 ['(4, 4)', '(5, 5)']"] + >>> print Series(Data(1,'d1')) + ["Group 1 ['d1: 1']"] + >>> print Series(Group([(1,2),(2,3)],'g1')) + ["g1 ['(1, 2)', '(2, 3)']"] + ''' + # Intial values + self.__group_list = [] + self.__name = None + self.__range = None + + # TODO: Implement colors with filling + self.__colors = None + + self.name = name + self.group_list = series + self.colors = colors + + # Name property + @apply + def name(): + doc = ''' + Name is a read/write property that controls the input of name. + - If passed an invalid value it cleans the name with None + + Usage: + >>> s = Series(13); s.name = 'name_test'; print s + name_test ["Group 1 ['13']"] + >>> s.name = 11; print s + ["Group 1 ['13']"] + >>> s.name = 'other_name'; print s + other_name ["Group 1 ['13']"] + >>> s.name = None; print s + ["Group 1 ['13']"] + >>> s.name = 'last_name'; print s + last_name ["Group 1 ['13']"] + >>> s.name = ''; print s + ["Group 1 ['13']"] + ''' + def fget(self): + ''' + Returns the name as a string + ''' + return self.__name + + def fset(self, name): + ''' + Sets the name of the Group + ''' + if type(name) in STRTYPES and len(name) > 0: + self.__name = name + else: + self.__name = None + + return property(**locals()) + + + + # Colors property + @apply + def colors(): + doc = ''' + >>> s = Series() + >>> s.colors = [[1,1,1],[2,2,2,'linear'],[3,3,3,'gradient']] + >>> print s.colors + {'Color 2': [2, 2, 2, 'linear'], 'Color 3': [3, 3, 3, 'gradient'], 'Color 1': [1, 1, 1, 'solid']} + >>> s.colors = [[1,1,1],(2,2,2,'solid'),(3,3,3,'linear')] + >>> print s.colors + {'Color 2': [2, 2, 2, 'solid'], 'Color 3': [3, 3, 3, 'linear'], 'Color 1': [1, 1, 1, 'solid']} + >>> s.colors = {'a':[1,1,1],'b':(2,2,2,'solid'),'c':(3,3,3,'linear'), 'd':(4,4,4)} + >>> print s.colors + {'a': [1, 1, 1, 'solid'], 'c': [3, 3, 3, 'linear'], 'b': [2, 2, 2, 'solid'], 'd': [4, 4, 4, 'solid']} + ''' + def fget(self): + ''' + Return the color list + ''' + return self.__colors.color_list + + def fset(self, colors): + ''' + Format the color list to a dictionary + ''' + self.__colors = Colors(colors) + + return property(**locals()) + + @apply + def range(): + doc = ''' + The range is a read/write property that generates a range of values + for the x axis of the functions. When passed a tuple it almost works + like the built-in range funtion: + - 1 item, represent the end of the range started from 0; + - 2 items, represents the start and the end, respectively; + - 3 items, the last one represents the step; + + When passed a list the range function understands as a valid range. + + Usage: + >>> s = Series(); s.range = 10; print s.range + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0] + >>> s = Series(); s.range = (5); print s.range + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0] + >>> s = Series(); s.range = (1,7); print s.range + [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0] + >>> s = Series(); s.range = (0,10,2); print s.range + [0.0, 2.0, 4.0, 6.0, 8.0, 10.0] + >>> + >>> s = Series(); s.range = [0]; print s.range + [0.0] + >>> s = Series(); s.range = [0,10,20]; print s.range + [0.0, 10.0, 20.0] + ''' + def fget(self): + ''' + Returns the range + ''' + return self.__range + + def fset(self, x_range): + ''' + Controls the input of a valid type and generate the range + ''' + # if passed a simple number convert to tuple + if type(x_range) in NUMTYPES: + x_range = (x_range,) + + # A list, just convert to float + if type(x_range) is list and len(x_range) > 0: + # Convert all to float + x_range = map(float, x_range) + # Prevents repeated values and convert back to list + self.__range = list(set(x_range[:])) + # Sort the list to ascending order + self.__range.sort() + + # A tuple, must check the lengths and generate the values + elif type(x_range) is tuple and len(x_range) in (1,2,3): + # Convert all to float + x_range = map(float, x_range) + + # Inital values + start = 0.0 + step = 1.0 + end = 0.0 + + # Only the end and it can't be less or iqual to 0 + if len(x_range) is 1 and x_range > 0: + end = x_range[0] + + # The start and the end but the start must be lesser then the end + elif len(x_range) is 2 and x_range[0] < x_range[1]: + start = x_range[0] + end = x_range[1] + + # All 3, but the start must be lesser then the end + elif x_range[0] < x_range[1]: + start = x_range[0] + end = x_range[1] + step = x_range[2] + + # Starts the range + self.__range = [] + # Generate the range + # Cnat use the range function becouse it don't suport float values + while start <= end: + self.__range.append(start) + start += step + + # Incorrect type + else: + raise Exception, "x_range must be a list with one or more item or a tuple with 2 or 3 items" + + return property(**locals()) + + @apply + def group_list(): + doc = ''' + The group_list is a read/write property used to pre-process the list + of Groups. + It can be: + - a single number, point or lambda, will be converted to a single + Group of one Data; + - a list of numbers, will be converted to a group of numbers; + - a list of tuples, will converted to a single Group of points; + - a list of lists of numbers, each 'sublist' will be converted to + a group of numbers; + - a list of lists of tuples, each 'sublist' will be converted to a + group of points; + - a list of lists of lists, the content of the 'sublist' will be + processed as coordinated lists and the result will be converted + to a group of points; + - a list of lambdas, each lambda represents a Group; + - a Dictionary where each item can be the same of the list: number, + point, list of numbers, list of points, list of lists + (coordinated lists) or lambdas + - an instance of Data; + - an instance of group. + + Usage: + >>> s = Series() + >>> s.group_list = [1,2,3,4]; print s + ["Group 1 ['1', '2', '3', '4']"] + >>> s.group_list = [[1,2,3],[4,5,6]]; print s + ["Group 1 ['1', '2', '3']", "Group 2 ['4', '5', '6']"] + >>> s.group_list = (1,2); print s + ["Group 1 ['(1, 2)']"] + >>> s.group_list = [(1,2),(2,3)]; print s + ["Group 1 ['(1, 2)', '(2, 3)']"] + >>> s.group_list = [[(1,2),(2,3)],[(4,5),(5,6)]]; print s + ["Group 1 ['(1, 2)', '(2, 3)']", "Group 2 ['(4, 5)', '(5, 6)']"] + >>> s.group_list = [[[1,2,3],[1,2,3],[1,2,3]]]; print s + ["Group 1 ['(1, 1, 1)', '(2, 2, 2)', '(3, 3, 3)']"] + >>> s.group_list = [(0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,9)]; print s + ["Group 1 ['(0.5, 5.5)']", "Group 2 ['(0, 4)', '(6, 8)']", "Group 3 ['(5.5, 7)']", "Group 4 ['(7, 9)']"] + >>> s.group_list = {'g1':[1,2,3], 'g2':[4,5,6]}; print s + ["g1 ['1', '2', '3']", "g2 ['4', '5', '6']"] + >>> s.group_list = {'g1':[(1,2),(2,3)], 'g2':[(4,5),(5,6)]}; print s + ["g1 ['(1, 2)', '(2, 3)']", "g2 ['(4, 5)', '(5, 6)']"] + >>> s.group_list = {'g1':[[1,2],[1,2]], 'g2':[[4,5],[4,5]]}; print s + ["g1 ['(1, 1)', '(2, 2)']", "g2 ['(4, 4)', '(5, 5)']"] + >>> s.range = 10 + >>> s.group_list = lambda x:x*2 + >>> s.group_list = [lambda x:x*2, lambda x:x**2, lambda x:x**3]; print s + ["Group 1 ['(0.0, 0.0)', '(1.0, 2.0)', '(2.0, 4.0)', '(3.0, 6.0)', '(4.0, 8.0)', '(5.0, 10.0)', '(6.0, 12.0)', '(7.0, 14.0)', '(8.0, 16.0)', '(9.0, 18.0)', '(10.0, 20.0)']", "Group 2 ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)', '(10.0, 100.0)']", "Group 3 ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 8.0)', '(3.0, 27.0)', '(4.0, 64.0)', '(5.0, 125.0)', '(6.0, 216.0)', '(7.0, 343.0)', '(8.0, 512.0)', '(9.0, 729.0)', '(10.0, 1000.0)']"] + >>> s.group_list = {'linear':lambda x:x*2, 'square':lambda x:x**2, 'cubic':lambda x:x**3}; print s + ["cubic ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 8.0)', '(3.0, 27.0)', '(4.0, 64.0)', '(5.0, 125.0)', '(6.0, 216.0)', '(7.0, 343.0)', '(8.0, 512.0)', '(9.0, 729.0)', '(10.0, 1000.0)']", "linear ['(0.0, 0.0)', '(1.0, 2.0)', '(2.0, 4.0)', '(3.0, 6.0)', '(4.0, 8.0)', '(5.0, 10.0)', '(6.0, 12.0)', '(7.0, 14.0)', '(8.0, 16.0)', '(9.0, 18.0)', '(10.0, 20.0)']", "square ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)', '(10.0, 100.0)']"] + >>> s.group_list = Data(1,'d1'); print s + ["Group 1 ['d1: 1']"] + >>> s.group_list = Group([(1,2),(2,3)],'g1'); print s + ["g1 ['(1, 2)', '(2, 3)']"] + ''' + def fget(self): + ''' + Return the group list. + ''' + return self.__group_list + + def fset(self, series): + ''' + Controls the input of a valid group list. + ''' + #TODO: Add support to the following strem of data: [ (0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,9)] + + # Type: None + if series is None: + self.__group_list = [] + + # List or Tuple + elif type(series) in LISTTYPES: + self.__group_list = [] + + is_function = lambda x: callable(x) + # Groups + if list in map(type, series) or max(map(is_function, series)): + for group in series: + self.add_group(group) + + # single group + else: + self.add_group(series) + + #old code + ## List of numbers + #if type(series[0]) in NUMTYPES or type(series[0]) is tuple: + # print series + # self.add_group(series) + # + ## List of anything else + #else: + # for group in series: + # self.add_group(group) + + # Dict representing series of groups + elif type(series) is dict: + self.__group_list = [] + names = series.keys() + names.sort() + for name in names: + self.add_group(Group(series[name],name,self)) + + # A single lambda + elif callable(series): + self.__group_list = [] + self.add_group(series) + + # Int/float, instance of Group or Data + elif type(series) in NUMTYPES or isinstance(series, Group) or isinstance(series, Data): + self.__group_list = [] + self.add_group(series) + + # Default + else: + raise TypeError, "Serie type not supported" + + return property(**locals()) + + def add_group(self, group, name=None): + ''' + Append a new group in group_list + ''' + if not isinstance(group, Group): + #Try to convert + group = Group(group, name, self) + + if len(group.data_list) is not 0: + # Auto naming groups + if group.name is None: + group.name = "Group "+str(len(self.__group_list)+1) + + self.__group_list.append(group) + self.__group_list[-1].parent = self + + def copy(self): + ''' + Returns a copy of the Series + ''' + new_series = Series() + new_series.__name = self.__name + if self.__range is not None: + new_series.__range = self.__range[:] + #Add color property in the copy method + #self.__colors = None + + for group in self: + new_series.add_group(group.copy()) + + return new_series + + def get_names(self): + ''' + Returns a list of the names of all groups in the Serie + ''' + names = [] + for group in self: + if group.name is None: + names.append('Group '+str(group.index()+1)) + else: + names.append(group.name) + + return names + + def to_list(self): + ''' + Returns a list with the content of all groups and data + ''' + big_list = [] + for group in self: + for data in group: + if type(data.content) in NUMTYPES: + big_list.append(data.content) + else: + big_list = big_list + list(data.content) + return big_list + + def __getitem__(self, key): + ''' + Makes the Series iterable, based in the group_list property + ''' + return self.__group_list[key] + + def __str__(self): + ''' + Returns a string that represents the Series + ''' + ret = "" + if self.name is not None: + ret += self.name + " " + if len(self) > 0: + list_str = [str(item) for item in self] + ret += str(list_str) + else: + ret += "[]" + return ret + + def __len__(self): + ''' + Returns the length of the Series, based in the group_lsit property + ''' + return len(self.group_list) + + +if __name__ == '__main__': + doctest.testmod() diff --git a/bindings/python/examples/sched_switch.py b/bindings/python/examples/sched_switch.py new file mode 100644 index 0000000..7ae834b --- /dev/null +++ b/bindings/python/examples/sched_switch.py @@ -0,0 +1,128 @@ +# sched_switch.py +# +# Babeltrace example script with sched_switch events +# +# Copyright 2012 EfficiOS Inc. +# +# Author: Danny Serres +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. + +# The script takes one optional argument (pid) +# The script will read events based on pid and +# print the scheduler switches happening with the process. +# If no arguments are passed, it displays all the scheduler switches. +# This can be used to understand which tasks schedule out the current +# process being traced, and when it gets scheduled in again. +# The trace needs PID context (lttng add-context -k -t pid) + +import sys +from babeltrace import * + +if len(sys.argv) < 2 or len(sys.argv) > 3: + raise TypeError("Usage: python sched_switch.py [pid] path/to/trace") +elif len(sys.argv) == 3: + usePID = True +else: + usePID = False + + +ctx = Context() +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +while event is not None: + while True: + if event.get_name() == "sched_switch": + # Getting scope definition + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) + if sco is None: + print("ERROR: Cannot get definition scope for sched_switch") + break # Next event + + # Getting PID + pid_field = event.get_field(sco, "_pid") + pid = pid_field.get_int64() + + if ctf.field_error(): + print("ERROR: Missing PID info for sched_switch") + break # Next event + + if usePID and (pid != long(sys.argv[1])): + break # Next event + + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + + # prev_comm + field = event.get_field(sco, "_prev_comm") + prev_comm = field.get_char_array() + if ctf.field_error(): + print("ERROR: Missing prev_comm context info") + + # prev_tid + field = event.get_field(sco, "_prev_tid") + prev_tid = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing prev_tid context info") + + # prev_prio + field = event.get_field(sco, "_prev_prio") + prev_prio = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing prev_prio context info") + + # prev_state + field = event.get_field(sco, "_prev_state") + prev_state = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing prev_state context info") + + # next_comm + field = event.get_field(sco, "_next_comm") + next_comm = field.get_char_array() + if ctf.field_error(): + print("ERROR: Missing next_comm context info") + + # next_tid + field = event.get_field(sco, "_next_tid") + next_tid = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing next_tid context info") + + # next_prio + field = event.get_field(sco, "_next_prio") + next_prio = field.get_int64() + if ctf.field_error(): + print("ERROR: Missing next_prio context info") + + # Output + print("sched_switch, pid = {}, TS = {}, prev_comm = {},\n\t" + "prev_tid = {}, prev_prio = {}, prev_state = {},\n\t" + "next_comm = {}, next_tid = {}, next_prio = {}".format( + pid, event.get_timestamp(), prev_comm, prev_tid, + prev_prio, prev_state, next_comm, next_tid, next_prio)) + + break # Next event + + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it diff --git a/bindings/python/examples/softirqtimes.py b/bindings/python/examples/softirqtimes.py new file mode 100644 index 0000000..903bf3e --- /dev/null +++ b/bindings/python/examples/softirqtimes.py @@ -0,0 +1,153 @@ +# softirqtimes.py +# +# Babeltrace time of softirqs example script +# +# Copyright 2012 EfficiOS Inc. +# +# Author: Danny Serres +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. + +# The script checks the number of events in the trace +# and outputs a table and a .svg histogram for the specified +# range (microseconds) or the total trace if no range specified. +# The graph is generated using the cairoplot module. + +# The script checks the trace for the amount of time +# spent from each softirq_raise to softirq_exit. +# It prints out the min, max (with timestamp), +# average times, the standard deviation and the total count. +# Using the cairoplot module, a .svg graph is also outputted +# showing the taken time in function of the time since the +# beginning of the trace. + +import sys, math +from output_format_modules import cairoplot +from babeltrace import * + +if len(sys.argv) < 2: + raise TypeError("Usage: python softirqtimes.py path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +time_taken = [] +graph_data = [] +max_time = (0.0, 0.0) # (val, ts) + +# tmp template: {(cpu_id, vec):TS raise} +tmp = {} +largest_val = 0 + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +start_time = event.get_timestamp() +while(event is not None): + + event_name = event.get_name() + error = True + appendNext = False + + if event_name == 'softirq_raise' or event_name == 'softirq_exit': + # Recover cpu_id and vec values to make a key to tmp + error = False + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) + field = event.get_field(scope, "cpu_id") + cpu_id = field.get_uint64() + if ctf.field_error(): + print("ERROR: Missing cpu_id info for {}".format( + event.get_name())) + error = True + + scope = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) + field = event.get_field(scope, "_vec") + vec = field.get_uint64() + if ctf.field_error(): + print("ERROR: Missing vec info for {}".format( + event.get_name())) + error = True + key = (cpu_id, vec) + + if event_name == 'softirq_raise' and not error: + # Add timestamp to tmp + if key in tmp: + # If key already exists + i = 0 + while True: + # Add index + key = (cpu_id, vec, i) + if key in tmp: + i += 1 + continue + if i > largest_val: + largest_val = i + break + + tmp[key] = event.get_timestamp() + + if event_name == 'softirq_exit' and not error: + # Saving data for output + # Key check + if not (key in tmp): + i = 0 + while i <= largest_val: + key = (key[0], key[1], i) + if key in tmp: + break + i += 1 + + raise_timestamp = tmp[key] + time_data = event.get_timestamp() - tmp.pop(key) + if time_data > max_time[0]: + # max_time = (val, ts) + max_time = (time_data, raise_timestamp) + time_taken.append(time_data) + graph_data.append((raise_timestamp - start_time, time_data)) + + # Next Event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + + +del ctf_it + +# Standard dev. calc. +try: + mean = sum(time_taken)/float(len(time_taken)) +except ZeroDivisionError: + raise TypeError("empty data") +deviations_squared = [] +for x in time_taken: + deviations_squared.append(math.pow((x - mean), 2)) +try: + stddev = math.sqrt(sum(deviations_squared) / (len(deviations_squared) - 1)) +except ZeroDivisionError: + stddev = '-' + +# Terminal output +print("AVG TIME: {} ns".format(mean)) +print("MIN TIME: {} ns".format(min(time_taken))) +print("MAX TIME: {} ns, TS: {}".format(max_time[0], max_time[1])) +print("STD DEV: {}".format(stddev)) +print("TOTAL COUNT: {}".format(len(time_taken))) + +# Graph output +cairoplot.scatter_plot ( 'softirqtimes.svg', data = graph_data, + width = 5000, height = 4000, border = 20, axis = True, + grid = True, series_colors = ["red"] ) diff --git a/bindings/python/examples/syscalls_by_pid.py b/bindings/python/examples/syscalls_by_pid.py new file mode 100644 index 0000000..3ae342e --- /dev/null +++ b/bindings/python/examples/syscalls_by_pid.py @@ -0,0 +1,84 @@ +# syscall_by_pid.py +# +# Babeltrace syscall by pid example script +# +# Copyright 2012 EfficiOS Inc. +# +# Author: Danny Serres +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. + +# The script checks the number of events in the trace +# and outputs a table and a .svg histogram for the specified +# range (microseconds) or the total trace if no range specified. +# The graph is generated using the cairoplot module. + +# The script checks all syscall in the trace and prints a list +# showing the number of systemcalls executed by each PID +# ordered from greatest to least number of syscalls. +# The trace needs PID context (lttng add-context -k -t pid) + +import sys +from babeltrace import * +from output_format_modules.pprint_table import pprint_table as pprint + +if len(sys.argv) < 2 : + raise TypeError("Usage: python syscalls_by_pid.py path/to/trace") + +ctx = Context() +ret = ctx.add_trace(sys.argv[1], "ctf") +if ret is None: + raise IOError("Error adding trace") + +data = {} + +# Setting iterator +bp = IterPos(SEEK_BEGIN) +ctf_it = ctf.Iterator(ctx, bp) + +# Reading events +event = ctf_it.read_event() +while event is not None: + if event.get_name().find("sys") >= 0: + # Getting scope definition + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) + if sco is None: + print("ERROR: Cannot get definition scope for {}".format( + event.get_name())) + else: + # Getting PID + pid_field = event.get_field(sco, "_pid") + pid = pid_field.get_int64() + + if ctf.field_error(): + print("ERROR: Missing PID info for sched_switch".format( + event.get_name())) + elif pid in data: + data[pid] += 1 + else: + data[pid] = 1 + # Next event + ret = ctf_it.next() + if ret < 0: + break + event = ctf_it.read_event() + +del ctf_it + +# Setting table for output +table = [] +for item in data: + table.append([data[item], item]) # [count, pid] +table.sort(reverse = True) # [big count first, pid] +for i in range(len(table)): + table[i].reverse() # [pid, big count first] +table.insert(0, ["PID", "SYSCALL COUNT"]) +pprint(table) diff --git a/bindings/python/python-complements.c b/bindings/python/python-complements.c new file mode 100644 index 0000000..a4ee37e --- /dev/null +++ b/bindings/python/python-complements.c @@ -0,0 +1,121 @@ +/* + * python-complements.c + * + * Babeltrace Python module complements, required for Python bindings + * + * Copyright 2012 EfficiOS Inc. + * + * Author: Danny Serres + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + */ + +#include "python-complements.h" + +/* FILE functions + ---------------------------------------------------- +*/ + +FILE *_bt_file_open(char *file_path, char *mode) +{ + FILE *fp = stdout; + if (file_path != NULL) + fp = fopen(file_path, mode); + return fp; +} + +void _bt_file_close(FILE *fp) +{ + if (fp != NULL) + fclose(fp); +} + + +/* List-related functions + ---------------------------------------------------- +*/ + +/* ctf-field-list */ +struct definition **_bt_python_field_listcaller( + const struct bt_ctf_event *ctf_event, + const struct definition *scope) +{ + struct definition **list; + unsigned int count; + int ret; + + ret = bt_ctf_get_field_list(ctf_event, scope, + (const struct definition * const **)&list, &count); + + if (ret < 0) /* For python to know an error occured */ + list = NULL; + else /* For python to know the end is reached */ + list[count] = NULL; + + return list; +} + +struct definition *_bt_python_field_one_from_list( + struct definition **list, int index) +{ + return list[index]; +} + +/* event_decl_list */ +struct bt_ctf_event_decl **_bt_python_event_decl_listcaller( + int handle_id, struct bt_context *ctx) +{ + struct bt_ctf_event_decl **list; + unsigned int count; + int ret; + + ret = bt_ctf_get_event_decl_list(handle_id, ctx, + (struct bt_ctf_event_decl * const **)&list, &count); + + if (ret < 0) /* For python to know an error occured */ + list = NULL; + else /* For python to know the end is reached */ + list[count] = NULL; + + return list; +} + +struct bt_ctf_event_decl *_bt_python_decl_one_from_list( + struct bt_ctf_event_decl **list, int index) +{ + return list[index]; +} + +/* decl_fields */ +struct bt_ctf_field_decl **_by_python_field_decl_listcaller( + struct bt_ctf_event_decl *event_decl, + enum bt_ctf_scope scope) +{ + struct bt_ctf_field_decl **list; + unsigned int count; + int ret; + + ret = bt_ctf_get_decl_fields(event_decl, scope, + (const struct bt_ctf_field_decl * const **)&list, &count); + + if (ret < 0) /* For python to know an error occured */ + list = NULL; + else /* For python to know the end is reached */ + list[count] = NULL; + + return list; +} + +struct bt_ctf_field_decl *_bt_python_field_decl_one_from_list( + struct bt_ctf_field_decl **list, int index) +{ + return list[index]; +} diff --git a/bindings/python/python-complements.h b/bindings/python/python-complements.h new file mode 100644 index 0000000..9597d70 --- /dev/null +++ b/bindings/python/python-complements.h @@ -0,0 +1,52 @@ +/* + * python-complements.h + * + * Babeltrace Python module complements header, required for Python bindings + * + * Copyright 2012 EfficiOS Inc. + * + * Author: Danny Serres + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +/* File */ +FILE *_bt_file_open(char *file_path, char *mode); +void _bt_file_close(FILE *fp); + +/* ctf-field-list */ +struct definition **_bt_python_field_listcaller( + const struct bt_ctf_event *ctf_event, + const struct definition *scope); +struct definition *_bt_python_field_one_from_list( + struct definition **list, int index); + +/* event_decl_list */ +struct bt_ctf_event_decl **_bt_python_event_decl_listcaller( + int handle_id, struct bt_context *ctx); +struct bt_ctf_event_decl *_bt_python_decl_one_from_list( + struct bt_ctf_event_decl **list, int index); + +/* decl_fields */ +struct bt_ctf_field_decl **_by_python_field_decl_listcaller( + struct bt_ctf_event_decl *event_decl, + enum bt_ctf_scope scope); +struct bt_ctf_field_decl *_bt_python_field_decl_one_from_list( + struct bt_ctf_field_decl **list, int index); diff --git a/bootstrap b/bootstrap index c507425..f6926ca 100755 --- a/bootstrap +++ b/bootstrap @@ -4,7 +4,7 @@ set -x if [ ! -e config ]; then mkdir config fi -aclocal +aclocal -I m4 libtoolize --force --copy autoheader automake --add-missing --copy diff --git a/configure.ac b/configure.ac index d90479d..f9cff9d 100644 --- a/configure.ac +++ b/configure.ac @@ -74,6 +74,41 @@ AC_CHECK_LIB([popt], [poptGetContext], [], [AC_MSG_ERROR([Cannot find popt.])] ) + +# For Python +# SWIG version needed or newer: +swig_version=2.0.0 + +AC_ARG_ENABLE([python], + [AC_HELP_STRING([--disable-python], + [do not compile Python bindings])], + [], [enable_python=yes]) + +AM_CONDITIONAL([USE_PYTHON], [test "x${enable_python:-yes}" = xyes]) + +if test "x${enable_python:-yes}" = xyes; then + AC_MSG_NOTICE([You may configure with --disable-python ]dnl +[if you do not want Python bindings.]) + + AX_PKG_SWIG($swig_version, [], [ AC_MSG_ERROR([SWIG $swig_version or newer is needed]) ]) + AM_PATH_PYTHON + + AC_ARG_VAR([PYTHON_INCLUDE], [Include flags for python, bypassing python-config]) + AC_ARG_VAR([PYTHON_CONFIG], [Path to python-config]) + AS_IF([test -z "$PYTHON_INCLUDE"], [ + AS_IF([test -z "$PYTHON_CONFIG"], [ + AC_PATH_PROGS([PYTHON_CONFIG], + [python$PYTHON_VERSION-config python-config], + [no], + [`dirname $PYTHON`]) + AS_IF([test "$PYTHON_CONFIG" = no], [AC_MSG_ERROR([cannot find python-config for $PYTHON.])]) + ]) + AC_MSG_CHECKING([python include flags]) + PYTHON_INCLUDE=`$PYTHON_CONFIG --includes` + AC_MSG_RESULT([$PYTHON_INCLUDE]) + ]) +fi + pkg_modules="gmodule-2.0 >= 2.0.0" PKG_CHECK_MODULES(GMODULE, [$pkg_modules]) AC_SUBST(PACKAGE_LIBS) @@ -103,6 +138,8 @@ AC_CONFIG_FILES([ lib/Makefile lib/prio_heap/Makefile include/Makefile + bindings/Makefile + bindings/python/Makefile tests/Makefile ]) AC_OUTPUT diff --git a/doc/python-howto.txt b/doc/python-howto.txt new file mode 100644 index 0000000..e2ed751 --- /dev/null +++ b/doc/python-howto.txt @@ -0,0 +1,70 @@ +PYTHON BINDINGS +---------------- + +This is a brief howto for using the Babeltrace Python module. + + +INSTALLATION: + +By default, the Python bindings are installed. +If you do not wish the Python bindings, you can configure with the +--disable-python option during the installation procedure: + + $ ./configure --disable-python + +The Python module is automatically generated using SWIG, therefore the +swig2.0 package on Debian/Ubuntu is requied. + + +USAGE: + +Once installed, the Python module can be used by importing it in Python. +In the Python interpreter: + + >>> import babeltrace + +Then the starting point is to create a context and add a trace to it. + + >>> ctx = babeltrace.Context() + >>> ctx.add_trace("path/to/trace", ) + +Where is a string containing the format name in which the trace +was produced. To print a list of available formats to the standard +output, it is possible to use the print_format_list function. + + >>> out = babeltrace.File(None) # This returns stdout + >>> babeltrace.print_format_list(out) + +When a trace is added to a context, it is opened and ready to read using +an iterator. While creating an iterator, optional starting and ending +position may be specified. So far, only ctf iterator are supported. + + >>> begin_pos = babeltrace.IterPos(babeltrace.SEEK_BEGIN) + >>> iterator = babeltrace.ctf.Iterator(ctx, begin_pos) + +From there, it is possible to read the events. + + >>> event = iterator.read_event() + +It is simple to obtain the timestamp of that event. + + >>> timestamp = event.get_timestamp() + +Let's say that we want to extract the prev_comm context info for a +sched_switch event. To do so, it is needed to set an event scope +with which we can obtain the field wanted. + + >>> if event.get_name == "sched_switch": + ... #prev_comm only for sched_switch events + ... scope = event.get_top_level_scope(babeltrace.ctf.scope.EVENT_FIELDS) + ... field = event.get_field(scope, "_prev_comm") + ... prev_comm = field.get_char_array() + +It is also possible to move on to the next event. + + >>> ret = iterator.next() # Move the iterator + >>> if ret == 0: # No error occured + ... event = iterator.read_event() # Read the next event + +For many usage script examples of the Babeltrace Python module, see the +bindings/python/examples directory. diff --git a/m4/ax_pkg_swig.m4 b/m4/ax_pkg_swig.m4 new file mode 100644 index 0000000..e112f3d --- /dev/null +++ b/m4/ax_pkg_swig.m4 @@ -0,0 +1,135 @@ +# =========================================================================== +# http://www.gnu.org/software/autoconf-archive/ax_pkg_swig.html +# =========================================================================== +# +# SYNOPSIS +# +# AX_PKG_SWIG([major.minor.micro], [action-if-found], [action-if-not-found]) +# +# DESCRIPTION +# +# This macro searches for a SWIG installation on your system. If found, +# then SWIG is AC_SUBST'd; if not found, then $SWIG is empty. If SWIG is +# found, then SWIG_LIB is set to the SWIG library path, and AC_SUBST'd. +# +# You can use the optional first argument to check if the version of the +# available SWIG is greater than or equal to the value of the argument. It +# should have the format: N[.N[.N]] (N is a number between 0 and 999. Only +# the first N is mandatory.) If the version argument is given (e.g. +# 1.3.17), AX_PKG_SWIG checks that the swig package is this version number +# or higher. +# +# As usual, action-if-found is executed if SWIG is found, otherwise +# action-if-not-found is executed. +# +# In configure.in, use as: +# +# AX_PKG_SWIG(1.3.17, [], [ AC_MSG_ERROR([SWIG is required to build..]) ]) +# AX_SWIG_ENABLE_CXX +# AX_SWIG_MULTI_MODULE_SUPPORT +# AX_SWIG_PYTHON +# +# LICENSE +# +# Copyright (c) 2008 Sebastian Huber +# Copyright (c) 2008 Alan W. Irwin +# Copyright (c) 2008 Rafael Laboissiere +# Copyright (c) 2008 Andrew Collier +# Copyright (c) 2011 Murray Cumming +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation; either version 2 of the License, or (at your +# option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program. If not, see . +# +# As a special exception, the respective Autoconf Macro's copyright owner +# gives unlimited permission to copy, distribute and modify the configure +# scripts that are the output of Autoconf when processing the Macro. You +# need not follow the terms of the GNU General Public License when using +# or distributing such scripts, even though portions of the text of the +# Macro appear in them. The GNU General Public License (GPL) does govern +# all other use of the material that constitutes the Autoconf Macro. +# +# This special exception to the GPL applies to versions of the Autoconf +# Macro released by the Autoconf Archive. When you make and distribute a +# modified version of the Autoconf Macro, you may extend this special +# exception to the GPL to apply to your modified version as well. + +#serial 8 + +AC_DEFUN([AX_PKG_SWIG],[ + # Ubuntu has swig 2.0 as /usr/bin/swig2.0 + AC_PATH_PROGS([SWIG],[swig swig2.0]) + if test -z "$SWIG" ; then + m4_ifval([$3],[$3],[:]) + elif test -n "$1" ; then + AC_MSG_CHECKING([SWIG version]) + [swig_version=`$SWIG -version 2>&1 | grep 'SWIG Version' | sed 's/.*\([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\).*/\1/g'`] + AC_MSG_RESULT([$swig_version]) + if test -n "$swig_version" ; then + # Calculate the required version number components + [required=$1] + [required_major=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_major" ; then + [required_major=0] + fi + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] + [required_minor=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_minor" ; then + [required_minor=0] + fi + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] + [required_patch=`echo $required | sed 's/[^0-9].*//'`] + if test -z "$required_patch" ; then + [required_patch=0] + fi + # Calculate the available version number components + [available=$swig_version] + [available_major=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_major" ; then + [available_major=0] + fi + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] + [available_minor=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_minor" ; then + [available_minor=0] + fi + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] + [available_patch=`echo $available | sed 's/[^0-9].*//'`] + if test -z "$available_patch" ; then + [available_patch=0] + fi + # Convert the version tuple into a single number for easier comparison. + # Using base 100 should be safe since SWIG internally uses BCD values + # to encode its version number. + required_swig_vernum=`expr $required_major \* 10000 \ + \+ $required_minor \* 100 \+ $required_patch` + available_swig_vernum=`expr $available_major \* 10000 \ + \+ $available_minor \* 100 \+ $available_patch` + + if test $available_swig_vernum -lt $required_swig_vernum; then + AC_MSG_WARN([SWIG version >= $1 is required. You have $swig_version.]) + SWIG='' + m4_ifval([$3],[$3],[]) + else + AC_MSG_CHECKING([for SWIG library]) + SWIG_LIB=`$SWIG -swiglib` + AC_MSG_RESULT([$SWIG_LIB]) + m4_ifval([$2],[$2],[]) + fi + else + AC_MSG_WARN([cannot determine SWIG version]) + SWIG='' + m4_ifval([$3],[$3],[]) + fi + fi + AC_SUBST([SWIG_LIB]) +]) diff --git a/tests/tests-python.py b/tests/tests-python.py new file mode 100644 index 0000000..0bd71c2 --- /dev/null +++ b/tests/tests-python.py @@ -0,0 +1,115 @@ +import unittest +import sys +from babeltrace import * + +class TestBabeltracePythonModule(unittest.TestCase): + + def test_handle_decl(self): + #Context creation, adding trace + ctx = Context() + trace_handle = ctx.add_trace( + "ctf-traces/succeed/lttng-modules-2.0-pre5", + "ctf") + self.assertIsNotNone(trace_handle, "Error adding trace") + + #TraceHandle test + ts_begin = trace_handle.get_timestamp_begin(ctx, CLOCK_REAL) + ts_end = trace_handle.get_timestamp_end(ctx, CLOCK_REAL) + self.assertGreater(ts_end, ts_begin, "Error get_timestamp from trace_handle") + + lst = ctf.get_event_decl_list(trace_handle, ctx) + self.assertIsNotNone(lst, "Error get_event_decl_list") + + name = lst[0].get_name() + self.assertIsNotNone(name) + + fields = lst[0].get_decl_fields(ctf.scope.EVENT_FIELDS) + self.assertIsNotNone(fields, "Error getting FieldDecl list") + + if len(fields) > 0: + self.assertIsNotNone(fields[0].get_name(), + "Error getting name from FieldDecl") + + #Remove trace + ctx.remove_trace(trace_handle) + del ctx + del trace_handle + + + def test_iterator_event(self): + #Context creation, adding trace + ctx = Context() + trace_handle = ctx.add_trace( + "ctf-traces/succeed/lttng-modules-2.0-pre5", + "ctf") + self.assertIsNotNone(trace_handle, "Error adding trace") + + begin_pos = IterPos(SEEK_BEGIN) + it = ctf.Iterator(ctx, begin_pos) + self.assertIsNotNone(it, "Error creating iterator") + + event = it.read_event() + self.assertIsNotNone(event, "Error reading event") + + handle = event.get_handle() + self.assertIsNotNone(handle, "Error getting handle") + + context = event.get_context() + self.assertIsNotNone(context, "Error getting context") + + name = "" + while event is not None and name != "sched_switch": + name = event.get_name() + self.assertIsNotNone(name, "Error getting event name") + + ts = event.get_timestamp() + self.assertGreaterEqual(ts, 0, "Error getting timestamp") + + cycles = event.get_cycles() + self.assertGreaterEqual(cycles, 0, "Error getting cycles") + + if name == "sched_switch": + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) + self.assertIsNotNone(scope, "Error getting scope definition") + + field = event.get_field(scope, "cpu_id") + prev_comm_str = field.get_uint64() + self.assertEqual(ctf.field_error(), 0, "Error getting vec info") + + field_lst = event.get_field_list(scope) + self.assertIsNotNone(field_lst, "Error getting field list") + + fname = field.field_name() + self.assertIsNotNone(fname, "Error getting field name") + + ftype = field.field_type() + self.assertIsNot(ftype, -1, "Error getting field type (unknown)") + + ret = it.next() + self.assertGreaterEqual(ret, 0, "Error moving iterator") + + event = it.read_event() + + pos = it.get_pos() + self.assertIsNotNone(pos._pos, "Error getting iterator position") + + ret = it.set_pos(pos) + self.assertEqual(ret, 0, "Error setting iterator position") + + pos = it.create_time_pos(ts) + self.assertIsNotNone(pos._pos, "Error creating time-based iterator position") + + del it, pos + set_pos = IterPos(SEEK_TIME, ts) + it = ctf.Iterator(ctx, begin_pos, set_pos) + self.assertIsNotNone(it, "Error creating iterator with end_pos") + + + def test_file_class(self): + f = File("test-bitfield.c") + self.assertIsNotNone(f, "Error opening file") + f.close() + + +if __name__ == "__main__": + unittest.main() -- 1.7.9.5 From mathieu.desnoyers at efficios.com Fri Aug 10 18:01:08 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 10 Aug 2012 18:01:08 -0400 Subject: [lttng-dev] [Babeltrace PATCH] Add BT_SEEK_LAST type to bt_iter_pos. In-Reply-To: <1344613794-17669-1-git-send-email-francis.deslauriers@polymtl.ca> References: <1344613794-17669-1-git-send-email-francis.deslauriers@polymtl.ca> Message-ID: <20120810220108.GC601@Krystal> * Francis Deslauriers (francis.deslauriers at polymtl.ca) wrote: > Signed-off-by: Francis Deslauriers > --- > include/babeltrace/iterator.h | 1 + > lib/iterator.c | 189 +++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 190 insertions(+) > > diff --git a/include/babeltrace/iterator.h b/include/babeltrace/iterator.h > index aa6470e..c13055d 100644 > --- a/include/babeltrace/iterator.h > +++ b/include/babeltrace/iterator.h > @@ -48,6 +48,7 @@ struct bt_iter_pos { > BT_SEEK_CUR, > BT_SEEK_BEGIN, > BT_SEEK_END, > + BT_SEEK_LAST, > } type; > union { > uint64_t seek_time; > diff --git a/lib/iterator.c b/lib/iterator.c > index bcb77d8..875c93d 100644 > --- a/lib/iterator.c > +++ b/lib/iterator.c > @@ -187,6 +187,176 @@ static int seek_ctf_trace_by_timestamp(struct ctf_trace *tin, > return found ? 0 : EOF; > } > Please document the return values. static > +int seek_last_element_ctf_file_stream(struct trace_collection *tc, > + struct ctf_file_stream **cfs, int max_packet, > + int max_stream_id, int max_stream_class_id, > + int max_trace_descriptor_id, uint64_t max_timestamp) > +{ > + int ret; > + struct trace_descriptor *max_td_read; > + struct ctf_trace *max_tin; > + struct ctf_stream_declaration *max_stream_class; > + struct ctf_stream_definition *max_stream; > + struct ctf_stream_pos *max_stream_pos; > + > + max_td_read = g_ptr_array_index(tc->array, > + max_trace_descriptor_id); > + max_tin = container_of(max_td_read, > + struct ctf_trace, parent); > + max_stream_class = g_ptr_array_index(max_tin->streams, > + max_stream_class_id); > + max_stream = g_ptr_array_index(max_stream_class->streams, > + max_stream_id); > + *cfs = container_of(max_stream, > + struct ctf_file_stream, parent); > + max_stream_pos = &(*cfs)->pos; > + /* we seek to the last packet of the stream.*/ > + max_stream_pos->packet_seek(&max_stream_pos->parent, > + max_packet, > + SEEK_SET); > + /* > + * iterate over every event until we reach on an event that > + * its timestamp correspond with the max saved previously. > + */ > + do { > + ret = stream_read_event(*cfs); > + } while ((*cfs)->parent.real_timestamp != max_timestamp && ret == 0); > + > + /* We insert the stream into the heap. */ > + return ret; > +} > + Please document the return values. static > +int find_last_event(struct ctf_file_stream *cfs, > + uint64_t *timestamp_end, > + int *packet) > +{ > + int ret = 0, count = 0, event_read = 0; move count into the for() loop. > + int i; move i with the other integers. > + uint64_t tmp = 0; > + struct ctf_stream_pos *stream_pos; > + stream_pos = &cfs->pos; > + > + /* > + * we start by the last packet as the current one. > + * If the current one is empty we go back one packet if possible. > + */ > + i = stream_pos->packet_real_index->len - 1; move to first argument of the for(). > + /* > + * Check if we have iterated on all the packets. > + * If we are not short on packets, we check what > + * made us leave the reading event loop. > + */ > + for (; i >= 0 && count == 0; i--) { two spaces here (one is enough) int count = 0; > + stream_pos->packet_seek(&stream_pos->parent, i, SEEK_SET); > + count = 0; remove. > + /* read each event until we reach the end of the packet */ > + do { > + tmp = cfs->parent.real_timestamp; > + ret = stream_read_event(cfs); > + if (ret == 0) { > + count++; > + } > + } while (ret == 0); > + increment event_read here. > + /* Error */ > + if (ret > 0) { > + goto end; > + } if (!count) break; > + event_read += count; > + } > + *packet = i; > + *timestamp_end = tmp; > + > + /* Check if we read at least one event on the stream */ > + if (event_read <= 1) { Not sure I understand the "at least" and <= 1. Did you mean >= ? > + ret = 1; > + goto end; > + } > + ret = 0; > + > +end: > + return ret; > +} > + static > +int find_max_timestamp_ctf_file_stream( > + struct ctf_stream_declaration *stream_class, int *max_stream_id, > + int *packet, uint64_t *max_timestamp, int *new_max) > +{ > + struct ctf_file_stream *cfs; > + int max_packet = 0, ret = 0, error = 1; > + int i; move i with other integers. > + uint64_t savedtime; > + > + for (i = 0; i < stream_class->streams->len; i++) { > + struct ctf_stream_definition *stream; > + stream = g_ptr_array_index( > + stream_class->streams, i); > + if (!stream) > + continue; > + cfs = container_of(stream, struct ctf_file_stream, parent); > + ret = find_last_event(cfs, &savedtime, &max_packet); > + /* Can return either EOF, 0, or error (> 0). */ > + if (ret == 0 && savedtime >= *max_timestamp) { > + *new_max = 1; > + *max_stream_id = i; > + *packet = max_packet; > + *max_timestamp = savedtime; > + } > + error = error & ret; what are you trying to do with this binary operator ? > + } > + return error; > +} > + static > +int seek_last_ctf_file_stream(struct trace_collection *tc, > + struct ctf_file_stream **cfs) > +{ Thanks, Mathieu > + int i, j, ret, new_max_found, max_packet, max_stream_id, > + max_stream_class_id, max_trace_descriptor_id; > + int found = 0; > + uint64_t max_timestamp = 0; > + > + /* For each trace in the trace_collection*/ > + for (i = 0; i < tc->array->len; i++) { > + struct ctf_trace *tin; > + struct trace_descriptor *td_read; > + td_read = g_ptr_array_index(tc->array, i); > + if (!td_read) > + continue; > + tin = container_of(td_read, struct ctf_trace, parent); > + /* For each stream_class in the trace*/ > + for (j = 0; j < tin->streams->len; j++) { > + struct ctf_stream_declaration *stream_class; > + > + stream_class = g_ptr_array_index(tin->streams, j); > + if (!stream_class) > + continue; > + /* For each file_stream in the stream_class */ > + new_max_found = 0; > + ret = find_max_timestamp_ctf_file_stream(stream_class, > + &max_stream_id, &max_packet, > + &max_timestamp, &new_max_found); > + if (!ret && new_max_found) { > + found = 1; > + max_trace_descriptor_id = i; > + max_stream_class_id = j; > + } > + } > + } > + /* > + * Now we know in which trace, stream_class and stream is the > + * last event of the trace_collection. > + * We can seek to this targeted event. > + */ > + if (!found) { > + ret = -1; > + } else { > + ret = seek_last_element_ctf_file_stream(tc, cfs, max_packet, > + max_stream_id, max_stream_class_id, > + max_trace_descriptor_id, max_timestamp); > + } > + return ret; > +} > + > int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) > { > struct trace_collection *tc; > @@ -326,6 +496,25 @@ int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) > } > } > break; > + case BT_SEEK_LAST: > + { > + struct ctf_file_stream *cfs; > + tc = iter->ctx->tc; > + > + ret = seek_last_ctf_file_stream(tc, &cfs); > + if (ret < 0) > + goto error; > + /* remove all stream from the heap*/ > + heap_free(iter->stream_heap); > + /* Create a new empty heap*/ > + ret = heap_init(iter->stream_heap, 0, stream_compare); > + if (ret < 0) > + goto error; > + /* Insert the stream that contains the last event. */ > + heap_insert(iter->stream_heap, cfs); > + > + return 0; > + } > default: > /* not implemented */ > return -EINVAL; > -- > 1.7.9.5 > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Fri Aug 10 18:17:07 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 10 Aug 2012 18:17:07 -0400 Subject: [lttng-dev] [babeltrace PATCH] Babeltrace python module v4 In-Reply-To: <1344635171-7654-1-git-send-email-danny.serres@efficios.com> References: <1344635171-7654-1-git-send-email-danny.serres@efficios.com> Message-ID: <20120810221707.GD601@Krystal> * Danny Serres (danny.serres at efficios.com) wrote: > The Babeltrace Python module can be used to directly control > the Babeltrace API inside Python, using 'import babeltrace'. > > Therefore, it becomes possible to create a Context, add a > trace to it, iterate on it, read events and so on from > within Python. > > SWIG >= 2.0 is used to create the wrapper and its > 'warning md variable unused' bug is patched in Makefile.am > > In the interface file, struct and enum are directly copied > from the include files. All changes to struct bt_iter_pos > and to enums in ctf/events.h and clock-types.h must also > be made in the interface file. > > Signed-off-by: Danny Serres > Signed-off-by: Yannick Brosseau Pulled into a bindings/python branch (for babeltrace 1.1) http://git.efficios.com/?p=babeltrace.git;a=shortlog;h=refs/heads/bindings/python Thanks, Mathieu > --- > .gitignore | 10 +- > Makefile.am | 2 +- > README | 6 + > bindings/Makefile.am | 3 + > bindings/python/Makefile.am | 28 + > bindings/python/babeltrace.i.in | 1098 +++++++++ > bindings/python/examples/babeltrace_and_lttng.py | 126 ++ > bindings/python/examples/eventcount.py | 84 + > bindings/python/examples/eventcountlist.py | 83 + > bindings/python/examples/events_per_cpu.py | 99 + > bindings/python/examples/example-api-test.py | 77 + > bindings/python/examples/histogram.py | 139 ++ > .../examples/output_format_modules/cairoplot.py | 2336 ++++++++++++++++++++ > .../examples/output_format_modules/pprint_table.py | 37 + > .../examples/output_format_modules/series.py | 1140 ++++++++++ > bindings/python/examples/sched_switch.py | 128 ++ > bindings/python/examples/softirqtimes.py | 153 ++ > bindings/python/examples/syscalls_by_pid.py | 84 + > bindings/python/python-complements.c | 121 + > bindings/python/python-complements.h | 52 + > bootstrap | 2 +- > configure.ac | 37 + > doc/python-howto.txt | 70 + > m4/ax_pkg_swig.m4 | 135 ++ > tests/tests-python.py | 115 + > 25 files changed, 6162 insertions(+), 3 deletions(-) > create mode 100644 bindings/Makefile.am > create mode 100644 bindings/python/Makefile.am > create mode 100644 bindings/python/babeltrace.i.in > create mode 100644 bindings/python/examples/babeltrace_and_lttng.py > create mode 100644 bindings/python/examples/eventcount.py > create mode 100644 bindings/python/examples/eventcountlist.py > create mode 100644 bindings/python/examples/events_per_cpu.py > create mode 100644 bindings/python/examples/example-api-test.py > create mode 100644 bindings/python/examples/histogram.py > create mode 100644 bindings/python/examples/output_format_modules/__init__.py > create mode 100755 bindings/python/examples/output_format_modules/cairoplot.py > create mode 100644 bindings/python/examples/output_format_modules/pprint_table.py > create mode 100755 bindings/python/examples/output_format_modules/series.py > create mode 100644 bindings/python/examples/sched_switch.py > create mode 100644 bindings/python/examples/softirqtimes.py > create mode 100644 bindings/python/examples/syscalls_by_pid.py > create mode 100644 bindings/python/python-complements.c > create mode 100644 bindings/python/python-complements.h > create mode 100644 doc/python-howto.txt > create mode 100644 m4/ax_pkg_swig.m4 > create mode 100644 tests/tests-python.py > > diff --git a/.gitignore b/.gitignore > index d6098ac..d7bce4d 100644 > --- a/.gitignore > +++ b/.gitignore > @@ -1,4 +1,5 @@ > /tests/test-bitfield > +*~ > *.o > *.a > *.la > @@ -15,7 +16,11 @@ ctf-parser-test > /config.h.in > /config.status > *.log > -*.m4 > +/m4/libtool.m4 > +/m4/lt~obsolete.m4 > +/m4/ltoptions.m4 > +/m4/ltsugar.m4 > +/m4/ltversion.m4 > libtool > /configure > Makefile > @@ -26,3 +31,6 @@ converter/babeltrace-log > core > formats/ctf/metadata/ctf-parser.output > stamp-h1 > +bindings/python/babeltrace.i > +bindings/python/babeltrace.py > +bindings/python/babeltrace_wrap.c > diff --git a/Makefile.am b/Makefile.am > index 308ee16..6584c5d 100644 > --- a/Makefile.am > +++ b/Makefile.am > @@ -2,7 +2,7 @@ AM_CFLAGS = $(PACKAGE_CFLAGS) -I$(top_srcdir)/include > > ACLOCAL_AMFLAGS = -I m4 > > -SUBDIRS = include types lib formats converter tests doc > +SUBDIRS = include types lib formats converter bindings tests doc > > dist_doc_DATA = ChangeLog LICENSE mit-license.txt gpl-2.0.txt \ > std-ext-lib.txt > diff --git a/README b/README > index 75bf0cf..1687075 100644 > --- a/README > +++ b/README > @@ -25,6 +25,7 @@ BUILDING > make install > ldconfig > > + If you do not want Python bindings, run ./configure --disable-python. > > DEPENDENCIES > ------------ > @@ -44,6 +45,11 @@ To compile Babeltrace, you will need: > libpopt >= 1.13 development libraries > (Debian : libpopt-dev) > (Fedora : popt) > + python headers (optional) > + (Debian/Ubuntu : python-dev) > + swig >= 2.0 (optional) > + (Debian/Ubuntu : swig2.0) > + > > For developers using the git tree: > > diff --git a/bindings/Makefile.am b/bindings/Makefile.am > new file mode 100644 > index 0000000..dcd868d > --- /dev/null > +++ b/bindings/Makefile.am > @@ -0,0 +1,3 @@ > +if USE_PYTHON > +SUBDIRS = python > +endif > diff --git a/bindings/python/Makefile.am b/bindings/python/Makefile.am > new file mode 100644 > index 0000000..579759f > --- /dev/null > +++ b/bindings/python/Makefile.am > @@ -0,0 +1,28 @@ > +babeltrace.i: babeltrace.i.in > + sed "s/BABELTRACE_VERSION_STR/Babeltrace $(PACKAGE_VERSION)/g" babeltrace.i > + > +AM_CFLAGS = -I$(PYTHON_INCLUDE) -I$(top_srcdir)/include/ > + > +EXTRA_DIST = babeltrace.i > +python_PYTHON = babeltrace.py > +pyexec_LTLIBRARIES = _babeltrace.la > + > +MAINTAINERCLEANFILES = babeltrace_wrap.c babeltrace.py > + > +_babeltrace_la_SOURCES = babeltrace_wrap.c python-complements.c > + > +_babeltrace_la_LDFLAGS = -module > + > +_babeltrace_la_CFLAGS = $(GLIB_CFLAGS) $(AM_CFLAGS) > + > +_babeltrace_la_LIBS = $(GLIB_LIBS) > + > +_babeltrace_la_LIBADD = $(top_srcdir)/formats/ctf/libbabeltrace-ctf.la \ > + $(top_srcdir)/formats/ctf-text/libbabeltrace-ctf-text.la > + > +# SWIG 'warning md variable unused' fixed after SWIG build: > +babeltrace_wrap.c: babeltrace.i > + $(SWIG) -python -Wall -I. -I$(top_srcdir)/include babeltrace.i > + sed -i "s/PyObject \*m, \*d, \*md;/PyObject \*m, \*d;\n#if defined(SWIGPYTHON_BUILTIN)\nPyObject *md;\n#endif/g" babeltrace_wrap.c > + sed -i "s/md = d/d/g" babeltrace_wrap.c > + sed -i "s/(void)public_symbol;/(void)public_symbol;\n md = d;/g" babeltrace_wrap.c > diff --git a/bindings/python/babeltrace.i.in b/bindings/python/babeltrace.i.in > new file mode 100644 > index 0000000..c8e4923 > --- /dev/null > +++ b/bindings/python/babeltrace.i.in > @@ -0,0 +1,1098 @@ > +/* > + * babeltrace.i.in > + * > + * Babeltrace Python Module interface file > + * > + * Copyright 2012 EfficiOS Inc. > + * > + * Author: Danny Serres > + * > + * Permission is hereby granted, free of charge, to any person obtaining a copy > + * of this software and associated documentation files (the "Software"), to deal > + * in the Software without restriction, including without limitation the rights > + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell > + * copies of the Software, and to permit persons to whom the Software is > + * furnished to do so, subject to the following conditions: > + * > + * The above copyright notice and this permission notice shall be included in > + * all copies or substantial portions of the Software. > + */ > + > + > +%define DOCSTRING > +"BABELTRACE_VERSION_STR > + > +Babeltrace is a trace viewer and converter reading and writing the > +Common Trace Format (CTF). Its main use is to pretty-print CTF > +traces into a human-readable text output. > + > +To use this module, the first step is to create a Context and add a > +trace to it." > +%enddef > + > +%module(docstring=DOCSTRING) babeltrace > + > +%include "typemaps.i" > +%{ > +#define SWIG_FILE_WITH_INIT > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include "python-complements.h" > +%} > + > +typedef unsigned long long uint64_t; > +typedef long long int64_t; > +typedef int bt_intern_str; > + > +/* ================================================================= > + CONTEXT.H, CONTEXT-INTERNAL.H > + ????????????????????????????? > +*/ > + > +%rename("_bt_context_create") bt_context_create(void); > +%rename("_bt_context_add_trace") bt_context_add_trace( > + struct bt_context *ctx, const char *path, const char *format, > + void (*packet_seek)(struct stream_pos *pos, size_t index, int whence), > + struct mmap_stream_list *stream_list, FILE *metadata); > +%rename("_bt_context_remove_trace") bt_context_remove_trace( > + struct bt_context *ctx, int trace_id); > +%rename("_bt_context_get") bt_context_get(struct bt_context *ctx); > +%rename("_bt_context_put") bt_context_put(struct bt_context *ctx); > +%rename("_bt_ctf_event_get_context") bt_ctf_event_get_context( > + const struct bt_ctf_event *event); > + > +struct bt_context *bt_context_create(void); > +int bt_context_add_trace(struct bt_context *ctx, const char *path, const char *format, > + void (*packet_seek)(struct stream_pos *pos, size_t index, int whence), > + struct mmap_stream_list *stream_list, FILE *metadata); > +void bt_context_remove_trace(struct bt_context *ctx, int trace_id); > +void bt_context_get(struct bt_context *ctx); > +void bt_context_put(struct bt_context *ctx); > +struct bt_context *bt_ctf_event_get_context(const struct bt_ctf_event *event); > + > +// class Context to prevent direct access to struct bt_context > +%pythoncode%{ > +class Context: > + """ > + The context represents the object in which a trace_collection is > + open. As long as this structure is allocated, the trace_collection > + is open and the traces it contains can be read and seeked by the > + iterators and callbacks. > + """ > + > + def __init__(self): > + self._c = _bt_context_create() > + > + def __del__(self): > + _bt_context_put(self._c) > + > + def add_trace(self, path, format_str, > + packet_seek=None, stream_list=None, metadata=None): > + """ > + Add a trace by path to the context. > + > + Open a trace. > + > + path is the path to the trace, it is not recursive. > + If "path" is None, stream_list is used instead as a list > + of mmap streams to open for the trace. > + > + format is a string containing the format name in which the trace was > + produced. > + > + packet_seek is not implemented for Python. Should be left None to > + use the default packet_seek handler provided by the trace format. > + > + stream_list is a linked list of streams, it is used to open a trace > + where the trace data is located in memory mapped areas instead of > + trace files, this argument should be None when path is not None. > + > + The metadata parameter acts as a metadata override when not None, > + otherwise the format handles the metadata opening. > + > + Return: the corresponding TraceHandle on success or None on error. > + """ > + if metadata is not None: > + metadata = metadata._file > + > + ret = _bt_context_add_trace(self._c, path, format_str, packet_seek, > + stream_list, metadata) > + if ret < 0: > + return None > + > + th = TraceHandle.__new__(TraceHandle) > + th._id = ret > + return th > + > + def add_traces_recursive(self, path, format_str): > + """ > + Open a trace recursively. > + > + Find each trace present in the subdirectory starting from the given > + path, and add them to the context. > + > + Return a dict of TraceHandle instances (the full path is the key). > + Return None on error. > + """ > + > + import os > + > + trace_handles = {} > + > + noTrace = True > + error = False > + > + for fullpath, dirs, files in os.walk(path): > + if "metadata" in files: > + trace_handle = self.add_trace(fullpath, format_str) > + if trace_handle is None: > + error = True > + continue > + > + trace_handles[fullpath] = trace_handle > + noTrace = False > + > + if noTrace and error: > + return None > + return trace_handles > + > + def remove_trace(self, trace_handle): > + """ > + Remove a trace from the context. > + Effectively closing the trace. > + """ > + try: > + _bt_context_remove_trace(self._c, trace_handle._id) > + except AttributeError: > + raise TypeError("in remove_trace, " > + "argument 2 must be a TraceHandle instance") > +%} > + > + > + > +/* ================================================================= > + FORMAT.H, REGISTRY > + ?????????????????? > +*/ > + > +%rename("lookup_format") bt_lookup_format(bt_intern_str qname); > +%rename("_bt_print_format_list") bt_fprintf_format_list(FILE *fp); > +%rename("register_format") bt_register_format(struct format *format); > + > +extern struct format *bt_lookup_format(bt_intern_str qname); > +extern void bt_fprintf_format_list(FILE *fp); > +extern int bt_register_format(struct format *format); > + > +void format_init(void); > +void format_finalize(void); > + > +%pythoncode %{ > + > +def print_format_list(babeltrace_file): > + """ > + Print a list of available formats to file. > + > + babeltrace_file must be a File instance opened in write mode. > + """ > + try: > + if babeltrace_file._file is not None: > + _bt_print_format_list(babeltrace_file._file) > + except AttributeError: > + raise TypeError("in print_format_list, " > + "argument 1 must be a File instance") > + > +%} > + > + > +/* ================================================================= > + ITERATOR.H, ITERATOR-INTERNAL.H > + ??????????????????????????????? > +*/ > + > +%rename("_bt_iter_create") bt_iter_create(struct bt_context *ctx, > + const struct bt_iter_pos *begin_pos, const struct bt_iter_pos *end_pos); > +%rename("_bt_iter_destroy") bt_iter_destroy(struct bt_iter *iter); > +%rename("_bt_iter_next") bt_iter_next(struct bt_iter *iter); > +%rename("_bt_iter_get_pos") bt_iter_get_pos(struct bt_iter *iter); > +%rename("_bt_iter_free_pos") bt_iter_free_pos(struct bt_iter_pos *pos); > +%rename("_bt_iter_set_pos") bt_iter_set_pos(struct bt_iter *iter, > + const struct bt_iter_pos *pos); > +%rename("_bt_iter_create_time_pos") bt_iter_create_time_pos(struct bt_iter *iter, > + uint64_t timestamp); > + > +struct bt_iter *bt_iter_create(struct bt_context *ctx, > + const struct bt_iter_pos *begin_pos, const struct bt_iter_pos *end_pos); > +void bt_iter_destroy(struct bt_iter *iter); > +int bt_iter_next(struct bt_iter *iter); > +struct bt_iter_pos *bt_iter_get_pos(struct bt_iter *iter); > +void bt_iter_free_pos(struct bt_iter_pos *pos); > +int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *pos); > +struct bt_iter_pos *bt_iter_create_time_pos(struct bt_iter *iter, uint64_t timestamp); > + > +%rename("_bt_iter_pos") bt_iter_pos; > +%rename("SEEK_TIME") BT_SEEK_TIME; > +%rename("SEEK_RESTORE") BT_SEEK_RESTORE; > +%rename("SEEK_CUR") BT_SEEK_CUR; > +%rename("SEEK_BEGIN") BT_SEEK_BEGIN; > +%rename("SEEK_END") BT_SEEK_END; > + > + > +// This struct is taken from iterator.h > +// All changes to the struct must also be made here > +struct bt_iter_pos { > + enum { > + BT_SEEK_TIME, /* uses u.seek_time */ > + BT_SEEK_RESTORE, /* uses u.restore */ > + BT_SEEK_CUR, > + BT_SEEK_BEGIN, > + BT_SEEK_END, > + } type; > + union { > + uint64_t seek_time; > + struct bt_saved_pos *restore; > + } u; > +}; > + > + > +%pythoncode%{ > + > +class IterPos: > + """This class represents the position where to set an iterator.""" > + > + __can_access = False > + > + def __init__(self, seek_type, seek_time = None): > + """ > + seek_type represents the type of seek to use. > + seek_time is the timestamp to seek to when using SEEK_TIME, it > + is expressed in nanoseconds > + Only use SEEK_RESTORE on IterPos obtained from the get_pos function > + in Iter class. > + """ > + > + self._pos = _bt_iter_pos() > + self._pos.type = seek_type > + if seek_time and seek_type == SEEK_TIME: > + self._pos.u.seek_time = seek_time > + self.__can_access = True > + > + def __del__(self): > + if not self.__can_access: > + _bt_iter_free_pos(self._pos) > + > + def _get_type(self): > + if not __can_access: > + raise AttributeError("seek_type is not available") > + return self._pos.type > + > + def _set_type(self, seek_type): > + if not __can_access: > + raise AttributeError("seek_type is not available") > + self._pos.type = seek_type > + > + def _get_time(self): > + if not __can_access: > + raise AttributeError("seek_time is not available") > + > + elif self._pos.type is not SEEK_TIME: > + raise TypeError("seek_type is not SEEK_TIME") > + > + return self._pos.u.seek_time > + > + def _set_time(self, time): > + if not __can_access: > + raise AttributeError("seek_time is not available") > + > + elif self._pos.type is not SEEK_TIME: > + raise TypeError("seek_type is not SEEK_TIME") > + > + self._pos.u.seek_time = time > + > + def _get_pos(self): > + return self._pos > + > + > + seek_type = property(_get_type, _set_type) > + seek_time = property(_get_time, _set_time) > + > + > +class Iterator: > + > + __with_init = False > + > + def __init__(self, context, begin_pos = None, end_pos = None, _no_init = None): > + """ > + Allocate a trace collection iterator. > + > + begin_pos and end_pos are optional parameters to specify the > + position at which the trace collection should be seeked upon > + iterator creation, and the position at which iteration will > + start returning "EOF". > + > + By default, if begin_pos is None, a BT_SEEK_CUR is performed at > + creation. By default, if end_pos is None, a BT_SEEK_END (end of > + trace) is the EOF criterion. > + """ > + if _no_init is None: > + if begin_pos is None: > + bp = None > + else: > + try: > + bp = begin_pos._pos > + except AttributeError: > + raise TypeError("in __init__, " > + "argument 3 must be a IterPos instance") > + > + if end_pos is None: > + ep = None > + else: > + try: > + ep = end_pos._pos > + except AttributeError: > + raise TypeError("in __init__, " > + "argument 4 must be a IterPos instance") > + > + try: > + self._bi = _bt_iter_create(context._c, bp, ep) > + except AttributeError: > + raise TypeError("in __init__, " > + "argument 2 must be a Context instance") > + > + self.__with_init = True > + > + else: > + self._bi = _no_init > + > + def __del__(self): > + if self.__with_init: > + _bt_iter_destroy(self._bi) > + > + def next(self): > + """ > + Move trace collection position to the next event. > + Returns 0 on success, a negative value on error. > + """ > + return _bt_iter_next(self._bi) > + > + def get_pos(self): > + """Return a IterPos class of the current iterator position.""" > + ret = IterPos(0) > + ret.__can_access = False > + ret._pos = _bt_iter_get_pos(self._bi) > + return ret > + > + def set_pos(self, pos): > + """ > + Move the iterator to a given position. > + > + On error, the stream_heap is reinitialized and returned empty. > + Return 0 for success. > + Return EOF if the position requested is after the last event of the > + trace collection. > + Return -EINVAL when called with invalid parameter. > + Return -ENOMEM if the stream_heap could not be properly initialized. > + """ > + try: > + return _bt_iter_set_pos(self._bi, pos._pos) > + except AttributeError: > + raise TypeError("in set_pos, " > + "argument 2 must be a IterPos instance") > + > + def create_time_pos(self, timestamp): > + """ > + Create a position based on time > + This function allocates and returns a new IterPos to be able to > + restore an iterator position based on a timestamp. > + """ > + > + if timestamp < 0: > + raise TypeError("timestamp must be an unsigned int") > + > + ret = IterPos(0) > + ret.__can_access = False > + ret._pos = _bt_iter_create_time_pos(self._bi, timestamp) > + return ret > +%} > + > + > +/* ================================================================= > + CLOCK-TYPE.H > + ???????????? > + *** Enum copied from clock-type.h? > + All changes must also be made here > +*/ > +%rename("CLOCK_CYCLES") BT_CLOCK_CYCLES; > +%rename("CLOCK_REAL") BT_CLOCK_REAL; > + > +enum bt_clock_type { > + BT_CLOCK_CYCLES = 0, > + BT_CLOCK_REAL > +}; > + > +/* ================================================================= > + TRACE-HANDLE.H, TRACE-HANDLE-INTERNAL.H > + ??????????????????????????????????????? > +*/ > + > +%rename("_bt_trace_handle_create") bt_trace_handle_create(struct bt_context *ctx); > +%rename("_bt_trace_handle_destroy") bt_trace_handle_destroy(struct bt_trace_handle *bt); > +struct bt_trace_handle *bt_trace_handle_create(struct bt_context *ctx); > +void bt_trace_handle_destroy(struct bt_trace_handle *bt); > + > +%rename("_bt_trace_handle_get_path") bt_trace_handle_get_path(struct bt_context *ctx, > + int handle_id); > +%rename("_bt_trace_handle_get_timestamp_begin") bt_trace_handle_get_timestamp_begin( > + struct bt_context *ctx, int handle_id, enum bt_clock_type type); > +%rename("_bt_trace_handle_get_timestamp_end") bt_trace_handle_get_timestamp_end( > + struct bt_context *ctx, int handle_id, enum bt_clock_type type); > +%rename("_bt_trace_handle_get_id") bt_trace_handle_get_id(struct bt_trace_handle *th); > +int bt_trace_handle_get_id(struct bt_trace_handle *th); > +const char *bt_trace_handle_get_path(struct bt_context *ctx, int handle_id); > +uint64_t bt_trace_handle_get_timestamp_begin(struct bt_context *ctx, int handle_id, > + enum bt_clock_type type); > +uint64_t bt_trace_handle_get_timestamp_end(struct bt_context *ctx, int handle_id, > + enum bt_clock_type type); > + > +%rename("_bt_ctf_event_get_handle_id") bt_ctf_event_get_handle_id( > + const struct bt_ctf_event *event); > +int bt_ctf_event_get_handle_id(const struct bt_ctf_event *event); > + > + > +%pythoncode%{ > + > +class TraceHandle(object): > + """ > + The TraceHandle allows the user to manipulate a trace file directly. > + It is a unique identifier representing a trace file. > + Do not instantiate. > + """ > + > + def __init__(self): > + raise NotImplementedError("TraceHandle cannot be instantiated") > + > + def __repr__(self): > + return "Babeltrace TraceHandle: trace_id('{}')".format(self._id) > + > + def get_id(self): > + """Return the TraceHandle id.""" > + return self._id > + > + def get_path(self, context): > + """Return the path of a TraceHandle.""" > + try: > + return _bt_trace_handle_get_path(context._c, self._id) > + except AttributeError: > + raise TypeError("in get_path, " > + "argument 2 must be a Context instance") > + > + def get_timestamp_begin(self, context, clock_type): > + """Return the creation time of the buffers of a trace.""" > + try: > + return _bt_trace_handle_get_timestamp_begin( > + context._c, self._id,clock_type) > + except AttributeError: > + raise TypeError("in get_timestamp_begin, " > + "argument 2 must be a Context instance") > + > + def get_timestamp_end(self, context, clock_type): > + """Return the destruction timestamp of the buffers of a trace.""" > + try: > + return _bt_trace_handle_get_timestamp_end( > + context._c, self._id, clock_type) > + except AttributeError: > + raise TypeError("in get_timestamp_end, " > + "argument 2 must be a Context instance") > + > +%} > + > + > + > +// ================================================================= > +// CTF > +// ================================================================= > + > +/* ================================================================= > + ITERATOR.H, EVENTS.H > + ???????????????????? > +*/ > + > +//Iterator > +%rename("_bt_ctf_iter_create") bt_ctf_iter_create(struct bt_context *ctx, > + const struct bt_iter_pos *begin_pos, > + const struct bt_iter_pos *end_pos); > +%rename("_bt_ctf_get_iter") bt_ctf_get_iter(struct bt_ctf_iter *iter); > +%rename("_bt_ctf_iter_destroy") bt_ctf_iter_destroy(struct bt_ctf_iter *iter); > +%rename("_bt_ctf_iter_read_event") bt_ctf_iter_read_event(struct bt_ctf_iter *iter); > + > +struct bt_ctf_iter *bt_ctf_iter_create(struct bt_context *ctx, > + const struct bt_iter_pos *begin_pos, > + const struct bt_iter_pos *end_pos); > +struct bt_iter *bt_ctf_get_iter(struct bt_ctf_iter *iter); > +void bt_ctf_iter_destroy(struct bt_ctf_iter *iter); > +struct bt_ctf_event *bt_ctf_iter_read_event(struct bt_ctf_iter *iter); > + > + > +//Events > + > +%rename("_bt_ctf_get_top_level_scope") bt_ctf_get_top_level_scope(const struct > + bt_ctf_event *event, enum bt_ctf_scope scope); > +%rename("_bt_ctf_event_name") bt_ctf_event_name(const struct bt_ctf_event *ctf_event); > +%rename("_bt_ctf_get_timestamp") bt_ctf_get_timestamp( > + const struct bt_ctf_event *ctf_event); > +%rename("_bt_ctf_get_cycles") bt_ctf_get_cycles( > + const struct bt_ctf_event *ctf_event); > + > +%rename("_bt_ctf_get_field") bt_ctf_get_field(const struct bt_ctf_event *ctf_event, > + const struct definition *scope, const char *field); > +%rename("_bt_ctf_get_index") bt_ctf_get_index(const struct bt_ctf_event *ctf_event, > + const struct definition *field, unsigned int index); > +%rename("_bt_ctf_field_name") bt_ctf_field_name(const struct definition *def); > +%rename("_bt_ctf_field_type") bt_ctf_field_type(const struct definition *def); > +%rename("_bt_ctf_get_int_signedness") bt_ctf_get_int_signedness( > + const struct definition *field); > +%rename("_bt_ctf_get_int_base") bt_ctf_get_int_base(const struct definition *field); > +%rename("_bt_ctf_get_int_byte_order") bt_ctf_get_int_byte_order( > + const struct definition *field); > +%rename("_bt_ctf_get_int_len") bt_ctf_get_int_len(const struct definition *field); > +%rename("_bt_ctf_get_encoding") bt_ctf_get_encoding(const struct definition *field); > +%rename("_bt_ctf_get_array_len") bt_ctf_get_array_len(const struct definition *field); > +%rename("_bt_ctf_get_uint64") bt_ctf_get_uint64(const struct definition *field); > +%rename("_bt_ctf_get_int64") bt_ctf_get_int64(const struct definition *field); > +%rename("_bt_ctf_get_char_array") bt_ctf_get_char_array(const struct definition *field); > +%rename("_bt_ctf_get_string") bt_ctf_get_string(const struct definition *field); > +%rename("_bt_ctf_field_get_error") bt_ctf_field_get_error(void); > +%rename("_bt_ctf_get_decl_event_name") bt_ctf_get_decl_event_name(const struct > + bt_ctf_event_decl *event); > +%rename("_bt_ctf_get_decl_field_name") bt_ctf_get_decl_field_name( > + const struct bt_ctf_field_decl *field); > + > +const struct definition *bt_ctf_get_top_level_scope(const struct bt_ctf_event *ctf_event, > + enum bt_ctf_scope scope); > +const char *bt_ctf_event_name(const struct bt_ctf_event *ctf_event); > +uint64_t bt_ctf_get_timestamp(const struct bt_ctf_event *ctf_event); > +uint64_t bt_ctf_get_cycles(const struct bt_ctf_event *ctf_event); > +const struct definition *bt_ctf_get_field(const struct bt_ctf_event *ctf_event, > + const struct definition *scope, > + const char *field); > +const struct definition *bt_ctf_get_index(const struct bt_ctf_event *ctf_event, > + const struct definition *field, > + unsigned int index); > +const char *bt_ctf_field_name(const struct definition *def); > +enum ctf_type_id bt_ctf_field_type(const struct definition *def); > +int bt_ctf_get_int_signedness(const struct definition *field); > +int bt_ctf_get_int_base(const struct definition *field); > +int bt_ctf_get_int_byte_order(const struct definition *field); > +ssize_t bt_ctf_get_int_len(const struct definition *field); > +enum ctf_string_encoding bt_ctf_get_encoding(const struct definition *field); > +int bt_ctf_get_array_len(const struct definition *field); > +uint64_t bt_ctf_get_uint64(const struct definition *field); > +int64_t bt_ctf_get_int64(const struct definition *field); > +char *bt_ctf_get_char_array(const struct definition *field); > +char *bt_ctf_get_string(const struct definition *field); > +int bt_ctf_field_get_error(void); > +const char *bt_ctf_get_decl_event_name(const struct bt_ctf_event_decl *event); > +const char *bt_ctf_get_decl_field_name(const struct bt_ctf_field_decl *field); > + > +%pythoncode%{ > + > +class ctf: > + > + #enum equivalent, accessible constants > + #These are taken directly from ctf/events.h > + #All changes to enums must also be made here > + class type_id: > + UNKNOWN = 0 > + INTEGER = 1 > + FLOAT = 2 > + ENUM = 3 > + STRING = 4 > + STRUCT = 5 > + UNTAGGED_VARIANT = 6 > + VARIANT = 7 > + ARRAY = 8 > + SEQUENCE = 9 > + NR_CTF_TYPES = 10 > + > + class scope: > + TRACE_PACKET_HEADER = 0 > + STREAM_PACKET_CONTEXT = 1 > + STREAM_EVENT_HEADER = 2 > + STREAM_EVENT_CONTEXT = 3 > + EVENT_CONTEXT = 4 > + EVENT_FIELDS = 5 > + > + class string_encoding: > + NONE = 0 > + UTF8 = 1 > + ASCII = 2 > + UNKNOWN = 3 > + > + class Iterator(Iterator, object): > + """ > + Allocate a CTF trace collection iterator. > + > + begin_pos and end_pos are optional parameters to specify the > + position at which the trace collection should be seeked upon > + iterator creation, and the position at which iteration will > + start returning "EOF". > + > + By default, if begin_pos is None, a SEEK_CUR is performed at > + creation. By default, if end_pos is None, a SEEK_END (end of > + trace) is the EOF criterion. > + > + Only one iterator can be created against a context. If more than one > + iterator is being created for the same context, the second creation > + will return None. The previous iterator must be destroyed before > + creation of the new iterator for this function to succeed. > + """ > + > + def __new__(cls, context, begin_pos = None, end_pos = None): > + # __new__ is used to control the return value > + # as the ctf.Iterator class should return None > + # if bt_ctf_iter_create returns NULL > + > + if begin_pos is None: > + bp = None > + else: > + bp = begin_pos._pos > + if end_pos is None: > + ep = None > + else: > + ep = end_pos._pos > + try: > + it = _bt_ctf_iter_create(context._c, bp, ep) > + except AttributeError: > + raise TypeError("in __init__, " > + "argument 2 must be a Context instance") > + if it is None: > + return None > + > + ret_class = super(ctf.Iterator, cls).__new__(cls) > + ret_class._i = it > + return ret_class > + > + def __init__(self, context, begin_pos = None, end_pos = None): > + Iterator.__init__(self, None, None, None, > + _bt_ctf_get_iter(self._i)) > + > + def __del__(self): > + _bt_ctf_iter_destroy(self._i) > + > + def read_event(self): > + """ > + Read the iterator's current event data. > + Return current event on success, None on end of trace. > + """ > + ret = _bt_ctf_iter_read_event(self._i) > + if ret is None: > + return ret > + ev = ctf.Event.__new__(ctf.Event) > + ev._e = ret > + return ev > + > + > + class Event(object): > + """ > + This class represents an event from the trace. > + It is obtained with read_event() from ctf.Iterator. > + Do not instantiate. > + """ > + > + def __init__(self): > + raise NotImplementedError("ctf.Event cannot be instantiated") > + > + def get_top_level_scope(self, scope): > + """ > + Return a definition of the top-level scope > + Top-level scopes are defined in ctf.scope. > + In order to get a field or a field list, the user needs to pass a > + scope as argument, this scope can be a top-level scope or a scope > + relative to an arbitrary field. This function provides the mapping > + between the scope and the actual definition of top-level scopes. > + On error return None. > + """ > + evDef = ctf.Definition.__new__(ctf.Definition) > + evDef._d = _bt_ctf_get_top_level_scope(self._e, scope) > + if evDef._d is None: > + return None > + return evDef > + > + def get_name(self): > + """Return the name of the event or None on error.""" > + return _bt_ctf_event_name(self._e) > + > + def get_cycles(self): > + """ > + Return the timestamp of the event as written in > + the packet (in cycles) or -1ULL on error. > + """ > + return _bt_ctf_get_cycles(self._e) > + > + def get_timestamp(self): > + """ > + Return the timestamp of the event offsetted with the > + system clock source or -1ULL on error. > + """ > + return _bt_ctf_get_timestamp(self._e) > + > + def get_field(self, scope, field): > + """Return the definition of a specific field.""" > + evDef = ctf.Definition.__new__(ctf.Definition) > + try: > + evDef._d = _bt_ctf_get_field(self._e, scope._d, field) > + except AttributeError: > + raise TypeError("in get_field, argument 2 must be a " > + "Definition (scope) instance") > + return evDef > + > + def get_field_list(self, scope): > + """ > + Return a list of Definitions > + Return None on error. > + """ > + try: > + field_lc = _bt_python_field_listcaller(self._e, scope._d) > + except AttributeError: > + raise TypeError("in get_field_list, argument 2 must be a " > + "Definition (scope) instance") > + > + if field_lc is None: > + return None > + > + def_list = [] > + i = 0 > + while True: > + tmp = ctf.Definition.__new__(ctf.Definition) > + tmp._d = _bt_python_field_one_from_list(field_lc, i) > + > + if tmp._d is None: > + #Last item of list is None, assured in > + #_bt_python_field_listcaller > + break > + > + def_list.append(tmp) > + i += 1 > + return def_list > + > + def get_index(self, field, index): > + """ > + If the field is an array or a sequence, return the element > + at position index, otherwise return None > + """ > + evDef = ctf.Definition.__new__(ctf.Definition) > + try: > + evDef._d = _bt_ctf_get_index(self._e, field._d, index) > + except AttributeError: > + raise TypeError("in get_index, argument 2 must be a " > + "Definition (field) instance") > + > + if evDef._d is None: > + return None > + return evDef > + > + def get_handle(self): > + """ > + Get the TraceHandle associated with an event > + Return None on error > + """ > + ret = _bt_ctf_event_get_handle_id(self._e) > + if ret < 0: > + return None > + > + th = TraceHandle.__new__(TraceHandle) > + th._id = ret > + return th > + > + def get_context(self): > + """ > + Get the context associated with an event. > + Return None on error. > + """ > + ctx = Context() > + ctx._c = _bt_ctf_event_get_context(self._e); > + if ctx._c is None: > + return None > + else: > + return ctx > + > + > + class Definition(object): > + """Definition class. Do not instantiate.""" > + > + def __init__(self): > + raise NotImplementedError("ctf.Definition cannot be instantiated") > + > + def __repr__(self): > + return "Babeltrace Definition: name('{}'), type({})".format( > + self.field_name(), self.field_type()) > + > + def field_name(self): > + """Return the name of a field or None on error.""" > + return _bt_ctf_field_name(self._d) > + > + def field_type(self): > + """Return the type of a field or -1 if unknown.""" > + return _bt_ctf_field_type(self._d) > + > + def get_int_signedness(self): > + """ > + Return the signedness of an integer: > + 0 if unsigned; 1 if signed; -1 on error. > + """ > + return _bt_ctf_get_int_signedness(self._d) > + > + def get_int_base(self): > + """Return the base of an int or a negative value on error.""" > + return _bt_ctf_get_int_base(self._d) > + > + def get_int_byte_order(self): > + """ > + Return the byte order of an int or a negative > + value on error. > + """ > + return _bt_ctf_get_int_byte_order(self._d) > + > + def get_int_len(self): > + """ > + Return the size, in bits, of an int or a negative > + value on error. > + """ > + return _bt_ctf_get_int_len(self._d) > + > + def get_encoding(self): > + """ > + Return the encoding of an int or a string. > + Return a negative value on error. > + """ > + return _bt_ctf_get_encoding(self._d) > + > + def get_array_len(self): > + """ > + Return the len of an array or a negative > + value on error. > + """ > + return _bt_ctf_get_array_len(self._d) > + > + def get_uint64(self): > + """ > + Return the value associated with the field. > + If the field does not exist or is not of the type requested, > + the value returned is undefined. To check if an error occured, > + use the ctf.field_error() function after accessing a field. > + """ > + return _bt_ctf_get_uint64(self._d) > + > + def get_int64(self): > + """ > + Return the value associated with the field. > + If the field does not exist or is not of the type requested, > + the value returned is undefined. To check if an error occured, > + use the ctf.field_error() function after accessing a field. > + """ > + return _bt_ctf_get_int64(self._d) > + > + def get_char_array(self): > + """ > + Return the value associated with the field. > + If the field does not exist or is not of the type requested, > + the value returned is undefined. To check if an error occured, > + use the ctf.field_error() function after accessing a field. > + """ > + return _bt_ctf_get_char_array(self._d) > + > + def get_str(self): > + """ > + Return the value associated with the field. > + If the field does not exist or is not of the type requested, > + the value returned is undefined. To check if an error occured, > + use the ctf.field_error() function after accessing a field. > + """ > + return _bt_ctf_get_string(self._d) > + > + > + class EventDecl(object): > + """Event declaration class. Do not instantiate.""" > + > + def __init__(self): > + raise NotImplementedError("ctf.EventDecl cannot be instantiated") > + > + def __repr__(self): > + return "Babeltrace EventDecl: name {}".format(self.get_name()) > + > + def get_name(self): > + """Return the name of the event or None on error""" > + return _bt_ctf_get_decl_event_name(self._d) > + > + def get_decl_fields(self, scope): > + """ > + Return a list of ctf.FieldDecl > + Return None on error. > + """ > + ptr_list = _by_python_field_decl_listcaller(self._d, scope) > + > + if ptr_list is None: > + return None > + > + decl_list = [] > + i = 0 > + while True: > + tmp = ctf.FieldDecl.__new__(ctf.FieldDecl) > + tmp._d = _bt_python_field_decl_one_from_list( > + ptr_list, i) > + > + if tmp._d is None: > + #Last item of list is None > + break > + > + decl_list.append(tmp) > + i += 1 > + return decl_list > + > + > + class FieldDecl(object): > + """Field declaration class. Do not instantiate.""" > + > + def __init__(self): > + raise NotImplementedError("ctf.FieldDecl cannot be instantiated") > + > + def __repr__(self): > + return "Babeltrace FieldDecl: name {}".format(self.get_name()) > + > + def get_name(self): > + """Return the name of a FieldDecl or None on error""" > + return _bt_ctf_get_decl_field_name(self._d) > + > + > + @staticmethod > + def field_error(): > + """ > + Return the last error code encountered while > + accessing a field and reset the error flag. > + Return 0 if no error, a negative value otherwise. > + """ > + return _bt_ctf_field_get_error() > + > + @staticmethod > + def get_event_decl_list(trace_handle, context): > + """ > + Return a list of ctf.EventDecl > + Return None on error. > + """ > + try: > + handle_id = trace_handle._id > + except AttributeError: > + raise TypeError("in get_event_decl_list, " > + "argument 1 must be a TraceHandle instance") > + try: > + ptr_list = _bt_python_event_decl_listcaller(handle_id, context._c) > + except AttributeError: > + raise TypeError("in get_event_decl_list, " > + "argument 2 must be a Context instance") > + > + if ptr_list is None: > + return None > + > + decl_list = [] > + i = 0 > + while True: > + tmp = ctf.EventDecl.__new__(ctf.EventDecl) > + tmp._d = _bt_python_decl_one_from_list(ptr_list, i) > + > + if tmp._d is None: > + #Last item of list is None > + break > + > + decl_list.append(tmp) > + i += 1 > + return decl_list > + > +%} > + > + > + > +// ================================================================= > +// NEW FUNCTIONS > +// File and list-related > +// python-complements.h > +// ================================================================= > + > +%include python-complements.c > + > +%pythoncode %{ > + > +class File(object): > + """ > + Open a file for babeltrace. > + > + file_path is a string containing the path or None to use the > + standard output in writing mode. > + > + The mode can be 'r', 'w' or 'a' for reading (default), writing or > + appending. The file will be created if it doesn't exist when > + opened for writing or appending; it will be truncated when opened > + for writing. Add a 'b' to the mode for binary files. Add a '+' > + to the mode to allow simultaneous reading and writing. > + """ > + > + def __new__(cls, file_path, mode='r'): > + # __new__ is used to control the return value > + # as the File class should return None > + # if _bt_file_open returns NULL > + > + # Type check > + if file_path is not None and type(file_path) is not str: > + raise TypeError("in method __init__, argument 2 of type 'str'") > + if type(mode) is not str: > + raise TypeError("in method __init__, argument 3 of type 'str'") > + > + # Opening file > + file_ptr = _bt_file_open(file_path, mode) > + if file_ptr is None: > + return None > + > + # Class instantiation > + file_inst = super(File, cls).__new__(cls) > + file_inst._file = file_ptr > + return file_inst > + > + def __init__(self, file_path, mode='r'): > + self._opened = True > + self._use_stdout = False > + > + if file_path is None: > + # use stdout > + file_path = "stdout" > + mode = 'w' > + self._use_stdout = True > + > + self._file_path = file_path > + self._mode = mode > + > + def __del__(self): > + self.close() > + > + def __repr__(self): > + if self._opened: > + stat = 'opened' > + else: > + stat = 'closed' > + return "{} babeltrace File; file_path('{}'), mode('{}')".format( > + stat, self._file_path, self._mode) > + > + def close(self): > + """Close the file. Is also called using del.""" > + if self._opened and not self._use_stdout: > + _bt_file_close(self._file) > + self._opened = False > +%} > diff --git a/bindings/python/examples/babeltrace_and_lttng.py b/bindings/python/examples/babeltrace_and_lttng.py > new file mode 100644 > index 0000000..cb44796 > --- /dev/null > +++ b/bindings/python/examples/babeltrace_and_lttng.py > @@ -0,0 +1,126 @@ > +# babeltrace_and_lttng.py > +# > +# Babeltrace and LTTng example script > +# > +# Copyright 2012 EfficiOS Inc. > +# > +# Author: Danny Serres > +# > +# Permission is hereby granted, free of charge, to any person obtaining a copy > +# of this software and associated documentation files (the "Software"), to deal > +# in the Software without restriction, including without limitation the rights > +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell > +# copies of the Software, and to permit persons to whom the Software is > +# furnished to do so, subject to the following conditions: > +# > +# The above copyright notice and this permission notice shall be included in > +# all copies or substantial portions of the Software. > + > + > +# This script uses both lttng-tools and babeltrace > +# python modules. It creates a session, enables > +# events, starts tracing for 2 seconds, stops tracing, > +# destroys the session and outputs the trace in the > +# specified output file. > +# > +# WARNING: will destroy any existing trace having > +# the same name as ses_name > + > + > +# ------------------------------------------------------ > +ses_name = "babeltrace-lttng-test" > +trace_path = "/lttng-traces/babeltrace-lttng-trace/" > +out_file = "babeltrace-lttng-trace-text-output.txt" > +# ------------------------------------------------------ > + > + > +import time > +try: > + import babeltrace, lttng > +except ImportError: > + raise ImportError( "both babeltrace and lttng-tools " > + "python modules must be installed" ) > + > + > +# Errors to raise if something goes wrong > +class LTTngError(Exception): > + pass > +class BabeltraceError(Exception): > + pass > + > + > +# LTTNG-TOOLS > + > +# Making sure session does not already exist > +lttng.destroy(ses_name) > + > +# Creating a new session and handle > +ret = lttng.create(ses_name,trace_path) > +if ret < 0: > + raise LTTngError(lttng.strerror(ret)) > + > +han = None > +han = lttng.Handle(ses_name, lttng.Domain()) > +if han is None: > + raise LTTngError("Handle not created") > + > + > +# Enabling all events > +ret = lttng.enable_event(han, lttng.Event(), None) > +if ret < 0: > + raise LTTngError(lttng.strerror(ret)) > + > + > +# Start, wait, stop > +ret = lttng.start(ses_name) > +if ret < 0: > + raise LTTngError(lttng.strerror(ret)) > +print("Tracing...") > +time.sleep(2) > +print("Stopped.") > +ret = lttng.stop(ses_name) > +if ret < 0: > + raise LTTngError(lttng.strerror(ret)) > + > + > +# Destroying tracing session > +ret = lttng.destroy(ses_name) > +if ret < 0: > + raise LTTngError(lttng.strerror(ret)) > + > + > +# BABELTRACE > + > +# Create context and add trace: > +ctx = babeltrace.Context() > +ret = ctx.add_trace(trace_path + "/kernel", "ctf") > +if ret is None: > + raise BabeltraceError("Error adding trace") > + > +# Iterator setup > +bp = babeltrace.IterPos(babeltrace.SEEK_BEGIN) > +ctf_it = babeltrace.ctf.Iterator(ctx,bp) > + > +# Reading events from trace > +# and outputting timestamps and event names > +# in out_file > +print("Writing trace file...") > +output = open(out_file, "wt") > + > +event = ctf_it.read_event() > +while(event is not None): > + output.write("TS: {}, {} : {}\n".format(event.get_timestamp(), > + event.get_cycles(), event.get_name())) > + > + # Next event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > +# Closing file > +output.close() > + > +# Destroying dynamic elements > +del ctf_it, han > +print("Done.") > diff --git a/bindings/python/examples/eventcount.py b/bindings/python/examples/eventcount.py > new file mode 100644 > index 0000000..5e96a43 > --- /dev/null > +++ b/bindings/python/examples/eventcount.py > @@ -0,0 +1,84 @@ > +# eventcount.py > +# > +# Babeltrace event count example script > +# > +# Copyright 2012 EfficiOS Inc. > +# > +# Author: Danny Serres > +# > +# Permission is hereby granted, free of charge, to any person obtaining a copy > +# of this software and associated documentation files (the "Software"), to deal > +# in the Software without restriction, including without limitation the rights > +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell > +# copies of the Software, and to permit persons to whom the Software is > +# furnished to do so, subject to the following conditions: > +# > +# The above copyright notice and this permission notice shall be included in > +# all copies or substantial portions of the Software. > + > +# The script prints a count of specified events and > +# their related tid's in a given trace. > +# The trace needs TID context (lttng add-context -k -t tid) > + > +import sys > +from babeltrace import * > +from output_format_modules.pprint_table import pprint_table as pprint > + > +if len(sys.argv) < 3: > + raise TypeError("Usage: python eventcount.py event1 [event2 ...] path/to/trace") > + > +ctx = Context() > +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +counts = {} > + > +# Setting iterator > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx, bp) > + > +# Reading events > +event = ctf_it.read_event() > +while(event is not None): > + for event_type in sys.argv[1:len(sys.argv)-1]: > + if event_type == event.get_name(): > + > + # Getting scope definition > + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) > + if sco is None: > + print("ERROR: Cannot get definition scope for {}".format( > + event.get_name())) > + continue > + > + # Getting TID > + tid_field = event.get_field(sco, "_tid") > + tid = tid_field.get_int64() > + > + if ctf.field_error(): > + print("ERROR: Missing TID info for {}".format( > + event.get_name())) > + continue > + > + tmp = (tid, event.get_name()) > + > + if tmp in counts: > + counts[tmp] += 1 > + else: > + counts[tmp] = 1 > + > + # Next event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > +del ctf_it > + > +# Appending data to table for output > +table = [] > +for item in counts: > + table.append([item[0], item[1], counts[item]]) > +table = sorted(table) > +table.insert(0,["TID", "EVENT", "COUNT"]) > +pprint(table, 2) > diff --git a/bindings/python/examples/eventcountlist.py b/bindings/python/examples/eventcountlist.py > new file mode 100644 > index 0000000..945a960 > --- /dev/null > +++ b/bindings/python/examples/eventcountlist.py > @@ -0,0 +1,83 @@ > +# eventcountlist.py > +# > +# Babeltrace event count list example script > +# > +# Copyright 2012 EfficiOS Inc. > +# > +# Author: Danny Serres > +# > +# Permission is hereby granted, free of charge, to any person obtaining a copy > +# of this software and associated documentation files (the "Software"), to deal > +# in the Software without restriction, including without limitation the rights > +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell > +# copies of the Software, and to permit persons to whom the Software is > +# furnished to do so, subject to the following conditions: > +# > +# The above copyright notice and this permission notice shall be included in > +# all copies or substantial portions of the Software. > + > +# The script prints a count and rate of events. > +# It also outputs a bar graph of count per event, using the cairoplot module. > + > +import sys > +from babeltrace import * > +from output_format_modules import cairoplot > +from output_format_modules.pprint_table import pprint_table as pprint > + > +# Check for path arg: > +if len(sys.argv) < 2: > + raise TypeError("Usage: python eventcountlist.py path/to/trace") > + > +ctx = Context() > +ret = ctx.add_trace(sys.argv[1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +# Events and their assossiated count > +# will be stored as a dict: > +events_count = {} > + > +# Setting iterator: > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx,bp) > + > +prev_event = None > +event = ctf_it.read_event() > + > +start_time = event.get_timestamp() > + > +# Reading events: > +while(event is not None): > + if event.get_name() in events_count: > + events_count[event.get_name()] += 1 > + else: > + events_count[event.get_name()] = 1 > + > + ret = ctf_it.next() > + if ret < 0: > + break > + else: > + prev_event = event > + event = ctf_it.read_event() > + > +if event: > + total_time = event.get_timestamp() - start_time > +else: > + total_time = prev_event.get_timestamp() - start_time > + > +del ctf_it > + > +# Printing encountered events with respective count and rate: > +print("Total time: {} ns".format(total_time)) > +table = [["EVENT", "COUNT", "RATE (Hz)"]] > +for item in sorted(events_count.iterkeys()): > + tmp = [item, events_count[item], > + events_count[item]/(total_time/1000000000.0)] > + table.append(tmp) > +pprint(table) > + > +# Exporting data as bar graph > +cairoplot.vertical_bar_plot ( 'eventcountlist.svg', events_count, 50+85*len(events_count), > + 800, border = 20, display_values = True, grid = True, > + rounded_corners = True, > + x_labels = sorted(events_count.keys()) ) > diff --git a/bindings/python/examples/events_per_cpu.py b/bindings/python/examples/events_per_cpu.py > new file mode 100644 > index 0000000..be497ec > --- /dev/null > +++ b/bindings/python/examples/events_per_cpu.py > @@ -0,0 +1,99 @@ > +# events_per_cpu.py > +# > +# Babeltrace events per cpu example script > +# > +# Copyright 2012 EfficiOS Inc. > +# > +# Author: Danny Serres > +# > +# Permission is hereby granted, free of charge, to any person obtaining a copy > +# of this software and associated documentation files (the "Software"), to deal > +# in the Software without restriction, including without limitation the rights > +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell > +# copies of the Software, and to permit persons to whom the Software is > +# furnished to do so, subject to the following conditions: > +# > +# The above copyright notice and this permission notice shall be included in > +# all copies or substantial portions of the Software. > + > +# The script opens a trace and prints out CPU statistics > +# for the given trace (event count per CPU, total active > +# time and % of time processing events). > +# It also outputs a .txt file showing each time interval > +# (since the beginning of the trace) in which each CPU > +# was active and the corresponding event. > + > +import sys, multiprocessing > +from output_format_modules.pprint_table import pprint_table as pprint > +from babeltrace import * > + > +if len(sys.argv) < 2: > + raise TypeError("Usage: python events_per_cpu.py path/to/trace") > + > +# Adding trace > +ctx = Context() > +ret = ctx.add_trace(sys.argv[1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +cpu_usage = [] > +nbEvents = 0 > +i = 0 > +while i < multiprocessing.cpu_count(): > + cpu_usage.append([]) > + i += 1 > + > +# Setting iterator > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx, bp) > + > +# Reading events > +event = ctf_it.read_event() > +start_time = event.get_timestamp() > + > +while(event is not None): > + > + event_name = event.get_name() > + ts = event.get_timestamp() > + > + # Getting cpu_id > + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) > + field = event.get_field(scope, "cpu_id") > + cpu_id = field.get_uint64() > + if ctf.field_error(): > + print("ERROR: Missing cpu_id info for {}".format(event.get_name())) > + else: > + cpu_usage[cpu_id].append( (int(ts), event_name) ) > + nbEvents += 1 > + > + # Next Event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > + > +# Outputting > +table = [] > +output = open("events_per_cpu.txt", "wt") > +output.write("(timestamp, event)\n") > + > +for cpu in range(len(cpu_usage)): > + # Setting table > + event_str = str(100.0 * len(cpu_usage[cpu]) / nbEvents) + '000' > + # % is printed with 2 decimals > + table.append([cpu, len(cpu_usage[cpu]), event_str[0:event_str.find('.') + 3] + ' %']) > + > + # Writing to file > + output.write("\n\n\n----------------------\n") > + output.write("CPU {}\n\n".format(cpu)) > + for event in cpu_usage[cpu]: > + output.write(str(event) + '\n') > + > +# Printing table > +table.insert(0, ["CPU ID", "EVENT COUNT", "TRACE EVENT %"]) > +pprint(table) > +print("Total event count: {}".format(nbEvents)) > +print("Total trace time: {} ns".format(ts - start_time)) > + > +output.close() > diff --git a/bindings/python/examples/example-api-test.py b/bindings/python/examples/example-api-test.py > new file mode 100644 > index 0000000..104f2d5 > --- /dev/null > +++ b/bindings/python/examples/example-api-test.py > @@ -0,0 +1,77 @@ > +# example_api_test.py > +# > +# Babeltrace example script based on the Babeltrace API test script > +# > +# Copyright 2012 EfficiOS Inc. > +# > +# Author: Danny Serres > +# > +# Permission is hereby granted, free of charge, to any person obtaining a copy > +# of this software and associated documentation files (the "Software"), to deal > +# in the Software without restriction, including without limitation the rights > +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell > +# copies of the Software, and to permit persons to whom the Software is > +# furnished to do so, subject to the following conditions: > +# > +# The above copyright notice and this permission notice shall be included in > +# all copies or substantial portions of the Software. > + > +# This example uses the babeltrace python module > +# to partially test the api. > + > +import sys > +from babeltrace import * > + > +# Check for path arg: > +if len(sys.argv) < 2: > + raise TypeError("Usage: python example-api-test.py path/to/file") > + > +# Create context and add trace: > +ctx = Context() > +trace_handle = ctx.add_trace(sys.argv[1], "ctf") > +if trace_handle is None: > + raise IOError("Error adding trace") > + > +# Listing events > +lst = ctf.get_event_decl_list(trace_handle, ctx) > +print("--- Event list ---") > +for item in lst: > + print("event : {}".format(item.get_name())) > +print("--- Done ---") > + > +# Iter trace > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx,bp) > +event = ctf_it.read_event() > + > +while(event is not None): > + print("TS: {}, {} : {}".format(event.get_timestamp(), > + event.get_cycles(), event.get_name())) > + > + if event.get_name() == "sched_switch": > + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) > + prev_field = event.get_field(sco, "_prev_comm") > + prev_comm = prev_field.get_char_array() > + > + if ctf.field_error(): > + print("ERROR: Missing prev_comm context info") > + else: > + print("sched_switch prev_comm: {}".format(prev_comm)) > + > + if event.get_name() == "exit_syscall": > + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) > + ret_field = event.get_field(sco, "_ret") > + ret_code = ret_field.get_int64() > + > + if ctf.field_error(): > + print("ERROR: Unable to extract ret") > + else: > + print("exit_syscall ret: {}".format(ret_code)) > + > + ret = ctf_it.next() > + if ret < 0: > + break > + else: > + event = ctf_it.read_event() > + > +del ctf_it > diff --git a/bindings/python/examples/histogram.py b/bindings/python/examples/histogram.py > new file mode 100644 > index 0000000..44616a6 > --- /dev/null > +++ b/bindings/python/examples/histogram.py > @@ -0,0 +1,139 @@ > +# histogram.py > +# > +# Babeltrace histogram example script > +# > +# Copyright 2012 EfficiOS Inc. > +# > +# Author: Danny Serres > +# > +# Permission is hereby granted, free of charge, to any person obtaining a copy > +# of this software and associated documentation files (the "Software"), to deal > +# in the Software without restriction, including without limitation the rights > +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell > +# copies of the Software, and to permit persons to whom the Software is > +# furnished to do so, subject to the following conditions: > +# > +# The above copyright notice and this permission notice shall be included in > +# all copies or substantial portions of the Software. > + > +# The script checks the number of events in the trace > +# and outputs a table and a .svg histogram for the specified > +# range (microseconds) or the total trace if no range specified. > +# The graph is generated using the cairoplot module. > + > +import sys > +from babeltrace import * > +from output_format_modules import cairoplot > +from output_format_modules.pprint_table import pprint_table as pprint > + > +# ------------------------------------------------ > +# Output settings > + > +# number of intervals: > +nbDiv = 25 # Should not be over 150 > + # for usable graph output > + > +# table output stream (file-like object): > +out = sys.stdout > +# ------------------------------------------------- > + > +if len(sys.argv) < 2 or len(sys.argv) > 4: > + raise TypeError("Usage: python histogram.py [ start_time [end_time] ] path/to/trace") > + > +ctx = Context() > +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +# Check when to start/stop graphing > +sinceBegin = True > +beginTime = 0.0 > +if len(sys.argv) > 2: > + sinceBegin = False > + beginTime = float(sys.argv[1]) > +untilEnd = True > +if len(sys.argv) == 4: > + untilEnd = False > + > +# Setting iterator > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx, bp) > + > +# Reading events > +event = ctf_it.read_event() > +start_time = event.get_timestamp() > +time = 0 > +count = {} > + > +while(event is not None): > + # Microsec. > + time = (event.get_timestamp() - start_time)/1000.0 > + > + # Check if in range > + if not sinceBegin: > + if time < beginTime: > + # Next Event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + continue > + if not untilEnd: > + if time > float(sys.argv[2]): > + break > + > + # Counting events per timestamp: > + if time in count: > + count[time] += 1 > + else: > + count[time] = 1 > + > + # Next Event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > +del ctf_it > + > +# Setting data for output > +interval = (time - beginTime)/nbDiv > +div_begin_time = beginTime > +div_end_time = beginTime + interval > +data = {} > + > +# Prefix for string sorting, considering > +# there should not be over 150 intervals. > +# This would work up to 9999 intervals. > +# If needed, add zeros. > +prefix = 0.0001 > + > +while div_end_time <= time: > + key = str(prefix) + '[' + str(div_begin_time) + ';' + str(div_end_time) + '[' > + for tmp in count: > + if tmp >= div_begin_time and tmp < div_end_time: > + if key in data: > + data[key] += count[tmp] > + else: > + data[key] = count[tmp] > + if not key in data: > + data[key] = 0 > + div_begin_time = div_end_time > + div_end_time += interval > + # Prefix increment > + prefix += 0.001 > + > +table = [] > +x_labels = [] > +for key in sorted(data): > + table.append([key[key.find('['):], data[key]]) > + x_labels.append(key[key.find('['):]) > + > +# Table output > +table.insert(0, ["INTERVAL (us)", "COUNT"]) > +pprint(table, 1, out) > + > +# Graph output > +cairoplot.vertical_bar_plot ( 'histogram.svg', data, 50 + 150*nbDiv, 50*nbDiv, > + border = 20, display_values = True, grid = True, > + x_labels = x_labels, rounded_corners = True ) > diff --git a/bindings/python/examples/output_format_modules/__init__.py b/bindings/python/examples/output_format_modules/__init__.py > new file mode 100644 > index 0000000..e69de29 > diff --git a/bindings/python/examples/output_format_modules/cairoplot.py b/bindings/python/examples/output_format_modules/cairoplot.py > new file mode 100755 > index 0000000..a27113f > --- /dev/null > +++ b/bindings/python/examples/output_format_modules/cairoplot.py > @@ -0,0 +1,2336 @@ > +?#!/usr/bin/env python > +# -*- coding: utf-8 -*- > + > +# CairoPlot.py > +# > +# Copyright (c) 2008 Rodrigo Moreira Ara?jo > +# > +# Author: Rodrigo Moreiro Araujo > +# > +# This program is free software; you can redistribute it and/or > +# modify it under the terms of the GNU Lesser General Public License > +# as published by the Free Software Foundation; either version 2 of > +# the License, or (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU Lesser General Public > +# License along with this program; if not, write to the Free Software > +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 > +# USA > + > +#Contributor: Jo?o S. O. Bueno > + > +#TODO: review BarPlot Code > +#TODO: x_label colision problem on Horizontal Bar Plot > +#TODO: y_label's eat too much space on HBP > + > + > +__version__ = 1.2 > + > +import cairo > +import math > +import random > +from series import Series, Group, Data > + > +HORZ = 0 > +VERT = 1 > +NORM = 2 > + > +COLORS = {"red" : (1.0,0.0,0.0,1.0), "lime" : (0.0,1.0,0.0,1.0), "blue" : (0.0,0.0,1.0,1.0), > + "maroon" : (0.5,0.0,0.0,1.0), "green" : (0.0,0.5,0.0,1.0), "navy" : (0.0,0.0,0.5,1.0), > + "yellow" : (1.0,1.0,0.0,1.0), "magenta" : (1.0,0.0,1.0,1.0), "cyan" : (0.0,1.0,1.0,1.0), > + "orange" : (1.0,0.5,0.0,1.0), "white" : (1.0,1.0,1.0,1.0), "black" : (0.0,0.0,0.0,1.0), > + "gray" : (0.5,0.5,0.5,1.0), "light_gray" : (0.9,0.9,0.9,1.0), > + "transparent" : (0.0,0.0,0.0,0.0)} > + > +THEMES = {"black_red" : [(0.0,0.0,0.0,1.0), (1.0,0.0,0.0,1.0)], > + "red_green_blue" : [(1.0,0.0,0.0,1.0), (0.0,1.0,0.0,1.0), (0.0,0.0,1.0,1.0)], > + "red_orange_yellow" : [(1.0,0.2,0.0,1.0), (1.0,0.7,0.0,1.0), (1.0,1.0,0.0,1.0)], > + "yellow_orange_red" : [(1.0,1.0,0.0,1.0), (1.0,0.7,0.0,1.0), (1.0,0.2,0.0,1.0)], > + "rainbow" : [(1.0,0.0,0.0,1.0), (1.0,0.5,0.0,1.0), (1.0,1.0,0.0,1.0), (0.0,1.0,0.0,1.0), (0.0,0.0,1.0,1.0), (0.3, 0.0, 0.5,1.0), (0.5, 0.0, 1.0, 1.0)]} > + > +def colors_from_theme( theme, series_length, mode = 'solid' ): > + colors = [] > + if theme not in THEMES.keys() : > + raise Exception, "Theme not defined" > + color_steps = THEMES[theme] > + n_colors = len(color_steps) > + if series_length <= n_colors: > + colors = [color + tuple([mode]) for color in color_steps[0:n_colors]] > + else: > + iterations = [(series_length - n_colors)/(n_colors - 1) for i in color_steps[:-1]] > + over_iterations = (series_length - n_colors) % (n_colors - 1) > + for i in range(n_colors - 1): > + if over_iterations <= 0: > + break > + iterations[i] += 1 > + over_iterations -= 1 > + for index,color in enumerate(color_steps[:-1]): > + colors.append(color + tuple([mode])) > + if iterations[index] == 0: > + continue > + next_color = color_steps[index+1] > + color_step = ((next_color[0] - color[0])/(iterations[index] + 1), > + (next_color[1] - color[1])/(iterations[index] + 1), > + (next_color[2] - color[2])/(iterations[index] + 1), > + (next_color[3] - color[3])/(iterations[index] + 1)) > + for i in range( iterations[index] ): > + colors.append((color[0] + color_step[0]*(i+1), > + color[1] + color_step[1]*(i+1), > + color[2] + color_step[2]*(i+1), > + color[3] + color_step[3]*(i+1), > + mode)) > + colors.append(color_steps[-1] + tuple([mode])) > + return colors > + > + > +def other_direction(direction): > + "explicit is better than implicit" > + if direction == HORZ: > + return VERT > + else: > + return HORZ > + > +#Class definition > + > +class Plot(object): > + def __init__(self, > + surface=None, > + data=None, > + width=640, > + height=480, > + background=None, > + border = 0, > + x_labels = None, > + y_labels = None, > + series_colors = None): > + random.seed(2) > + self.create_surface(surface, width, height) > + self.dimensions = {} > + self.dimensions[HORZ] = width > + self.dimensions[VERT] = height > + self.context = cairo.Context(self.surface) > + self.labels={} > + self.labels[HORZ] = x_labels > + self.labels[VERT] = y_labels > + self.load_series(data, x_labels, y_labels, series_colors) > + self.font_size = 10 > + self.set_background (background) > + self.border = border > + self.borders = {} > + self.line_color = (0.5, 0.5, 0.5) > + self.line_width = 0.5 > + self.label_color = (0.0, 0.0, 0.0) > + self.grid_color = (0.8, 0.8, 0.8) > + > + def create_surface(self, surface, width=None, height=None): > + self.filename = None > + if isinstance(surface, cairo.Surface): > + self.surface = surface > + return > + if not type(surface) in (str, unicode): > + raise TypeError("Surface should be either a Cairo surface or a filename, not %s" % surface) > + sufix = surface.rsplit(".")[-1].lower() > + self.filename = surface > + if sufix == "png": > + self.surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, width, height) > + elif sufix == "ps": > + self.surface = cairo.PSSurface(surface, width, height) > + elif sufix == "pdf": > + self.surface = cairo.PSSurface(surface, width, height) > + else: > + if sufix != "svg": > + self.filename += ".svg" > + self.surface = cairo.SVGSurface(self.filename, width, height) > + > + def commit(self): > + try: > + self.context.show_page() > + if self.filename and self.filename.endswith(".png"): > + self.surface.write_to_png(self.filename) > + else: > + self.surface.finish() > + except cairo.Error: > + pass > + > + def load_series (self, data, x_labels=None, y_labels=None, series_colors=None): > + self.series_labels = [] > + self.series = None > + > + #The pretty way > + #if not isinstance(data, Series): > + # # Not an instance of Series > + # self.series = Series(data) > + #else: > + # self.series = data > + # > + #self.series_labels = self.series.get_names() > + > + #TODO: Remove on next version > + # The ugly way, keeping retrocompatibility... > + if callable(data) or type(data) is list and callable(data[0]): # Lambda or List of lambdas > + self.series = data > + self.series_labels = None > + elif isinstance(data, Series): # Instance of Series > + self.series = data > + self.series_labels = data.get_names() > + else: # Anything else > + self.series = Series(data) > + self.series_labels = self.series.get_names() > + > + #TODO: allow user passed series_widths > + self.series_widths = [1.0 for group in self.series] > + > + #TODO: Remove on next version > + self.process_colors( series_colors ) > + > + def process_colors( self, series_colors, length = None, mode = 'solid' ): > + #series_colors might be None, a theme, a string of colors names or a list of color tuples > + if length is None : > + length = len( self.series.to_list() ) > + > + #no colors passed > + if not series_colors: > + #Randomize colors > + self.series_colors = [ [random.random() for i in range(3)] + [1.0, mode] for series in range( length ) ] > + else: > + #Just theme pattern > + if not hasattr( series_colors, "__iter__" ): > + theme = series_colors > + self.series_colors = colors_from_theme( theme.lower(), length ) > + > + #Theme pattern and mode > + elif not hasattr(series_colors, '__delitem__') and not hasattr( series_colors[0], "__iter__" ): > + theme = series_colors[0] > + mode = series_colors[1] > + self.series_colors = colors_from_theme( theme.lower(), length, mode ) > + > + #List > + else: > + self.series_colors = series_colors > + for index, color in enumerate( self.series_colors ): > + #element is a color name > + if not hasattr(color, "__iter__"): > + self.series_colors[index] = COLORS[color.lower()] + tuple([mode]) > + #element is rgb tuple instead of rgba > + elif len( color ) == 3 : > + self.series_colors[index] += (1.0,mode) > + #element has 4 elements, might be rgba tuple or rgb tuple with mode > + elif len( color ) == 4 : > + #last element is mode > + if not hasattr(color[3], "__iter__"): > + self.series_colors[index] += tuple([color[3]]) > + self.series_colors[index][3] = 1.0 > + #last element is alpha > + else: > + self.series_colors[index] += tuple([mode]) > + > + def get_width(self): > + return self.surface.get_width() > + > + def get_height(self): > + return self.surface.get_height() > + > + def set_background(self, background): > + if background is None: > + self.background = (0.0,0.0,0.0,0.0) > + elif type(background) in (cairo.LinearGradient, tuple): > + self.background = background > + elif not hasattr(background,"__iter__"): > + colors = background.split(" ") > + if len(colors) == 1 and colors[0] in COLORS: > + self.background = COLORS[background] > + elif len(colors) > 1: > + self.background = cairo.LinearGradient(self.dimensions[HORZ] / 2, 0, self.dimensions[HORZ] / 2, self.dimensions[VERT]) > + for index,color in enumerate(colors): > + self.background.add_color_stop_rgba(float(index)/(len(colors)-1),*COLORS[color]) > + else: > + raise TypeError ("Background should be either cairo.LinearGradient or a 3/4-tuple, not %s" % type(background)) > + > + def render_background(self): > + if isinstance(self.background, cairo.LinearGradient): > + self.context.set_source(self.background) > + else: > + self.context.set_source_rgba(*self.background) > + self.context.rectangle(0,0, self.dimensions[HORZ], self.dimensions[VERT]) > + self.context.fill() > + > + def render_bounding_box(self): > + self.context.set_source_rgba(*self.line_color) > + self.context.set_line_width(self.line_width) > + self.context.rectangle(self.border, self.border, > + self.dimensions[HORZ] - 2 * self.border, > + self.dimensions[VERT] - 2 * self.border) > + self.context.stroke() > + > + def render(self): > + pass > + > +class ScatterPlot( Plot ): > + def __init__(self, > + surface=None, > + data=None, > + errorx=None, > + errory=None, > + width=640, > + height=480, > + background=None, > + border=0, > + axis = False, > + dash = False, > + discrete = False, > + dots = 0, > + grid = False, > + series_legend = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + z_bounds = None, > + x_title = None, > + y_title = None, > + series_colors = None, > + circle_colors = None ): > + > + self.bounds = {} > + self.bounds[HORZ] = x_bounds > + self.bounds[VERT] = y_bounds > + self.bounds[NORM] = z_bounds > + self.titles = {} > + self.titles[HORZ] = x_title > + self.titles[VERT] = y_title > + self.max_value = {} > + self.axis = axis > + self.discrete = discrete > + self.dots = dots > + self.grid = grid > + self.series_legend = series_legend > + self.variable_radius = False > + self.x_label_angle = math.pi / 2.5 > + self.circle_colors = circle_colors > + > + Plot.__init__(self, surface, data, width, height, background, border, x_labels, y_labels, series_colors) > + > + self.dash = None > + if dash: > + if hasattr(dash, "keys"): > + self.dash = [dash[key] for key in self.series_labels] > + elif max([hasattr(item,'__delitem__') for item in data]) : > + self.dash = dash > + else: > + self.dash = [dash] > + > + self.load_errors(errorx, errory) > + > + def convert_list_to_tuple(self, data): > + #Data must be converted from lists of coordinates to a single > + # list of tuples > + out_data = zip(*data) > + if len(data) == 3: > + self.variable_radius = True > + return out_data > + > + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): > + #TODO: In cairoplot 2.0 keep only the Series instances > + > + # Convert Data and Group to Series > + if isinstance(data, Data) or isinstance(data, Group): > + data = Series(data) > + > + # Series > + if isinstance(data, Series): > + for group in data: > + for item in group: > + if len(item) is 3: > + self.variable_radius = True > + > + #Dictionary with lists > + if hasattr(data, "keys") : > + if hasattr( data.values()[0][0], "__delitem__" ) : > + for key in data.keys() : > + data[key] = self.convert_list_to_tuple(data[key]) > + elif len(data.values()[0][0]) == 3: > + self.variable_radius = True > + #List > + elif hasattr(data[0], "__delitem__") : > + #List of lists > + if hasattr(data[0][0], "__delitem__") : > + for index,value in enumerate(data) : > + data[index] = self.convert_list_to_tuple(value) > + #List > + elif type(data[0][0]) != type((0,0)): > + data = self.convert_list_to_tuple(data) > + #Three dimensional data > + elif len(data[0][0]) == 3: > + self.variable_radius = True > + > + #List with three dimensional tuples > + elif len(data[0]) == 3: > + self.variable_radius = True > + Plot.load_series(self, data, x_labels, y_labels, series_colors) > + self.calc_boundaries() > + self.calc_labels() > + > + def load_errors(self, errorx, errory): > + self.errors = None > + if errorx == None and errory == None: > + return > + self.errors = {} > + self.errors[HORZ] = None > + self.errors[VERT] = None > + #asimetric errors > + if errorx and hasattr(errorx[0], "__delitem__"): > + self.errors[HORZ] = errorx > + #simetric errors > + elif errorx: > + self.errors[HORZ] = [errorx] > + #asimetric errors > + if errory and hasattr(errory[0], "__delitem__"): > + self.errors[VERT] = errory > + #simetric errors > + elif errory: > + self.errors[VERT] = [errory] > + > + def calc_labels(self): > + if not self.labels[HORZ]: > + amplitude = self.bounds[HORZ][1] - self.bounds[HORZ][0] > + if amplitude % 10: #if horizontal labels need floating points > + self.labels[HORZ] = ["%.2lf" % (float(self.bounds[HORZ][0] + (amplitude * i / 10.0))) for i in range(11) ] > + else: > + self.labels[HORZ] = ["%d" % (int(self.bounds[HORZ][0] + (amplitude * i / 10.0))) for i in range(11) ] > + if not self.labels[VERT]: > + amplitude = self.bounds[VERT][1] - self.bounds[VERT][0] > + if amplitude % 10: #if vertical labels need floating points > + self.labels[VERT] = ["%.2lf" % (float(self.bounds[VERT][0] + (amplitude * i / 10.0))) for i in range(11) ] > + else: > + self.labels[VERT] = ["%d" % (int(self.bounds[VERT][0] + (amplitude * i / 10.0))) for i in range(11) ] > + > + def calc_extents(self, direction): > + self.context.set_font_size(self.font_size * 0.8) > + self.max_value[direction] = max(self.context.text_extents(item)[2] for item in self.labels[direction]) > + self.borders[other_direction(direction)] = self.max_value[direction] + self.border + 20 > + > + def calc_boundaries(self): > + #HORZ = 0, VERT = 1, NORM = 2 > + min_data_value = [0,0,0] > + max_data_value = [0,0,0] > + > + for group in self.series: > + if type(group[0].content) in (int, float, long): > + group = [Data((index, item.content)) for index,item in enumerate(group)] > + > + for point in group: > + for index, item in enumerate(point.content): > + if item > max_data_value[index]: > + max_data_value[index] = item > + elif item < min_data_value[index]: > + min_data_value[index] = item > + > + if not self.bounds[HORZ]: > + self.bounds[HORZ] = (min_data_value[HORZ], max_data_value[HORZ]) > + if not self.bounds[VERT]: > + self.bounds[VERT] = (min_data_value[VERT], max_data_value[VERT]) > + if not self.bounds[NORM]: > + self.bounds[NORM] = (min_data_value[NORM], max_data_value[NORM]) > + > + def calc_all_extents(self): > + self.calc_extents(HORZ) > + self.calc_extents(VERT) > + > + self.plot_height = self.dimensions[VERT] - 2 * self.borders[VERT] > + self.plot_width = self.dimensions[HORZ] - 2* self.borders[HORZ] > + > + self.plot_top = self.dimensions[VERT] - self.borders[VERT] > + > + def calc_steps(self): > + #Calculates all the x, y, z and color steps > + series_amplitude = [self.bounds[index][1] - self.bounds[index][0] for index in range(3)] > + > + if series_amplitude[HORZ]: > + self.horizontal_step = float (self.plot_width) / series_amplitude[HORZ] > + else: > + self.horizontal_step = 0.00 > + > + if series_amplitude[VERT]: > + self.vertical_step = float (self.plot_height) / series_amplitude[VERT] > + else: > + self.vertical_step = 0.00 > + > + if series_amplitude[NORM]: > + if self.variable_radius: > + self.z_step = float (self.bounds[NORM][1]) / series_amplitude[NORM] > + if self.circle_colors: > + self.circle_color_step = tuple([float(self.circle_colors[1][i]-self.circle_colors[0][i])/series_amplitude[NORM] for i in range(4)]) > + else: > + self.z_step = 0.00 > + self.circle_color_step = ( 0.0, 0.0, 0.0, 0.0 ) > + > + def get_circle_color(self, value): > + return tuple( [self.circle_colors[0][i] + value*self.circle_color_step[i] for i in range(4)] ) > + > + def render(self): > + self.calc_all_extents() > + self.calc_steps() > + self.render_background() > + self.render_bounding_box() > + if self.axis: > + self.render_axis() > + if self.grid: > + self.render_grid() > + self.render_labels() > + self.render_plot() > + if self.errors: > + self.render_errors() > + if self.series_legend and self.series_labels: > + self.render_legend() > + > + def render_axis(self): > + #Draws both the axis lines and their titles > + cr = self.context > + cr.set_source_rgba(*self.line_color) > + cr.move_to(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) > + cr.line_to(self.borders[HORZ], self.borders[VERT]) > + cr.stroke() > + > + cr.move_to(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) > + cr.line_to(self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT]) > + cr.stroke() > + > + cr.set_source_rgba(*self.label_color) > + self.context.set_font_size( 1.2 * self.font_size ) > + if self.titles[HORZ]: > + title_width,title_height = cr.text_extents(self.titles[HORZ])[2:4] > + cr.move_to( self.dimensions[HORZ]/2 - title_width/2, self.borders[VERT] - title_height/2 ) > + cr.show_text( self.titles[HORZ] ) > + > + if self.titles[VERT]: > + title_width,title_height = cr.text_extents(self.titles[VERT])[2:4] > + cr.move_to( self.dimensions[HORZ] - self.borders[HORZ] + title_height/2, self.dimensions[VERT]/2 - title_width/2) > + cr.save() > + cr.rotate( math.pi/2 ) > + cr.show_text( self.titles[VERT] ) > + cr.restore() > + > + def render_grid(self): > + cr = self.context > + horizontal_step = float( self.plot_height ) / ( len( self.labels[VERT] ) - 1 ) > + vertical_step = float( self.plot_width ) / ( len( self.labels[HORZ] ) - 1 ) > + > + x = self.borders[HORZ] + vertical_step > + y = self.plot_top - horizontal_step > + > + for label in self.labels[HORZ][:-1]: > + cr.set_source_rgba(*self.grid_color) > + cr.move_to(x, self.dimensions[VERT] - self.borders[VERT]) > + cr.line_to(x, self.borders[VERT]) > + cr.stroke() > + x += vertical_step > + for label in self.labels[VERT][:-1]: > + cr.set_source_rgba(*self.grid_color) > + cr.move_to(self.borders[HORZ], y) > + cr.line_to(self.dimensions[HORZ] - self.borders[HORZ], y) > + cr.stroke() > + y -= horizontal_step > + > + def render_labels(self): > + self.context.set_font_size(self.font_size * 0.8) > + self.render_horz_labels() > + self.render_vert_labels() > + > + def render_horz_labels(self): > + cr = self.context > + step = float( self.plot_width ) / ( len( self.labels[HORZ] ) - 1 ) > + x = self.borders[HORZ] > + y = self.dimensions[VERT] - self.borders[VERT] + 5 > + > + # store rotation matrix from the initial state > + rotation_matrix = cr.get_matrix() > + rotation_matrix.rotate(self.x_label_angle) > + > + cr.set_source_rgba(*self.label_color) > + > + for item in self.labels[HORZ]: > + width = cr.text_extents(item)[2] > + cr.move_to(x, y) > + cr.save() > + cr.set_matrix(rotation_matrix) > + cr.show_text(item) > + cr.restore() > + x += step > + > + def render_vert_labels(self): > + cr = self.context > + step = ( self.plot_height ) / ( len( self.labels[VERT] ) - 1 ) > + y = self.plot_top > + cr.set_source_rgba(*self.label_color) > + for item in self.labels[VERT]: > + width = cr.text_extents(item)[2] > + cr.move_to(self.borders[HORZ] - width - 5,y) > + cr.show_text(item) > + y -= step > + > + def render_legend(self): > + cr = self.context > + cr.set_font_size(self.font_size) > + cr.set_line_width(self.line_width) > + > + widest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[2]) > + tallest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[3]) > + max_width = self.context.text_extents(widest_word)[2] > + max_height = self.context.text_extents(tallest_word)[3] * 1.1 > + > + color_box_height = max_height / 2 > + color_box_width = color_box_height * 2 > + > + #Draw a bounding box > + bounding_box_width = max_width + color_box_width + 15 > + bounding_box_height = (len(self.series_labels)+0.5) * max_height > + cr.set_source_rgba(1,1,1) > + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - bounding_box_width, self.borders[VERT], > + bounding_box_width, bounding_box_height) > + cr.fill() > + > + cr.set_source_rgba(*self.line_color) > + cr.set_line_width(self.line_width) > + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - bounding_box_width, self.borders[VERT], > + bounding_box_width, bounding_box_height) > + cr.stroke() > + > + for idx,key in enumerate(self.series_labels): > + #Draw color box > + cr.set_source_rgba(*self.series_colors[idx][:4]) > + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - max_width - color_box_width - 10, > + self.borders[VERT] + color_box_height + (idx*max_height) , > + color_box_width, color_box_height) > + cr.fill() > + > + cr.set_source_rgba(0, 0, 0) > + cr.rectangle(self.dimensions[HORZ] - self.borders[HORZ] - max_width - color_box_width - 10, > + self.borders[VERT] + color_box_height + (idx*max_height), > + color_box_width, color_box_height) > + cr.stroke() > + > + #Draw series labels > + cr.set_source_rgba(0, 0, 0) > + cr.move_to(self.dimensions[HORZ] - self.borders[HORZ] - max_width - 5, self.borders[VERT] + ((idx+1)*max_height)) > + cr.show_text(key) > + > + def render_errors(self): > + cr = self.context > + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) > + cr.clip() > + radius = self.dots > + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step > + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step > + for index, group in enumerate(self.series): > + cr.set_source_rgba(*self.series_colors[index][:4]) > + for number, data in enumerate(group): > + x = x0 + self.horizontal_step * data.content[0] > + y = self.dimensions[VERT] - y0 - self.vertical_step * data.content[1] > + if self.errors[HORZ]: > + cr.move_to(x, y) > + x1 = x - self.horizontal_step * self.errors[HORZ][0][number] > + cr.line_to(x1, y) > + cr.line_to(x1, y - radius) > + cr.line_to(x1, y + radius) > + cr.stroke() > + if self.errors[HORZ] and len(self.errors[HORZ]) == 2: > + cr.move_to(x, y) > + x1 = x + self.horizontal_step * self.errors[HORZ][1][number] > + cr.line_to(x1, y) > + cr.line_to(x1, y - radius) > + cr.line_to(x1, y + radius) > + cr.stroke() > + if self.errors[VERT]: > + cr.move_to(x, y) > + y1 = y + self.vertical_step * self.errors[VERT][0][number] > + cr.line_to(x, y1) > + cr.line_to(x - radius, y1) > + cr.line_to(x + radius, y1) > + cr.stroke() > + if self.errors[VERT] and len(self.errors[VERT]) == 2: > + cr.move_to(x, y) > + y1 = y - self.vertical_step * self.errors[VERT][1][number] > + cr.line_to(x, y1) > + cr.line_to(x - radius, y1) > + cr.line_to(x + radius, y1) > + cr.stroke() > + > + > + def render_plot(self): > + cr = self.context > + if self.discrete: > + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) > + cr.clip() > + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step > + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step > + radius = self.dots > + for number, group in enumerate (self.series): > + cr.set_source_rgba(*self.series_colors[number][:4]) > + for data in group : > + if self.variable_radius: > + radius = data.content[2]*self.z_step > + if self.circle_colors: > + cr.set_source_rgba( *self.get_circle_color( data.content[2]) ) > + x = x0 + self.horizontal_step*data.content[0] > + y = y0 + self.vertical_step*data.content[1] > + cr.arc(x, self.dimensions[VERT] - y, radius, 0, 2*math.pi) > + cr.fill() > + else: > + cr.rectangle(self.borders[HORZ], self.borders[VERT], self.plot_width, self.plot_height) > + cr.clip() > + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step > + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step > + radius = self.dots > + for number, group in enumerate (self.series): > + last_data = None > + cr.set_source_rgba(*self.series_colors[number][:4]) > + for data in group : > + x = x0 + self.horizontal_step*data.content[0] > + y = y0 + self.vertical_step*data.content[1] > + if self.dots: > + if self.variable_radius: > + radius = data.content[2]*self.z_step > + cr.arc(x, self.dimensions[VERT] - y, radius, 0, 2*math.pi) > + cr.fill() > + if last_data : > + old_x = x0 + self.horizontal_step*last_data.content[0] > + old_y = y0 + self.vertical_step*last_data.content[1] > + cr.move_to( old_x, self.dimensions[VERT] - old_y ) > + cr.line_to( x, self.dimensions[VERT] - y) > + cr.set_line_width(self.series_widths[number]) > + > + #?Display line as dash line > + if self.dash and self.dash[number]: > + s = self.series_widths[number] > + cr.set_dash([s*3, s*3], 0) > + > + cr.stroke() > + cr.set_dash([]) > + last_data = data > + > +class DotLinePlot(ScatterPlot): > + def __init__(self, > + surface=None, > + data=None, > + width=640, > + height=480, > + background=None, > + border=0, > + axis = False, > + dash = False, > + dots = 0, > + grid = False, > + series_legend = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + x_title = None, > + y_title = None, > + series_colors = None): > + > + ScatterPlot.__init__(self, surface, data, None, None, width, height, background, border, > + axis, dash, False, dots, grid, series_legend, x_labels, y_labels, > + x_bounds, y_bounds, None, x_title, y_title, series_colors, None ) > + > + > + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): > + Plot.load_series(self, data, x_labels, y_labels, series_colors) > + for group in self.series : > + for index,data in enumerate(group): > + group[index].content = (index, data.content) > + > + self.calc_boundaries() > + self.calc_labels() > + > +class FunctionPlot(ScatterPlot): > + def __init__(self, > + surface=None, > + data=None, > + width=640, > + height=480, > + background=None, > + border=0, > + axis = False, > + discrete = False, > + dots = 0, > + grid = False, > + series_legend = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + x_title = None, > + y_title = None, > + series_colors = None, > + step = 1): > + > + self.function = data > + self.step = step > + self.discrete = discrete > + > + data, x_bounds = self.load_series_from_function( self.function, x_bounds ) > + > + ScatterPlot.__init__(self, surface, data, None, None, width, height, background, border, > + axis, False, discrete, dots, grid, series_legend, x_labels, y_labels, > + x_bounds, y_bounds, None, x_title, y_title, series_colors, None ) > + > + def load_series(self, data, x_labels = None, y_labels = None, series_colors=None): > + Plot.load_series(self, data, x_labels, y_labels, series_colors) > + > + if len(self.series[0][0]) is 1: > + for group_id, group in enumerate(self.series) : > + for index,data in enumerate(group): > + group[index].content = (self.bounds[HORZ][0] + self.step*index, data.content) > + > + self.calc_boundaries() > + self.calc_labels() > + > + def load_series_from_function( self, function, x_bounds ): > + #TODO: Add the possibility for the user to define multiple functions with different discretization parameters > + > + #This function converts a function, a list of functions or a dictionary > + #of functions into its corresponding array of data > + series = Series() > + > + if isinstance(function, Group) or isinstance(function, Data): > + function = Series(function) > + > + # If is instance of Series > + if isinstance(function, Series): > + # Overwrite any bounds passed by the function > + x_bounds = (function.range[0],function.range[-1]) > + > + #if no bounds are provided > + if x_bounds == None: > + x_bounds = (0,10) > + > + > + #TODO: Finish the dict translation > + if hasattr(function, "keys"): #dictionary: > + for key in function.keys(): > + group = Group(name=key) > + #data[ key ] = [] > + i = x_bounds[0] > + while i <= x_bounds[1] : > + group.add_data(function[ key ](i)) > + #data[ key ].append( function[ key ](i) ) > + i += self.step > + series.add_group(group) > + > + elif hasattr(function, "__delitem__"): #list of functions > + for index,f in enumerate( function ) : > + group = Group() > + #data.append( [] ) > + i = x_bounds[0] > + while i <= x_bounds[1] : > + group.add_data(f(i)) > + #data[ index ].append( f(i) ) > + i += self.step > + series.add_group(group) > + > + elif isinstance(function, Series): # instance of Series > + series = function > + > + else: #function > + group = Group() > + i = x_bounds[0] > + while i <= x_bounds[1] : > + group.add_data(function(i)) > + i += self.step > + series.add_group(group) > + > + > + return series, x_bounds > + > + def calc_labels(self): > + if not self.labels[HORZ]: > + self.labels[HORZ] = [] > + i = self.bounds[HORZ][0] > + while i<=self.bounds[HORZ][1]: > + self.labels[HORZ].append(str(i)) > + i += float(self.bounds[HORZ][1] - self.bounds[HORZ][0])/10 > + ScatterPlot.calc_labels(self) > + > + def render_plot(self): > + if not self.discrete: > + ScatterPlot.render_plot(self) > + else: > + last = None > + cr = self.context > + for number, group in enumerate (self.series): > + cr.set_source_rgba(*self.series_colors[number][:4]) > + x0 = self.borders[HORZ] - self.bounds[HORZ][0]*self.horizontal_step > + y0 = self.borders[VERT] - self.bounds[VERT][0]*self.vertical_step > + for data in group: > + x = x0 + self.horizontal_step * data.content[0] > + y = y0 + self.vertical_step * data.content[1] > + cr.move_to(x, self.dimensions[VERT] - y) > + cr.line_to(x, self.plot_top) > + cr.set_line_width(self.series_widths[number]) > + cr.stroke() > + if self.dots: > + cr.new_path() > + cr.arc(x, self.dimensions[VERT] - y, 3, 0, 2.1 * math.pi) > + cr.close_path() > + cr.fill() > + > +class BarPlot(Plot): > + def __init__(self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + border = 0, > + display_values = False, > + grid = False, > + rounded_corners = False, > + stack = False, > + three_dimension = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + series_colors = None, > + main_dir = None): > + > + self.bounds = {} > + self.bounds[HORZ] = x_bounds > + self.bounds[VERT] = y_bounds > + self.display_values = display_values > + self.grid = grid > + self.rounded_corners = rounded_corners > + self.stack = stack > + self.three_dimension = three_dimension > + self.x_label_angle = math.pi / 2.5 > + self.main_dir = main_dir > + self.max_value = {} > + self.plot_dimensions = {} > + self.steps = {} > + self.value_label_color = (0.5,0.5,0.5,1.0) > + > + Plot.__init__(self, surface, data, width, height, background, border, x_labels, y_labels, series_colors) > + > + def load_series(self, data, x_labels = None, y_labels = None, series_colors = None): > + Plot.load_series(self, data, x_labels, y_labels, series_colors) > + self.calc_boundaries() > + > + def process_colors(self, series_colors): > + #Data for a BarPlot might be a List or a List of Lists. > + #On the first case, colors must be generated for all bars, > + #On the second, colors must be generated for each of the inner lists. > + > + #TODO: Didn't get it... > + #if hasattr(self.data[0], '__getitem__'): > + # length = max(len(series) for series in self.data) > + #else: > + # length = len( self.data ) > + > + length = max(len(group) for group in self.series) > + > + Plot.process_colors( self, series_colors, length, 'linear') > + > + def calc_boundaries(self): > + if not self.bounds[self.main_dir]: > + if self.stack: > + max_data_value = max(sum(group.to_list()) for group in self.series) > + else: > + max_data_value = max(max(group.to_list()) for group in self.series) > + self.bounds[self.main_dir] = (0, max_data_value) > + if not self.bounds[other_direction(self.main_dir)]: > + self.bounds[other_direction(self.main_dir)] = (0, len(self.series)) > + > + def calc_extents(self, direction): > + self.max_value[direction] = 0 > + if self.labels[direction]: > + widest_word = max(self.labels[direction], key = lambda item: self.context.text_extents(item)[2]) > + self.max_value[direction] = self.context.text_extents(widest_word)[3 - direction] > + self.borders[other_direction(direction)] = (2-direction)*self.max_value[direction] + self.border + direction*(5) > + else: > + self.borders[other_direction(direction)] = self.border > + > + def calc_horz_extents(self): > + self.calc_extents(HORZ) > + > + def calc_vert_extents(self): > + self.calc_extents(VERT) > + > + def calc_all_extents(self): > + self.calc_horz_extents() > + self.calc_vert_extents() > + other_dir = other_direction(self.main_dir) > + self.value_label = 0 > + if self.display_values: > + if self.stack: > + self.value_label = self.context.text_extents(str(max(sum(group.to_list()) for group in self.series)))[2 + self.main_dir] > + else: > + self.value_label = self.context.text_extents(str(max(max(group.to_list()) for group in self.series)))[2 + self.main_dir] > + if self.labels[self.main_dir]: > + self.plot_dimensions[self.main_dir] = self.dimensions[self.main_dir] - 2*self.borders[self.main_dir] - self.value_label > + else: > + self.plot_dimensions[self.main_dir] = self.dimensions[self.main_dir] - self.borders[self.main_dir] - 1.2*self.border - self.value_label > + self.plot_dimensions[other_dir] = self.dimensions[other_dir] - self.borders[other_dir] - self.border > + self.plot_top = self.dimensions[VERT] - self.borders[VERT] > + > + def calc_steps(self): > + other_dir = other_direction(self.main_dir) > + self.series_amplitude = self.bounds[self.main_dir][1] - self.bounds[self.main_dir][0] > + if self.series_amplitude: > + self.steps[self.main_dir] = float(self.plot_dimensions[self.main_dir])/self.series_amplitude > + else: > + self.steps[self.main_dir] = 0.00 > + series_length = len(self.series) > + self.steps[other_dir] = float(self.plot_dimensions[other_dir])/(series_length + 0.1*(series_length + 1)) > + self.space = 0.1*self.steps[other_dir] > + > + def render(self): > + self.calc_all_extents() > + self.calc_steps() > + self.render_background() > + self.render_bounding_box() > + if self.grid: > + self.render_grid() > + if self.three_dimension: > + self.render_ground() > + if self.display_values: > + self.render_values() > + self.render_labels() > + self.render_plot() > + if self.series_labels: > + self.render_legend() > + > + def draw_3d_rectangle_front(self, x0, y0, x1, y1, shift): > + self.context.rectangle(x0-shift, y0+shift, x1-x0, y1-y0) > + > + def draw_3d_rectangle_side(self, x0, y0, x1, y1, shift): > + self.context.move_to(x1-shift,y0+shift) > + self.context.line_to(x1, y0) > + self.context.line_to(x1, y1) > + self.context.line_to(x1-shift, y1+shift) > + self.context.line_to(x1-shift, y0+shift) > + self.context.close_path() > + > + def draw_3d_rectangle_top(self, x0, y0, x1, y1, shift): > + self.context.move_to(x0-shift,y0+shift) > + self.context.line_to(x0, y0) > + self.context.line_to(x1, y0) > + self.context.line_to(x1-shift, y0+shift) > + self.context.line_to(x0-shift, y0+shift) > + self.context.close_path() > + > + def draw_round_rectangle(self, x0, y0, x1, y1): > + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) > + self.context.line_to(x1-5, y0) > + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) > + self.context.line_to(x1, y1-5) > + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) > + self.context.line_to(x0+5, y1) > + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) > + self.context.line_to(x0, y0+5) > + self.context.close_path() > + > + def render_ground(self): > + self.draw_3d_rectangle_front(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], > + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) > + self.context.fill() > + > + self.draw_3d_rectangle_side (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], > + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) > + self.context.fill() > + > + self.draw_3d_rectangle_top (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], > + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) > + self.context.fill() > + > + def render_labels(self): > + self.context.set_font_size(self.font_size * 0.8) > + if self.labels[HORZ]: > + self.render_horz_labels() > + if self.labels[VERT]: > + self.render_vert_labels() > + > + def render_legend(self): > + cr = self.context > + cr.set_font_size(self.font_size) > + cr.set_line_width(self.line_width) > + > + widest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[2]) > + tallest_word = max(self.series_labels, key = lambda item: self.context.text_extents(item)[3]) > + max_width = self.context.text_extents(widest_word)[2] > + max_height = self.context.text_extents(tallest_word)[3] * 1.1 + 5 > + > + color_box_height = max_height / 2 > + color_box_width = color_box_height * 2 > + > + #Draw a bounding box > + bounding_box_width = max_width + color_box_width + 15 > + bounding_box_height = (len(self.series_labels)+0.5) * max_height > + cr.set_source_rgba(1,1,1) > + cr.rectangle(self.dimensions[HORZ] - self.border - bounding_box_width, self.border, > + bounding_box_width, bounding_box_height) > + cr.fill() > + > + cr.set_source_rgba(*self.line_color) > + cr.set_line_width(self.line_width) > + cr.rectangle(self.dimensions[HORZ] - self.border - bounding_box_width, self.border, > + bounding_box_width, bounding_box_height) > + cr.stroke() > + > + for idx,key in enumerate(self.series_labels): > + #Draw color box > + cr.set_source_rgba(*self.series_colors[idx][:4]) > + cr.rectangle(self.dimensions[HORZ] - self.border - max_width - color_box_width - 10, > + self.border + color_box_height + (idx*max_height) , > + color_box_width, color_box_height) > + cr.fill() > + > + cr.set_source_rgba(0, 0, 0) > + cr.rectangle(self.dimensions[HORZ] - self.border - max_width - color_box_width - 10, > + self.border + color_box_height + (idx*max_height), > + color_box_width, color_box_height) > + cr.stroke() > + > + #Draw series labels > + cr.set_source_rgba(0, 0, 0) > + cr.move_to(self.dimensions[HORZ] - self.border - max_width - 5, self.border + ((idx+1)*max_height)) > + cr.show_text(key) > + > + > +class HorizontalBarPlot(BarPlot): > + def __init__(self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + border = 0, > + display_values = False, > + grid = False, > + rounded_corners = False, > + stack = False, > + three_dimension = False, > + series_labels = None, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + series_colors = None): > + > + BarPlot.__init__(self, surface, data, width, height, background, border, > + display_values, grid, rounded_corners, stack, three_dimension, > + x_labels, y_labels, x_bounds, y_bounds, series_colors, HORZ) > + self.series_labels = series_labels > + > + def calc_vert_extents(self): > + self.calc_extents(VERT) > + if self.labels[HORZ] and not self.labels[VERT]: > + self.borders[HORZ] += 10 > + > + def draw_rectangle_bottom(self, x0, y0, x1, y1): > + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) > + self.context.line_to(x0, y0+5) > + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) > + self.context.line_to(x1, y0) > + self.context.line_to(x1, y1) > + self.context.line_to(x0+5, y1) > + self.context.close_path() > + > + def draw_rectangle_top(self, x0, y0, x1, y1): > + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) > + self.context.line_to(x1, y1-5) > + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) > + self.context.line_to(x0, y1) > + self.context.line_to(x0, y0) > + self.context.line_to(x1, y0) > + self.context.close_path() > + > + def draw_rectangle(self, index, length, x0, y0, x1, y1): > + if length == 1: > + BarPlot.draw_rectangle(self, x0, y0, x1, y1) > + elif index == 0: > + self.draw_rectangle_bottom(x0, y0, x1, y1) > + elif index == length-1: > + self.draw_rectangle_top(x0, y0, x1, y1) > + else: > + self.context.rectangle(x0, y0, x1-x0, y1-y0) > + > + #TODO: Review BarPlot.render_grid code > + def render_grid(self): > + self.context.set_source_rgba(0.8, 0.8, 0.8) > + if self.labels[HORZ]: > + self.context.set_font_size(self.font_size * 0.8) > + step = (self.dimensions[HORZ] - 2*self.borders[HORZ] - self.value_label)/(len(self.labels[HORZ])-1) > + x = self.borders[HORZ] > + next_x = 0 > + for item in self.labels[HORZ]: > + width = self.context.text_extents(item)[2] > + if x - width/2 > next_x and x - width/2 > self.border: > + self.context.move_to(x, self.border) > + self.context.line_to(x, self.dimensions[VERT] - self.borders[VERT]) > + self.context.stroke() > + next_x = x + width/2 > + x += step > + else: > + lines = 11 > + horizontal_step = float(self.plot_dimensions[HORZ])/(lines-1) > + x = self.borders[HORZ] > + for y in xrange(0, lines): > + self.context.move_to(x, self.border) > + self.context.line_to(x, self.dimensions[VERT] - self.borders[VERT]) > + self.context.stroke() > + x += horizontal_step > + > + def render_horz_labels(self): > + step = (self.dimensions[HORZ] - 2*self.borders[HORZ])/(len(self.labels[HORZ])-1) > + x = self.borders[HORZ] > + next_x = 0 > + > + for item in self.labels[HORZ]: > + self.context.set_source_rgba(*self.label_color) > + width = self.context.text_extents(item)[2] > + if x - width/2 > next_x and x - width/2 > self.border: > + self.context.move_to(x - width/2, self.dimensions[VERT] - self.borders[VERT] + self.max_value[HORZ] + 3) > + self.context.show_text(item) > + next_x = x + width/2 > + x += step > + > + def render_vert_labels(self): > + series_length = len(self.labels[VERT]) > + step = (self.plot_dimensions[VERT] - (series_length + 1)*self.space)/(len(self.labels[VERT])) > + y = self.border + step/2 + self.space > + > + for item in self.labels[VERT]: > + self.context.set_source_rgba(*self.label_color) > + width, height = self.context.text_extents(item)[2:4] > + self.context.move_to(self.borders[HORZ] - width - 5, y + height/2) > + self.context.show_text(item) > + y += step + self.space > + self.labels[VERT].reverse() > + > + def render_values(self): > + self.context.set_source_rgba(*self.value_label_color) > + self.context.set_font_size(self.font_size * 0.8) > + if self.stack: > + for i,group in enumerate(self.series): > + value = sum(group.to_list()) > + height = self.context.text_extents(str(value))[3] > + x = self.borders[HORZ] + value*self.steps[HORZ] + 2 > + y = self.borders[VERT] + (i+0.5)*self.steps[VERT] + (i+1)*self.space + height/2 > + self.context.move_to(x, y) > + self.context.show_text(str(value)) > + else: > + for i,group in enumerate(self.series): > + inner_step = self.steps[VERT]/len(group) > + y0 = self.border + i*self.steps[VERT] + (i+1)*self.space > + for number,data in enumerate(group): > + height = self.context.text_extents(str(data.content))[3] > + self.context.move_to(self.borders[HORZ] + data.content*self.steps[HORZ] + 2, y0 + 0.5*inner_step + height/2, ) > + self.context.show_text(str(data.content)) > + y0 += inner_step > + > + def render_plot(self): > + if self.stack: > + for i,group in enumerate(self.series): > + x0 = self.borders[HORZ] > + y0 = self.borders[VERT] + i*self.steps[VERT] + (i+1)*self.space > + for number,data in enumerate(group): > + if self.series_colors[number][4] in ('radial','linear') : > + linear = cairo.LinearGradient( data.content*self.steps[HORZ]/2, y0, data.content*self.steps[HORZ]/2, y0 + self.steps[VERT] ) > + color = self.series_colors[number] > + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) > + linear.add_color_stop_rgba(1.0, *color[:4]) > + self.context.set_source(linear) > + elif self.series_colors[number][4] == 'solid': > + self.context.set_source_rgba(*self.series_colors[number][:4]) > + if self.rounded_corners: > + self.draw_rectangle(number, len(group), x0, y0, x0+data.content*self.steps[HORZ], y0+self.steps[VERT]) > + self.context.fill() > + else: > + self.context.rectangle(x0, y0, data.content*self.steps[HORZ], self.steps[VERT]) > + self.context.fill() > + x0 += data.content*self.steps[HORZ] > + else: > + for i,group in enumerate(self.series): > + inner_step = self.steps[VERT]/len(group) > + x0 = self.borders[HORZ] > + y0 = self.border + i*self.steps[VERT] + (i+1)*self.space > + for number,data in enumerate(group): > + linear = cairo.LinearGradient(data.content*self.steps[HORZ]/2, y0, data.content*self.steps[HORZ]/2, y0 + inner_step) > + color = self.series_colors[number] > + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) > + linear.add_color_stop_rgba(1.0, *color[:4]) > + self.context.set_source(linear) > + if self.rounded_corners and data.content != 0: > + BarPlot.draw_round_rectangle(self,x0, y0, x0 + data.content*self.steps[HORZ], y0 + inner_step) > + self.context.fill() > + else: > + self.context.rectangle(x0, y0, data.content*self.steps[HORZ], inner_step) > + self.context.fill() > + y0 += inner_step > + > +class VerticalBarPlot(BarPlot): > + def __init__(self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + border = 0, > + display_values = False, > + grid = False, > + rounded_corners = False, > + stack = False, > + three_dimension = False, > + series_labels = None, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + series_colors = None): > + > + BarPlot.__init__(self, surface, data, width, height, background, border, > + display_values, grid, rounded_corners, stack, three_dimension, > + x_labels, y_labels, x_bounds, y_bounds, series_colors, VERT) > + self.series_labels = series_labels > + > + def calc_vert_extents(self): > + self.calc_extents(VERT) > + if self.labels[VERT] and not self.labels[HORZ]: > + self.borders[VERT] += 10 > + > + def draw_rectangle_bottom(self, x0, y0, x1, y1): > + self.context.move_to(x1,y1) > + self.context.arc(x1-5, y1-5, 5, 0, math.pi/2) > + self.context.line_to(x0+5, y1) > + self.context.arc(x0+5, y1-5, 5, math.pi/2, math.pi) > + self.context.line_to(x0, y0) > + self.context.line_to(x1, y0) > + self.context.line_to(x1, y1) > + self.context.close_path() > + > + def draw_rectangle_top(self, x0, y0, x1, y1): > + self.context.arc(x0+5, y0+5, 5, -math.pi, -math.pi/2) > + self.context.line_to(x1-5, y0) > + self.context.arc(x1-5, y0+5, 5, -math.pi/2, 0) > + self.context.line_to(x1, y1) > + self.context.line_to(x0, y1) > + self.context.line_to(x0, y0) > + self.context.close_path() > + > + def draw_rectangle(self, index, length, x0, y0, x1, y1): > + if length == 1: > + BarPlot.draw_rectangle(self, x0, y0, x1, y1) > + elif index == 0: > + self.draw_rectangle_bottom(x0, y0, x1, y1) > + elif index == length-1: > + self.draw_rectangle_top(x0, y0, x1, y1) > + else: > + self.context.rectangle(x0, y0, x1-x0, y1-y0) > + > + def render_grid(self): > + self.context.set_source_rgba(0.8, 0.8, 0.8) > + if self.labels[VERT]: > + lines = len(self.labels[VERT]) > + vertical_step = float(self.plot_dimensions[self.main_dir])/(lines-1) > + y = self.borders[VERT] + self.value_label > + else: > + lines = 11 > + vertical_step = float(self.plot_dimensions[self.main_dir])/(lines-1) > + y = 1.2*self.border + self.value_label > + for x in xrange(0, lines): > + self.context.move_to(self.borders[HORZ], y) > + self.context.line_to(self.dimensions[HORZ] - self.border, y) > + self.context.stroke() > + y += vertical_step > + > + def render_ground(self): > + self.draw_3d_rectangle_front(self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], > + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) > + self.context.fill() > + > + self.draw_3d_rectangle_side (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], > + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) > + self.context.fill() > + > + self.draw_3d_rectangle_top (self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT], > + self.dimensions[HORZ] - self.borders[HORZ], self.dimensions[VERT] - self.borders[VERT] + 5, 10) > + self.context.fill() > + > + def render_horz_labels(self): > + series_length = len(self.labels[HORZ]) > + step = float (self.plot_dimensions[HORZ] - (series_length + 1)*self.space)/len(self.labels[HORZ]) > + x = self.borders[HORZ] + step/2 + self.space > + next_x = 0 > + > + for item in self.labels[HORZ]: > + self.context.set_source_rgba(*self.label_color) > + width = self.context.text_extents(item)[2] > + if x - width/2 > next_x and x - width/2 > self.borders[HORZ]: > + self.context.move_to(x - width/2, self.dimensions[VERT] - self.borders[VERT] + self.max_value[HORZ] + 3) > + self.context.show_text(item) > + next_x = x + width/2 > + x += step + self.space > + > + def render_vert_labels(self): > + self.context.set_source_rgba(*self.label_color) > + y = self.borders[VERT] + self.value_label > + step = (self.dimensions[VERT] - 2*self.borders[VERT] - self.value_label)/(len(self.labels[VERT]) - 1) > + self.labels[VERT].reverse() > + for item in self.labels[VERT]: > + width, height = self.context.text_extents(item)[2:4] > + self.context.move_to(self.borders[HORZ] - width - 5, y + height/2) > + self.context.show_text(item) > + y += step > + self.labels[VERT].reverse() > + > + def render_values(self): > + self.context.set_source_rgba(*self.value_label_color) > + self.context.set_font_size(self.font_size * 0.8) > + if self.stack: > + for i,group in enumerate(self.series): > + value = sum(group.to_list()) > + width = self.context.text_extents(str(value))[2] > + x = self.borders[HORZ] + (i+0.5)*self.steps[HORZ] + (i+1)*self.space - width/2 > + y = value*self.steps[VERT] + 2 > + self.context.move_to(x, self.plot_top-y) > + self.context.show_text(str(value)) > + else: > + for i,group in enumerate(self.series): > + inner_step = self.steps[HORZ]/len(group) > + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space > + for number,data in enumerate(group): > + width = self.context.text_extents(str(data.content))[2] > + self.context.move_to(x0 + 0.5*inner_step - width/2, self.plot_top - data.content*self.steps[VERT] - 2) > + self.context.show_text(str(data.content)) > + x0 += inner_step > + > + def render_plot(self): > + if self.stack: > + for i,group in enumerate(self.series): > + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space > + y0 = 0 > + for number,data in enumerate(group): > + if self.series_colors[number][4] in ('linear','radial'): > + linear = cairo.LinearGradient( x0, data.content*self.steps[VERT]/2, x0 + self.steps[HORZ], data.content*self.steps[VERT]/2 ) > + color = self.series_colors[number] > + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) > + linear.add_color_stop_rgba(1.0, *color[:4]) > + self.context.set_source(linear) > + elif self.series_colors[number][4] == 'solid': > + self.context.set_source_rgba(*self.series_colors[number][:4]) > + if self.rounded_corners: > + self.draw_rectangle(number, len(group), x0, self.plot_top - y0 - data.content*self.steps[VERT], x0 + self.steps[HORZ], self.plot_top - y0) > + self.context.fill() > + else: > + self.context.rectangle(x0, self.plot_top - y0 - data.content*self.steps[VERT], self.steps[HORZ], data.content*self.steps[VERT]) > + self.context.fill() > + y0 += data.content*self.steps[VERT] > + else: > + for i,group in enumerate(self.series): > + inner_step = self.steps[HORZ]/len(group) > + y0 = self.borders[VERT] > + x0 = self.borders[HORZ] + i*self.steps[HORZ] + (i+1)*self.space > + for number,data in enumerate(group): > + if self.series_colors[number][4] == 'linear': > + linear = cairo.LinearGradient( x0, data.content*self.steps[VERT]/2, x0 + inner_step, data.content*self.steps[VERT]/2 ) > + color = self.series_colors[number] > + linear.add_color_stop_rgba(0.0, 3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) > + linear.add_color_stop_rgba(1.0, *color[:4]) > + self.context.set_source(linear) > + elif self.series_colors[number][4] == 'solid': > + self.context.set_source_rgba(*self.series_colors[number][:4]) > + if self.rounded_corners and data.content != 0: > + BarPlot.draw_round_rectangle(self, x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top) > + self.context.fill() > + elif self.three_dimension: > + self.draw_3d_rectangle_front(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) > + self.context.fill() > + self.draw_3d_rectangle_side(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) > + self.context.fill() > + self.draw_3d_rectangle_top(x0, self.plot_top - data.content*self.steps[VERT], x0+inner_step, self.plot_top, 5) > + self.context.fill() > + else: > + self.context.rectangle(x0, self.plot_top - data.content*self.steps[VERT], inner_step, data.content*self.steps[VERT]) > + self.context.fill() > + > + x0 += inner_step > + > +class StreamChart(VerticalBarPlot): > + def __init__(self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + border = 0, > + grid = False, > + series_legend = None, > + x_labels = None, > + x_bounds = None, > + y_bounds = None, > + series_colors = None): > + > + VerticalBarPlot.__init__(self, surface, data, width, height, background, border, > + False, grid, False, True, False, > + None, x_labels, None, x_bounds, y_bounds, series_colors) > + > + def calc_steps(self): > + other_dir = other_direction(self.main_dir) > + self.series_amplitude = self.bounds[self.main_dir][1] - self.bounds[self.main_dir][0] > + if self.series_amplitude: > + self.steps[self.main_dir] = float(self.plot_dimensions[self.main_dir])/self.series_amplitude > + else: > + self.steps[self.main_dir] = 0.00 > + series_length = len(self.data) > + self.steps[other_dir] = float(self.plot_dimensions[other_dir])/series_length > + > + def render_legend(self): > + pass > + > + def ground(self, index): > + sum_values = sum(self.data[index]) > + return -0.5*sum_values > + > + def calc_angles(self): > + middle = self.plot_top - self.plot_dimensions[VERT]/2.0 > + self.angles = [tuple([0.0 for x in range(len(self.data)+1)])] > + for x_index in range(1, len(self.data)-1): > + t = [] > + x0 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] > + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] > + y0 = middle - self.ground(x_index-1)*self.steps[VERT] > + y2 = middle - self.ground(x_index+1)*self.steps[VERT] > + t.append(math.atan(float(y0-y2)/(x0-x2))) > + for data_index in range(len(self.data[x_index])): > + x0 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] > + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] > + y0 = middle - self.ground(x_index-1)*self.steps[VERT] - self.data[x_index-1][data_index]*self.steps[VERT] > + y2 = middle - self.ground(x_index+1)*self.steps[VERT] - self.data[x_index+1][data_index]*self.steps[VERT] > + > + for i in range(0,data_index): > + y0 -= self.data[x_index-1][i]*self.steps[VERT] > + y2 -= self.data[x_index+1][i]*self.steps[VERT] > + > + if data_index == len(self.data[0])-1 and False: > + self.context.set_source_rgba(0.0,0.0,0.0,0.3) > + self.context.move_to(x0,y0) > + self.context.line_to(x2,y2) > + self.context.stroke() > + self.context.arc(x0,y0,2,0,2*math.pi) > + self.context.fill() > + t.append(math.atan(float(y0-y2)/(x0-x2))) > + self.angles.append(tuple(t)) > + self.angles.append(tuple([0.0 for x in range(len(self.data)+1)])) > + > + def render_plot(self): > + self.calc_angles() > + middle = self.plot_top - self.plot_dimensions[VERT]/2.0 > + p = 0.4*self.steps[HORZ] > + for data_index in range(len(self.data[0])-1,-1,-1): > + self.context.set_source_rgba(*self.series_colors[data_index][:4]) > + > + #draw the upper line > + for x_index in range(len(self.data)-1) : > + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] > + y1 = middle - self.ground(x_index)*self.steps[VERT] - self.data[x_index][data_index]*self.steps[VERT] > + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] > + y2 = middle - self.ground(x_index + 1)*self.steps[VERT] - self.data[x_index + 1][data_index]*self.steps[VERT] > + > + for i in range(0,data_index): > + y1 -= self.data[x_index][i]*self.steps[VERT] > + y2 -= self.data[x_index+1][i]*self.steps[VERT] > + > + if x_index == 0: > + self.context.move_to(x1,y1) > + > + ang1 = self.angles[x_index][data_index+1] > + ang2 = self.angles[x_index+1][data_index+1] + math.pi > + self.context.curve_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1), > + x2+p*math.cos(ang2),y2+p*math.sin(ang2), > + x2,y2) > + > + for x_index in range(len(self.data)-1,0,-1) : > + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] > + y1 = middle - self.ground(x_index)*self.steps[VERT] > + x2 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] > + y2 = middle - self.ground(x_index - 1)*self.steps[VERT] > + > + for i in range(0,data_index): > + y1 -= self.data[x_index][i]*self.steps[VERT] > + y2 -= self.data[x_index-1][i]*self.steps[VERT] > + > + if x_index == len(self.data)-1: > + self.context.line_to(x1,y1+2) > + > + #revert angles by pi degrees to take the turn back > + ang1 = self.angles[x_index][data_index] + math.pi > + ang2 = self.angles[x_index-1][data_index] > + self.context.curve_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1), > + x2+p*math.cos(ang2),y2+p*math.sin(ang2), > + x2,y2+2) > + > + self.context.close_path() > + self.context.fill() > + > + if False: > + self.context.move_to(self.borders[HORZ] + 0.5*self.steps[HORZ], middle) > + for x_index in range(len(self.data)-1) : > + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] > + y1 = middle - self.ground(x_index)*self.steps[VERT] - self.data[x_index][data_index]*self.steps[VERT] > + x2 = self.borders[HORZ] + (0.5 + x_index + 1)*self.steps[HORZ] > + y2 = middle - self.ground(x_index + 1)*self.steps[VERT] - self.data[x_index + 1][data_index]*self.steps[VERT] > + > + for i in range(0,data_index): > + y1 -= self.data[x_index][i]*self.steps[VERT] > + y2 -= self.data[x_index+1][i]*self.steps[VERT] > + > + ang1 = self.angles[x_index][data_index+1] > + ang2 = self.angles[x_index+1][data_index+1] + math.pi > + self.context.set_source_rgba(1.0,0.0,0.0) > + self.context.arc(x1+p*math.cos(ang1),y1+p*math.sin(ang1),2,0,2*math.pi) > + self.context.fill() > + self.context.set_source_rgba(0.0,0.0,0.0) > + self.context.arc(x2+p*math.cos(ang2),y2+p*math.sin(ang2),2,0,2*math.pi) > + self.context.fill() > + '''self.context.set_source_rgba(0.0,0.0,0.0,0.3) > + self.context.arc(x2,y2,2,0,2*math.pi) > + self.context.fill()''' > + self.context.move_to(x1,y1) > + self.context.line_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1)) > + self.context.stroke() > + self.context.move_to(x2,y2) > + self.context.line_to(x2+p*math.cos(ang2),y2+p*math.sin(ang2)) > + self.context.stroke() > + if False: > + for x_index in range(len(self.data)-1,0,-1) : > + x1 = self.borders[HORZ] + (0.5 + x_index)*self.steps[HORZ] > + y1 = middle - self.ground(x_index)*self.steps[VERT] > + x2 = self.borders[HORZ] + (0.5 + x_index - 1)*self.steps[HORZ] > + y2 = middle - self.ground(x_index - 1)*self.steps[VERT] > + > + for i in range(0,data_index): > + y1 -= self.data[x_index][i]*self.steps[VERT] > + y2 -= self.data[x_index-1][i]*self.steps[VERT] > + > + #revert angles by pi degrees to take the turn back > + ang1 = self.angles[x_index][data_index] + math.pi > + ang2 = self.angles[x_index-1][data_index] > + self.context.set_source_rgba(0.0,1.0,0.0) > + self.context.arc(x1+p*math.cos(ang1),y1+p*math.sin(ang1),2,0,2*math.pi) > + self.context.fill() > + self.context.set_source_rgba(0.0,0.0,1.0) > + self.context.arc(x2+p*math.cos(ang2),y2+p*math.sin(ang2),2,0,2*math.pi) > + self.context.fill() > + '''self.context.set_source_rgba(0.0,0.0,0.0,0.3) > + self.context.arc(x2,y2,2,0,2*math.pi) > + self.context.fill()''' > + self.context.move_to(x1,y1) > + self.context.line_to(x1+p*math.cos(ang1),y1+p*math.sin(ang1)) > + self.context.stroke() > + self.context.move_to(x2,y2) > + self.context.line_to(x2+p*math.cos(ang2),y2+p*math.sin(ang2)) > + self.context.stroke() > + #break > + > + #self.context.arc(self.dimensions[HORZ]/2, self.dimensions[VERT]/2,50,0,3*math.pi/2) > + #self.context.fill() > + > + > +class PiePlot(Plot): > + #TODO: Check the old cairoplot, graphs aren't matching > + def __init__ (self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + gradient = False, > + shadow = False, > + colors = None): > + > + Plot.__init__( self, surface, data, width, height, background, series_colors = colors ) > + self.center = (self.dimensions[HORZ]/2, self.dimensions[VERT]/2) > + self.total = sum( self.series.to_list() ) > + self.radius = min(self.dimensions[HORZ]/3,self.dimensions[VERT]/3) > + self.gradient = gradient > + self.shadow = shadow > + > + def sort_function(x,y): > + return x.content - y.content > + > + def load_series(self, data, x_labels=None, y_labels=None, series_colors=None): > + Plot.load_series(self, data, x_labels, y_labels, series_colors) > + # Already done inside series > + #self.data = sorted(self.data) > + > + def draw_piece(self, angle, next_angle): > + self.context.move_to(self.center[0],self.center[1]) > + self.context.line_to(self.center[0] + self.radius*math.cos(angle), self.center[1] + self.radius*math.sin(angle)) > + self.context.arc(self.center[0], self.center[1], self.radius, angle, next_angle) > + self.context.line_to(self.center[0], self.center[1]) > + self.context.close_path() > + > + def render(self): > + self.render_background() > + self.render_bounding_box() > + if self.shadow: > + self.render_shadow() > + self.render_plot() > + self.render_series_labels() > + > + def render_shadow(self): > + horizontal_shift = 3 > + vertical_shift = 3 > + self.context.set_source_rgba(0, 0, 0, 0.5) > + self.context.arc(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.radius, 0, 2*math.pi) > + self.context.fill() > + > + def render_series_labels(self): > + angle = 0 > + next_angle = 0 > + x0,y0 = self.center > + cr = self.context > + for number,key in enumerate(self.series_labels): > + # self.data[number] should be just a number > + data = sum(self.series[number].to_list()) > + > + next_angle = angle + 2.0*math.pi*data/self.total > + cr.set_source_rgba(*self.series_colors[number][:4]) > + w = cr.text_extents(key)[2] > + if (angle + next_angle)/2 < math.pi/2 or (angle + next_angle)/2 > 3*math.pi/2: > + cr.move_to(x0 + (self.radius+10)*math.cos((angle+next_angle)/2), y0 + (self.radius+10)*math.sin((angle+next_angle)/2) ) > + else: > + cr.move_to(x0 + (self.radius+10)*math.cos((angle+next_angle)/2) - w, y0 + (self.radius+10)*math.sin((angle+next_angle)/2) ) > + cr.show_text(key) > + angle = next_angle > + > + def render_plot(self): > + angle = 0 > + next_angle = 0 > + x0,y0 = self.center > + cr = self.context > + for number,group in enumerate(self.series): > + # Group should be just a number > + data = sum(group.to_list()) > + next_angle = angle + 2.0*math.pi*data/self.total > + if self.gradient or self.series_colors[number][4] in ('linear','radial'): > + gradient_color = cairo.RadialGradient(self.center[0], self.center[1], 0, self.center[0], self.center[1], self.radius) > + gradient_color.add_color_stop_rgba(0.3, *self.series_colors[number][:4]) > + gradient_color.add_color_stop_rgba(1, self.series_colors[number][0]*0.7, > + self.series_colors[number][1]*0.7, > + self.series_colors[number][2]*0.7, > + self.series_colors[number][3]) > + cr.set_source(gradient_color) > + else: > + cr.set_source_rgba(*self.series_colors[number][:4]) > + > + self.draw_piece(angle, next_angle) > + cr.fill() > + > + cr.set_source_rgba(1.0, 1.0, 1.0) > + self.draw_piece(angle, next_angle) > + cr.stroke() > + > + angle = next_angle > + > +class DonutPlot(PiePlot): > + def __init__ (self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + gradient = False, > + shadow = False, > + colors = None, > + inner_radius=-1): > + > + Plot.__init__( self, surface, data, width, height, background, series_colors = colors ) > + > + self.center = ( self.dimensions[HORZ]/2, self.dimensions[VERT]/2 ) > + self.total = sum( self.series.to_list() ) > + self.radius = min( self.dimensions[HORZ]/3,self.dimensions[VERT]/3 ) > + self.inner_radius = inner_radius*self.radius > + > + if inner_radius == -1: > + self.inner_radius = self.radius/3 > + > + self.gradient = gradient > + self.shadow = shadow > + > + def draw_piece(self, angle, next_angle): > + self.context.move_to(self.center[0] + (self.inner_radius)*math.cos(angle), self.center[1] + (self.inner_radius)*math.sin(angle)) > + self.context.line_to(self.center[0] + self.radius*math.cos(angle), self.center[1] + self.radius*math.sin(angle)) > + self.context.arc(self.center[0], self.center[1], self.radius, angle, next_angle) > + self.context.line_to(self.center[0] + (self.inner_radius)*math.cos(next_angle), self.center[1] + (self.inner_radius)*math.sin(next_angle)) > + self.context.arc_negative(self.center[0], self.center[1], self.inner_radius, next_angle, angle) > + self.context.close_path() > + > + def render_shadow(self): > + horizontal_shift = 3 > + vertical_shift = 3 > + self.context.set_source_rgba(0, 0, 0, 0.5) > + self.context.arc(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.inner_radius, 0, 2*math.pi) > + self.context.arc_negative(self.center[0] + horizontal_shift, self.center[1] + vertical_shift, self.radius, 0, -2*math.pi) > + self.context.fill() > + > +class GanttChart (Plot) : > + def __init__(self, > + surface = None, > + data = None, > + width = 640, > + height = 480, > + x_labels = None, > + y_labels = None, > + colors = None): > + self.bounds = {} > + self.max_value = {} > + Plot.__init__(self, surface, data, width, height, x_labels = x_labels, y_labels = y_labels, series_colors = colors) > + > + def load_series(self, data, x_labels=None, y_labels=None, series_colors=None): > + Plot.load_series(self, data, x_labels, y_labels, series_colors) > + self.calc_boundaries() > + > + def calc_boundaries(self): > + self.bounds[HORZ] = (0,len(self.series)) > + end_pos = max(self.series.to_list()) > + > + #for group in self.series: > + # if hasattr(item, "__delitem__"): > + # for sub_item in item: > + # end_pos = max(sub_item) > + # else: > + # end_pos = max(item) > + self.bounds[VERT] = (0,end_pos) > + > + def calc_extents(self, direction): > + self.max_value[direction] = 0 > + if self.labels[direction]: > + self.max_value[direction] = max(self.context.text_extents(item)[2] for item in self.labels[direction]) > + else: > + self.max_value[direction] = self.context.text_extents( str(self.bounds[direction][1] + 1) )[2] > + > + def calc_horz_extents(self): > + self.calc_extents(HORZ) > + self.borders[HORZ] = 100 + self.max_value[HORZ] > + > + def calc_vert_extents(self): > + self.calc_extents(VERT) > + self.borders[VERT] = self.dimensions[VERT]/(self.bounds[HORZ][1] + 1) > + > + def calc_steps(self): > + self.horizontal_step = (self.dimensions[HORZ] - self.borders[HORZ])/(len(self.labels[VERT])) > + self.vertical_step = self.borders[VERT] > + > + def render(self): > + self.calc_horz_extents() > + self.calc_vert_extents() > + self.calc_steps() > + self.render_background() > + > + self.render_labels() > + self.render_grid() > + self.render_plot() > + > + def render_background(self): > + cr = self.context > + cr.set_source_rgba(255,255,255) > + cr.rectangle(0,0,self.dimensions[HORZ], self.dimensions[VERT]) > + cr.fill() > + for number,group in enumerate(self.series): > + linear = cairo.LinearGradient(self.dimensions[HORZ]/2, self.borders[VERT] + number*self.vertical_step, > + self.dimensions[HORZ]/2, self.borders[VERT] + (number+1)*self.vertical_step) > + linear.add_color_stop_rgba(0,1.0,1.0,1.0,1.0) > + linear.add_color_stop_rgba(1.0,0.9,0.9,0.9,1.0) > + cr.set_source(linear) > + cr.rectangle(0,self.borders[VERT] + number*self.vertical_step,self.dimensions[HORZ],self.vertical_step) > + cr.fill() > + > + def render_grid(self): > + cr = self.context > + cr.set_source_rgba(0.7, 0.7, 0.7) > + cr.set_dash((1,0,0,0,0,0,1)) > + cr.set_line_width(0.5) > + for number,label in enumerate(self.labels[VERT]): > + h = cr.text_extents(label)[3] > + cr.move_to(self.borders[HORZ] + number*self.horizontal_step, self.vertical_step/2 + h) > + cr.line_to(self.borders[HORZ] + number*self.horizontal_step, self.dimensions[VERT]) > + cr.stroke() > + > + def render_labels(self): > + self.context.set_font_size(0.02 * self.dimensions[HORZ]) > + > + self.render_horz_labels() > + self.render_vert_labels() > + > + def render_horz_labels(self): > + cr = self.context > + labels = self.labels[HORZ] > + if not labels: > + labels = [str(i) for i in range(1, self.bounds[HORZ][1] + 1) ] > + for number,label in enumerate(labels): > + if label != None: > + cr.set_source_rgba(0.5, 0.5, 0.5) > + w,h = cr.text_extents(label)[2], cr.text_extents(label)[3] > + cr.move_to(40,self.borders[VERT] + number*self.vertical_step + self.vertical_step/2 + h/2) > + cr.show_text(label) > + > + def render_vert_labels(self): > + cr = self.context > + labels = self.labels[VERT] > + if not labels: > + labels = [str(i) for i in range(1, self.bounds[VERT][1] + 1) ] > + for number,label in enumerate(labels): > + w,h = cr.text_extents(label)[2], cr.text_extents(label)[3] > + cr.move_to(self.borders[HORZ] + number*self.horizontal_step - w/2, self.vertical_step/2) > + cr.show_text(label) > + > + def render_rectangle(self, x0, y0, x1, y1, color): > + self.draw_shadow(x0, y0, x1, y1) > + self.draw_rectangle(x0, y0, x1, y1, color) > + > + def draw_rectangular_shadow(self, gradient, x0, y0, w, h): > + self.context.set_source(gradient) > + self.context.rectangle(x0,y0,w,h) > + self.context.fill() > + > + def draw_circular_shadow(self, x, y, radius, ang_start, ang_end, mult, shadow): > + gradient = cairo.RadialGradient(x, y, 0, x, y, 2*radius) > + gradient.add_color_stop_rgba(0, 0, 0, 0, shadow) > + gradient.add_color_stop_rgba(1, 0, 0, 0, 0) > + self.context.set_source(gradient) > + self.context.move_to(x,y) > + self.context.line_to(x + mult[0]*radius,y + mult[1]*radius) > + self.context.arc(x, y, 8, ang_start, ang_end) > + self.context.line_to(x,y) > + self.context.close_path() > + self.context.fill() > + > + def draw_rectangle(self, x0, y0, x1, y1, color): > + cr = self.context > + middle = (x0+x1)/2 > + linear = cairo.LinearGradient(middle,y0,middle,y1) > + linear.add_color_stop_rgba(0,3.5*color[0]/5.0, 3.5*color[1]/5.0, 3.5*color[2]/5.0,1.0) > + linear.add_color_stop_rgba(1,*color[:4]) > + cr.set_source(linear) > + > + cr.arc(x0+5, y0+5, 5, 0, 2*math.pi) > + cr.arc(x1-5, y0+5, 5, 0, 2*math.pi) > + cr.arc(x0+5, y1-5, 5, 0, 2*math.pi) > + cr.arc(x1-5, y1-5, 5, 0, 2*math.pi) > + cr.rectangle(x0+5,y0,x1-x0-10,y1-y0) > + cr.rectangle(x0,y0+5,x1-x0,y1-y0-10) > + cr.fill() > + > + def draw_shadow(self, x0, y0, x1, y1): > + shadow = 0.4 > + h_mid = (x0+x1)/2 > + v_mid = (y0+y1)/2 > + h_linear_1 = cairo.LinearGradient(h_mid,y0-4,h_mid,y0+4) > + h_linear_2 = cairo.LinearGradient(h_mid,y1-4,h_mid,y1+4) > + v_linear_1 = cairo.LinearGradient(x0-4,v_mid,x0+4,v_mid) > + v_linear_2 = cairo.LinearGradient(x1-4,v_mid,x1+4,v_mid) > + > + h_linear_1.add_color_stop_rgba( 0, 0, 0, 0, 0) > + h_linear_1.add_color_stop_rgba( 1, 0, 0, 0, shadow) > + h_linear_2.add_color_stop_rgba( 0, 0, 0, 0, shadow) > + h_linear_2.add_color_stop_rgba( 1, 0, 0, 0, 0) > + v_linear_1.add_color_stop_rgba( 0, 0, 0, 0, 0) > + v_linear_1.add_color_stop_rgba( 1, 0, 0, 0, shadow) > + v_linear_2.add_color_stop_rgba( 0, 0, 0, 0, shadow) > + v_linear_2.add_color_stop_rgba( 1, 0, 0, 0, 0) > + > + self.draw_rectangular_shadow(h_linear_1,x0+4,y0-4,x1-x0-8,8) > + self.draw_rectangular_shadow(h_linear_2,x0+4,y1-4,x1-x0-8,8) > + self.draw_rectangular_shadow(v_linear_1,x0-4,y0+4,8,y1-y0-8) > + self.draw_rectangular_shadow(v_linear_2,x1-4,y0+4,8,y1-y0-8) > + > + self.draw_circular_shadow(x0+4, y0+4, 4, math.pi, 3*math.pi/2, (-1,0), shadow) > + self.draw_circular_shadow(x1-4, y0+4, 4, 3*math.pi/2, 2*math.pi, (0,-1), shadow) > + self.draw_circular_shadow(x0+4, y1-4, 4, math.pi/2, math.pi, (0,1), shadow) > + self.draw_circular_shadow(x1-4, y1-4, 4, 0, math.pi/2, (1,0), shadow) > + > + def render_plot(self): > + for index,group in enumerate(self.series): > + for data in group: > + self.render_rectangle(self.borders[HORZ] + data.content[0]*self.horizontal_step, > + self.borders[VERT] + index*self.vertical_step + self.vertical_step/4.0, > + self.borders[HORZ] + data.content[1]*self.horizontal_step, > + self.borders[VERT] + index*self.vertical_step + 3.0*self.vertical_step/4.0, > + self.series_colors[index]) > + > +# Function definition > + > +def scatter_plot(name, > + data = None, > + errorx = None, > + errory = None, > + width = 640, > + height = 480, > + background = "white light_gray", > + border = 0, > + axis = False, > + dash = False, > + discrete = False, > + dots = False, > + grid = False, > + series_legend = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + z_bounds = None, > + x_title = None, > + y_title = None, > + series_colors = None, > + circle_colors = None): > + > + ''' > + - Function to plot scatter data. > + > + - Parameters > + > + data - The values to be ploted might be passed in a two basic: > + list of points: [(0,0), (0,1), (0,2)] or [(0,0,1), (0,1,4), (0,2,1)] > + lists of coordinates: [ [0,0,0] , [0,1,2] ] or [ [0,0,0] , [0,1,2] , [1,4,1] ] > + Notice that these kinds of that can be grouped in order to form more complex data > + using lists of lists or dictionaries; > + series_colors - Define color values for each of the series > + circle_colors - Define a lower and an upper bound for the circle colors for variable radius > + (3 dimensions) series > + ''' > + > + plot = ScatterPlot( name, data, errorx, errory, width, height, background, border, > + axis, dash, discrete, dots, grid, series_legend, x_labels, y_labels, > + x_bounds, y_bounds, z_bounds, x_title, y_title, series_colors, circle_colors ) > + plot.render() > + plot.commit() > + > +def dot_line_plot(name, > + data, > + width, > + height, > + background = "white light_gray", > + border = 0, > + axis = False, > + dash = False, > + dots = False, > + grid = False, > + series_legend = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + x_title = None, > + y_title = None, > + series_colors = None): > + ''' > + - Function to plot graphics using dots and lines. > + > + dot_line_plot (name, data, width, height, background = "white light_gray", border = 0, axis = False, grid = False, x_labels = None, y_labels = None, x_bounds = None, y_bounds = None) > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; > + data - The list, list of lists or dictionary holding the data to be plotted; > + width, height - Dimensions of the output image; > + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. > + If left None, a gray to white gradient will be generated; > + border - Distance in pixels of a square border into which the graphics will be drawn; > + axis - Whether or not the axis are to be drawn; > + dash - Boolean or a list or a dictionary of booleans indicating whether or not the associated series should be drawn in dashed mode; > + dots - Whether or not dots should be drawn on each point; > + grid - Whether or not the gris is to be drawn; > + series_legend - Whether or not the legend is to be drawn; > + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; > + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; > + x_title - Whether or not to plot a title over the x axis. > + y_title - Whether or not to plot a title over the y axis. > + > + - Examples of use > + > + data = [0, 1, 3, 8, 9, 0, 10, 10, 2, 1] > + CairoPlot.dot_line_plot('teste', data, 400, 300) > + > + data = { "john" : [10, 10, 10, 10, 30], "mary" : [0, 0, 3, 5, 15], "philip" : [13, 32, 11, 25, 2] } > + x_labels = ["jan/2008", "feb/2008", "mar/2008", "apr/2008", "may/2008" ] > + CairoPlot.dot_line_plot( 'test', data, 400, 300, axis = True, grid = True, > + series_legend = True, x_labels = x_labels ) > + ''' > + plot = DotLinePlot( name, data, width, height, background, border, > + axis, dash, dots, grid, series_legend, x_labels, y_labels, > + x_bounds, y_bounds, x_title, y_title, series_colors ) > + plot.render() > + plot.commit() > + > +def function_plot(name, > + data, > + width, > + height, > + background = "white light_gray", > + border = 0, > + axis = True, > + dots = False, > + discrete = False, > + grid = False, > + series_legend = False, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + x_title = None, > + y_title = None, > + series_colors = None, > + step = 1): > + > + ''' > + - Function to plot functions. > + > + function_plot(name, data, width, height, background = "white light_gray", border = 0, axis = True, grid = False, dots = False, x_labels = None, y_labels = None, x_bounds = None, y_bounds = None, step = 1, discrete = False) > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; > + data - The list, list of lists or dictionary holding the data to be plotted; > + width, height - Dimensions of the output image; > + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. > + If left None, a gray to white gradient will be generated; > + border - Distance in pixels of a square border into which the graphics will be drawn; > + axis - Whether or not the axis are to be drawn; > + grid - Whether or not the gris is to be drawn; > + dots - Whether or not dots should be shown at each point; > + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; > + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; > + step - the horizontal distance from one point to the other. The smaller, the smoother the curve will be; > + discrete - whether or not the function should be plotted in discrete format. > + > + - Example of use > + > + data = lambda x : x**2 > + CairoPlot.function_plot('function4', data, 400, 300, grid = True, x_bounds=(-10,10), step = 0.1) > + ''' > + > + plot = FunctionPlot( name, data, width, height, background, border, > + axis, discrete, dots, grid, series_legend, x_labels, y_labels, > + x_bounds, y_bounds, x_title, y_title, series_colors, step ) > + plot.render() > + plot.commit() > + > +def pie_plot( name, data, width, height, background = "white light_gray", gradient = False, shadow = False, colors = None ): > + > + ''' > + - Function to plot pie graphics. > + > + pie_plot(name, data, width, height, background = "white light_gray", gradient = False, colors = None) > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; > + data - The list, list of lists or dictionary holding the data to be plotted; > + width, height - Dimensions of the output image; > + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. > + If left None, a gray to white gradient will be generated; > + gradient - Whether or not the pie color will be painted with a gradient; > + shadow - Whether or not there will be a shadow behind the pie; > + colors - List of slices colors. > + > + - Example of use > + > + teste_data = {"john" : 123, "mary" : 489, "philip" : 890 , "suzy" : 235} > + CairoPlot.pie_plot("pie_teste", teste_data, 500, 500) > + ''' > + > + plot = PiePlot( name, data, width, height, background, gradient, shadow, colors ) > + plot.render() > + plot.commit() > + > +def donut_plot(name, data, width, height, background = "white light_gray", gradient = False, shadow = False, colors = None, inner_radius = -1): > + > + ''' > + - Function to plot donut graphics. > + > + donut_plot(name, data, width, height, background = "white light_gray", gradient = False, inner_radius = -1) > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; > + data - The list, list of lists or dictionary holding the data to be plotted; > + width, height - Dimensions of the output image; > + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. > + If left None, a gray to white gradient will be generated; > + shadow - Whether or not there will be a shadow behind the donut; > + gradient - Whether or not the donut color will be painted with a gradient; > + colors - List of slices colors; > + inner_radius - The radius of the donut's inner circle. > + > + - Example of use > + > + teste_data = {"john" : 123, "mary" : 489, "philip" : 890 , "suzy" : 235} > + CairoPlot.donut_plot("donut_teste", teste_data, 500, 500) > + ''' > + > + plot = DonutPlot(name, data, width, height, background, gradient, shadow, colors, inner_radius) > + plot.render() > + plot.commit() > + > +def gantt_chart(name, pieces, width, height, x_labels, y_labels, colors): > + > + ''' > + - Function to generate Gantt Charts. > + > + gantt_chart(name, pieces, width, height, x_labels, y_labels, colors): > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtim; > + pieces - A list defining the spaces to be drawn. The user must pass, for each line, the index of its start and the index of its end. If a line must have two or more spaces, they must be passed inside a list; > + width, height - Dimensions of the output image; > + x_labels - A list of names for each of the vertical lines; > + y_labels - A list of names for each of the horizontal spaces; > + colors - List containing the colors expected for each of the horizontal spaces > + > + - Example of use > + > + pieces = [ (0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,8)] > + x_labels = [ 'teste01', 'teste02', 'teste03', 'teste04'] > + y_labels = [ '0001', '0002', '0003', '0004', '0005', '0006', '0007', '0008', '0009', '0010' ] > + colors = [ (1.0, 0.0, 0.0), (1.0, 0.7, 0.0), (1.0, 1.0, 0.0), (0.0, 1.0, 0.0) ] > + CairoPlot.gantt_chart('gantt_teste', pieces, 600, 300, x_labels, y_labels, colors) > + ''' > + > + plot = GanttChart(name, pieces, width, height, x_labels, y_labels, colors) > + plot.render() > + plot.commit() > + > +def vertical_bar_plot(name, > + data, > + width, > + height, > + background = "white light_gray", > + border = 0, > + display_values = False, > + grid = False, > + rounded_corners = False, > + stack = False, > + three_dimension = False, > + series_labels = None, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + colors = None): > + #TODO: Fix docstring for vertical_bar_plot > + ''' > + - Function to generate vertical Bar Plot Charts. > + > + bar_plot(name, data, width, height, background, border, grid, rounded_corners, three_dimension, > + x_labels, y_labels, x_bounds, y_bounds, colors): > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtime; > + data - The list, list of lists or dictionary holding the data to be plotted; > + width, height - Dimensions of the output image; > + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. > + If left None, a gray to white gradient will be generated; > + border - Distance in pixels of a square border into which the graphics will be drawn; > + grid - Whether or not the gris is to be drawn; > + rounded_corners - Whether or not the bars should have rounded corners; > + three_dimension - Whether or not the bars should be drawn in pseudo 3D; > + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; > + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; > + colors - List containing the colors expected for each of the bars. > + > + - Example of use > + > + data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] > + CairoPlot.vertical_bar_plot ('bar2', data, 400, 300, border = 20, grid = True, rounded_corners = False) > + ''' > + > + plot = VerticalBarPlot(name, data, width, height, background, border, > + display_values, grid, rounded_corners, stack, three_dimension, > + series_labels, x_labels, y_labels, x_bounds, y_bounds, colors) > + plot.render() > + plot.commit() > + > +def horizontal_bar_plot(name, > + data, > + width, > + height, > + background = "white light_gray", > + border = 0, > + display_values = False, > + grid = False, > + rounded_corners = False, > + stack = False, > + three_dimension = False, > + series_labels = None, > + x_labels = None, > + y_labels = None, > + x_bounds = None, > + y_bounds = None, > + colors = None): > + > + #TODO: Fix docstring for horizontal_bar_plot > + ''' > + - Function to generate Horizontal Bar Plot Charts. > + > + bar_plot(name, data, width, height, background, border, grid, rounded_corners, three_dimension, > + x_labels, y_labels, x_bounds, y_bounds, colors): > + > + - Parameters > + > + name - Name of the desired output file, no need to input the .svg as it will be added at runtime; > + data - The list, list of lists or dictionary holding the data to be plotted; > + width, height - Dimensions of the output image; > + background - A 3 element tuple representing the rgb color expected for the background or a new cairo linear gradient. > + If left None, a gray to white gradient will be generated; > + border - Distance in pixels of a square border into which the graphics will be drawn; > + grid - Whether or not the gris is to be drawn; > + rounded_corners - Whether or not the bars should have rounded corners; > + three_dimension - Whether or not the bars should be drawn in pseudo 3D; > + x_labels, y_labels - lists of strings containing the horizontal and vertical labels for the axis; > + x_bounds, y_bounds - tuples containing the lower and upper value bounds for the data to be plotted; > + colors - List containing the colors expected for each of the bars. > + > + - Example of use > + > + data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] > + CairoPlot.bar_plot ('bar2', data, 400, 300, border = 20, grid = True, rounded_corners = False) > + ''' > + > + plot = HorizontalBarPlot(name, data, width, height, background, border, > + display_values, grid, rounded_corners, stack, three_dimension, > + series_labels, x_labels, y_labels, x_bounds, y_bounds, colors) > + plot.render() > + plot.commit() > + > +def stream_chart(name, > + data, > + width, > + height, > + background = "white light_gray", > + border = 0, > + grid = False, > + series_legend = None, > + x_labels = None, > + x_bounds = None, > + y_bounds = None, > + colors = None): > + > + #TODO: Fix docstring for horizontal_bar_plot > + plot = StreamChart(name, data, width, height, background, border, > + grid, series_legend, x_labels, x_bounds, y_bounds, colors) > + plot.render() > + plot.commit() > + > + > +if __name__ == "__main__": > + import tests > + import seriestests > diff --git a/bindings/python/examples/output_format_modules/pprint_table.py b/bindings/python/examples/output_format_modules/pprint_table.py > new file mode 100644 > index 0000000..a7e8255 > --- /dev/null > +++ b/bindings/python/examples/output_format_modules/pprint_table.py > @@ -0,0 +1,37 @@ > +# pprint_table.py > +# > +# This module is used to pretty-print a table > +# Adapted from > +# http://ginstrom.com/scribbles/2007/09/04/pretty-printing-a-table-in-python/ > + > +import sys > + > +def get_max_width(table, index): > + """Get the maximum width of the given column index""" > + > + return max([len(str(row[index])) for row in table]) > + > + > +def pprint_table(table, nbLeft=1, out=sys.stdout): > + """ > + Prints out a table of data, padded for alignment > + @param table: The table to print. A list of lists. > + Each row must have the same number of columns. > + @param nbLeft: The number of columns aligned left > + @param out: Output stream (file-like object) > + """ > + > + col_paddings = [] > + > + for i in range(len(table[0])): > + col_paddings.append(get_max_width(table, i)) > + > + for row in table: > + # left cols > + for i in range(nbLeft): > + print >> out, str(row[i]).ljust(col_paddings[i] + 1), > + # rest of the cols > + for i in range(nbLeft, len(row)): > + col = str(row[i]).rjust(col_paddings[i] + 2) > + print >> out, col, > + print >> out > diff --git a/bindings/python/examples/output_format_modules/series.py b/bindings/python/examples/output_format_modules/series.py > new file mode 100755 > index 0000000..8e8b236 > --- /dev/null > +++ b/bindings/python/examples/output_format_modules/series.py > @@ -0,0 +1,1140 @@ > +#!/usr/bin/env python > +# -*- coding: utf-8 -*- > + > +# Serie.py > +# > +# Copyright (c) 2008 Magnun Leno da Silva > +# > +# Author: Magnun Leno da Silva > +# > +# This program is free software; you can redistribute it and/or > +# modify it under the terms of the GNU Lesser General Public License > +# as published by the Free Software Foundation; either version 2 of > +# the License, or (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU Lesser General Public > +# License along with this program; if not, write to the Free Software > +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 > +# USA > + > +# Contributor: Rodrigo Moreiro Araujo > + > +#import cairoplot > +import doctest > + > +NUMTYPES = (int, float, long) > +LISTTYPES = (list, tuple) > +STRTYPES = (str, unicode) > +FILLING_TYPES = ['linear', 'solid', 'gradient'] > +DEFAULT_COLOR_FILLING = 'solid' > +#TODO: Define default color list > +DEFAULT_COLOR_LIST = None > + > +class Data(object): > + ''' > + Class that models the main data structure. > + It can hold: > + - a number type (int, float or long) > + - a tuple, witch represents a point and can have 2 or 3 items (x,y,z) > + - if a list is passed it will be converted to a tuple. > + > + obs: In case a tuple is passed it will convert to tuple > + ''' > + def __init__(self, data=None, name=None, parent=None): > + ''' > + Starts main atributes from the Data class > + @name - Name for each point; > + @content - The real data, can be an int, float, long or tuple, which > + represents a point (x,y) or (x,y,z); > + @parent - A pointer that give the data access to it's parent. > + > + Usage: > + >>> d = Data(name='empty'); print d > + empty: () > + >>> d = Data((1,1),'point a'); print d > + point a: (1, 1) > + >>> d = Data((1,2,3),'point b'); print d > + point b: (1, 2, 3) > + >>> d = Data([2,3],'point c'); print d > + point c: (2, 3) > + >>> d = Data(12, 'simple value'); print d > + simple value: 12 > + ''' > + # Initial values > + self.__content = None > + self.__name = None > + > + # Setting passed values > + self.parent = parent > + self.name = name > + self.content = data > + > + # Name property > + @apply > + def name(): > + doc = ''' > + Name is a read/write property that controls the input of name. > + - If passed an invalid value it cleans the name with None > + > + Usage: > + >>> d = Data(13); d.name = 'name_test'; print d > + name_test: 13 > + >>> d.name = 11; print d > + 13 > + >>> d.name = 'other_name'; print d > + other_name: 13 > + >>> d.name = None; print d > + 13 > + >>> d.name = 'last_name'; print d > + last_name: 13 > + >>> d.name = ''; print d > + 13 > + ''' > + def fget(self): > + ''' > + returns the name as a string > + ''' > + return self.__name > + > + def fset(self, name): > + ''' > + Sets the name of the Data > + ''' > + if type(name) in STRTYPES and len(name) > 0: > + self.__name = name > + else: > + self.__name = None > + > + > + > + return property(**locals()) > + > + # Content property > + @apply > + def content(): > + doc = ''' > + Content is a read/write property that validate the data passed > + and return it. > + > + Usage: > + >>> d = Data(); d.content = 13; d.content > + 13 > + >>> d = Data(); d.content = (1,2); d.content > + (1, 2) > + >>> d = Data(); d.content = (1,2,3); d.content > + (1, 2, 3) > + >>> d = Data(); d.content = [1,2,3]; d.content > + (1, 2, 3) > + >>> d = Data(); d.content = [1.5,.2,3.3]; d.content > + (1.5, 0.20000000000000001, 3.2999999999999998) > + ''' > + def fget(self): > + ''' > + Return the content of Data > + ''' > + return self.__content > + > + def fset(self, data): > + ''' > + Ensures that data is a valid tuple/list or a number (int, float > + or long) > + ''' > + # Type: None > + if data is None: > + self.__content = None > + return > + > + # Type: Int or Float > + elif type(data) in NUMTYPES: > + self.__content = data > + > + # Type: List or Tuple > + elif type(data) in LISTTYPES: > + # Ensures the correct size > + if len(data) not in (2, 3): > + raise TypeError, "Data (as list/tuple) must have 2 or 3 items" > + return > + > + # Ensures that all items in list/tuple is a number > + isnum = lambda x : type(x) not in NUMTYPES > + > + if max(map(isnum, data)): > + # An item in data isn't an int or a float > + raise TypeError, "All content of data must be a number (int or float)" > + > + # Convert the tuple to list > + if type(data) is list: > + data = tuple(data) > + > + # Append a copy and sets the type > + self.__content = data[:] > + > + # Unknown type! > + else: > + self.__content = None > + raise TypeError, "Data must be an int, float or a tuple with two or three items" > + return > + > + return property(**locals()) > + > + > + def clear(self): > + ''' > + Clear the all Data (content, name and parent) > + ''' > + self.content = None > + self.name = None > + self.parent = None > + > + def copy(self): > + ''' > + Returns a copy of the Data structure > + ''' > + # The copy > + new_data = Data() > + if self.content is not None: > + # If content is a point > + if type(self.content) is tuple: > + new_data.__content = self.content[:] > + > + # If content is a number > + else: > + new_data.__content = self.content > + > + # If it has a name > + if self.name is not None: > + new_data.__name = self.name > + > + return new_data > + > + def __str__(self): > + ''' > + Return a string representation of the Data structure > + ''' > + if self.name is None: > + if self.content is None: > + return '' > + return str(self.content) > + else: > + if self.content is None: > + return self.name+": ()" > + return self.name+": "+str(self.content) > + > + def __len__(self): > + ''' > + Return the length of the Data. > + - If it's a number return 1; > + - If it's a list return it's length; > + - If its None return 0. > + ''' > + if self.content is None: > + return 0 > + elif type(self.content) in NUMTYPES: > + return 1 > + return len(self.content) > + > + > + > + > +class Group(object): > + ''' > + Class that models a group of data. Every value (int, float, long, tuple > + or list) passed is converted to a list of Data. > + It can receive: > + - A single number (int, float, long); > + - A list of numbers; > + - A tuple of numbers; > + - An instance of Data; > + - A list of Data; > + > + Obs: If a tuple with 2 or 3 items is passed it is converted to a point. > + If a tuple with only 1 item is passed it's converted to a number; > + If a tuple with more than 2 items is passed it's converted to a > + list of numbers > + ''' > + def __init__(self, group=None, name=None, parent=None): > + ''' > + Starts main atributes in Group instance. > + @data_list - a list of data which forms the group; > + @range - a range that represent the x axis of possible functions; > + @name - name of the data group; > + @parent - the Serie parent of this group. > + > + Usage: > + >>> g = Group(13, 'simple number'); print g > + simple number ['13'] > + >>> g = Group((1,2), 'simple point'); print g > + simple point ['(1, 2)'] > + >>> g = Group([1,2,3,4], 'list of numbers'); print g > + list of numbers ['1', '2', '3', '4'] > + >>> g = Group((1,2,3,4),'int in tuple'); print g > + int in tuple ['1', '2', '3', '4'] > + >>> g = Group([(1,2),(2,3),(3,4)], 'list of points'); print g > + list of points ['(1, 2)', '(2, 3)', '(3, 4)'] > + >>> g = Group([[1,2,3],[1,2,3]], '2D coordinate lists'); print g > + 2D coordinated lists ['(1, 1)', '(2, 2)', '(3, 3)'] > + >>> g = Group([[1,2],[1,2],[1,2]], '3D coordinate lists'); print g > + 3D coordinated lists ['(1, 1, 1)', '(2, 2, 2)'] > + ''' > + # Initial values > + self.__data_list = [] > + self.__range = [] > + self.__name = None > + > + > + self.parent = parent > + self.name = name > + self.data_list = group > + > + # Name property > + @apply > + def name(): > + doc = ''' > + Name is a read/write property that controls the input of name. > + - If passed an invalid value it cleans the name with None > + > + Usage: > + >>> g = Group(13); g.name = 'name_test'; print g > + name_test ['13'] > + >>> g.name = 11; print g > + ['13'] > + >>> g.name = 'other_name'; print g > + other_name ['13'] > + >>> g.name = None; print g > + ['13'] > + >>> g.name = 'last_name'; print g > + last_name ['13'] > + >>> g.name = ''; print g > + ['13'] > + ''' > + def fget(self): > + ''' > + Returns the name as a string > + ''' > + return self.__name > + > + def fset(self, name): > + ''' > + Sets the name of the Group > + ''' > + if type(name) in STRTYPES and len(name) > 0: > + self.__name = name > + else: > + self.__name = None > + > + return property(**locals()) > + > + # data_list property > + @apply > + def data_list(): > + doc = ''' > + The data_list is a read/write property that can be a list of > + numbers, a list of points or a list of 2 or 3 coordinate lists. This > + property uses mainly the self.add_data method. > + > + Usage: > + >>> g = Group(); g.data_list = 13; print g > + ['13'] > + >>> g.data_list = (1,2); print g > + ['(1, 2)'] > + >>> g.data_list = Data((1,2),'point a'); print g > + ['point a: (1, 2)'] > + >>> g.data_list = [1,2,3]; print g > + ['1', '2', '3'] > + >>> g.data_list = (1,2,3,4); print g > + ['1', '2', '3', '4'] > + >>> g.data_list = [(1,2),(2,3),(3,4)]; print g > + ['(1, 2)', '(2, 3)', '(3, 4)'] > + >>> g.data_list = [[1,2],[1,2]]; print g > + ['(1, 1)', '(2, 2)'] > + >>> g.data_list = [[1,2],[1,2],[1,2]]; print g > + ['(1, 1, 1)', '(2, 2, 2)'] > + >>> g.range = (10); g.data_list = lambda x:x**2; print g > + ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)'] > + ''' > + def fget(self): > + ''' > + Returns the value of data_list > + ''' > + return self.__data_list > + > + def fset(self, group): > + ''' > + Ensures that group is valid. > + ''' > + # None > + if group is None: > + self.__data_list = [] > + > + # Int/float/long or Instance of Data > + elif type(group) in NUMTYPES or isinstance(group, Data): > + # Clean data_list > + self.__data_list = [] > + self.add_data(group) > + > + # One point > + elif type(group) is tuple and len(group) in (2,3): > + self.__data_list = [] > + self.add_data(group) > + > + # list of items > + elif type(group) in LISTTYPES and type(group[0]) is not list: > + # Clean data_list > + self.__data_list = [] > + for item in group: > + # try to append and catch an exception > + self.add_data(item) > + > + # function lambda > + elif callable(group): > + # Explicit is better than implicit > + function = group > + # Has range > + if len(self.range) is not 0: > + # Clean data_list > + self.__data_list = [] > + # Generate values for the lambda function > + for x in self.range: > + #self.add_data((x,round(group(x),2))) > + self.add_data((x,function(x))) > + > + # Only have range in parent > + elif self.parent is not None and len(self.parent.range) is not 0: > + # Copy parent range > + self.__range = self.parent.range[:] > + # Clean data_list > + self.__data_list = [] > + # Generate values for the lambda function > + for x in self.range: > + #self.add_data((x,round(group(x),2))) > + self.add_data((x,function(x))) > + > + # Don't have range anywhere > + else: > + # x_data don't exist > + raise Exception, "Data argument is valid but to use function type please set x_range first" > + > + # Coordinate Lists > + elif type(group) in LISTTYPES and type(group[0]) is list: > + # Clean data_list > + self.__data_list = [] > + data = [] > + if len(group) == 3: > + data = zip(group[0], group[1], group[2]) > + elif len(group) == 2: > + data = zip(group[0], group[1]) > + else: > + raise TypeError, "Only one list of coordinates was received." > + > + for item in data: > + self.add_data(item) > + > + else: > + raise TypeError, "Group type not supported" > + > + return property(**locals()) > + > + @apply > + def range(): > + doc = ''' > + The range is a read/write property that generates a range of values > + for the x axis of the functions. When passed a tuple it almost works > + like the built-in range funtion: > + - 1 item, represent the end of the range started from 0; > + - 2 items, represents the start and the end, respectively; > + - 3 items, the last one represents the step; > + > + When passed a list the range function understands as a valid range. > + > + Usage: > + >>> g = Group(); g.range = 10; print g.range > + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0] > + >>> g = Group(); g.range = (5); print g.range > + [0.0, 1.0, 2.0, 3.0, 4.0] > + >>> g = Group(); g.range = (1,7); print g.range > + [1.0, 2.0, 3.0, 4.0, 5.0, 6.0] > + >>> g = Group(); g.range = (0,10,2); print g.range > + [0.0, 2.0, 4.0, 6.0, 8.0] > + >>> > + >>> g = Group(); g.range = [0]; print g.range > + [0.0] > + >>> g = Group(); g.range = [0,10,20]; print g.range > + [0.0, 10.0, 20.0] > + ''' > + def fget(self): > + ''' > + Returns the range > + ''' > + return self.__range > + > + def fset(self, x_range): > + ''' > + Controls the input of a valid type and generate the range > + ''' > + # if passed a simple number convert to tuple > + if type(x_range) in NUMTYPES: > + x_range = (x_range,) > + > + # A list, just convert to float > + if type(x_range) is list and len(x_range) > 0: > + # Convert all to float > + x_range = map(float, x_range) > + # Prevents repeated values and convert back to list > + self.__range = list(set(x_range[:])) > + # Sort the list to ascending order > + self.__range.sort() > + > + # A tuple, must check the lengths and generate the values > + elif type(x_range) is tuple and len(x_range) in (1,2,3): > + # Convert all to float > + x_range = map(float, x_range) > + > + # Inital values > + start = 0.0 > + step = 1.0 > + end = 0.0 > + > + # Only the end and it can't be less or iqual to 0 > + if len(x_range) is 1 and x_range > 0: > + end = x_range[0] > + > + # The start and the end but the start must be less then the end > + elif len(x_range) is 2 and x_range[0] < x_range[1]: > + start = x_range[0] > + end = x_range[1] > + > + # All 3, but the start must be less then the end > + elif x_range[0] <= x_range[1]: > + start = x_range[0] > + end = x_range[1] > + step = x_range[2] > + > + # Starts the range > + self.__range = [] > + # Generate the range > + # Can't use the range function because it doesn't support float values > + while start < end: > + self.__range.append(start) > + start += step > + > + # Incorrect type > + else: > + raise Exception, "x_range must be a list with one or more items or a tuple with 2 or 3 items" > + > + return property(**locals()) > + > + def add_data(self, data, name=None): > + ''' > + Append a new data to the data_list. > + - If data is an instance of Data, append it > + - If it's an int, float, tuple or list create an instance of Data and append it > + > + Usage: > + >>> g = Group() > + >>> g.add_data(12); print g > + ['12'] > + >>> g.add_data(7,'other'); print g > + ['12', 'other: 7'] > + >>> > + >>> g = Group() > + >>> g.add_data((1,1),'a'); print g > + ['a: (1, 1)'] > + >>> g.add_data((2,2),'b'); print g > + ['a: (1, 1)', 'b: (2, 2)'] > + >>> > + >>> g.add_data(Data((1,2),'c')); print g > + ['a: (1, 1)', 'b: (2, 2)', 'c: (1, 2)'] > + ''' > + if not isinstance(data, Data): > + # Try to convert > + data = Data(data,name,self) > + > + if data.content is not None: > + self.__data_list.append(data.copy()) > + self.__data_list[-1].parent = self > + > + > + def to_list(self): > + ''' > + Returns the group as a list of numbers (int, float or long) or a > + list of tuples (points 2D or 3D). > + > + Usage: > + >>> g = Group([1,2,3,4],'g1'); g.to_list() > + [1, 2, 3, 4] > + >>> g = Group([(1,2),(2,3),(3,4)],'g2'); g.to_list() > + [(1, 2), (2, 3), (3, 4)] > + >>> g = Group([(1,2,3),(3,4,5)],'g2'); g.to_list() > + [(1, 2, 3), (3, 4, 5)] > + ''' > + return [data.content for data in self] > + > + def copy(self): > + ''' > + Returns a copy of this group > + ''' > + new_group = Group() > + new_group.__name = self.__name > + if self.__range is not None: > + new_group.__range = self.__range[:] > + for data in self: > + new_group.add_data(data.copy()) > + return new_group > + > + def get_names(self): > + ''' > + Return a list with the names of all data in this group > + ''' > + names = [] > + for data in self: > + if data.name is None: > + names.append('Data '+str(data.index()+1)) > + else: > + names.append(data.name) > + return names > + > + > + def __str__ (self): > + ''' > + Returns a string representing the Group > + ''' > + ret = "" > + if self.name is not None: > + ret += self.name + " " > + if len(self) > 0: > + list_str = [str(item) for item in self] > + ret += str(list_str) > + else: > + ret += "[]" > + return ret > + > + def __getitem__(self, key): > + ''' > + Makes a Group iterable, based in the data_list property > + ''' > + return self.data_list[key] > + > + def __len__(self): > + ''' > + Returns the length of the Group, based in the data_list property > + ''' > + return len(self.data_list) > + > + > +class Colors(object): > + ''' > + Class that models the colors its labels (names) and its properties, RGB > + and filling type. > + > + It can receive: > + - A list where each item is a list with 3 or 4 items. The > + first 3 items represent the RGB values and the last argument > + defines the filling type. The list will be converted to a dict > + and each color will receve a name based in its position in the > + list. > + - A dictionary where each key will be the color name and its item > + can be a list with 3 or 4 items. The first 3 items represent > + the RGB colors and the last argument defines the filling type. > + ''' > + def __init__(self, color_list=None): > + ''' > + Start the color_list property > + @ color_list - the list or dict contaning the colors properties. > + ''' > + self.__color_list = None > + > + self.color_list = color_list > + > + @apply > + def color_list(): > + doc = ''' > + >>> c = Colors([[1,1,1],[2,2,2,'linear'],[3,3,3,'gradient']]) > + >>> print c.color_list > + {'Color 2': [2, 2, 2, 'linear'], 'Color 3': [3, 3, 3, 'gradient'], 'Color 1': [1, 1, 1, 'solid']} > + >>> c.color_list = [[1,1,1],(2,2,2,'solid'),(3,3,3,'linear')] > + >>> print c.color_list > + {'Color 2': [2, 2, 2, 'solid'], 'Color 3': [3, 3, 3, 'linear'], 'Color 1': [1, 1, 1, 'solid']} > + >>> c.color_list = {'a':[1,1,1],'b':(2,2,2,'solid'),'c':(3,3,3,'linear'), 'd':(4,4,4)} > + >>> print c.color_list > + {'a': [1, 1, 1, 'solid'], 'c': [3, 3, 3, 'linear'], 'b': [2, 2, 2, 'solid'], 'd': [4, 4, 4, 'solid']} > + ''' > + def fget(self): > + ''' > + Return the color list > + ''' > + return self.__color_list > + > + def fset(self, color_list): > + ''' > + Format the color list to a dictionary > + ''' > + if color_list is None: > + self.__color_list = None > + return > + > + if type(color_list) in LISTTYPES and type(color_list[0]) in LISTTYPES: > + old_color_list = color_list[:] > + color_list = {} > + for index, color in enumerate(old_color_list): > + if len(color) is 3 and max(map(type, color)) in NUMTYPES: > + color_list['Color '+str(index+1)] = list(color)+[DEFAULT_COLOR_FILLING] > + elif len(color) is 4 and max(map(type, color[:-1])) in NUMTYPES and color[-1] in FILLING_TYPES: > + color_list['Color '+str(index+1)] = list(color) > + else: > + raise TypeError, "Unsuported color format" > + elif type(color_list) is not dict: > + raise TypeError, "Unsuported color format" > + > + for name, color in color_list.items(): > + if len(color) is 3: > + if max(map(type, color)) in NUMTYPES: > + color_list[name] = list(color)+[DEFAULT_COLOR_FILLING] > + else: > + raise TypeError, "Unsuported color format" > + elif len(color) is 4: > + if max(map(type, color[:-1])) in NUMTYPES and color[-1] in FILLING_TYPES: > + color_list[name] = list(color) > + else: > + raise TypeError, "Unsuported color format" > + self.__color_list = color_list.copy() > + > + return property(**locals()) > + > + > +class Series(object): > + ''' > + Class that models a Series (group of groups). Every value (int, float, > + long, tuple or list) passed is converted to a list of Group or Data. > + It can receive: > + - a single number or point, will be converted to a Group of one Data; > + - a list of numbers, will be converted to a group of numbers; > + - a list of tuples, will converted to a single Group of points; > + - a list of lists of numbers, each 'sublist' will be converted to a > + group of numbers; > + - a list of lists of tuples, each 'sublist' will be converted to a > + group of points; > + - a list of lists of lists, the content of the 'sublist' will be > + processed as coordinated lists and the result will be converted to > + a group of points; > + - a Dictionary where each item can be the same of the list: number, > + point, list of numbers, list of points or list of lists (coordinated > + lists); > + - an instance of Data; > + - an instance of group. > + ''' > + def __init__(self, series=None, name=None, property=[], colors=None): > + ''' > + Starts main atributes in Group instance. > + @series - a list, dict of data of which the series is composed; > + @name - name of the series; > + @property - a list/dict of properties to be used in the plots of > + this Series > + > + Usage: > + >>> print Series([1,2,3,4]) > + ["Group 1 ['1', '2', '3', '4']"] > + >>> print Series([[1,2,3],[4,5,6]]) > + ["Group 1 ['1', '2', '3']", "Group 2 ['4', '5', '6']"] > + >>> print Series((1,2)) > + ["Group 1 ['(1, 2)']"] > + >>> print Series([(1,2),(2,3)]) > + ["Group 1 ['(1, 2)', '(2, 3)']"] > + >>> print Series([[(1,2),(2,3)],[(4,5),(5,6)]]) > + ["Group 1 ['(1, 2)', '(2, 3)']", "Group 2 ['(4, 5)', '(5, 6)']"] > + >>> print Series([[[1,2,3],[1,2,3],[1,2,3]]]) > + ["Group 1 ['(1, 1, 1)', '(2, 2, 2)', '(3, 3, 3)']"] > + >>> print Series({'g1':[1,2,3], 'g2':[4,5,6]}) > + ["g1 ['1', '2', '3']", "g2 ['4', '5', '6']"] > + >>> print Series({'g1':[(1,2),(2,3)], 'g2':[(4,5),(5,6)]}) > + ["g1 ['(1, 2)', '(2, 3)']", "g2 ['(4, 5)', '(5, 6)']"] > + >>> print Series({'g1':[[1,2],[1,2]], 'g2':[[4,5],[4,5]]}) > + ["g1 ['(1, 1)', '(2, 2)']", "g2 ['(4, 4)', '(5, 5)']"] > + >>> print Series(Data(1,'d1')) > + ["Group 1 ['d1: 1']"] > + >>> print Series(Group([(1,2),(2,3)],'g1')) > + ["g1 ['(1, 2)', '(2, 3)']"] > + ''' > + # Intial values > + self.__group_list = [] > + self.__name = None > + self.__range = None > + > + # TODO: Implement colors with filling > + self.__colors = None > + > + self.name = name > + self.group_list = series > + self.colors = colors > + > + # Name property > + @apply > + def name(): > + doc = ''' > + Name is a read/write property that controls the input of name. > + - If passed an invalid value it cleans the name with None > + > + Usage: > + >>> s = Series(13); s.name = 'name_test'; print s > + name_test ["Group 1 ['13']"] > + >>> s.name = 11; print s > + ["Group 1 ['13']"] > + >>> s.name = 'other_name'; print s > + other_name ["Group 1 ['13']"] > + >>> s.name = None; print s > + ["Group 1 ['13']"] > + >>> s.name = 'last_name'; print s > + last_name ["Group 1 ['13']"] > + >>> s.name = ''; print s > + ["Group 1 ['13']"] > + ''' > + def fget(self): > + ''' > + Returns the name as a string > + ''' > + return self.__name > + > + def fset(self, name): > + ''' > + Sets the name of the Group > + ''' > + if type(name) in STRTYPES and len(name) > 0: > + self.__name = name > + else: > + self.__name = None > + > + return property(**locals()) > + > + > + > + # Colors property > + @apply > + def colors(): > + doc = ''' > + >>> s = Series() > + >>> s.colors = [[1,1,1],[2,2,2,'linear'],[3,3,3,'gradient']] > + >>> print s.colors > + {'Color 2': [2, 2, 2, 'linear'], 'Color 3': [3, 3, 3, 'gradient'], 'Color 1': [1, 1, 1, 'solid']} > + >>> s.colors = [[1,1,1],(2,2,2,'solid'),(3,3,3,'linear')] > + >>> print s.colors > + {'Color 2': [2, 2, 2, 'solid'], 'Color 3': [3, 3, 3, 'linear'], 'Color 1': [1, 1, 1, 'solid']} > + >>> s.colors = {'a':[1,1,1],'b':(2,2,2,'solid'),'c':(3,3,3,'linear'), 'd':(4,4,4)} > + >>> print s.colors > + {'a': [1, 1, 1, 'solid'], 'c': [3, 3, 3, 'linear'], 'b': [2, 2, 2, 'solid'], 'd': [4, 4, 4, 'solid']} > + ''' > + def fget(self): > + ''' > + Return the color list > + ''' > + return self.__colors.color_list > + > + def fset(self, colors): > + ''' > + Format the color list to a dictionary > + ''' > + self.__colors = Colors(colors) > + > + return property(**locals()) > + > + @apply > + def range(): > + doc = ''' > + The range is a read/write property that generates a range of values > + for the x axis of the functions. When passed a tuple it almost works > + like the built-in range funtion: > + - 1 item, represent the end of the range started from 0; > + - 2 items, represents the start and the end, respectively; > + - 3 items, the last one represents the step; > + > + When passed a list the range function understands as a valid range. > + > + Usage: > + >>> s = Series(); s.range = 10; print s.range > + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0] > + >>> s = Series(); s.range = (5); print s.range > + [0.0, 1.0, 2.0, 3.0, 4.0, 5.0] > + >>> s = Series(); s.range = (1,7); print s.range > + [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0] > + >>> s = Series(); s.range = (0,10,2); print s.range > + [0.0, 2.0, 4.0, 6.0, 8.0, 10.0] > + >>> > + >>> s = Series(); s.range = [0]; print s.range > + [0.0] > + >>> s = Series(); s.range = [0,10,20]; print s.range > + [0.0, 10.0, 20.0] > + ''' > + def fget(self): > + ''' > + Returns the range > + ''' > + return self.__range > + > + def fset(self, x_range): > + ''' > + Controls the input of a valid type and generate the range > + ''' > + # if passed a simple number convert to tuple > + if type(x_range) in NUMTYPES: > + x_range = (x_range,) > + > + # A list, just convert to float > + if type(x_range) is list and len(x_range) > 0: > + # Convert all to float > + x_range = map(float, x_range) > + # Prevents repeated values and convert back to list > + self.__range = list(set(x_range[:])) > + # Sort the list to ascending order > + self.__range.sort() > + > + # A tuple, must check the lengths and generate the values > + elif type(x_range) is tuple and len(x_range) in (1,2,3): > + # Convert all to float > + x_range = map(float, x_range) > + > + # Inital values > + start = 0.0 > + step = 1.0 > + end = 0.0 > + > + # Only the end and it can't be less or iqual to 0 > + if len(x_range) is 1 and x_range > 0: > + end = x_range[0] > + > + # The start and the end but the start must be lesser then the end > + elif len(x_range) is 2 and x_range[0] < x_range[1]: > + start = x_range[0] > + end = x_range[1] > + > + # All 3, but the start must be lesser then the end > + elif x_range[0] < x_range[1]: > + start = x_range[0] > + end = x_range[1] > + step = x_range[2] > + > + # Starts the range > + self.__range = [] > + # Generate the range > + # Cnat use the range function becouse it don't suport float values > + while start <= end: > + self.__range.append(start) > + start += step > + > + # Incorrect type > + else: > + raise Exception, "x_range must be a list with one or more item or a tuple with 2 or 3 items" > + > + return property(**locals()) > + > + @apply > + def group_list(): > + doc = ''' > + The group_list is a read/write property used to pre-process the list > + of Groups. > + It can be: > + - a single number, point or lambda, will be converted to a single > + Group of one Data; > + - a list of numbers, will be converted to a group of numbers; > + - a list of tuples, will converted to a single Group of points; > + - a list of lists of numbers, each 'sublist' will be converted to > + a group of numbers; > + - a list of lists of tuples, each 'sublist' will be converted to a > + group of points; > + - a list of lists of lists, the content of the 'sublist' will be > + processed as coordinated lists and the result will be converted > + to a group of points; > + - a list of lambdas, each lambda represents a Group; > + - a Dictionary where each item can be the same of the list: number, > + point, list of numbers, list of points, list of lists > + (coordinated lists) or lambdas > + - an instance of Data; > + - an instance of group. > + > + Usage: > + >>> s = Series() > + >>> s.group_list = [1,2,3,4]; print s > + ["Group 1 ['1', '2', '3', '4']"] > + >>> s.group_list = [[1,2,3],[4,5,6]]; print s > + ["Group 1 ['1', '2', '3']", "Group 2 ['4', '5', '6']"] > + >>> s.group_list = (1,2); print s > + ["Group 1 ['(1, 2)']"] > + >>> s.group_list = [(1,2),(2,3)]; print s > + ["Group 1 ['(1, 2)', '(2, 3)']"] > + >>> s.group_list = [[(1,2),(2,3)],[(4,5),(5,6)]]; print s > + ["Group 1 ['(1, 2)', '(2, 3)']", "Group 2 ['(4, 5)', '(5, 6)']"] > + >>> s.group_list = [[[1,2,3],[1,2,3],[1,2,3]]]; print s > + ["Group 1 ['(1, 1, 1)', '(2, 2, 2)', '(3, 3, 3)']"] > + >>> s.group_list = [(0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,9)]; print s > + ["Group 1 ['(0.5, 5.5)']", "Group 2 ['(0, 4)', '(6, 8)']", "Group 3 ['(5.5, 7)']", "Group 4 ['(7, 9)']"] > + >>> s.group_list = {'g1':[1,2,3], 'g2':[4,5,6]}; print s > + ["g1 ['1', '2', '3']", "g2 ['4', '5', '6']"] > + >>> s.group_list = {'g1':[(1,2),(2,3)], 'g2':[(4,5),(5,6)]}; print s > + ["g1 ['(1, 2)', '(2, 3)']", "g2 ['(4, 5)', '(5, 6)']"] > + >>> s.group_list = {'g1':[[1,2],[1,2]], 'g2':[[4,5],[4,5]]}; print s > + ["g1 ['(1, 1)', '(2, 2)']", "g2 ['(4, 4)', '(5, 5)']"] > + >>> s.range = 10 > + >>> s.group_list = lambda x:x*2 > + >>> s.group_list = [lambda x:x*2, lambda x:x**2, lambda x:x**3]; print s > + ["Group 1 ['(0.0, 0.0)', '(1.0, 2.0)', '(2.0, 4.0)', '(3.0, 6.0)', '(4.0, 8.0)', '(5.0, 10.0)', '(6.0, 12.0)', '(7.0, 14.0)', '(8.0, 16.0)', '(9.0, 18.0)', '(10.0, 20.0)']", "Group 2 ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)', '(10.0, 100.0)']", "Group 3 ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 8.0)', '(3.0, 27.0)', '(4.0, 64.0)', '(5.0, 125.0)', '(6.0, 216.0)', '(7.0, 343.0)', '(8.0, 512.0)', '(9.0, 729.0)', '(10.0, 1000.0)']"] > + >>> s.group_list = {'linear':lambda x:x*2, 'square':lambda x:x**2, 'cubic':lambda x:x**3}; print s > + ["cubic ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 8.0)', '(3.0, 27.0)', '(4.0, 64.0)', '(5.0, 125.0)', '(6.0, 216.0)', '(7.0, 343.0)', '(8.0, 512.0)', '(9.0, 729.0)', '(10.0, 1000.0)']", "linear ['(0.0, 0.0)', '(1.0, 2.0)', '(2.0, 4.0)', '(3.0, 6.0)', '(4.0, 8.0)', '(5.0, 10.0)', '(6.0, 12.0)', '(7.0, 14.0)', '(8.0, 16.0)', '(9.0, 18.0)', '(10.0, 20.0)']", "square ['(0.0, 0.0)', '(1.0, 1.0)', '(2.0, 4.0)', '(3.0, 9.0)', '(4.0, 16.0)', '(5.0, 25.0)', '(6.0, 36.0)', '(7.0, 49.0)', '(8.0, 64.0)', '(9.0, 81.0)', '(10.0, 100.0)']"] > + >>> s.group_list = Data(1,'d1'); print s > + ["Group 1 ['d1: 1']"] > + >>> s.group_list = Group([(1,2),(2,3)],'g1'); print s > + ["g1 ['(1, 2)', '(2, 3)']"] > + ''' > + def fget(self): > + ''' > + Return the group list. > + ''' > + return self.__group_list > + > + def fset(self, series): > + ''' > + Controls the input of a valid group list. > + ''' > + #TODO: Add support to the following strem of data: [ (0.5,5.5) , [(0,4),(6,8)] , (5.5,7) , (7,9)] > + > + # Type: None > + if series is None: > + self.__group_list = [] > + > + # List or Tuple > + elif type(series) in LISTTYPES: > + self.__group_list = [] > + > + is_function = lambda x: callable(x) > + # Groups > + if list in map(type, series) or max(map(is_function, series)): > + for group in series: > + self.add_group(group) > + > + # single group > + else: > + self.add_group(series) > + > + #old code > + ## List of numbers > + #if type(series[0]) in NUMTYPES or type(series[0]) is tuple: > + # print series > + # self.add_group(series) > + # > + ## List of anything else > + #else: > + # for group in series: > + # self.add_group(group) > + > + # Dict representing series of groups > + elif type(series) is dict: > + self.__group_list = [] > + names = series.keys() > + names.sort() > + for name in names: > + self.add_group(Group(series[name],name,self)) > + > + # A single lambda > + elif callable(series): > + self.__group_list = [] > + self.add_group(series) > + > + # Int/float, instance of Group or Data > + elif type(series) in NUMTYPES or isinstance(series, Group) or isinstance(series, Data): > + self.__group_list = [] > + self.add_group(series) > + > + # Default > + else: > + raise TypeError, "Serie type not supported" > + > + return property(**locals()) > + > + def add_group(self, group, name=None): > + ''' > + Append a new group in group_list > + ''' > + if not isinstance(group, Group): > + #Try to convert > + group = Group(group, name, self) > + > + if len(group.data_list) is not 0: > + # Auto naming groups > + if group.name is None: > + group.name = "Group "+str(len(self.__group_list)+1) > + > + self.__group_list.append(group) > + self.__group_list[-1].parent = self > + > + def copy(self): > + ''' > + Returns a copy of the Series > + ''' > + new_series = Series() > + new_series.__name = self.__name > + if self.__range is not None: > + new_series.__range = self.__range[:] > + #Add color property in the copy method > + #self.__colors = None > + > + for group in self: > + new_series.add_group(group.copy()) > + > + return new_series > + > + def get_names(self): > + ''' > + Returns a list of the names of all groups in the Serie > + ''' > + names = [] > + for group in self: > + if group.name is None: > + names.append('Group '+str(group.index()+1)) > + else: > + names.append(group.name) > + > + return names > + > + def to_list(self): > + ''' > + Returns a list with the content of all groups and data > + ''' > + big_list = [] > + for group in self: > + for data in group: > + if type(data.content) in NUMTYPES: > + big_list.append(data.content) > + else: > + big_list = big_list + list(data.content) > + return big_list > + > + def __getitem__(self, key): > + ''' > + Makes the Series iterable, based in the group_list property > + ''' > + return self.__group_list[key] > + > + def __str__(self): > + ''' > + Returns a string that represents the Series > + ''' > + ret = "" > + if self.name is not None: > + ret += self.name + " " > + if len(self) > 0: > + list_str = [str(item) for item in self] > + ret += str(list_str) > + else: > + ret += "[]" > + return ret > + > + def __len__(self): > + ''' > + Returns the length of the Series, based in the group_lsit property > + ''' > + return len(self.group_list) > + > + > +if __name__ == '__main__': > + doctest.testmod() > diff --git a/bindings/python/examples/sched_switch.py b/bindings/python/examples/sched_switch.py > new file mode 100644 > index 0000000..7ae834b > --- /dev/null > +++ b/bindings/python/examples/sched_switch.py > @@ -0,0 +1,128 @@ > +# sched_switch.py > +# > +# Babeltrace example script with sched_switch events > +# > +# Copyright 2012 EfficiOS Inc. > +# > +# Author: Danny Serres > +# > +# Permission is hereby granted, free of charge, to any person obtaining a copy > +# of this software and associated documentation files (the "Software"), to deal > +# in the Software without restriction, including without limitation the rights > +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell > +# copies of the Software, and to permit persons to whom the Software is > +# furnished to do so, subject to the following conditions: > +# > +# The above copyright notice and this permission notice shall be included in > +# all copies or substantial portions of the Software. > + > +# The script takes one optional argument (pid) > +# The script will read events based on pid and > +# print the scheduler switches happening with the process. > +# If no arguments are passed, it displays all the scheduler switches. > +# This can be used to understand which tasks schedule out the current > +# process being traced, and when it gets scheduled in again. > +# The trace needs PID context (lttng add-context -k -t pid) > + > +import sys > +from babeltrace import * > + > +if len(sys.argv) < 2 or len(sys.argv) > 3: > + raise TypeError("Usage: python sched_switch.py [pid] path/to/trace") > +elif len(sys.argv) == 3: > + usePID = True > +else: > + usePID = False > + > + > +ctx = Context() > +ret = ctx.add_trace(sys.argv[len(sys.argv)-1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +# Setting iterator > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx, bp) > + > +# Reading events > +event = ctf_it.read_event() > +while event is not None: > + while True: > + if event.get_name() == "sched_switch": > + # Getting scope definition > + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) > + if sco is None: > + print("ERROR: Cannot get definition scope for sched_switch") > + break # Next event > + > + # Getting PID > + pid_field = event.get_field(sco, "_pid") > + pid = pid_field.get_int64() > + > + if ctf.field_error(): > + print("ERROR: Missing PID info for sched_switch") > + break # Next event > + > + if usePID and (pid != long(sys.argv[1])): > + break # Next event > + > + sco = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) > + > + # prev_comm > + field = event.get_field(sco, "_prev_comm") > + prev_comm = field.get_char_array() > + if ctf.field_error(): > + print("ERROR: Missing prev_comm context info") > + > + # prev_tid > + field = event.get_field(sco, "_prev_tid") > + prev_tid = field.get_int64() > + if ctf.field_error(): > + print("ERROR: Missing prev_tid context info") > + > + # prev_prio > + field = event.get_field(sco, "_prev_prio") > + prev_prio = field.get_int64() > + if ctf.field_error(): > + print("ERROR: Missing prev_prio context info") > + > + # prev_state > + field = event.get_field(sco, "_prev_state") > + prev_state = field.get_int64() > + if ctf.field_error(): > + print("ERROR: Missing prev_state context info") > + > + # next_comm > + field = event.get_field(sco, "_next_comm") > + next_comm = field.get_char_array() > + if ctf.field_error(): > + print("ERROR: Missing next_comm context info") > + > + # next_tid > + field = event.get_field(sco, "_next_tid") > + next_tid = field.get_int64() > + if ctf.field_error(): > + print("ERROR: Missing next_tid context info") > + > + # next_prio > + field = event.get_field(sco, "_next_prio") > + next_prio = field.get_int64() > + if ctf.field_error(): > + print("ERROR: Missing next_prio context info") > + > + # Output > + print("sched_switch, pid = {}, TS = {}, prev_comm = {},\n\t" > + "prev_tid = {}, prev_prio = {}, prev_state = {},\n\t" > + "next_comm = {}, next_tid = {}, next_prio = {}".format( > + pid, event.get_timestamp(), prev_comm, prev_tid, > + prev_prio, prev_state, next_comm, next_tid, next_prio)) > + > + break # Next event > + > + # Next event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > +del ctf_it > diff --git a/bindings/python/examples/softirqtimes.py b/bindings/python/examples/softirqtimes.py > new file mode 100644 > index 0000000..903bf3e > --- /dev/null > +++ b/bindings/python/examples/softirqtimes.py > @@ -0,0 +1,153 @@ > +# softirqtimes.py > +# > +# Babeltrace time of softirqs example script > +# > +# Copyright 2012 EfficiOS Inc. > +# > +# Author: Danny Serres > +# > +# Permission is hereby granted, free of charge, to any person obtaining a copy > +# of this software and associated documentation files (the "Software"), to deal > +# in the Software without restriction, including without limitation the rights > +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell > +# copies of the Software, and to permit persons to whom the Software is > +# furnished to do so, subject to the following conditions: > +# > +# The above copyright notice and this permission notice shall be included in > +# all copies or substantial portions of the Software. > + > +# The script checks the number of events in the trace > +# and outputs a table and a .svg histogram for the specified > +# range (microseconds) or the total trace if no range specified. > +# The graph is generated using the cairoplot module. > + > +# The script checks the trace for the amount of time > +# spent from each softirq_raise to softirq_exit. > +# It prints out the min, max (with timestamp), > +# average times, the standard deviation and the total count. > +# Using the cairoplot module, a .svg graph is also outputted > +# showing the taken time in function of the time since the > +# beginning of the trace. > + > +import sys, math > +from output_format_modules import cairoplot > +from babeltrace import * > + > +if len(sys.argv) < 2: > + raise TypeError("Usage: python softirqtimes.py path/to/trace") > + > +ctx = Context() > +ret = ctx.add_trace(sys.argv[1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +time_taken = [] > +graph_data = [] > +max_time = (0.0, 0.0) # (val, ts) > + > +# tmp template: {(cpu_id, vec):TS raise} > +tmp = {} > +largest_val = 0 > + > +# Setting iterator > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx, bp) > + > +# Reading events > +event = ctf_it.read_event() > +start_time = event.get_timestamp() > +while(event is not None): > + > + event_name = event.get_name() > + error = True > + appendNext = False > + > + if event_name == 'softirq_raise' or event_name == 'softirq_exit': > + # Recover cpu_id and vec values to make a key to tmp > + error = False > + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) > + field = event.get_field(scope, "cpu_id") > + cpu_id = field.get_uint64() > + if ctf.field_error(): > + print("ERROR: Missing cpu_id info for {}".format( > + event.get_name())) > + error = True > + > + scope = event.get_top_level_scope(ctf.scope.EVENT_FIELDS) > + field = event.get_field(scope, "_vec") > + vec = field.get_uint64() > + if ctf.field_error(): > + print("ERROR: Missing vec info for {}".format( > + event.get_name())) > + error = True > + key = (cpu_id, vec) > + > + if event_name == 'softirq_raise' and not error: > + # Add timestamp to tmp > + if key in tmp: > + # If key already exists > + i = 0 > + while True: > + # Add index > + key = (cpu_id, vec, i) > + if key in tmp: > + i += 1 > + continue > + if i > largest_val: > + largest_val = i > + break > + > + tmp[key] = event.get_timestamp() > + > + if event_name == 'softirq_exit' and not error: > + # Saving data for output > + # Key check > + if not (key in tmp): > + i = 0 > + while i <= largest_val: > + key = (key[0], key[1], i) > + if key in tmp: > + break > + i += 1 > + > + raise_timestamp = tmp[key] > + time_data = event.get_timestamp() - tmp.pop(key) > + if time_data > max_time[0]: > + # max_time = (val, ts) > + max_time = (time_data, raise_timestamp) > + time_taken.append(time_data) > + graph_data.append((raise_timestamp - start_time, time_data)) > + > + # Next Event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > + > +del ctf_it > + > +# Standard dev. calc. > +try: > + mean = sum(time_taken)/float(len(time_taken)) > +except ZeroDivisionError: > + raise TypeError("empty data") > +deviations_squared = [] > +for x in time_taken: > + deviations_squared.append(math.pow((x - mean), 2)) > +try: > + stddev = math.sqrt(sum(deviations_squared) / (len(deviations_squared) - 1)) > +except ZeroDivisionError: > + stddev = '-' > + > +# Terminal output > +print("AVG TIME: {} ns".format(mean)) > +print("MIN TIME: {} ns".format(min(time_taken))) > +print("MAX TIME: {} ns, TS: {}".format(max_time[0], max_time[1])) > +print("STD DEV: {}".format(stddev)) > +print("TOTAL COUNT: {}".format(len(time_taken))) > + > +# Graph output > +cairoplot.scatter_plot ( 'softirqtimes.svg', data = graph_data, > + width = 5000, height = 4000, border = 20, axis = True, > + grid = True, series_colors = ["red"] ) > diff --git a/bindings/python/examples/syscalls_by_pid.py b/bindings/python/examples/syscalls_by_pid.py > new file mode 100644 > index 0000000..3ae342e > --- /dev/null > +++ b/bindings/python/examples/syscalls_by_pid.py > @@ -0,0 +1,84 @@ > +# syscall_by_pid.py > +# > +# Babeltrace syscall by pid example script > +# > +# Copyright 2012 EfficiOS Inc. > +# > +# Author: Danny Serres > +# > +# Permission is hereby granted, free of charge, to any person obtaining a copy > +# of this software and associated documentation files (the "Software"), to deal > +# in the Software without restriction, including without limitation the rights > +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell > +# copies of the Software, and to permit persons to whom the Software is > +# furnished to do so, subject to the following conditions: > +# > +# The above copyright notice and this permission notice shall be included in > +# all copies or substantial portions of the Software. > + > +# The script checks the number of events in the trace > +# and outputs a table and a .svg histogram for the specified > +# range (microseconds) or the total trace if no range specified. > +# The graph is generated using the cairoplot module. > + > +# The script checks all syscall in the trace and prints a list > +# showing the number of systemcalls executed by each PID > +# ordered from greatest to least number of syscalls. > +# The trace needs PID context (lttng add-context -k -t pid) > + > +import sys > +from babeltrace import * > +from output_format_modules.pprint_table import pprint_table as pprint > + > +if len(sys.argv) < 2 : > + raise TypeError("Usage: python syscalls_by_pid.py path/to/trace") > + > +ctx = Context() > +ret = ctx.add_trace(sys.argv[1], "ctf") > +if ret is None: > + raise IOError("Error adding trace") > + > +data = {} > + > +# Setting iterator > +bp = IterPos(SEEK_BEGIN) > +ctf_it = ctf.Iterator(ctx, bp) > + > +# Reading events > +event = ctf_it.read_event() > +while event is not None: > + if event.get_name().find("sys") >= 0: > + # Getting scope definition > + sco = event.get_top_level_scope(ctf.scope.STREAM_EVENT_CONTEXT) > + if sco is None: > + print("ERROR: Cannot get definition scope for {}".format( > + event.get_name())) > + else: > + # Getting PID > + pid_field = event.get_field(sco, "_pid") > + pid = pid_field.get_int64() > + > + if ctf.field_error(): > + print("ERROR: Missing PID info for sched_switch".format( > + event.get_name())) > + elif pid in data: > + data[pid] += 1 > + else: > + data[pid] = 1 > + # Next event > + ret = ctf_it.next() > + if ret < 0: > + break > + event = ctf_it.read_event() > + > +del ctf_it > + > +# Setting table for output > +table = [] > +for item in data: > + table.append([data[item], item]) # [count, pid] > +table.sort(reverse = True) # [big count first, pid] > +for i in range(len(table)): > + table[i].reverse() # [pid, big count first] > +table.insert(0, ["PID", "SYSCALL COUNT"]) > +pprint(table) > diff --git a/bindings/python/python-complements.c b/bindings/python/python-complements.c > new file mode 100644 > index 0000000..a4ee37e > --- /dev/null > +++ b/bindings/python/python-complements.c > @@ -0,0 +1,121 @@ > +/* > + * python-complements.c > + * > + * Babeltrace Python module complements, required for Python bindings > + * > + * Copyright 2012 EfficiOS Inc. > + * > + * Author: Danny Serres > + * > + * Permission is hereby granted, free of charge, to any person obtaining a copy > + * of this software and associated documentation files (the "Software"), to deal > + * in the Software without restriction, including without limitation the rights > + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell > + * copies of the Software, and to permit persons to whom the Software is > + * furnished to do so, subject to the following conditions: > + * > + * The above copyright notice and this permission notice shall be included in > + * all copies or substantial portions of the Software. > + */ > + > +#include "python-complements.h" > + > +/* FILE functions > + ---------------------------------------------------- > +*/ > + > +FILE *_bt_file_open(char *file_path, char *mode) > +{ > + FILE *fp = stdout; > + if (file_path != NULL) > + fp = fopen(file_path, mode); > + return fp; > +} > + > +void _bt_file_close(FILE *fp) > +{ > + if (fp != NULL) > + fclose(fp); > +} > + > + > +/* List-related functions > + ---------------------------------------------------- > +*/ > + > +/* ctf-field-list */ > +struct definition **_bt_python_field_listcaller( > + const struct bt_ctf_event *ctf_event, > + const struct definition *scope) > +{ > + struct definition **list; > + unsigned int count; > + int ret; > + > + ret = bt_ctf_get_field_list(ctf_event, scope, > + (const struct definition * const **)&list, &count); > + > + if (ret < 0) /* For python to know an error occured */ > + list = NULL; > + else /* For python to know the end is reached */ > + list[count] = NULL; > + > + return list; > +} > + > +struct definition *_bt_python_field_one_from_list( > + struct definition **list, int index) > +{ > + return list[index]; > +} > + > +/* event_decl_list */ > +struct bt_ctf_event_decl **_bt_python_event_decl_listcaller( > + int handle_id, struct bt_context *ctx) > +{ > + struct bt_ctf_event_decl **list; > + unsigned int count; > + int ret; > + > + ret = bt_ctf_get_event_decl_list(handle_id, ctx, > + (struct bt_ctf_event_decl * const **)&list, &count); > + > + if (ret < 0) /* For python to know an error occured */ > + list = NULL; > + else /* For python to know the end is reached */ > + list[count] = NULL; > + > + return list; > +} > + > +struct bt_ctf_event_decl *_bt_python_decl_one_from_list( > + struct bt_ctf_event_decl **list, int index) > +{ > + return list[index]; > +} > + > +/* decl_fields */ > +struct bt_ctf_field_decl **_by_python_field_decl_listcaller( > + struct bt_ctf_event_decl *event_decl, > + enum bt_ctf_scope scope) > +{ > + struct bt_ctf_field_decl **list; > + unsigned int count; > + int ret; > + > + ret = bt_ctf_get_decl_fields(event_decl, scope, > + (const struct bt_ctf_field_decl * const **)&list, &count); > + > + if (ret < 0) /* For python to know an error occured */ > + list = NULL; > + else /* For python to know the end is reached */ > + list[count] = NULL; > + > + return list; > +} > + > +struct bt_ctf_field_decl *_bt_python_field_decl_one_from_list( > + struct bt_ctf_field_decl **list, int index) > +{ > + return list[index]; > +} > diff --git a/bindings/python/python-complements.h b/bindings/python/python-complements.h > new file mode 100644 > index 0000000..9597d70 > --- /dev/null > +++ b/bindings/python/python-complements.h > @@ -0,0 +1,52 @@ > +/* > + * python-complements.h > + * > + * Babeltrace Python module complements header, required for Python bindings > + * > + * Copyright 2012 EfficiOS Inc. > + * > + * Author: Danny Serres > + * > + * Permission is hereby granted, free of charge, to any person obtaining a copy > + * of this software and associated documentation files (the "Software"), to deal > + * in the Software without restriction, including without limitation the rights > + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell > + * copies of the Software, and to permit persons to whom the Software is > + * furnished to do so, subject to the following conditions: > + * > + * The above copyright notice and this permission notice shall be included in > + * all copies or substantial portions of the Software. > + */ > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +/* File */ > +FILE *_bt_file_open(char *file_path, char *mode); > +void _bt_file_close(FILE *fp); > + > +/* ctf-field-list */ > +struct definition **_bt_python_field_listcaller( > + const struct bt_ctf_event *ctf_event, > + const struct definition *scope); > +struct definition *_bt_python_field_one_from_list( > + struct definition **list, int index); > + > +/* event_decl_list */ > +struct bt_ctf_event_decl **_bt_python_event_decl_listcaller( > + int handle_id, struct bt_context *ctx); > +struct bt_ctf_event_decl *_bt_python_decl_one_from_list( > + struct bt_ctf_event_decl **list, int index); > + > +/* decl_fields */ > +struct bt_ctf_field_decl **_by_python_field_decl_listcaller( > + struct bt_ctf_event_decl *event_decl, > + enum bt_ctf_scope scope); > +struct bt_ctf_field_decl *_bt_python_field_decl_one_from_list( > + struct bt_ctf_field_decl **list, int index); > diff --git a/bootstrap b/bootstrap > index c507425..f6926ca 100755 > --- a/bootstrap > +++ b/bootstrap > @@ -4,7 +4,7 @@ set -x > if [ ! -e config ]; then > mkdir config > fi > -aclocal > +aclocal -I m4 > libtoolize --force --copy > autoheader > automake --add-missing --copy > diff --git a/configure.ac b/configure.ac > index d90479d..f9cff9d 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -74,6 +74,41 @@ AC_CHECK_LIB([popt], [poptGetContext], [], > [AC_MSG_ERROR([Cannot find popt.])] > ) > > + > +# For Python > +# SWIG version needed or newer: > +swig_version=2.0.0 > + > +AC_ARG_ENABLE([python], > + [AC_HELP_STRING([--disable-python], > + [do not compile Python bindings])], > + [], [enable_python=yes]) > + > +AM_CONDITIONAL([USE_PYTHON], [test "x${enable_python:-yes}" = xyes]) > + > +if test "x${enable_python:-yes}" = xyes; then > + AC_MSG_NOTICE([You may configure with --disable-python ]dnl > +[if you do not want Python bindings.]) > + > + AX_PKG_SWIG($swig_version, [], [ AC_MSG_ERROR([SWIG $swig_version or newer is needed]) ]) > + AM_PATH_PYTHON > + > + AC_ARG_VAR([PYTHON_INCLUDE], [Include flags for python, bypassing python-config]) > + AC_ARG_VAR([PYTHON_CONFIG], [Path to python-config]) > + AS_IF([test -z "$PYTHON_INCLUDE"], [ > + AS_IF([test -z "$PYTHON_CONFIG"], [ > + AC_PATH_PROGS([PYTHON_CONFIG], > + [python$PYTHON_VERSION-config python-config], > + [no], > + [`dirname $PYTHON`]) > + AS_IF([test "$PYTHON_CONFIG" = no], [AC_MSG_ERROR([cannot find python-config for $PYTHON.])]) > + ]) > + AC_MSG_CHECKING([python include flags]) > + PYTHON_INCLUDE=`$PYTHON_CONFIG --includes` > + AC_MSG_RESULT([$PYTHON_INCLUDE]) > + ]) > +fi > + > pkg_modules="gmodule-2.0 >= 2.0.0" > PKG_CHECK_MODULES(GMODULE, [$pkg_modules]) > AC_SUBST(PACKAGE_LIBS) > @@ -103,6 +138,8 @@ AC_CONFIG_FILES([ > lib/Makefile > lib/prio_heap/Makefile > include/Makefile > + bindings/Makefile > + bindings/python/Makefile > tests/Makefile > ]) > AC_OUTPUT > diff --git a/doc/python-howto.txt b/doc/python-howto.txt > new file mode 100644 > index 0000000..e2ed751 > --- /dev/null > +++ b/doc/python-howto.txt > @@ -0,0 +1,70 @@ > +PYTHON BINDINGS > +---------------- > + > +This is a brief howto for using the Babeltrace Python module. > + > + > +INSTALLATION: > + > +By default, the Python bindings are installed. > +If you do not wish the Python bindings, you can configure with the > +--disable-python option during the installation procedure: > + > + $ ./configure --disable-python > + > +The Python module is automatically generated using SWIG, therefore the > +swig2.0 package on Debian/Ubuntu is requied. > + > + > +USAGE: > + > +Once installed, the Python module can be used by importing it in Python. > +In the Python interpreter: > + > + >>> import babeltrace > + > +Then the starting point is to create a context and add a trace to it. > + > + >>> ctx = babeltrace.Context() > + >>> ctx.add_trace("path/to/trace", ) > + > +Where is a string containing the format name in which the trace > +was produced. To print a list of available formats to the standard > +output, it is possible to use the print_format_list function. > + > + >>> out = babeltrace.File(None) # This returns stdout > + >>> babeltrace.print_format_list(out) > + > +When a trace is added to a context, it is opened and ready to read using > +an iterator. While creating an iterator, optional starting and ending > +position may be specified. So far, only ctf iterator are supported. > + > + >>> begin_pos = babeltrace.IterPos(babeltrace.SEEK_BEGIN) > + >>> iterator = babeltrace.ctf.Iterator(ctx, begin_pos) > + > +From there, it is possible to read the events. > + > + >>> event = iterator.read_event() > + > +It is simple to obtain the timestamp of that event. > + > + >>> timestamp = event.get_timestamp() > + > +Let's say that we want to extract the prev_comm context info for a > +sched_switch event. To do so, it is needed to set an event scope > +with which we can obtain the field wanted. > + > + >>> if event.get_name == "sched_switch": > + ... #prev_comm only for sched_switch events > + ... scope = event.get_top_level_scope(babeltrace.ctf.scope.EVENT_FIELDS) > + ... field = event.get_field(scope, "_prev_comm") > + ... prev_comm = field.get_char_array() > + > +It is also possible to move on to the next event. > + > + >>> ret = iterator.next() # Move the iterator > + >>> if ret == 0: # No error occured > + ... event = iterator.read_event() # Read the next event > + > +For many usage script examples of the Babeltrace Python module, see the > +bindings/python/examples directory. > diff --git a/m4/ax_pkg_swig.m4 b/m4/ax_pkg_swig.m4 > new file mode 100644 > index 0000000..e112f3d > --- /dev/null > +++ b/m4/ax_pkg_swig.m4 > @@ -0,0 +1,135 @@ > +# =========================================================================== > +# http://www.gnu.org/software/autoconf-archive/ax_pkg_swig.html > +# =========================================================================== > +# > +# SYNOPSIS > +# > +# AX_PKG_SWIG([major.minor.micro], [action-if-found], [action-if-not-found]) > +# > +# DESCRIPTION > +# > +# This macro searches for a SWIG installation on your system. If found, > +# then SWIG is AC_SUBST'd; if not found, then $SWIG is empty. If SWIG is > +# found, then SWIG_LIB is set to the SWIG library path, and AC_SUBST'd. > +# > +# You can use the optional first argument to check if the version of the > +# available SWIG is greater than or equal to the value of the argument. It > +# should have the format: N[.N[.N]] (N is a number between 0 and 999. Only > +# the first N is mandatory.) If the version argument is given (e.g. > +# 1.3.17), AX_PKG_SWIG checks that the swig package is this version number > +# or higher. > +# > +# As usual, action-if-found is executed if SWIG is found, otherwise > +# action-if-not-found is executed. > +# > +# In configure.in, use as: > +# > +# AX_PKG_SWIG(1.3.17, [], [ AC_MSG_ERROR([SWIG is required to build..]) ]) > +# AX_SWIG_ENABLE_CXX > +# AX_SWIG_MULTI_MODULE_SUPPORT > +# AX_SWIG_PYTHON > +# > +# LICENSE > +# > +# Copyright (c) 2008 Sebastian Huber > +# Copyright (c) 2008 Alan W. Irwin > +# Copyright (c) 2008 Rafael Laboissiere > +# Copyright (c) 2008 Andrew Collier > +# Copyright (c) 2011 Murray Cumming > +# > +# This program is free software; you can redistribute it and/or modify it > +# under the terms of the GNU General Public License as published by the > +# Free Software Foundation; either version 2 of the License, or (at your > +# option) any later version. > +# > +# This program is distributed in the hope that it will be useful, but > +# WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General > +# Public License for more details. > +# > +# You should have received a copy of the GNU General Public License along > +# with this program. If not, see . > +# > +# As a special exception, the respective Autoconf Macro's copyright owner > +# gives unlimited permission to copy, distribute and modify the configure > +# scripts that are the output of Autoconf when processing the Macro. You > +# need not follow the terms of the GNU General Public License when using > +# or distributing such scripts, even though portions of the text of the > +# Macro appear in them. The GNU General Public License (GPL) does govern > +# all other use of the material that constitutes the Autoconf Macro. > +# > +# This special exception to the GPL applies to versions of the Autoconf > +# Macro released by the Autoconf Archive. When you make and distribute a > +# modified version of the Autoconf Macro, you may extend this special > +# exception to the GPL to apply to your modified version as well. > + > +#serial 8 > + > +AC_DEFUN([AX_PKG_SWIG],[ > + # Ubuntu has swig 2.0 as /usr/bin/swig2.0 > + AC_PATH_PROGS([SWIG],[swig swig2.0]) > + if test -z "$SWIG" ; then > + m4_ifval([$3],[$3],[:]) > + elif test -n "$1" ; then > + AC_MSG_CHECKING([SWIG version]) > + [swig_version=`$SWIG -version 2>&1 | grep 'SWIG Version' | sed 's/.*\([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\).*/\1/g'`] > + AC_MSG_RESULT([$swig_version]) > + if test -n "$swig_version" ; then > + # Calculate the required version number components > + [required=$1] > + [required_major=`echo $required | sed 's/[^0-9].*//'`] > + if test -z "$required_major" ; then > + [required_major=0] > + fi > + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] > + [required_minor=`echo $required | sed 's/[^0-9].*//'`] > + if test -z "$required_minor" ; then > + [required_minor=0] > + fi > + [required=`echo $required | sed 's/[0-9]*[^0-9]//'`] > + [required_patch=`echo $required | sed 's/[^0-9].*//'`] > + if test -z "$required_patch" ; then > + [required_patch=0] > + fi > + # Calculate the available version number components > + [available=$swig_version] > + [available_major=`echo $available | sed 's/[^0-9].*//'`] > + if test -z "$available_major" ; then > + [available_major=0] > + fi > + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] > + [available_minor=`echo $available | sed 's/[^0-9].*//'`] > + if test -z "$available_minor" ; then > + [available_minor=0] > + fi > + [available=`echo $available | sed 's/[0-9]*[^0-9]//'`] > + [available_patch=`echo $available | sed 's/[^0-9].*//'`] > + if test -z "$available_patch" ; then > + [available_patch=0] > + fi > + # Convert the version tuple into a single number for easier comparison. > + # Using base 100 should be safe since SWIG internally uses BCD values > + # to encode its version number. > + required_swig_vernum=`expr $required_major \* 10000 \ > + \+ $required_minor \* 100 \+ $required_patch` > + available_swig_vernum=`expr $available_major \* 10000 \ > + \+ $available_minor \* 100 \+ $available_patch` > + > + if test $available_swig_vernum -lt $required_swig_vernum; then > + AC_MSG_WARN([SWIG version >= $1 is required. You have $swig_version.]) > + SWIG='' > + m4_ifval([$3],[$3],[]) > + else > + AC_MSG_CHECKING([for SWIG library]) > + SWIG_LIB=`$SWIG -swiglib` > + AC_MSG_RESULT([$SWIG_LIB]) > + m4_ifval([$2],[$2],[]) > + fi > + else > + AC_MSG_WARN([cannot determine SWIG version]) > + SWIG='' > + m4_ifval([$3],[$3],[]) > + fi > + fi > + AC_SUBST([SWIG_LIB]) > +]) > diff --git a/tests/tests-python.py b/tests/tests-python.py > new file mode 100644 > index 0000000..0bd71c2 > --- /dev/null > +++ b/tests/tests-python.py > @@ -0,0 +1,115 @@ > +import unittest > +import sys > +from babeltrace import * > + > +class TestBabeltracePythonModule(unittest.TestCase): > + > + def test_handle_decl(self): > + #Context creation, adding trace > + ctx = Context() > + trace_handle = ctx.add_trace( > + "ctf-traces/succeed/lttng-modules-2.0-pre5", > + "ctf") > + self.assertIsNotNone(trace_handle, "Error adding trace") > + > + #TraceHandle test > + ts_begin = trace_handle.get_timestamp_begin(ctx, CLOCK_REAL) > + ts_end = trace_handle.get_timestamp_end(ctx, CLOCK_REAL) > + self.assertGreater(ts_end, ts_begin, "Error get_timestamp from trace_handle") > + > + lst = ctf.get_event_decl_list(trace_handle, ctx) > + self.assertIsNotNone(lst, "Error get_event_decl_list") > + > + name = lst[0].get_name() > + self.assertIsNotNone(name) > + > + fields = lst[0].get_decl_fields(ctf.scope.EVENT_FIELDS) > + self.assertIsNotNone(fields, "Error getting FieldDecl list") > + > + if len(fields) > 0: > + self.assertIsNotNone(fields[0].get_name(), > + "Error getting name from FieldDecl") > + > + #Remove trace > + ctx.remove_trace(trace_handle) > + del ctx > + del trace_handle > + > + > + def test_iterator_event(self): > + #Context creation, adding trace > + ctx = Context() > + trace_handle = ctx.add_trace( > + "ctf-traces/succeed/lttng-modules-2.0-pre5", > + "ctf") > + self.assertIsNotNone(trace_handle, "Error adding trace") > + > + begin_pos = IterPos(SEEK_BEGIN) > + it = ctf.Iterator(ctx, begin_pos) > + self.assertIsNotNone(it, "Error creating iterator") > + > + event = it.read_event() > + self.assertIsNotNone(event, "Error reading event") > + > + handle = event.get_handle() > + self.assertIsNotNone(handle, "Error getting handle") > + > + context = event.get_context() > + self.assertIsNotNone(context, "Error getting context") > + > + name = "" > + while event is not None and name != "sched_switch": > + name = event.get_name() > + self.assertIsNotNone(name, "Error getting event name") > + > + ts = event.get_timestamp() > + self.assertGreaterEqual(ts, 0, "Error getting timestamp") > + > + cycles = event.get_cycles() > + self.assertGreaterEqual(cycles, 0, "Error getting cycles") > + > + if name == "sched_switch": > + scope = event.get_top_level_scope(ctf.scope.STREAM_PACKET_CONTEXT) > + self.assertIsNotNone(scope, "Error getting scope definition") > + > + field = event.get_field(scope, "cpu_id") > + prev_comm_str = field.get_uint64() > + self.assertEqual(ctf.field_error(), 0, "Error getting vec info") > + > + field_lst = event.get_field_list(scope) > + self.assertIsNotNone(field_lst, "Error getting field list") > + > + fname = field.field_name() > + self.assertIsNotNone(fname, "Error getting field name") > + > + ftype = field.field_type() > + self.assertIsNot(ftype, -1, "Error getting field type (unknown)") > + > + ret = it.next() > + self.assertGreaterEqual(ret, 0, "Error moving iterator") > + > + event = it.read_event() > + > + pos = it.get_pos() > + self.assertIsNotNone(pos._pos, "Error getting iterator position") > + > + ret = it.set_pos(pos) > + self.assertEqual(ret, 0, "Error setting iterator position") > + > + pos = it.create_time_pos(ts) > + self.assertIsNotNone(pos._pos, "Error creating time-based iterator position") > + > + del it, pos > + set_pos = IterPos(SEEK_TIME, ts) > + it = ctf.Iterator(ctx, begin_pos, set_pos) > + self.assertIsNotNone(it, "Error creating iterator with end_pos") > + > + > + def test_file_class(self): > + f = File("test-bitfield.c") > + self.assertIsNotNone(f, "Error opening file") > + f.close() > + > + > +if __name__ == "__main__": > + unittest.main() > -- > 1.7.9.5 > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From eag0628 at gmail.com Fri Aug 10 21:46:07 2012 From: eag0628 at gmail.com (Lai Jiangshan) Date: Sat, 11 Aug 2012 09:46:07 +0800 Subject: [lttng-dev] [PATCH 2/2] urcu: new wfqueue implementation In-Reply-To: <20120810182840.GA31156@Krystal> References: <1344501986-23151-1-git-send-email-laijs@cn.fujitsu.com> <1344501986-23151-2-git-send-email-laijs@cn.fujitsu.com> <20120810182840.GA31156@Krystal> Message-ID: On Sat, Aug 11, 2012 at 2:28 AM, Mathieu Desnoyers wrote: > * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: >> Some guys would be surprised by this fact: >> There are already TWO implementations of wfqueue in urcu. >> >> The first one is in urcu/static/wfqueue.h: >> 1) enqueue: exchange the tail and then update previous->next >> 2) dequeue: wait for first node's next pointer and them shift, a dummy node >> is introduced to avoid the queue->tail become NULL when shift. >> >> The second one shares some code with the first one, and the left code >> are spreading in urcu-call-rcu-impl.h: >> 1) enqueue: share with the first one >> 2) no dequeue operation: and no shift, so it don't need dummy node, >> Although the dummy node is queued when initialization, but it is removed >> after the first dequeue_all operation in call_rcu_thread(). >> call_rcu_data_free() forgets to handle the dummy node if it is not removed. >> 3)dequeue_all: record the old head and tail, and queue->head become the special >> tail node.(atomic record the tail and change the tail). >> >> The second implementation's code are spreading, bad for review, and it is not >> tested by tests/test_urcu_wfq. >> >> So we need a better implementation avoid the dummy node dancing and can service >> both generic wfqueue APIs and dequeue_all API for call rcu. >> >> The new implementation: >> 1) enqueue: share with the first one/original implementation. >> 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. >> no dummy node, save memory. >> 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail >> and return the old head.next. > > Hi Lai, > > Your approach is very interesting! It is indeed good for testing and > maintenance if we can keep all the queue code within the API. > > I am concerned about the following scenario in your new implementation, > I would like to know your thoughts on this. It could happen on > architectures reordering loads (DEC Alpha, AMD64, IA64, PA-RISC, POWER, > SPARC RMO, x86 TSO, and x86 OOStore): > > init state: list is empty > > CPU 0 CPU 1 > > ___cds_wfq_append_list() (append newtail) > oldtail = uatomic_xchg(&q->tail, newtail); (A) > CMM_STORE_SHARED(oldtail->next, head); (B) > > (B) is observable by cpu 1, but not (A) yet > > ___cds_wfq_dequeue_blocking() > _cds_wfq_empty(q) > return q->head.next == NULL > && CMM_LOAD_SHARED(q->tail) == &q->head; > -> false (q->tail != &q->head) > node = ___cds_wfq_node_sync_next(&q->head); > -> node is newtail > if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { > -> taken, newtail->next is indeed NULL > * (see note below) > if (CMM_LOAD_SHARED(q->tail) == node) { > -> not taken, since q->tail still appears as &q->head > } > next = ___cds_wfq_node_sync_next(node); > -> endless loop if no other enqueue is performed. (BUG) > } > > (A) is observable by cpu 1 > > * note: I think we should add a cmm_smp_rmb() here to fix this issue. It > would force CPU 1 to necessarily see store (A) if store (B) is seen. > This would be matching the full memory barrier implied after > uatomic_xchg(). > You are right. Can you add this line of code after merge this patch if there is no other problem. Lai > Thanks, > > Mathieu > >> >> More implementation details are in the code. >> tests/test_urcu_wfq will be update in future for testing new APIs. >> >> >> Signed-off-by: Lai Jiangshan >> --- >> urcu-call-rcu-impl.h | 50 ++++++++++-------------- >> urcu/static/wfqueue.h | 104 ++++++++++++++++++++++++++++++++++++------------ >> urcu/wfqueue.h | 25 ++++++++++-- >> wfqueue.c | 29 ++++++++++++++ >> 4 files changed, 149 insertions(+), 59 deletions(-) >> >> diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h >> index 13b24ff..dbfb410 100644 >> --- a/urcu-call-rcu-impl.h >> +++ b/urcu-call-rcu-impl.h >> @@ -221,7 +221,7 @@ static void *call_rcu_thread(void *arg) >> { >> unsigned long cbcount; >> struct cds_wfq_node *cbs; >> - struct cds_wfq_node **cbs_tail; >> + struct cds_wfq_node *cbs_tail; >> struct call_rcu_data *crdp = (struct call_rcu_data *)arg; >> struct rcu_head *rhp; >> int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); >> @@ -243,24 +243,18 @@ static void *call_rcu_thread(void *arg) >> cmm_smp_mb(); >> } >> for (;;) { >> - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { >> - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) >> - poll(NULL, 0, 1); >> - _CMM_STORE_SHARED(crdp->cbs.head, NULL); >> - cbs_tail = (struct cds_wfq_node **) >> - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); >> + cbs = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &cbs_tail); >> + if (cbs) { >> synchronize_rcu(); >> cbcount = 0; >> do { >> - while (cbs->next == NULL && >> - &cbs->next != cbs_tail) >> - poll(NULL, 0, 1); >> - if (cbs == &crdp->cbs.dummy) { >> - cbs = cbs->next; >> - continue; >> - } >> rhp = (struct rcu_head *)cbs; >> - cbs = cbs->next; >> + >> + if (cbs != cbs_tail) >> + cbs = __cds_wfq_node_sync_next(cbs); >> + else >> + cbs = NULL; >> + >> rhp->func(rhp); >> cbcount++; >> } while (cbs != NULL); >> @@ -270,8 +264,7 @@ static void *call_rcu_thread(void *arg) >> break; >> rcu_thread_offline(); >> if (!rt) { >> - if (&crdp->cbs.head >> - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { >> + if (cds_wfq_empty(&crdp->cbs)) { >> call_rcu_wait(crdp); >> poll(NULL, 0, 10); >> uatomic_dec(&crdp->futex); >> @@ -625,32 +618,31 @@ void call_rcu(struct rcu_head *head, >> */ >> void call_rcu_data_free(struct call_rcu_data *crdp) >> { >> - struct cds_wfq_node *cbs; >> - struct cds_wfq_node **cbs_tail; >> - struct cds_wfq_node **cbs_endprev; >> + struct cds_wfq_node *head, *tail; >> >> if (crdp == NULL || crdp == default_call_rcu_data) { >> return; >> } >> + >> if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { >> uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); >> wake_call_rcu_thread(crdp); >> while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) >> poll(NULL, 0, 1); >> } >> - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { >> - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) >> - poll(NULL, 0, 1); >> - _CMM_STORE_SHARED(crdp->cbs.head, NULL); >> - cbs_tail = (struct cds_wfq_node **) >> - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); >> + >> + if (!cds_wfq_empty(&crdp->cbs)) { >> + head = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &tail); >> + assert(head); >> + >> /* Create default call rcu data if need be */ >> (void) get_default_call_rcu_data(); >> - cbs_endprev = (struct cds_wfq_node **) >> - uatomic_xchg(&default_call_rcu_data, cbs_tail); >> - *cbs_endprev = cbs; >> + >> + __cds_wfq_append_list(&default_call_rcu_data->cbs, head, tail); >> + >> uatomic_add(&default_call_rcu_data->qlen, >> uatomic_read(&crdp->qlen)); >> + >> wake_call_rcu_thread(default_call_rcu_data); >> } >> >> diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h >> index 636e1af..15ea9fc 100644 >> --- a/urcu/static/wfqueue.h >> +++ b/urcu/static/wfqueue.h >> @@ -10,6 +10,7 @@ >> * dynamically with the userspace rcu library. >> * >> * Copyright 2010 - Mathieu Desnoyers >> + * Copyright 2011-2012 - Lai Jiangshan >> * >> * This library is free software; you can redistribute it and/or >> * modify it under the terms of the GNU Lesser General Public >> @@ -29,6 +30,7 @@ >> #include >> #include >> #include >> +#include >> #include >> #include >> >> @@ -38,8 +40,6 @@ extern "C" { >> >> /* >> * Queue with wait-free enqueue/blocking dequeue. >> - * This implementation adds a dummy head node when the queue is empty to ensure >> - * we can always update the queue locklessly. >> * >> * Inspired from half-wait-free/half-blocking queue implementation done by >> * Paul E. McKenney. >> @@ -57,31 +57,43 @@ static inline void _cds_wfq_init(struct cds_wfq_queue *q) >> { >> int ret; >> >> - _cds_wfq_node_init(&q->dummy); >> /* Set queue head and tail */ >> - q->head = &q->dummy; >> - q->tail = &q->dummy.next; >> + _cds_wfq_node_init(&q->head); >> + q->tail = &q->head; >> ret = pthread_mutex_init(&q->lock, NULL); >> assert(!ret); >> } >> >> -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, >> - struct cds_wfq_node *node) >> +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) >> { >> - struct cds_wfq_node **old_tail; >> + /* >> + * Queue is empty if no node is pointed by q->head.next nor q->tail. >> + */ >> + return q->head.next == NULL && CMM_LOAD_SHARED(q->tail) == &q->head; >> +} >> >> +static inline void ___cds_wfq_append_list(struct cds_wfq_queue *q, >> + struct cds_wfq_node *head, struct cds_wfq_node *tail) >> +{ >> /* >> * uatomic_xchg() implicit memory barrier orders earlier stores to data >> * structure containing node and setting node->next to NULL before >> * publication. >> */ >> - old_tail = uatomic_xchg(&q->tail, &node->next); >> + tail = uatomic_xchg(&q->tail, tail); >> + >> /* >> - * At this point, dequeuers see a NULL old_tail->next, which indicates >> + * At this point, dequeuers see a NULL tail->next, which indicates >> * that the queue is being appended to. The following store will append >> * "node" to the queue from a dequeuer perspective. >> */ >> - CMM_STORE_SHARED(*old_tail, node); >> + CMM_STORE_SHARED(tail->next, head); >> +} >> + >> +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, >> + struct cds_wfq_node *node) >> +{ >> + ___cds_wfq_append_list(q, node, node); >> } >> >> /* >> @@ -120,27 +132,46 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) >> { >> struct cds_wfq_node *node, *next; >> >> - /* >> - * Queue is empty if it only contains the dummy node. >> - */ >> - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) >> + if (_cds_wfq_empty(q)) >> return NULL; >> - node = q->head; >> >> - next = ___cds_wfq_node_sync_next(node); >> + node = ___cds_wfq_node_sync_next(&q->head); >> + >> + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { >> + if (CMM_LOAD_SHARED(q->tail) == node) { >> + /* >> + * @node is the only node in the queue. >> + * Try to move the tail to &q->head >> + */ >> + _cds_wfq_node_init(&q->head); >> + if (uatomic_cmpxchg(&q->tail, node, &q->head) == node) >> + return node; >> + } >> + next = ___cds_wfq_node_sync_next(node); >> + } >> >> /* >> * Move queue head forward. >> */ >> - q->head = next; >> - /* >> - * Requeue dummy node if we just dequeued it. >> - */ >> - if (node == &q->dummy) { >> - _cds_wfq_node_init(node); >> - _cds_wfq_enqueue(q, node); >> - return ___cds_wfq_dequeue_blocking(q); >> - } >> + q->head.next = next; >> + >> + return node; >> +} >> + >> +/* dequeue all nodes, the nodes are not synchronized for the next pointer */ >> +static inline struct cds_wfq_node * >> +___cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, >> + struct cds_wfq_node **tail) >> +{ >> + struct cds_wfq_node *node; >> + >> + if (_cds_wfq_empty(q)) >> + return NULL; >> + >> + node = ___cds_wfq_node_sync_next(&q->head); >> + _cds_wfq_node_init(&q->head); >> + *tail = uatomic_xchg(&q->tail, &q->head); >> + >> return node; >> } >> >> @@ -158,6 +189,27 @@ _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) >> return retnode; >> } >> >> +static inline struct cds_wfq_node * >> +_cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, >> + struct cds_wfq_node **tail) >> +{ >> + struct cds_wfq_node *node, *next; >> + int ret; >> + >> + ret = pthread_mutex_lock(&q->lock); >> + assert(!ret); >> + node = ___cds_wfq_dequeue_all_blocking(q, tail); >> + ret = pthread_mutex_unlock(&q->lock); >> + assert(!ret); >> + >> + /* synchronize all nodes' next pointer */ >> + next = node; >> + while (next != *tail) >> + next = ___cds_wfq_node_sync_next(next); >> + >> + return node; >> +} >> + >> #ifdef __cplusplus >> } >> #endif >> diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h >> index 03a73f1..985f540 100644 >> --- a/urcu/wfqueue.h >> +++ b/urcu/wfqueue.h >> @@ -7,6 +7,7 @@ >> * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue >> * >> * Copyright 2010 - Mathieu Desnoyers >> + * Copyright 2011-2012 - Lai Jiangshan >> * >> * This library is free software; you can redistribute it and/or >> * modify it under the terms of the GNU Lesser General Public >> @@ -25,6 +26,7 @@ >> >> #include >> #include >> +#include >> #include >> >> #ifdef __cplusplus >> @@ -33,8 +35,6 @@ extern "C" { >> >> /* >> * Queue with wait-free enqueue/blocking dequeue. >> - * This implementation adds a dummy head node when the queue is empty to ensure >> - * we can always update the queue locklessly. >> * >> * Inspired from half-wait-free/half-blocking queue implementation done by >> * Paul E. McKenney. >> @@ -45,8 +45,7 @@ struct cds_wfq_node { >> }; >> >> struct cds_wfq_queue { >> - struct cds_wfq_node *head, **tail; >> - struct cds_wfq_node dummy; /* Dummy node */ >> + struct cds_wfq_node head, *tail; >> pthread_mutex_t lock; >> }; >> >> @@ -56,18 +55,36 @@ struct cds_wfq_queue { >> >> #define cds_wfq_node_init _cds_wfq_node_init >> #define cds_wfq_init _cds_wfq_init >> +#define cds_wfq_empty _cds_wfq_empty >> +#define __cds_wfq_append_list ___cds_wfq_append_list >> #define cds_wfq_enqueue _cds_wfq_enqueue >> #define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking >> #define cds_wfq_dequeue_blocking _cds_wfq_dequeue_blocking >> +#define __cds_wfq_node_sync_next ___cds_wfq_node_sync_next >> +#define __cds_wfq_dequeue_all_blocking ___cds_wfq_dequeue_all_blocking >> +#define cds_wfq_dequeue_all_blocking _cds_wfq_dequeue_all_blocking >> >> #else /* !_LGPL_SOURCE */ >> >> extern void cds_wfq_node_init(struct cds_wfq_node *node); >> extern void cds_wfq_init(struct cds_wfq_queue *q); >> +extern bool cds_wfq_empty(struct cds_wfq_queue *q); >> +/* __cds_wfq_append_list: caller ensures mutual exclusion between dequeues */ >> +extern void __cds_wfq_append_list(struct cds_wfq_queue *q, >> + struct cds_wfq_node *head, struct cds_wfq_node *tail); >> extern void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node); >> /* __cds_wfq_dequeue_blocking: caller ensures mutual exclusion between dequeues */ >> extern struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); >> extern struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); >> +extern struct cds_wfq_node *__cds_wfq_node_sync_next(struct cds_wfq_node *node); >> +/* >> + * __cds_wfq_dequeue_all_blocking: caller ensures mutual exclusion between >> + * dequeues, and need synchronize next pointer berfore use it. >> + */ >> +extern struct cds_wfq_node *__cds_wfq_dequeue_all_blocking( >> + struct cds_wfq_queue *q, struct cds_wfq_node **tail); >> +extern struct cds_wfq_node *cds_wfq_dequeue_all_blocking( >> + struct cds_wfq_queue *q, struct cds_wfq_node **tail); >> >> #endif /* !_LGPL_SOURCE */ >> >> diff --git a/wfqueue.c b/wfqueue.c >> index 3337171..28a7b58 100644 >> --- a/wfqueue.c >> +++ b/wfqueue.c >> @@ -4,6 +4,7 @@ >> * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue >> * >> * Copyright 2010 - Mathieu Desnoyers >> + * Copyright 2011-2012 - Lai Jiangshan >> * >> * This library is free software; you can redistribute it and/or >> * modify it under the terms of the GNU Lesser General Public >> @@ -38,6 +39,17 @@ void cds_wfq_init(struct cds_wfq_queue *q) >> _cds_wfq_init(q); >> } >> >> +bool cds_wfq_empty(struct cds_wfq_queue *q) >> +{ >> + return _cds_wfq_empty(q); >> +} >> + >> +void __cds_wfq_append_list(struct cds_wfq_queue *q, >> + struct cds_wfq_node *head, struct cds_wfq_node *tail) >> +{ >> + return ___cds_wfq_append_list(q, head, tail); >> +} >> + >> void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node) >> { >> _cds_wfq_enqueue(q, node); >> @@ -52,3 +64,20 @@ struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) >> { >> return _cds_wfq_dequeue_blocking(q); >> } >> + >> +struct cds_wfq_node *__cds_wfq_node_sync_next(struct cds_wfq_node *node) >> +{ >> + return ___cds_wfq_node_sync_next(node); >> +} >> + >> +struct cds_wfq_node *__cds_wfq_dequeue_all_blocking( >> + struct cds_wfq_queue *q, struct cds_wfq_node **tail) >> +{ >> + return ___cds_wfq_dequeue_all_blocking(q, tail); >> +} >> + >> +struct cds_wfq_node *cds_wfq_dequeue_all_blocking( >> + struct cds_wfq_queue *q, struct cds_wfq_node **tail) >> +{ >> + return _cds_wfq_dequeue_all_blocking(q, tail); >> +} >> -- >> 1.7.7 >> >> >> _______________________________________________ >> lttng-dev mailing list >> lttng-dev at lists.lttng.org >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com From mathieu.desnoyers at efficios.com Sat Aug 11 09:14:17 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Sat, 11 Aug 2012 09:14:17 -0400 Subject: [lttng-dev] [PATCH 2/2] urcu: new wfqueue implementation In-Reply-To: <1344501986-23151-2-git-send-email-laijs@cn.fujitsu.com> References: <1344501986-23151-1-git-send-email-laijs@cn.fujitsu.com> <1344501986-23151-2-git-send-email-laijs@cn.fujitsu.com> Message-ID: <20120811131417.GA13886@Krystal> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > Some guys would be surprised by this fact: > There are already TWO implementations of wfqueue in urcu. > > The first one is in urcu/static/wfqueue.h: > 1) enqueue: exchange the tail and then update previous->next > 2) dequeue: wait for first node's next pointer and them shift, a dummy node > is introduced to avoid the queue->tail become NULL when shift. > > The second one shares some code with the first one, and the left code > are spreading in urcu-call-rcu-impl.h: > 1) enqueue: share with the first one > 2) no dequeue operation: and no shift, so it don't need dummy node, > Although the dummy node is queued when initialization, but it is removed > after the first dequeue_all operation in call_rcu_thread(). > call_rcu_data_free() forgets to handle the dummy node if it is not removed. > 3)dequeue_all: record the old head and tail, and queue->head become the special > tail node.(atomic record the tail and change the tail). > > The second implementation's code are spreading, bad for review, and it is not > tested by tests/test_urcu_wfq. > > So we need a better implementation avoid the dummy node dancing and can service > both generic wfqueue APIs and dequeue_all API for call rcu. > > The new implementation: > 1) enqueue: share with the first one/original implementation. > 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. > no dummy node, save memory. > 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail > and return the old head.next. > > More implementation details are in the code. > tests/test_urcu_wfq will be update in future for testing new APIs. Hi Lai, Some other style-related questions below, > > > Signed-off-by: Lai Jiangshan > --- > urcu-call-rcu-impl.h | 50 ++++++++++-------------- > urcu/static/wfqueue.h | 104 ++++++++++++++++++++++++++++++++++++------------ > urcu/wfqueue.h | 25 ++++++++++-- > wfqueue.c | 29 ++++++++++++++ > 4 files changed, 149 insertions(+), 59 deletions(-) > > diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h > index 13b24ff..dbfb410 100644 > --- a/urcu-call-rcu-impl.h > +++ b/urcu-call-rcu-impl.h > @@ -221,7 +221,7 @@ static void *call_rcu_thread(void *arg) > { > unsigned long cbcount; > struct cds_wfq_node *cbs; > - struct cds_wfq_node **cbs_tail; > + struct cds_wfq_node *cbs_tail; > struct call_rcu_data *crdp = (struct call_rcu_data *)arg; > struct rcu_head *rhp; > int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); > @@ -243,24 +243,18 @@ static void *call_rcu_thread(void *arg) > cmm_smp_mb(); > } > for (;;) { > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > - poll(NULL, 0, 1); > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > - cbs_tail = (struct cds_wfq_node **) > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > + cbs = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &cbs_tail); > + if (cbs) { > synchronize_rcu(); > cbcount = 0; > do { > - while (cbs->next == NULL && > - &cbs->next != cbs_tail) > - poll(NULL, 0, 1); > - if (cbs == &crdp->cbs.dummy) { > - cbs = cbs->next; > - continue; > - } > rhp = (struct rcu_head *)cbs; > - cbs = cbs->next; > + > + if (cbs != cbs_tail) > + cbs = __cds_wfq_node_sync_next(cbs); > + else > + cbs = NULL; > + > rhp->func(rhp); > cbcount++; > } while (cbs != NULL); > @@ -270,8 +264,7 @@ static void *call_rcu_thread(void *arg) > break; > rcu_thread_offline(); > if (!rt) { > - if (&crdp->cbs.head > - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { > + if (cds_wfq_empty(&crdp->cbs)) { > call_rcu_wait(crdp); > poll(NULL, 0, 10); > uatomic_dec(&crdp->futex); > @@ -625,32 +618,31 @@ void call_rcu(struct rcu_head *head, > */ > void call_rcu_data_free(struct call_rcu_data *crdp) > { > - struct cds_wfq_node *cbs; > - struct cds_wfq_node **cbs_tail; > - struct cds_wfq_node **cbs_endprev; > + struct cds_wfq_node *head, *tail; > > if (crdp == NULL || crdp == default_call_rcu_data) { > return; > } > + > if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { > uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); > wake_call_rcu_thread(crdp); > while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) > poll(NULL, 0, 1); > } > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > - poll(NULL, 0, 1); > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > - cbs_tail = (struct cds_wfq_node **) > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > + > + if (!cds_wfq_empty(&crdp->cbs)) { > + head = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &tail); > + assert(head); > + > /* Create default call rcu data if need be */ > (void) get_default_call_rcu_data(); > - cbs_endprev = (struct cds_wfq_node **) > - uatomic_xchg(&default_call_rcu_data, cbs_tail); > - *cbs_endprev = cbs; > + > + __cds_wfq_append_list(&default_call_rcu_data->cbs, head, tail); > + > uatomic_add(&default_call_rcu_data->qlen, > uatomic_read(&crdp->qlen)); > + > wake_call_rcu_thread(default_call_rcu_data); > } > > diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h > index 636e1af..15ea9fc 100644 > --- a/urcu/static/wfqueue.h > +++ b/urcu/static/wfqueue.h > @@ -10,6 +10,7 @@ > * dynamically with the userspace rcu library. > * > * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -29,6 +30,7 @@ > #include > #include > #include > +#include > #include > #include > > @@ -38,8 +40,6 @@ extern "C" { > > /* > * Queue with wait-free enqueue/blocking dequeue. > - * This implementation adds a dummy head node when the queue is empty to ensure > - * we can always update the queue locklessly. > * > * Inspired from half-wait-free/half-blocking queue implementation done by > * Paul E. McKenney. > @@ -57,31 +57,43 @@ static inline void _cds_wfq_init(struct cds_wfq_queue *q) > { > int ret; > > - _cds_wfq_node_init(&q->dummy); > /* Set queue head and tail */ > - q->head = &q->dummy; > - q->tail = &q->dummy.next; > + _cds_wfq_node_init(&q->head); > + q->tail = &q->head; > ret = pthread_mutex_init(&q->lock, NULL); > assert(!ret); > } > > -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > - struct cds_wfq_node *node) > +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) > { > - struct cds_wfq_node **old_tail; > + /* > + * Queue is empty if no node is pointed by q->head.next nor q->tail. > + */ > + return q->head.next == NULL && CMM_LOAD_SHARED(q->tail) == &q->head; > +} > > +static inline void ___cds_wfq_append_list(struct cds_wfq_queue *q, > + struct cds_wfq_node *head, struct cds_wfq_node *tail) > +{ > /* > * uatomic_xchg() implicit memory barrier orders earlier stores to data > * structure containing node and setting node->next to NULL before > * publication. > */ > - old_tail = uatomic_xchg(&q->tail, &node->next); > + tail = uatomic_xchg(&q->tail, tail); I'd prefer to keep "old_tail" here, because it becomes clearer to anyone reviewing that uatomic_xchg() returns the old tail (and this extra clarity comes without any overhead). > + > /* > - * At this point, dequeuers see a NULL old_tail->next, which indicates > + * At this point, dequeuers see a NULL tail->next, which indicates > * that the queue is being appended to. The following store will append > * "node" to the queue from a dequeuer perspective. > */ > - CMM_STORE_SHARED(*old_tail, node); > + CMM_STORE_SHARED(tail->next, head); > +} > + > +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > + struct cds_wfq_node *node) > +{ > + ___cds_wfq_append_list(q, node, node); > } Why not keep ___cds_wfq_append_list() merged into _cds_wfq_enqueue() ? This would keep the number of symbols exported minimal. So if I get it right, one "_" prefix is the "normally used" functions (exposed through the LGPL symbol API). The "__" prefix are somewhat more internal, but can also be used externally. Finally, the "___" prefix seem to be quite similar to the double-underscores. We might need more consistency, I'm not sure the triple-underscores are needed. Also, I'm not sure should export the double-underscore functions outside of LGPL use (in other words, maybe we should not expose them to !LGPL_SOURCE code). So we would emit the static inlines, but no symbols for those. This covers ___cds_wfq_node_sync_next(), and ___cds_wfq_dequeue_all_blocking (which requires the caller to use sync_next). Currently, all code that needs to fine-grained integration is within the userspace RCU tree, which defines LGPL_SOURCE. > > /* > @@ -120,27 +132,46 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > { > struct cds_wfq_node *node, *next; > > - /* > - * Queue is empty if it only contains the dummy node. > - */ > - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) > + if (_cds_wfq_empty(q)) > return NULL; > - node = q->head; > > - next = ___cds_wfq_node_sync_next(node); > + node = ___cds_wfq_node_sync_next(&q->head); > + > + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { > + if (CMM_LOAD_SHARED(q->tail) == node) { > + /* > + * @node is the only node in the queue. > + * Try to move the tail to &q->head > + */ > + _cds_wfq_node_init(&q->head); > + if (uatomic_cmpxchg(&q->tail, node, &q->head) == node) > + return node; > + } > + next = ___cds_wfq_node_sync_next(node); > + } > > /* > * Move queue head forward. > */ > - q->head = next; > - /* > - * Requeue dummy node if we just dequeued it. > - */ > - if (node == &q->dummy) { > - _cds_wfq_node_init(node); > - _cds_wfq_enqueue(q, node); > - return ___cds_wfq_dequeue_blocking(q); > - } > + q->head.next = next; > + > + return node; > +} > + > +/* dequeue all nodes, the nodes are not synchronized for the next pointer */ > +static inline struct cds_wfq_node * > +___cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, > + struct cds_wfq_node **tail) > +{ > + struct cds_wfq_node *node; > + > + if (_cds_wfq_empty(q)) > + return NULL; > + > + node = ___cds_wfq_node_sync_next(&q->head); > + _cds_wfq_node_init(&q->head); > + *tail = uatomic_xchg(&q->tail, &q->head); > + > return node; > } > > @@ -158,6 +189,27 @@ _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > return retnode; > } > > +static inline struct cds_wfq_node * > +_cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, > + struct cds_wfq_node **tail) > +{ > + struct cds_wfq_node *node, *next; > + int ret; > + > + ret = pthread_mutex_lock(&q->lock); > + assert(!ret); > + node = ___cds_wfq_dequeue_all_blocking(q, tail); > + ret = pthread_mutex_unlock(&q->lock); > + assert(!ret); So we take the queue lock on dequeue_all, but not on dequeue. It might be good to have a consistent behavior: either we lock dequeue and dequeue_all, or leave the lock entirely to the caller (and document it). Thoughts ? Thanks! Mathieu > + > + /* synchronize all nodes' next pointer */ > + next = node; > + while (next != *tail) > + next = ___cds_wfq_node_sync_next(next); > + > + return node; > +} > + > #ifdef __cplusplus > } > #endif > diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h > index 03a73f1..985f540 100644 > --- a/urcu/wfqueue.h > +++ b/urcu/wfqueue.h > @@ -7,6 +7,7 @@ > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > * > * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -25,6 +26,7 @@ > > #include > #include > +#include > #include > > #ifdef __cplusplus > @@ -33,8 +35,6 @@ extern "C" { > > /* > * Queue with wait-free enqueue/blocking dequeue. > - * This implementation adds a dummy head node when the queue is empty to ensure > - * we can always update the queue locklessly. > * > * Inspired from half-wait-free/half-blocking queue implementation done by > * Paul E. McKenney. > @@ -45,8 +45,7 @@ struct cds_wfq_node { > }; > > struct cds_wfq_queue { > - struct cds_wfq_node *head, **tail; > - struct cds_wfq_node dummy; /* Dummy node */ > + struct cds_wfq_node head, *tail; > pthread_mutex_t lock; > }; > > @@ -56,18 +55,36 @@ struct cds_wfq_queue { > > #define cds_wfq_node_init _cds_wfq_node_init > #define cds_wfq_init _cds_wfq_init > +#define cds_wfq_empty _cds_wfq_empty > +#define __cds_wfq_append_list ___cds_wfq_append_list > #define cds_wfq_enqueue _cds_wfq_enqueue > #define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking > #define cds_wfq_dequeue_blocking _cds_wfq_dequeue_blocking > +#define __cds_wfq_node_sync_next ___cds_wfq_node_sync_next > +#define __cds_wfq_dequeue_all_blocking ___cds_wfq_dequeue_all_blocking > +#define cds_wfq_dequeue_all_blocking _cds_wfq_dequeue_all_blocking > > #else /* !_LGPL_SOURCE */ > > extern void cds_wfq_node_init(struct cds_wfq_node *node); > extern void cds_wfq_init(struct cds_wfq_queue *q); > +extern bool cds_wfq_empty(struct cds_wfq_queue *q); > +/* __cds_wfq_append_list: caller ensures mutual exclusion between dequeues */ > +extern void __cds_wfq_append_list(struct cds_wfq_queue *q, > + struct cds_wfq_node *head, struct cds_wfq_node *tail); > extern void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node); > /* __cds_wfq_dequeue_blocking: caller ensures mutual exclusion between dequeues */ > extern struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); > extern struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); > +extern struct cds_wfq_node *__cds_wfq_node_sync_next(struct cds_wfq_node *node); > +/* > + * __cds_wfq_dequeue_all_blocking: caller ensures mutual exclusion between > + * dequeues, and need synchronize next pointer berfore use it. > + */ > +extern struct cds_wfq_node *__cds_wfq_dequeue_all_blocking( > + struct cds_wfq_queue *q, struct cds_wfq_node **tail); > +extern struct cds_wfq_node *cds_wfq_dequeue_all_blocking( > + struct cds_wfq_queue *q, struct cds_wfq_node **tail); > > #endif /* !_LGPL_SOURCE */ > > diff --git a/wfqueue.c b/wfqueue.c > index 3337171..28a7b58 100644 > --- a/wfqueue.c > +++ b/wfqueue.c > @@ -4,6 +4,7 @@ > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > * > * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -38,6 +39,17 @@ void cds_wfq_init(struct cds_wfq_queue *q) > _cds_wfq_init(q); > } > > +bool cds_wfq_empty(struct cds_wfq_queue *q) > +{ > + return _cds_wfq_empty(q); > +} > + > +void __cds_wfq_append_list(struct cds_wfq_queue *q, > + struct cds_wfq_node *head, struct cds_wfq_node *tail) > +{ > + return ___cds_wfq_append_list(q, head, tail); > +} > + > void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node) > { > _cds_wfq_enqueue(q, node); > @@ -52,3 +64,20 @@ struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > { > return _cds_wfq_dequeue_blocking(q); > } > + > +struct cds_wfq_node *__cds_wfq_node_sync_next(struct cds_wfq_node *node) > +{ > + return ___cds_wfq_node_sync_next(node); > +} > + > +struct cds_wfq_node *__cds_wfq_dequeue_all_blocking( > + struct cds_wfq_queue *q, struct cds_wfq_node **tail) > +{ > + return ___cds_wfq_dequeue_all_blocking(q, tail); > +} > + > +struct cds_wfq_node *cds_wfq_dequeue_all_blocking( > + struct cds_wfq_queue *q, struct cds_wfq_node **tail) > +{ > + return _cds_wfq_dequeue_all_blocking(q, tail); > +} > -- > 1.7.7 > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Sat Aug 11 22:48:47 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Sat, 11 Aug 2012 22:48:47 -0400 Subject: [lttng-dev] [PATCH 2/2] urcu: new wfqueue implementation In-Reply-To: <20120811131417.GA13886@Krystal> References: <1344501986-23151-1-git-send-email-laijs@cn.fujitsu.com> <1344501986-23151-2-git-send-email-laijs@cn.fujitsu.com> <20120811131417.GA13886@Krystal> Message-ID: <20120812024847.GA24711@Krystal> * Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote: > * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > > Some guys would be surprised by this fact: > > There are already TWO implementations of wfqueue in urcu. > > > > The first one is in urcu/static/wfqueue.h: > > 1) enqueue: exchange the tail and then update previous->next > > 2) dequeue: wait for first node's next pointer and them shift, a dummy node > > is introduced to avoid the queue->tail become NULL when shift. > > > > The second one shares some code with the first one, and the left code > > are spreading in urcu-call-rcu-impl.h: > > 1) enqueue: share with the first one > > 2) no dequeue operation: and no shift, so it don't need dummy node, > > Although the dummy node is queued when initialization, but it is removed > > after the first dequeue_all operation in call_rcu_thread(). > > call_rcu_data_free() forgets to handle the dummy node if it is not removed. > > 3)dequeue_all: record the old head and tail, and queue->head become the special > > tail node.(atomic record the tail and change the tail). > > > > The second implementation's code are spreading, bad for review, and it is not > > tested by tests/test_urcu_wfq. > > > > So we need a better implementation avoid the dummy node dancing and can service > > both generic wfqueue APIs and dequeue_all API for call rcu. > > > > The new implementation: > > 1) enqueue: share with the first one/original implementation. > > 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. > > no dummy node, save memory. > > 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail > > and return the old head.next. > > > > More implementation details are in the code. > > tests/test_urcu_wfq will be update in future for testing new APIs. > > Hi Lai, > > Some other style-related questions below, FYI, I'm preparing a patch implementing those and some other ideas regarding the API. I plan to post it tomorrow. Thanks, Mathieu > > > > > > > Signed-off-by: Lai Jiangshan > > --- > > urcu-call-rcu-impl.h | 50 ++++++++++-------------- > > urcu/static/wfqueue.h | 104 ++++++++++++++++++++++++++++++++++++------------ > > urcu/wfqueue.h | 25 ++++++++++-- > > wfqueue.c | 29 ++++++++++++++ > > 4 files changed, 149 insertions(+), 59 deletions(-) > > > > diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h > > index 13b24ff..dbfb410 100644 > > --- a/urcu-call-rcu-impl.h > > +++ b/urcu-call-rcu-impl.h > > @@ -221,7 +221,7 @@ static void *call_rcu_thread(void *arg) > > { > > unsigned long cbcount; > > struct cds_wfq_node *cbs; > > - struct cds_wfq_node **cbs_tail; > > + struct cds_wfq_node *cbs_tail; > > struct call_rcu_data *crdp = (struct call_rcu_data *)arg; > > struct rcu_head *rhp; > > int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); > > @@ -243,24 +243,18 @@ static void *call_rcu_thread(void *arg) > > cmm_smp_mb(); > > } > > for (;;) { > > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > > - poll(NULL, 0, 1); > > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > > - cbs_tail = (struct cds_wfq_node **) > > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > > + cbs = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &cbs_tail); > > + if (cbs) { > > synchronize_rcu(); > > cbcount = 0; > > do { > > - while (cbs->next == NULL && > > - &cbs->next != cbs_tail) > > - poll(NULL, 0, 1); > > - if (cbs == &crdp->cbs.dummy) { > > - cbs = cbs->next; > > - continue; > > - } > > rhp = (struct rcu_head *)cbs; > > - cbs = cbs->next; > > + > > + if (cbs != cbs_tail) > > + cbs = __cds_wfq_node_sync_next(cbs); > > + else > > + cbs = NULL; > > + > > rhp->func(rhp); > > cbcount++; > > } while (cbs != NULL); > > @@ -270,8 +264,7 @@ static void *call_rcu_thread(void *arg) > > break; > > rcu_thread_offline(); > > if (!rt) { > > - if (&crdp->cbs.head > > - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { > > + if (cds_wfq_empty(&crdp->cbs)) { > > call_rcu_wait(crdp); > > poll(NULL, 0, 10); > > uatomic_dec(&crdp->futex); > > @@ -625,32 +618,31 @@ void call_rcu(struct rcu_head *head, > > */ > > void call_rcu_data_free(struct call_rcu_data *crdp) > > { > > - struct cds_wfq_node *cbs; > > - struct cds_wfq_node **cbs_tail; > > - struct cds_wfq_node **cbs_endprev; > > + struct cds_wfq_node *head, *tail; > > > > if (crdp == NULL || crdp == default_call_rcu_data) { > > return; > > } > > + > > if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { > > uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); > > wake_call_rcu_thread(crdp); > > while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) > > poll(NULL, 0, 1); > > } > > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > > - poll(NULL, 0, 1); > > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > > - cbs_tail = (struct cds_wfq_node **) > > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > > + > > + if (!cds_wfq_empty(&crdp->cbs)) { > > + head = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &tail); > > + assert(head); > > + > > /* Create default call rcu data if need be */ > > (void) get_default_call_rcu_data(); > > - cbs_endprev = (struct cds_wfq_node **) > > - uatomic_xchg(&default_call_rcu_data, cbs_tail); > > - *cbs_endprev = cbs; > > + > > + __cds_wfq_append_list(&default_call_rcu_data->cbs, head, tail); > > + > > uatomic_add(&default_call_rcu_data->qlen, > > uatomic_read(&crdp->qlen)); > > + > > wake_call_rcu_thread(default_call_rcu_data); > > } > > > > diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h > > index 636e1af..15ea9fc 100644 > > --- a/urcu/static/wfqueue.h > > +++ b/urcu/static/wfqueue.h > > @@ -10,6 +10,7 @@ > > * dynamically with the userspace rcu library. > > * > > * Copyright 2010 - Mathieu Desnoyers > > + * Copyright 2011-2012 - Lai Jiangshan > > * > > * This library is free software; you can redistribute it and/or > > * modify it under the terms of the GNU Lesser General Public > > @@ -29,6 +30,7 @@ > > #include > > #include > > #include > > +#include > > #include > > #include > > > > @@ -38,8 +40,6 @@ extern "C" { > > > > /* > > * Queue with wait-free enqueue/blocking dequeue. > > - * This implementation adds a dummy head node when the queue is empty to ensure > > - * we can always update the queue locklessly. > > * > > * Inspired from half-wait-free/half-blocking queue implementation done by > > * Paul E. McKenney. > > @@ -57,31 +57,43 @@ static inline void _cds_wfq_init(struct cds_wfq_queue *q) > > { > > int ret; > > > > - _cds_wfq_node_init(&q->dummy); > > /* Set queue head and tail */ > > - q->head = &q->dummy; > > - q->tail = &q->dummy.next; > > + _cds_wfq_node_init(&q->head); > > + q->tail = &q->head; > > ret = pthread_mutex_init(&q->lock, NULL); > > assert(!ret); > > } > > > > -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > > - struct cds_wfq_node *node) > > +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) > > { > > - struct cds_wfq_node **old_tail; > > + /* > > + * Queue is empty if no node is pointed by q->head.next nor q->tail. > > + */ > > + return q->head.next == NULL && CMM_LOAD_SHARED(q->tail) == &q->head; > > +} > > > > +static inline void ___cds_wfq_append_list(struct cds_wfq_queue *q, > > + struct cds_wfq_node *head, struct cds_wfq_node *tail) > > +{ > > /* > > * uatomic_xchg() implicit memory barrier orders earlier stores to data > > * structure containing node and setting node->next to NULL before > > * publication. > > */ > > - old_tail = uatomic_xchg(&q->tail, &node->next); > > + tail = uatomic_xchg(&q->tail, tail); > > I'd prefer to keep "old_tail" here, because it becomes clearer to anyone > reviewing that uatomic_xchg() returns the old tail (and this extra > clarity comes without any overhead). > > > + > > /* > > - * At this point, dequeuers see a NULL old_tail->next, which indicates > > + * At this point, dequeuers see a NULL tail->next, which indicates > > * that the queue is being appended to. The following store will append > > * "node" to the queue from a dequeuer perspective. > > */ > > - CMM_STORE_SHARED(*old_tail, node); > > + CMM_STORE_SHARED(tail->next, head); > > +} > > + > > +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > > + struct cds_wfq_node *node) > > +{ > > + ___cds_wfq_append_list(q, node, node); > > } > > Why not keep ___cds_wfq_append_list() merged into _cds_wfq_enqueue() ? > > This would keep the number of symbols exported minimal. > > So if I get it right, one "_" prefix is the "normally used" functions > (exposed through the LGPL symbol API). > > The "__" prefix are somewhat more internal, but can also be used > externally. > > Finally, the "___" prefix seem to be quite similar to the > double-underscores. > > We might need more consistency, I'm not sure the triple-underscores are > needed. Also, I'm not sure should export the double-underscore functions > outside of LGPL use (in other words, maybe we should not expose them to > !LGPL_SOURCE code). So we would emit the static inlines, but no symbols > for those. This covers ___cds_wfq_node_sync_next(), and > ___cds_wfq_dequeue_all_blocking (which requires the caller to use > sync_next). Currently, all code that needs to fine-grained integration > is within the userspace RCU tree, which defines LGPL_SOURCE. > > > > > /* > > @@ -120,27 +132,46 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > > { > > struct cds_wfq_node *node, *next; > > > > - /* > > - * Queue is empty if it only contains the dummy node. > > - */ > > - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) > > + if (_cds_wfq_empty(q)) > > return NULL; > > - node = q->head; > > > > - next = ___cds_wfq_node_sync_next(node); > > + node = ___cds_wfq_node_sync_next(&q->head); > > + > > + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { > > + if (CMM_LOAD_SHARED(q->tail) == node) { > > + /* > > + * @node is the only node in the queue. > > + * Try to move the tail to &q->head > > + */ > > + _cds_wfq_node_init(&q->head); > > + if (uatomic_cmpxchg(&q->tail, node, &q->head) == node) > > + return node; > > + } > > + next = ___cds_wfq_node_sync_next(node); > > + } > > > > /* > > * Move queue head forward. > > */ > > - q->head = next; > > - /* > > - * Requeue dummy node if we just dequeued it. > > - */ > > - if (node == &q->dummy) { > > - _cds_wfq_node_init(node); > > - _cds_wfq_enqueue(q, node); > > - return ___cds_wfq_dequeue_blocking(q); > > - } > > + q->head.next = next; > > + > > + return node; > > +} > > + > > +/* dequeue all nodes, the nodes are not synchronized for the next pointer */ > > +static inline struct cds_wfq_node * > > +___cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, > > + struct cds_wfq_node **tail) > > +{ > > + struct cds_wfq_node *node; > > + > > + if (_cds_wfq_empty(q)) > > + return NULL; > > + > > + node = ___cds_wfq_node_sync_next(&q->head); > > + _cds_wfq_node_init(&q->head); > > + *tail = uatomic_xchg(&q->tail, &q->head); > > + > > return node; > > } > > > > @@ -158,6 +189,27 @@ _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > > return retnode; > > } > > > > +static inline struct cds_wfq_node * > > +_cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, > > + struct cds_wfq_node **tail) > > +{ > > + struct cds_wfq_node *node, *next; > > + int ret; > > + > > + ret = pthread_mutex_lock(&q->lock); > > + assert(!ret); > > + node = ___cds_wfq_dequeue_all_blocking(q, tail); > > + ret = pthread_mutex_unlock(&q->lock); > > + assert(!ret); > > So we take the queue lock on dequeue_all, but not on dequeue. It might > be good to have a consistent behavior: either we lock dequeue and > dequeue_all, or leave the lock entirely to the caller (and document it). > > Thoughts ? > > Thanks! > > Mathieu > > > + > > + /* synchronize all nodes' next pointer */ > > + next = node; > > + while (next != *tail) > > + next = ___cds_wfq_node_sync_next(next); > > + > > + return node; > > +} > > + > > #ifdef __cplusplus > > } > > #endif > > diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h > > index 03a73f1..985f540 100644 > > --- a/urcu/wfqueue.h > > +++ b/urcu/wfqueue.h > > @@ -7,6 +7,7 @@ > > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > > * > > * Copyright 2010 - Mathieu Desnoyers > > + * Copyright 2011-2012 - Lai Jiangshan > > * > > * This library is free software; you can redistribute it and/or > > * modify it under the terms of the GNU Lesser General Public > > @@ -25,6 +26,7 @@ > > > > #include > > #include > > +#include > > #include > > > > #ifdef __cplusplus > > @@ -33,8 +35,6 @@ extern "C" { > > > > /* > > * Queue with wait-free enqueue/blocking dequeue. > > - * This implementation adds a dummy head node when the queue is empty to ensure > > - * we can always update the queue locklessly. > > * > > * Inspired from half-wait-free/half-blocking queue implementation done by > > * Paul E. McKenney. > > @@ -45,8 +45,7 @@ struct cds_wfq_node { > > }; > > > > struct cds_wfq_queue { > > - struct cds_wfq_node *head, **tail; > > - struct cds_wfq_node dummy; /* Dummy node */ > > + struct cds_wfq_node head, *tail; > > pthread_mutex_t lock; > > }; > > > > @@ -56,18 +55,36 @@ struct cds_wfq_queue { > > > > #define cds_wfq_node_init _cds_wfq_node_init > > #define cds_wfq_init _cds_wfq_init > > +#define cds_wfq_empty _cds_wfq_empty > > +#define __cds_wfq_append_list ___cds_wfq_append_list > > #define cds_wfq_enqueue _cds_wfq_enqueue > > #define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking > > #define cds_wfq_dequeue_blocking _cds_wfq_dequeue_blocking > > +#define __cds_wfq_node_sync_next ___cds_wfq_node_sync_next > > +#define __cds_wfq_dequeue_all_blocking ___cds_wfq_dequeue_all_blocking > > +#define cds_wfq_dequeue_all_blocking _cds_wfq_dequeue_all_blocking > > > > #else /* !_LGPL_SOURCE */ > > > > extern void cds_wfq_node_init(struct cds_wfq_node *node); > > extern void cds_wfq_init(struct cds_wfq_queue *q); > > +extern bool cds_wfq_empty(struct cds_wfq_queue *q); > > +/* __cds_wfq_append_list: caller ensures mutual exclusion between dequeues */ > > +extern void __cds_wfq_append_list(struct cds_wfq_queue *q, > > + struct cds_wfq_node *head, struct cds_wfq_node *tail); > > extern void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node); > > /* __cds_wfq_dequeue_blocking: caller ensures mutual exclusion between dequeues */ > > extern struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); > > extern struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); > > +extern struct cds_wfq_node *__cds_wfq_node_sync_next(struct cds_wfq_node *node); > > +/* > > + * __cds_wfq_dequeue_all_blocking: caller ensures mutual exclusion between > > + * dequeues, and need synchronize next pointer berfore use it. > > + */ > > +extern struct cds_wfq_node *__cds_wfq_dequeue_all_blocking( > > + struct cds_wfq_queue *q, struct cds_wfq_node **tail); > > +extern struct cds_wfq_node *cds_wfq_dequeue_all_blocking( > > + struct cds_wfq_queue *q, struct cds_wfq_node **tail); > > > > #endif /* !_LGPL_SOURCE */ > > > > diff --git a/wfqueue.c b/wfqueue.c > > index 3337171..28a7b58 100644 > > --- a/wfqueue.c > > +++ b/wfqueue.c > > @@ -4,6 +4,7 @@ > > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > > * > > * Copyright 2010 - Mathieu Desnoyers > > + * Copyright 2011-2012 - Lai Jiangshan > > * > > * This library is free software; you can redistribute it and/or > > * modify it under the terms of the GNU Lesser General Public > > @@ -38,6 +39,17 @@ void cds_wfq_init(struct cds_wfq_queue *q) > > _cds_wfq_init(q); > > } > > > > +bool cds_wfq_empty(struct cds_wfq_queue *q) > > +{ > > + return _cds_wfq_empty(q); > > +} > > + > > +void __cds_wfq_append_list(struct cds_wfq_queue *q, > > + struct cds_wfq_node *head, struct cds_wfq_node *tail) > > +{ > > + return ___cds_wfq_append_list(q, head, tail); > > +} > > + > > void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node) > > { > > _cds_wfq_enqueue(q, node); > > @@ -52,3 +64,20 @@ struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > > { > > return _cds_wfq_dequeue_blocking(q); > > } > > + > > +struct cds_wfq_node *__cds_wfq_node_sync_next(struct cds_wfq_node *node) > > +{ > > + return ___cds_wfq_node_sync_next(node); > > +} > > + > > +struct cds_wfq_node *__cds_wfq_dequeue_all_blocking( > > + struct cds_wfq_queue *q, struct cds_wfq_node **tail) > > +{ > > + return ___cds_wfq_dequeue_all_blocking(q, tail); > > +} > > + > > +struct cds_wfq_node *cds_wfq_dequeue_all_blocking( > > + struct cds_wfq_queue *q, struct cds_wfq_node **tail) > > +{ > > + return _cds_wfq_dequeue_all_blocking(q, tail); > > +} > > -- > > 1.7.7 > > > > > > _______________________________________________ > > lttng-dev mailing list > > lttng-dev at lists.lttng.org > > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Sun Aug 12 10:50:00 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Sun, 12 Aug 2012 10:50:00 -0400 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost Message-ID: <20120812145000.GA32747@Krystal> This work is derived from the patch from Lai Jiangshan submitted as "urcu: new wfqueue implementation" (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) Its changelog: > Some guys would be surprised by this fact: > There are already TWO implementations of wfqueue in urcu. > > The first one is in urcu/static/wfqueue.h: > 1) enqueue: exchange the tail and then update previous->next > 2) dequeue: wait for first node's next pointer and them shift, a dummy node > is introduced to avoid the queue->tail become NULL when shift. > > The second one shares some code with the first one, and the left code > are spreading in urcu-call-rcu-impl.h: > 1) enqueue: share with the first one > 2) no dequeue operation: and no shift, so it don't need dummy node, > Although the dummy node is queued when initialization, but it is removed > after the first dequeue_all operation in call_rcu_thread(). > call_rcu_data_free() forgets to handle the dummy node if it is not removed. > 3)dequeue_all: record the old head and tail, and queue->head become the special > tail node.(atomic record the tail and change the tail). > > The second implementation's code are spreading, bad for review, and it is not > tested by tests/test_urcu_wfq. > > So we need a better implementation avoid the dummy node dancing and can service > both generic wfqueue APIs and dequeue_all API for call rcu. > > The new implementation: > 1) enqueue: share with the first one/original implementation. > 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. > no dummy node, save memory. > 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail > and return the old head.next. > > More implementation details are in the code. > tests/test_urcu_wfq will be update in future for testing new APIs. The patch proposed by Lai brings a very interesting simplification to the single-node handling (which is kept here), and moves all queue handling code away from call_rcu implementation, back into the wfqueue code. This has the benefit to allow testing enhancements. I modified it so the API does not expose implementation details to the user (e.g. ___cds_wfq_node_sync_next). I added a "splice" operation and a for loop iterator which should allow wfqueue users to use the list very efficiently both from LGPL/GPL code and from non-LGPL-compatible code. Benchmarks performed on Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz (dual-core, with hyperthreading) Benchmark invoked: test_urcu_wfq 2 2 10 Only did 2 runs, but a small improvement seems to be clear for the dequeue speed: Before patch: testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 136251248 nr_dequeues 54694027 successful enqueues 136251248 successful dequeues 54693904 end_dequeues 81557344 nr_ops 190945275 testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 137258881 nr_dequeues 54463340 successful enqueues 137258881 successful dequeues 54463238 end_dequeues 82795643 nr_ops 191722221 After patch: testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 138589301 nr_dequeues 56911253 successful enqueues 138589301 successful dequeues 56910916 end_dequeues 81678385 nr_ops 195500554 testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 139007622 nr_dequeues 57281502 successful enqueues 139007622 successful dequeues 57281348 end_dequeues 81726274 nr_ops 196289124 Summary: Number of enqueues is slightly lower, probably due to higher dequeue rate. Number of dequeue increased. Respective rate change is within 1% (slowdown) for enqueue, 2% (performance improvement) for dequeue. Overall number of operations (dequeue+enqueue) increased with the patch. We can verify that: successful enqueues - successful dequeues = end_dequeues For all runs (ensures correctness: no lost node). CC: Lai Jiangshan CC: Paul McKenney Signed-off-by: Mathieu Desnoyers --- diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h index 13b24ff..5363fe0 100644 --- a/urcu-call-rcu-impl.h +++ b/urcu-call-rcu-impl.h @@ -21,6 +21,7 @@ */ #define _GNU_SOURCE +#define _LGPL_SOURCE #include #include #include @@ -220,10 +221,7 @@ static void call_rcu_wake_up(struct call_rcu_data *crdp) static void *call_rcu_thread(void *arg) { unsigned long cbcount; - struct cds_wfq_node *cbs; - struct cds_wfq_node **cbs_tail; - struct call_rcu_data *crdp = (struct call_rcu_data *)arg; - struct rcu_head *rhp; + struct call_rcu_data *crdp = (struct call_rcu_data *) arg; int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); int ret; @@ -243,35 +241,29 @@ static void *call_rcu_thread(void *arg) cmm_smp_mb(); } for (;;) { - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) - poll(NULL, 0, 1); - _CMM_STORE_SHARED(crdp->cbs.head, NULL); - cbs_tail = (struct cds_wfq_node **) - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); + struct cds_wfq_queue cbs_tmp; + struct cds_wfq_node *cbs; + + cds_wfq_init(&cbs_tmp); + __cds_wfq_splice_blocking(&cbs_tmp, &crdp->cbs); + if (!cds_wfq_empty(&cbs_tmp)) { synchronize_rcu(); cbcount = 0; - do { - while (cbs->next == NULL && - &cbs->next != cbs_tail) - poll(NULL, 0, 1); - if (cbs == &crdp->cbs.dummy) { - cbs = cbs->next; - continue; - } - rhp = (struct rcu_head *)cbs; - cbs = cbs->next; + __cds_wfq_for_each_blocking(&cbs_tmp, cbs) { + struct rcu_head *rhp; + + rhp = caa_container_of(cbs, + struct rcu_head, next); rhp->func(rhp); cbcount++; - } while (cbs != NULL); + } uatomic_sub(&crdp->qlen, cbcount); } if (uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOP) break; rcu_thread_offline(); if (!rt) { - if (&crdp->cbs.head - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { + if (cds_wfq_empty(&crdp->cbs)) { call_rcu_wait(crdp); poll(NULL, 0, 10); uatomic_dec(&crdp->futex); @@ -625,32 +617,32 @@ void call_rcu(struct rcu_head *head, */ void call_rcu_data_free(struct call_rcu_data *crdp) { - struct cds_wfq_node *cbs; - struct cds_wfq_node **cbs_tail; - struct cds_wfq_node **cbs_endprev; - if (crdp == NULL || crdp == default_call_rcu_data) { return; } + if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); wake_call_rcu_thread(crdp); while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) poll(NULL, 0, 1); } - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) - poll(NULL, 0, 1); - _CMM_STORE_SHARED(crdp->cbs.head, NULL); - cbs_tail = (struct cds_wfq_node **) - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); + + if (!cds_wfq_empty(&crdp->cbs)) { + struct cds_wfq_queue cbs_tmp; + + cds_wfq_init(&cbs_tmp); + __cds_wfq_splice_blocking(&cbs_tmp, &crdp->cbs); + /* Create default call rcu data if need be */ (void) get_default_call_rcu_data(); - cbs_endprev = (struct cds_wfq_node **) - uatomic_xchg(&default_call_rcu_data, cbs_tail); - *cbs_endprev = cbs; + + __cds_wfq_splice_blocking(&default_call_rcu_data->cbs, + &cbs_tmp); + uatomic_add(&default_call_rcu_data->qlen, uatomic_read(&crdp->qlen)); + wake_call_rcu_thread(default_call_rcu_data); } diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h index 636e1af..08d8d52 100644 --- a/urcu/static/wfqueue.h +++ b/urcu/static/wfqueue.h @@ -9,7 +9,8 @@ * TO BE INCLUDED ONLY IN LGPL-COMPATIBLE CODE. See wfqueue.h for linking * dynamically with the userspace rcu library. * - * Copyright 2010 - Mathieu Desnoyers + * Copyright 2010-2012 - Mathieu Desnoyers + * Copyright 2011-2012 - Lai Jiangshan * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -29,6 +30,7 @@ #include #include #include +#include #include #include @@ -38,11 +40,16 @@ extern "C" { /* * Queue with wait-free enqueue/blocking dequeue. - * This implementation adds a dummy head node when the queue is empty to ensure - * we can always update the queue locklessly. * * Inspired from half-wait-free/half-blocking queue implementation done by * Paul E. McKenney. + * + * Caller must ensure mutual exclusion of queue update operations + * "dequeue" and "splice" source queue. Queue read operations "first" + * and "next" need to be protected against concurrent "dequeue" and + * "splice" (for source queue) by the caller. "enqueue", "splice" + * (destination queue), and "empty" are the only operations that can be + * used without any mutual exclusion. */ #define WFQ_ADAPT_ATTEMPTS 10 /* Retry if being set */ @@ -57,31 +64,51 @@ static inline void _cds_wfq_init(struct cds_wfq_queue *q) { int ret; - _cds_wfq_node_init(&q->dummy); /* Set queue head and tail */ - q->head = &q->dummy; - q->tail = &q->dummy.next; + _cds_wfq_node_init(&q->head); + q->tail = &q->head; ret = pthread_mutex_init(&q->lock, NULL); assert(!ret); } -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, - struct cds_wfq_node *node) +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) +{ + /* + * Queue is empty if no node is pointed by q->head.next nor q->tail. + */ + return CMM_LOAD_SHARED(q->head.next) == NULL + && CMM_LOAD_SHARED(q->tail) == &q->head; +} + +static inline void ___cds_wfq_append(struct cds_wfq_queue *q, + struct cds_wfq_node *new_head, + struct cds_wfq_node *new_tail) { - struct cds_wfq_node **old_tail; + struct cds_wfq_node *old_tail; /* - * uatomic_xchg() implicit memory barrier orders earlier stores to data - * structure containing node and setting node->next to NULL before - * publication. + * Implicit memory barrier before uatomic_xchg() orders earlier + * stores to data structure containing node and setting + * node->next to NULL before publication. */ - old_tail = uatomic_xchg(&q->tail, &node->next); + old_tail = uatomic_xchg(&q->tail, new_tail); + /* - * At this point, dequeuers see a NULL old_tail->next, which indicates - * that the queue is being appended to. The following store will append - * "node" to the queue from a dequeuer perspective. + * Implicit memory barrier after uatomic_xchg() orders store to + * q->tail before store to old_tail->next. + * + * At this point, dequeuers see a NULL q->tail->next, which + * indicates that the queue is being appended to. The following + * store will append "node" to the queue from a dequeuer + * perspective. */ - CMM_STORE_SHARED(*old_tail, node); + CMM_STORE_SHARED(old_tail->next, new_head); +} + +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, + struct cds_wfq_node *new_tail) +{ + ___cds_wfq_append(q, new_tail, new_tail); } /* @@ -100,14 +127,45 @@ ___cds_wfq_node_sync_next(struct cds_wfq_node *node) if (++attempt >= WFQ_ADAPT_ATTEMPTS) { poll(NULL, 0, WFQ_WAIT); /* Wait for 10ms */ attempt = 0; - } else + } else { caa_cpu_relax(); + } } return next; } /* + * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. + * + * Mutual exclusion with "dequeue" and "splice" operations must be ensured + * by the caller. + */ +static inline struct cds_wfq_node * +___cds_wfq_first_blocking(struct cds_wfq_queue *q) +{ + if (_cds_wfq_empty(q)) + return NULL; + return ___cds_wfq_node_sync_next(&q->head); +} + +/* + * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. + * + * Mutual exclusion with "dequeue" and "splice" operations must be ensured + * by the caller. + */ +static inline struct cds_wfq_node * +___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) +{ + if (CMM_LOAD_SHARED(q->tail) == node) + return NULL; + return ___cds_wfq_node_sync_next(node); +} + +/* + * ___cds_wfq_dequeue_blocking: dequeue a node from the queue. + * * It is valid to reuse and free a dequeued node immediately. * * No need to go on a waitqueue here, as there is no possible state in which the @@ -120,42 +178,123 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) { struct cds_wfq_node *node, *next; - /* - * Queue is empty if it only contains the dummy node. - */ - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) + if (_cds_wfq_empty(q)) return NULL; - node = q->head; - next = ___cds_wfq_node_sync_next(node); + node = ___cds_wfq_node_sync_next(&q->head); + + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { + /* Load node->next before q->tail */ + cmm_smp_rmb(); + if (CMM_LOAD_SHARED(q->tail) == node) { + /* + * @node is the only node in the queue. + * Try to move the tail to &q->head + */ + _cds_wfq_node_init(&q->head); + if (uatomic_cmpxchg(&q->tail, node, &q->head) == node) + return node; + } + next = ___cds_wfq_node_sync_next(node); + } /* * Move queue head forward. */ - q->head = next; + q->head.next = next; + + return node; +} + +/* + * ___cds_wfq_splice_blocking: enqueue all src_q nodes at the end of dest_q. + * + * Dequeue all nodes from src_q. + * dest_q must be already initialized. + * caller ensures mutual exclusion of dequeue and splice operations on + * src_q. + */ +static inline void +___cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q) +{ + struct cds_wfq_node *head, *tail; + + if (_cds_wfq_empty(src_q)) + return; + + head = ___cds_wfq_node_sync_next(&src_q->head); + _cds_wfq_node_init(&src_q->head); + + /* + * Memory barrier implied before uatomic_xchg() orders store to + * src_q->head before store to src_q->tail. This is required by + * concurrent enqueue on src_q, which exchanges the tail before + * updating the previous tail's next pointer. + */ + tail = uatomic_xchg(&src_q->tail, &src_q->head); + /* - * Requeue dummy node if we just dequeued it. + * Append the spliced content of src_q into dest_q. Does not + * require mutual exclusion on dest_q (wait-free). */ - if (node == &q->dummy) { - _cds_wfq_node_init(node); - _cds_wfq_enqueue(q, node); - return ___cds_wfq_dequeue_blocking(q); - } - return node; + ___cds_wfq_append(dest_q, head, tail); +} + +/* Locking performed within cds_wfq calls. */ +static inline struct cds_wfq_node * +_cds_wfq_first_blocking(struct cds_wfq_queue *q) +{ + struct cds_wfq_node *retval; + int ret; + + ret = pthread_mutex_lock(&q->lock); + assert(!ret); + retval = ___cds_wfq_first_blocking(q); + ret = pthread_mutex_unlock(&q->lock); + assert(!ret); + return retval; +} + +static inline struct cds_wfq_node * +_cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) +{ + struct cds_wfq_node *retval; + int ret; + + ret = pthread_mutex_lock(&q->lock); + assert(!ret); + retval = ___cds_wfq_next_blocking(q, node); + ret = pthread_mutex_unlock(&q->lock); + assert(!ret); + return retval; } static inline struct cds_wfq_node * _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) { - struct cds_wfq_node *retnode; + struct cds_wfq_node *retval; int ret; ret = pthread_mutex_lock(&q->lock); assert(!ret); - retnode = ___cds_wfq_dequeue_blocking(q); + retval = ___cds_wfq_dequeue_blocking(q); ret = pthread_mutex_unlock(&q->lock); assert(!ret); - return retnode; + return retval; +} + +static inline void +_cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q) +{ + int ret; + + ret = pthread_mutex_lock(&src_q->lock); + assert(!ret); + ___cds_wfq_splice_blocking(dest_q, src_q); + ret = pthread_mutex_unlock(&src_q->lock); + assert(!ret); } #ifdef __cplusplus diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h index 03a73f1..d33d47a 100644 --- a/urcu/wfqueue.h +++ b/urcu/wfqueue.h @@ -6,7 +6,8 @@ * * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue * - * Copyright 2010 - Mathieu Desnoyers + * Copyright 2010-2012 - Mathieu Desnoyers + * Copyright 2011-2012 - Lai Jiangshan * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -25,6 +26,7 @@ #include #include +#include #include #ifdef __cplusplus @@ -33,8 +35,6 @@ extern "C" { /* * Queue with wait-free enqueue/blocking dequeue. - * This implementation adds a dummy head node when the queue is empty to ensure - * we can always update the queue locklessly. * * Inspired from half-wait-free/half-blocking queue implementation done by * Paul E. McKenney. @@ -45,8 +45,8 @@ struct cds_wfq_node { }; struct cds_wfq_queue { - struct cds_wfq_node *head, **tail; - struct cds_wfq_node dummy; /* Dummy node */ + struct cds_wfq_node head, *tail; + struct cds_wfq_node padding; /* unused */ pthread_mutex_t lock; }; @@ -55,22 +55,90 @@ struct cds_wfq_queue { #include #define cds_wfq_node_init _cds_wfq_node_init -#define cds_wfq_init _cds_wfq_init -#define cds_wfq_enqueue _cds_wfq_enqueue -#define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking +#define cds_wfq_init _cds_wfq_init +#define cds_wfq_empty _cds_wfq_empty +#define cds_wfq_enqueue _cds_wfq_enqueue + +/* Locking performed within cds_wfq calls. */ #define cds_wfq_dequeue_blocking _cds_wfq_dequeue_blocking +#define cds_wfq_splice_blocking _cds_wfq_splice_blocking +#define cds_wfq_first_blocking _cds_wfq_first_blocking +#define cds_wfq_next_blocking _cds_wfq_next_blocking + +/* Locking ensured by caller */ +#define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking +#define __cds_wfq_splice_blocking ___cds_wfq_splice_blocking +#define __cds_wfq_first_blocking ___cds_wfq_first_blocking +#define __cds_wfq_next_blocking ___cds_wfq_next_blocking #else /* !_LGPL_SOURCE */ extern void cds_wfq_node_init(struct cds_wfq_node *node); extern void cds_wfq_init(struct cds_wfq_queue *q); +extern bool cds_wfq_empty(struct cds_wfq_queue *q); extern void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node); -/* __cds_wfq_dequeue_blocking: caller ensures mutual exclusion between dequeues */ -extern struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); + +/* Locking performed within cds_wfq calls. */ extern struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); +extern void cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q); +extern struct cds_wfq_node *cds_wfq_first_blocking(struct cds_wfq_queue *q); +extern struct cds_wfq_node *cds_wfq_next_blocking(struct cds_wfq_queue *q, + struct cds_wfq_node *node); + +/* + * __cds_wfq_dequeue_blocking: caller ensures mutual exclusion of dequeue + * and splice operations. + */ +extern struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); + +/* + * __cds_wfq_splice_blocking: caller ensures mutual exclusion of dequeue and + * splice operations on src_q. dest_q must be already initialized. + */ +extern void __cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q); + +/* + * __cds_wfq_first_blocking: mutual exclusion with "dequeue" and + * "splice" operations must be ensured by the caller. + */ +extern struct cds_wfq_node *__cds_wfq_first_blocking(struct cds_wfq_queue *q); + +/* + * __cds_wfq_next_blocking: mutual exclusion with "dequeue" and "splice" + * operations must be ensured by the caller. + */ +extern struct cds_wfq_node *__cds_wfq_next_blocking(struct cds_wfq_queue *q, + struct cds_wfq_node *node); #endif /* !_LGPL_SOURCE */ +/* + * cds_wfq_for_each_blocking: Iterate over all nodes in a queue, without + * dequeuing them. + * + * cds_wfq_for_each_blocking: mutual exclusion is performed within the + * cds_wfq calls. + */ +#define cds_wfq_for_each_blocking(q, node) \ + for (node = cds_wfq_first_blocking(q); \ + node != NULL; \ + node = cds_wfq_next_blocking(q, node)) + +/* + * __cds_wfq_for_each_blocking: Iterate over all nodes in a queue, + * without dequeuing them. + * + * Mutual exclusion with "dequeue" and "splice" operations must be + * ensured by the caller. + */ + +#define __cds_wfq_for_each_blocking(q, node) \ + for (node = __cds_wfq_first_blocking(q); \ + node != NULL; \ + node = __cds_wfq_next_blocking(q, node)) + #ifdef __cplusplus } #endif diff --git a/wfqueue.c b/wfqueue.c index 3337171..cf3dae6 100644 --- a/wfqueue.c +++ b/wfqueue.c @@ -3,7 +3,8 @@ * * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue * - * Copyright 2010 - Mathieu Desnoyers + * Copyright 2010-2012 - Mathieu Desnoyers + * Copyright 2011-2012 - Lai Jiangshan * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -38,17 +39,56 @@ void cds_wfq_init(struct cds_wfq_queue *q) _cds_wfq_init(q); } +bool cds_wfq_empty(struct cds_wfq_queue *q) +{ + return _cds_wfq_empty(q); +} + void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node) { _cds_wfq_enqueue(q, node); } +struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) +{ + return _cds_wfq_dequeue_blocking(q); +} + +void cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q) +{ + _cds_wfq_splice_blocking(dest_q, src_q); +} + +struct cds_wfq_node *cds_wfq_first_blocking(struct cds_wfq_queue *q) +{ + return _cds_wfq_first_blocking(q); +} + +struct cds_wfq_node *cds_wfq_next_blocking(struct cds_wfq_queue *q, + struct cds_wfq_node *node) +{ + return _cds_wfq_next_blocking(q, node); +} + struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) { return ___cds_wfq_dequeue_blocking(q); } -struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) +void __cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q) { - return _cds_wfq_dequeue_blocking(q); + ___cds_wfq_splice_blocking(dest_q, src_q); +} + +struct cds_wfq_node *__cds_wfq_first_blocking(struct cds_wfq_queue *q) +{ + return ___cds_wfq_first_blocking(q); +} + +struct cds_wfq_node *__cds_wfq_next_blocking(struct cds_wfq_queue *q, + struct cds_wfq_node *node) +{ + return ___cds_wfq_next_blocking(q, node); } -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From karim.yaghmour at opersys.com Sun Aug 12 16:08:59 2012 From: karim.yaghmour at opersys.com (Karim Yaghmour) Date: Sun, 12 Aug 2012 16:08:59 -0400 Subject: [lttng-dev] LTTng / Android questions In-Reply-To: <20120802215250.GB29960@Krystal> References: <20120724182031.GB2285@Krystal> <20120802215250.GB29960@Krystal> Message-ID: <50280D5B.6070100@opersys.com> Hi Mathieu, Apologies for the delayed response, I was in Bangalore for a few weeks with limited bandwidth (time-wise.) I haven't had the chance to try getting LTTng 2.0 to work on Android. The most up-to-date pointer I would've had was the Ericsson work which was pointed to in a later email in this thread. I remain, however, interested in any work in this direction, please keep me posted. Thanks, -- Karim Yaghmour CEO - Opersys inc. / www.opersys.com http://twitter.com/karimyaghmour On 12-08-02 05:52 PM, Mathieu Desnoyers wrote: > Hi Karim, > > Any word of advice about building LTTng 2.0 on Android ? See the > question below. > > Thanks, > > Mathieu > > * Glossop, Kent (kent.glossop at intel.com) wrote: >> Thanks. >> >> I've been trying to understand how to do the lttng cross build for >> Android. Is there a particular approach that you would expect people >> to most naturally use (e.g. if I were to write up a "how-to")? >> It seems like one approach is to use a full Android build area, adding >> the lttng components to the "external" directory, requiring Android.mk >> files. If the files aren't generated automatically, this adds >> maintenance. (There is a tool called androgenizer that can be used >> assist, but that seems undesirable to include for a "general" >> solution.) Another approach would be to build some of components with >> options to configure, in conjunction with an Android build (or NDK?) >> area. Would you view one of these (or something else), a "preferred" >> way? >> >> Thanks, >> Kent >> >> -----Original Message----- >> From: Mathieu Desnoyers [mailto:mathieu.desnoyers at efficios.com] >> Sent: Tuesday, July 24, 2012 2:21 PM >> To: Glossop, Kent >> Cc: christian.babeux at efficios.com >> Subject: Re: LTTng / Android questions >> >> Hi Kent, >> >> It should work, theoretically. Testing would be welcome. We did port of the lttng tools (except for the kernel tracer) to NetBSD/FreeBSD recently, where we had to circumvent lack of TLS support, so this requirement on the glibc is now gone from the 2.0 lttng series. >> >> If Android kernel is close enough to mainline, lttng-modules should work too. >> >> I'm CCing christian, who is currently looking at the embedded aspect of the continuous integration heterogenous cluster we are currently building. >> >> Thanks, >> >> Mathieu >> >> >> * Glossop, Kent (kent.glossop at intel.com) wrote: >>> Mathieu, >>> >>> I'm interested in LTTng for Android. If you have time, a few >>> questions... (Let me know if this would be better posted to >>> lttng-dev.) >>> >>> From what I can tell from previous postings to lttng-dev and other places: >>> >>> - People apparently used a previous version with a 2.x kernel with some changes and building it in >>> >>> - There was/is an issue with glibc vs. bionic for lttng's use of shared memory >>> >>> - Back in Feb. there was a demo done of LTTng on Android done using a copy of glibc >>> >>> - There are at least some changes being made that mention Android (e.g. a TLS change that mentioned Android about 2 months ago) >>> >>> What I'm interested in: >>> >>> - Are there directions somewhere for building LTTng for use with Android? (e.g. how to configure component and reference an Android build area rather than things for the native host?) >>> >>> - How much is expected to work on Android? >>> >>> - Is glibc still needed? >>> >>> - Are there people actively working on Android support (if it doesn't already work)? >>> >>> - Do you have a rough idea what might be involved for an Android x86 version beyond arm? >>> >>> Ideally, I would like to be able to install a minimal set of pieces, preferably on to a stock phone, collect LTTng traces, and use the viewer on linux (or maybe even windows if linux-tools works there.) Then, to do that with x86 Android in addition to arm... >>> >>> Thanks, >>> Kent Glossop, Intel >>> >> >> -- >> Mathieu Desnoyers >> Operating System Efficiency R&D Consultant EfficiOS Inc. >> http://www.efficios.com > From mathieu.desnoyers at efficios.com Mon Aug 13 09:22:31 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Mon, 13 Aug 2012 09:22:31 -0400 Subject: [lttng-dev] LTTng: problem with lttng-sessiond. In-Reply-To: References: Message-ID: <20120813132231.GA24219@Krystal> CCing David Goulet, maintainer of lttng-tools. He will look into your issue. Thanks, Mathieu * Henrik Hautakoski (henrik at fiktivkod.org) wrote: > Hi, We have some problems with spawning a session daemon. running the > command "lttng-sessiond --vvv --no-kernel" and > get "Illegal instruction" See output: > > DEBUG3: Creating LTTng run directory: /var/run/lttng [in > create_lttng_rundir() at main.c:5323] > DEBUG2: Kernel consumer err path: /var/run/lttng/kconsumerd/error [in > main() at main.c:5571] > DEBUG2: Kernel consumer cmd path: /var/run/lttng/kconsumerd/command > [in main() at main.c:5573] > DEBUG1: Client socket path /var/run/lttng/client-lttng-sessiond [in > main() at main.c:5626] > DEBUG1: Application socket path /var/run/lttng/apps-lttng-sessiond [in > main() at main.c:5627] > DEBUG1: LTTng run directory path: /var/run/lttng [in main() at main.c:5628] > DEBUG2: UST consumer 32 bits err path: > /var/run/lttng/ustconsumerd32/error [in main() at main.c:5637] > DEBUG2: UST consumer 32 bits cmd path: > /var/run/lttng/ustconsumerd32/command [in main() at main.c:5639] > DEBUG2: UST consumer 64 bits err path: > /var/run/lttng/ustconsumerd64/error [in main() at main.c:5648] > DEBUG2: UST consumer 64 bits cmd path: > /var/run/lttng/ustconsumerd64/command [in main() at main.c:5650] > DEBUG3: Created hashtable size 4 at 0x10064080 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG3: Created hashtable size 4 at 0x10064168 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG2: Creating consumer directory: /var/run/lttng/kconsumerd [in > set_consumer_sockets() at main.c:5365] > DEBUG2: Creating consumer directory: /var/run/lttng/ustconsumerd64 [in > set_consumer_sockets() at main.c:5365] > DEBUG2: Creating consumer directory: /var/run/lttng/ustconsumerd32 [in > set_consumer_sockets() at main.c:5365] > DEBUG1: Signal handler set for SIGTERM, SIGPIPE and SIGINT [in > set_signal_handler() at main.c:5457] > DEBUG1: All permissions are set [in set_permissions() at main.c:5310] > DEBUG1: epoll set max size is 180337 [in compat_epoll_set_max_size() > at compat-epoll.c:224] > Illegal instruction > > What to do? :P > > -- > Henrik Hautakoski > henrik at fiktivkod.org -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From eag0628 at gmail.com Mon Aug 13 09:59:18 2012 From: eag0628 at gmail.com (Lai Jiangshan) Date: Mon, 13 Aug 2012 21:59:18 +0800 Subject: [lttng-dev] [PATCH 2/2] urcu: new wfqueue implementation In-Reply-To: <20120811131417.GA13886@Krystal> References: <1344501986-23151-1-git-send-email-laijs@cn.fujitsu.com> <1344501986-23151-2-git-send-email-laijs@cn.fujitsu.com> <20120811131417.GA13886@Krystal> Message-ID: On Sat, Aug 11, 2012 at 9:14 PM, Mathieu Desnoyers wrote: > * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: >> Some guys would be surprised by this fact: >> There are already TWO implementations of wfqueue in urcu. >> >> The first one is in urcu/static/wfqueue.h: >> 1) enqueue: exchange the tail and then update previous->next >> 2) dequeue: wait for first node's next pointer and them shift, a dummy node >> is introduced to avoid the queue->tail become NULL when shift. >> >> The second one shares some code with the first one, and the left code >> are spreading in urcu-call-rcu-impl.h: >> 1) enqueue: share with the first one >> 2) no dequeue operation: and no shift, so it don't need dummy node, >> Although the dummy node is queued when initialization, but it is removed >> after the first dequeue_all operation in call_rcu_thread(). >> call_rcu_data_free() forgets to handle the dummy node if it is not removed. >> 3)dequeue_all: record the old head and tail, and queue->head become the special >> tail node.(atomic record the tail and change the tail). >> >> The second implementation's code are spreading, bad for review, and it is not >> tested by tests/test_urcu_wfq. >> >> So we need a better implementation avoid the dummy node dancing and can service >> both generic wfqueue APIs and dequeue_all API for call rcu. >> >> The new implementation: >> 1) enqueue: share with the first one/original implementation. >> 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. >> no dummy node, save memory. >> 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail >> and return the old head.next. >> >> More implementation details are in the code. >> tests/test_urcu_wfq will be update in future for testing new APIs. > > Hi Lai, > > Some other style-related questions below, > >> >> >> Signed-off-by: Lai Jiangshan >> --- >> urcu-call-rcu-impl.h | 50 ++++++++++-------------- >> urcu/static/wfqueue.h | 104 ++++++++++++++++++++++++++++++++++++------------ >> urcu/wfqueue.h | 25 ++++++++++-- >> wfqueue.c | 29 ++++++++++++++ >> 4 files changed, 149 insertions(+), 59 deletions(-) >> >> diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h >> index 13b24ff..dbfb410 100644 >> --- a/urcu-call-rcu-impl.h >> +++ b/urcu-call-rcu-impl.h >> @@ -221,7 +221,7 @@ static void *call_rcu_thread(void *arg) >> { >> unsigned long cbcount; >> struct cds_wfq_node *cbs; >> - struct cds_wfq_node **cbs_tail; >> + struct cds_wfq_node *cbs_tail; >> struct call_rcu_data *crdp = (struct call_rcu_data *)arg; >> struct rcu_head *rhp; >> int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); >> @@ -243,24 +243,18 @@ static void *call_rcu_thread(void *arg) >> cmm_smp_mb(); >> } >> for (;;) { >> - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { >> - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) >> - poll(NULL, 0, 1); >> - _CMM_STORE_SHARED(crdp->cbs.head, NULL); >> - cbs_tail = (struct cds_wfq_node **) >> - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); >> + cbs = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &cbs_tail); >> + if (cbs) { >> synchronize_rcu(); >> cbcount = 0; >> do { >> - while (cbs->next == NULL && >> - &cbs->next != cbs_tail) >> - poll(NULL, 0, 1); >> - if (cbs == &crdp->cbs.dummy) { >> - cbs = cbs->next; >> - continue; >> - } >> rhp = (struct rcu_head *)cbs; >> - cbs = cbs->next; >> + >> + if (cbs != cbs_tail) >> + cbs = __cds_wfq_node_sync_next(cbs); >> + else >> + cbs = NULL; >> + >> rhp->func(rhp); >> cbcount++; >> } while (cbs != NULL); >> @@ -270,8 +264,7 @@ static void *call_rcu_thread(void *arg) >> break; >> rcu_thread_offline(); >> if (!rt) { >> - if (&crdp->cbs.head >> - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { >> + if (cds_wfq_empty(&crdp->cbs)) { >> call_rcu_wait(crdp); >> poll(NULL, 0, 10); >> uatomic_dec(&crdp->futex); >> @@ -625,32 +618,31 @@ void call_rcu(struct rcu_head *head, >> */ >> void call_rcu_data_free(struct call_rcu_data *crdp) >> { >> - struct cds_wfq_node *cbs; >> - struct cds_wfq_node **cbs_tail; >> - struct cds_wfq_node **cbs_endprev; >> + struct cds_wfq_node *head, *tail; >> >> if (crdp == NULL || crdp == default_call_rcu_data) { >> return; >> } >> + >> if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { >> uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); >> wake_call_rcu_thread(crdp); >> while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) >> poll(NULL, 0, 1); >> } >> - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { >> - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) >> - poll(NULL, 0, 1); >> - _CMM_STORE_SHARED(crdp->cbs.head, NULL); >> - cbs_tail = (struct cds_wfq_node **) >> - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); >> + >> + if (!cds_wfq_empty(&crdp->cbs)) { >> + head = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &tail); >> + assert(head); >> + >> /* Create default call rcu data if need be */ >> (void) get_default_call_rcu_data(); >> - cbs_endprev = (struct cds_wfq_node **) >> - uatomic_xchg(&default_call_rcu_data, cbs_tail); >> - *cbs_endprev = cbs; >> + >> + __cds_wfq_append_list(&default_call_rcu_data->cbs, head, tail); >> + >> uatomic_add(&default_call_rcu_data->qlen, >> uatomic_read(&crdp->qlen)); >> + >> wake_call_rcu_thread(default_call_rcu_data); >> } >> >> diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h >> index 636e1af..15ea9fc 100644 >> --- a/urcu/static/wfqueue.h >> +++ b/urcu/static/wfqueue.h >> @@ -10,6 +10,7 @@ >> * dynamically with the userspace rcu library. >> * >> * Copyright 2010 - Mathieu Desnoyers >> + * Copyright 2011-2012 - Lai Jiangshan >> * >> * This library is free software; you can redistribute it and/or >> * modify it under the terms of the GNU Lesser General Public >> @@ -29,6 +30,7 @@ >> #include >> #include >> #include >> +#include >> #include >> #include >> >> @@ -38,8 +40,6 @@ extern "C" { >> >> /* >> * Queue with wait-free enqueue/blocking dequeue. >> - * This implementation adds a dummy head node when the queue is empty to ensure >> - * we can always update the queue locklessly. >> * >> * Inspired from half-wait-free/half-blocking queue implementation done by >> * Paul E. McKenney. >> @@ -57,31 +57,43 @@ static inline void _cds_wfq_init(struct cds_wfq_queue *q) >> { >> int ret; >> >> - _cds_wfq_node_init(&q->dummy); >> /* Set queue head and tail */ >> - q->head = &q->dummy; >> - q->tail = &q->dummy.next; >> + _cds_wfq_node_init(&q->head); >> + q->tail = &q->head; >> ret = pthread_mutex_init(&q->lock, NULL); >> assert(!ret); >> } >> >> -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, >> - struct cds_wfq_node *node) >> +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) >> { >> - struct cds_wfq_node **old_tail; >> + /* >> + * Queue is empty if no node is pointed by q->head.next nor q->tail. >> + */ >> + return q->head.next == NULL && CMM_LOAD_SHARED(q->tail) == &q->head; >> +} >> >> +static inline void ___cds_wfq_append_list(struct cds_wfq_queue *q, >> + struct cds_wfq_node *head, struct cds_wfq_node *tail) >> +{ >> /* >> * uatomic_xchg() implicit memory barrier orders earlier stores to data >> * structure containing node and setting node->next to NULL before >> * publication. >> */ >> - old_tail = uatomic_xchg(&q->tail, &node->next); >> + tail = uatomic_xchg(&q->tail, tail); > > I'd prefer to keep "old_tail" here, because it becomes clearer to anyone > reviewing that uatomic_xchg() returns the old tail (and this extra > clarity comes without any overhead). > >> + >> /* >> - * At this point, dequeuers see a NULL old_tail->next, which indicates >> + * At this point, dequeuers see a NULL tail->next, which indicates >> * that the queue is being appended to. The following store will append >> * "node" to the queue from a dequeuer perspective. >> */ >> - CMM_STORE_SHARED(*old_tail, node); >> + CMM_STORE_SHARED(tail->next, head); >> +} >> + >> +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, >> + struct cds_wfq_node *node) >> +{ >> + ___cds_wfq_append_list(q, node, node); >> } > > Why not keep ___cds_wfq_append_list() merged into _cds_wfq_enqueue() ? > > This would keep the number of symbols exported minimal. > > So if I get it right, one "_" prefix is the "normally used" functions > (exposed through the LGPL symbol API). > > The "__" prefix are somewhat more internal, but can also be used > externally. > > Finally, the "___" prefix seem to be quite similar to the > double-underscores. > > We might need more consistency, I'm not sure the triple-underscores are > needed. Also, I'm not sure should export the double-underscore functions > outside of LGPL use (in other words, maybe we should not expose them to > !LGPL_SOURCE code). So we would emit the static inlines, but no symbols > for those. This covers ___cds_wfq_node_sync_next(), and > ___cds_wfq_dequeue_all_blocking (which requires the caller to use > sync_next). Currently, all code that needs to fine-grained integration > is within the userspace RCU tree, which defines LGPL_SOURCE. The mean of _xfunction() is not defined by me, I guess it is a function implemented in urcu/static/ which don't care about LGPL_SOURCE. (xfunction() is the same as _xfunction() but different wrapped-way with LGPL_SOURCE or not) __xfunction() (double-underscores) in my patch means "this function will not call pthread_mutex_lock(&q->lock), it is caller's responsibility for synchronization" if xfunction() is already underscored, __xfunction() will become triple-underscores. > >> >> /* >> @@ -120,27 +132,46 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) >> { >> struct cds_wfq_node *node, *next; >> >> - /* >> - * Queue is empty if it only contains the dummy node. >> - */ >> - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) >> + if (_cds_wfq_empty(q)) >> return NULL; >> - node = q->head; >> >> - next = ___cds_wfq_node_sync_next(node); >> + node = ___cds_wfq_node_sync_next(&q->head); >> + >> + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { >> + if (CMM_LOAD_SHARED(q->tail) == node) { >> + /* >> + * @node is the only node in the queue. >> + * Try to move the tail to &q->head >> + */ >> + _cds_wfq_node_init(&q->head); >> + if (uatomic_cmpxchg(&q->tail, node, &q->head) == node) >> + return node; >> + } >> + next = ___cds_wfq_node_sync_next(node); >> + } >> >> /* >> * Move queue head forward. >> */ >> - q->head = next; >> - /* >> - * Requeue dummy node if we just dequeued it. >> - */ >> - if (node == &q->dummy) { >> - _cds_wfq_node_init(node); >> - _cds_wfq_enqueue(q, node); >> - return ___cds_wfq_dequeue_blocking(q); >> - } >> + q->head.next = next; >> + >> + return node; >> +} >> + >> +/* dequeue all nodes, the nodes are not synchronized for the next pointer */ >> +static inline struct cds_wfq_node * >> +___cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, >> + struct cds_wfq_node **tail) >> +{ >> + struct cds_wfq_node *node; >> + >> + if (_cds_wfq_empty(q)) >> + return NULL; >> + >> + node = ___cds_wfq_node_sync_next(&q->head); >> + _cds_wfq_node_init(&q->head); >> + *tail = uatomic_xchg(&q->tail, &q->head); >> + >> return node; >> } >> >> @@ -158,6 +189,27 @@ _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) >> return retnode; >> } >> >> +static inline struct cds_wfq_node * >> +_cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, >> + struct cds_wfq_node **tail) >> +{ >> + struct cds_wfq_node *node, *next; >> + int ret; >> + >> + ret = pthread_mutex_lock(&q->lock); >> + assert(!ret); >> + node = ___cds_wfq_dequeue_all_blocking(q, tail); >> + ret = pthread_mutex_unlock(&q->lock); >> + assert(!ret); > > So we take the queue lock on dequeue_all, but not on dequeue. ?! dequeue operation still takes the lock, I didn't change _dequeue() function, so it does not appear in the patch. > It might > be good to have a consistent behavior: either we lock dequeue and > dequeue_all, or leave the lock entirely to the caller (and document it). _dequeue() and _dequeue_all() take the queue lock. ___dequeue() and ___dequeue_all() don't. thanks, Lai From eag0628 at gmail.com Mon Aug 13 11:13:20 2012 From: eag0628 at gmail.com (Lai Jiangshan) Date: Mon, 13 Aug 2012 23:13:20 +0800 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost In-Reply-To: <20120812145000.GA32747@Krystal> References: <20120812145000.GA32747@Krystal> Message-ID: On Sun, Aug 12, 2012 at 10:50 PM, Mathieu Desnoyers wrote: > This work is derived from the patch from Lai Jiangshan submitted as > "urcu: new wfqueue implementation" > (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) > > Its changelog: > >> Some guys would be surprised by this fact: >> There are already TWO implementations of wfqueue in urcu. >> >> The first one is in urcu/static/wfqueue.h: >> 1) enqueue: exchange the tail and then update previous->next >> 2) dequeue: wait for first node's next pointer and them shift, a dummy node >> is introduced to avoid the queue->tail become NULL when shift. >> >> The second one shares some code with the first one, and the left code >> are spreading in urcu-call-rcu-impl.h: >> 1) enqueue: share with the first one >> 2) no dequeue operation: and no shift, so it don't need dummy node, >> Although the dummy node is queued when initialization, but it is removed >> after the first dequeue_all operation in call_rcu_thread(). >> call_rcu_data_free() forgets to handle the dummy node if it is not removed. >> 3)dequeue_all: record the old head and tail, and queue->head become the special >> tail node.(atomic record the tail and change the tail). >> >> The second implementation's code are spreading, bad for review, and it is not >> tested by tests/test_urcu_wfq. >> >> So we need a better implementation avoid the dummy node dancing and can service >> both generic wfqueue APIs and dequeue_all API for call rcu. >> >> The new implementation: >> 1) enqueue: share with the first one/original implementation. >> 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. >> no dummy node, save memory. >> 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail >> and return the old head.next. >> >> More implementation details are in the code. >> tests/test_urcu_wfq will be update in future for testing new APIs. > > The patch proposed by Lai brings a very interesting simplification to > the single-node handling (which is kept here), and moves all queue > handling code away from call_rcu implementation, back into the wfqueue > code. This has the benefit to allow testing enhancements. > > I modified it so the API does not expose implementation details to the > user (e.g. ___cds_wfq_node_sync_next). I added a "splice" operation and > a for loop iterator which should allow wfqueue users to use the list > very efficiently both from LGPL/GPL code and from non-LGPL-compatible > code. > > Benchmarks performed on Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz > (dual-core, with hyperthreading) > > Benchmark invoked: > test_urcu_wfq 2 2 10 > > Only did 2 runs, but a small improvement seems to be clear for the > dequeue speed: > > Before patch: > > testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 136251248 nr_dequeues 54694027 successful enqueues 136251248 successful dequeues 54693904 end_dequeues 81557344 nr_ops 190945275 > testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 137258881 nr_dequeues 54463340 successful enqueues 137258881 successful dequeues 54463238 end_dequeues 82795643 nr_ops 191722221 > > After patch: > > testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 138589301 nr_dequeues 56911253 successful enqueues 138589301 successful dequeues 56910916 end_dequeues 81678385 nr_ops 195500554 > testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 139007622 nr_dequeues 57281502 successful enqueues 139007622 successful dequeues 57281348 end_dequeues 81726274 nr_ops 196289124 > > Summary: Number of enqueues is slightly lower, ?! I see the nr_enqueues and successful enqueues are both increased after after patch. > probably due to higher > dequeue rate. Number of dequeue increased. Respective rate change is > within 1% (slowdown) for enqueue, 2% (performance improvement) for > dequeue. Overall number of operations (dequeue+enqueue) increased with > the patch. > > We can verify that: > successful enqueues - successful dequeues = end_dequeues > > For all runs (ensures correctness: no lost node). > > CC: Lai Jiangshan > CC: Paul McKenney > Signed-off-by: Mathieu Desnoyers > --- > diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h > index 13b24ff..5363fe0 100644 > --- a/urcu-call-rcu-impl.h > +++ b/urcu-call-rcu-impl.h > @@ -21,6 +21,7 @@ > */ > > #define _GNU_SOURCE > +#define _LGPL_SOURCE > #include > #include > #include > @@ -220,10 +221,7 @@ static void call_rcu_wake_up(struct call_rcu_data *crdp) > static void *call_rcu_thread(void *arg) > { > unsigned long cbcount; > - struct cds_wfq_node *cbs; > - struct cds_wfq_node **cbs_tail; > - struct call_rcu_data *crdp = (struct call_rcu_data *)arg; > - struct rcu_head *rhp; > + struct call_rcu_data *crdp = (struct call_rcu_data *) arg; > int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); > int ret; > > @@ -243,35 +241,29 @@ static void *call_rcu_thread(void *arg) > cmm_smp_mb(); > } > for (;;) { > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > - poll(NULL, 0, 1); > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > - cbs_tail = (struct cds_wfq_node **) > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > + struct cds_wfq_queue cbs_tmp; > + struct cds_wfq_node *cbs; > + > + cds_wfq_init(&cbs_tmp); > + __cds_wfq_splice_blocking(&cbs_tmp, &crdp->cbs); > + if (!cds_wfq_empty(&cbs_tmp)) { > synchronize_rcu(); > cbcount = 0; > - do { > - while (cbs->next == NULL && > - &cbs->next != cbs_tail) > - poll(NULL, 0, 1); > - if (cbs == &crdp->cbs.dummy) { > - cbs = cbs->next; > - continue; > - } > - rhp = (struct rcu_head *)cbs; > - cbs = cbs->next; > + __cds_wfq_for_each_blocking(&cbs_tmp, cbs) { > + struct rcu_head *rhp; > + > + rhp = caa_container_of(cbs, > + struct rcu_head, next); > rhp->func(rhp); cbs is freed hear, but it will be used in __cds_wfq_next_blocking(). Introduce __cds_wfq_for_each_blocking_safe() ? > cbcount++; > - } while (cbs != NULL); > + } > uatomic_sub(&crdp->qlen, cbcount); > } > if (uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOP) > break; > rcu_thread_offline(); > if (!rt) { > - if (&crdp->cbs.head > - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { > + if (cds_wfq_empty(&crdp->cbs)) { > call_rcu_wait(crdp); > poll(NULL, 0, 10); > uatomic_dec(&crdp->futex); > @@ -625,32 +617,32 @@ void call_rcu(struct rcu_head *head, > */ > void call_rcu_data_free(struct call_rcu_data *crdp) > { > - struct cds_wfq_node *cbs; > - struct cds_wfq_node **cbs_tail; > - struct cds_wfq_node **cbs_endprev; > - > if (crdp == NULL || crdp == default_call_rcu_data) { > return; > } > + > if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { > uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); > wake_call_rcu_thread(crdp); > while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) > poll(NULL, 0, 1); > } > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > - poll(NULL, 0, 1); > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > - cbs_tail = (struct cds_wfq_node **) > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > + > + if (!cds_wfq_empty(&crdp->cbs)) { > + struct cds_wfq_queue cbs_tmp; > + > + cds_wfq_init(&cbs_tmp); > + __cds_wfq_splice_blocking(&cbs_tmp, &crdp->cbs); > + > /* Create default call rcu data if need be */ > (void) get_default_call_rcu_data(); > - cbs_endprev = (struct cds_wfq_node **) > - uatomic_xchg(&default_call_rcu_data, cbs_tail); > - *cbs_endprev = cbs; > + > + __cds_wfq_splice_blocking(&default_call_rcu_data->cbs, > + &cbs_tmp); > + Too much code to me, cbs_tmp is not required here. /* Create default call rcu data if need be */ (void) get_default_call_rcu_data(); + __cds_wfq_splice_blocking(&default_call_rcu_data->cbs,&crdp->cbs); > uatomic_add(&default_call_rcu_data->qlen, > uatomic_read(&crdp->qlen)); > + > wake_call_rcu_thread(default_call_rcu_data); > } > > diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h > index 636e1af..08d8d52 100644 > --- a/urcu/static/wfqueue.h > +++ b/urcu/static/wfqueue.h > @@ -9,7 +9,8 @@ > * TO BE INCLUDED ONLY IN LGPL-COMPATIBLE CODE. See wfqueue.h for linking > * dynamically with the userspace rcu library. > * > - * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2010-2012 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -29,6 +30,7 @@ > #include > #include > #include > +#include > #include > #include > > @@ -38,11 +40,16 @@ extern "C" { > > /* > * Queue with wait-free enqueue/blocking dequeue. > - * This implementation adds a dummy head node when the queue is empty to ensure > - * we can always update the queue locklessly. > * > * Inspired from half-wait-free/half-blocking queue implementation done by > * Paul E. McKenney. > + * > + * Caller must ensure mutual exclusion of queue update operations > + * "dequeue" and "splice" source queue. Queue read operations "first" > + * and "next" need to be protected against concurrent "dequeue" and > + * "splice" (for source queue) by the caller. "enqueue", "splice" > + * (destination queue), and "empty" are the only operations that can be > + * used without any mutual exclusion. > */ > > #define WFQ_ADAPT_ATTEMPTS 10 /* Retry if being set */ > @@ -57,31 +64,51 @@ static inline void _cds_wfq_init(struct cds_wfq_queue *q) > { > int ret; > > - _cds_wfq_node_init(&q->dummy); > /* Set queue head and tail */ > - q->head = &q->dummy; > - q->tail = &q->dummy.next; > + _cds_wfq_node_init(&q->head); > + q->tail = &q->head; > ret = pthread_mutex_init(&q->lock, NULL); > assert(!ret); > } > > -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > - struct cds_wfq_node *node) > +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) > +{ > + /* > + * Queue is empty if no node is pointed by q->head.next nor q->tail. > + */ > + return CMM_LOAD_SHARED(q->head.next) == NULL > + && CMM_LOAD_SHARED(q->tail) == &q->head; > +} > + > +static inline void ___cds_wfq_append(struct cds_wfq_queue *q, > + struct cds_wfq_node *new_head, > + struct cds_wfq_node *new_tail) > { > - struct cds_wfq_node **old_tail; > + struct cds_wfq_node *old_tail; > > /* > - * uatomic_xchg() implicit memory barrier orders earlier stores to data > - * structure containing node and setting node->next to NULL before > - * publication. > + * Implicit memory barrier before uatomic_xchg() orders earlier > + * stores to data structure containing node and setting > + * node->next to NULL before publication. > */ > - old_tail = uatomic_xchg(&q->tail, &node->next); > + old_tail = uatomic_xchg(&q->tail, new_tail); > + > /* > - * At this point, dequeuers see a NULL old_tail->next, which indicates > - * that the queue is being appended to. The following store will append > - * "node" to the queue from a dequeuer perspective. > + * Implicit memory barrier after uatomic_xchg() orders store to > + * q->tail before store to old_tail->next. > + * > + * At this point, dequeuers see a NULL q->tail->next, which > + * indicates that the queue is being appended to. The following > + * store will append "node" to the queue from a dequeuer > + * perspective. > */ > - CMM_STORE_SHARED(*old_tail, node); > + CMM_STORE_SHARED(old_tail->next, new_head); > +} > + > +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > + struct cds_wfq_node *new_tail) > +{ > + ___cds_wfq_append(q, new_tail, new_tail); > } > > /* > @@ -100,14 +127,45 @@ ___cds_wfq_node_sync_next(struct cds_wfq_node *node) > if (++attempt >= WFQ_ADAPT_ATTEMPTS) { > poll(NULL, 0, WFQ_WAIT); /* Wait for 10ms */ > attempt = 0; > - } else > + } else { > caa_cpu_relax(); > + } > } > > return next; > } > > /* > + * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. > + * > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured > + * by the caller. > + */ > +static inline struct cds_wfq_node * > +___cds_wfq_first_blocking(struct cds_wfq_queue *q) > +{ > + if (_cds_wfq_empty(q)) > + return NULL; > + return ___cds_wfq_node_sync_next(&q->head); > +} > + > +/* > + * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. > + * > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured > + * by the caller. > + */ > +static inline struct cds_wfq_node * > +___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > +{ > + if (CMM_LOAD_SHARED(q->tail) == node) > + return NULL; > + return ___cds_wfq_node_sync_next(node); > +} The same BUG as you told me. If q has only one node just enqueued by other thread. but if q->head.next is seen, ___cds_wfq_first_blocking() returns a node, And the update of q->tail is not seen, it is still &q->head, ___cds_wfq_node_sync_next(node) will be loop for every if there is no other enqueue. static inline struct cds_wfq_node * ___cds_wfq_first_blocking(struct cds_wfq_queue *q) { + struct cds_wfq_node *ret. if (_cds_wfq_empty(q)) return NULL; ret = ___cds_wfq_node_sync_next(&q->head); + cmm_smp_rmb(); + return ret; } > + > +/* > + * ___cds_wfq_dequeue_blocking: dequeue a node from the queue. > + * > * It is valid to reuse and free a dequeued node immediately. > * > * No need to go on a waitqueue here, as there is no possible state in which the > @@ -120,42 +178,123 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > { > struct cds_wfq_node *node, *next; > > - /* > - * Queue is empty if it only contains the dummy node. > - */ > - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) > + if (_cds_wfq_empty(q)) > return NULL; > - node = q->head; > > - next = ___cds_wfq_node_sync_next(node); > + node = ___cds_wfq_node_sync_next(&q->head); > + > + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { > + /* Load node->next before q->tail */ > + cmm_smp_rmb(); > + if (CMM_LOAD_SHARED(q->tail) == node) { I don't know why I added this "if" since it is likely true. Could you remove the above 3 lines? (I remember there is a mb() before uatomic_cmpxchg() which means this mb() is before the test in uatomic_cmpxchg()) > + /* > + * @node is the only node in the queue. > + * Try to move the tail to &q->head > + */ > + _cds_wfq_node_init(&q->head); > + if (uatomic_cmpxchg(&q->tail, node, &q->head) == node) > + return node; > + } > + next = ___cds_wfq_node_sync_next(node); > + } > > /* > * Move queue head forward. > */ > - q->head = next; > + q->head.next = next; > + > + return node; > +} > + > +/* > + * ___cds_wfq_splice_blocking: enqueue all src_q nodes at the end of dest_q. > + * > + * Dequeue all nodes from src_q. > + * dest_q must be already initialized. > + * caller ensures mutual exclusion of dequeue and splice operations on > + * src_q. > + */ > +static inline void > +___cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, > + struct cds_wfq_queue *src_q) > +{ > + struct cds_wfq_node *head, *tail; > + > + if (_cds_wfq_empty(src_q)) > + return; > + > + head = ___cds_wfq_node_sync_next(&src_q->head); > + _cds_wfq_node_init(&src_q->head); > + > + /* > + * Memory barrier implied before uatomic_xchg() orders store to > + * src_q->head before store to src_q->tail. This is required by > + * concurrent enqueue on src_q, which exchanges the tail before > + * updating the previous tail's next pointer. > + */ > + tail = uatomic_xchg(&src_q->tail, &src_q->head); > + > /* > - * Requeue dummy node if we just dequeued it. > + * Append the spliced content of src_q into dest_q. Does not > + * require mutual exclusion on dest_q (wait-free). > */ > - if (node == &q->dummy) { > - _cds_wfq_node_init(node); > - _cds_wfq_enqueue(q, node); > - return ___cds_wfq_dequeue_blocking(q); > - } > - return node; > + ___cds_wfq_append(dest_q, head, tail); > +} > + > +/* Locking performed within cds_wfq calls. */ > +static inline struct cds_wfq_node * > +_cds_wfq_first_blocking(struct cds_wfq_queue *q) > +{ > + struct cds_wfq_node *retval; > + int ret; > + > + ret = pthread_mutex_lock(&q->lock); > + assert(!ret); > + retval = ___cds_wfq_first_blocking(q); > + ret = pthread_mutex_unlock(&q->lock); > + assert(!ret); > + return retval; > +} > + > +static inline struct cds_wfq_node * > +_cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > +{ > + struct cds_wfq_node *retval; > + int ret; > + > + ret = pthread_mutex_lock(&q->lock); > + assert(!ret); > + retval = ___cds_wfq_next_blocking(q, node); > + ret = pthread_mutex_unlock(&q->lock); > + assert(!ret); > + return retval; > } I reject these _cds_wfq_first_blocking(), _cds_wfq_next_blocking() and cds_wfq_for_each_blocking(), because the claimed "Locking" makes no sense: 1. It protects nothing in _cds_wfq_next_blocking(). 2. There is no "Locking" in the loop body, @node is not dequeued, it will be invalid if some other dequeue it, and _cds_wfq_next_blocking() results BUG. > > static inline struct cds_wfq_node * > _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > { > - struct cds_wfq_node *retnode; > + struct cds_wfq_node *retval; > int ret; > > ret = pthread_mutex_lock(&q->lock); > assert(!ret); > - retnode = ___cds_wfq_dequeue_blocking(q); > + retval = ___cds_wfq_dequeue_blocking(q); > ret = pthread_mutex_unlock(&q->lock); > assert(!ret); > - return retnode; > + return retval; > +} > + > +static inline void > +_cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, > + struct cds_wfq_queue *src_q) > +{ > + int ret; > + > + ret = pthread_mutex_lock(&src_q->lock); > + assert(!ret); > + ___cds_wfq_splice_blocking(dest_q, src_q); > + ret = pthread_mutex_unlock(&src_q->lock); > + assert(!ret); > } > > #ifdef __cplusplus > diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h > index 03a73f1..d33d47a 100644 > --- a/urcu/wfqueue.h > +++ b/urcu/wfqueue.h > @@ -6,7 +6,8 @@ > * > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > * > - * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2010-2012 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -25,6 +26,7 @@ > > #include > #include > +#include > #include > > #ifdef __cplusplus > @@ -33,8 +35,6 @@ extern "C" { > > /* > * Queue with wait-free enqueue/blocking dequeue. > - * This implementation adds a dummy head node when the queue is empty to ensure > - * we can always update the queue locklessly. > * > * Inspired from half-wait-free/half-blocking queue implementation done by > * Paul E. McKenney. > @@ -45,8 +45,8 @@ struct cds_wfq_node { > }; > > struct cds_wfq_queue { > - struct cds_wfq_node *head, **tail; > - struct cds_wfq_node dummy; /* Dummy node */ > + struct cds_wfq_node head, *tail; > + struct cds_wfq_node padding; /* unused */ > pthread_mutex_t lock; > }; Why keep the padding? > > @@ -55,22 +55,90 @@ struct cds_wfq_queue { > #include > > #define cds_wfq_node_init _cds_wfq_node_init > -#define cds_wfq_init _cds_wfq_init > -#define cds_wfq_enqueue _cds_wfq_enqueue > -#define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking > +#define cds_wfq_init _cds_wfq_init > +#define cds_wfq_empty _cds_wfq_empty > +#define cds_wfq_enqueue _cds_wfq_enqueue > + > +/* Locking performed within cds_wfq calls. */ > #define cds_wfq_dequeue_blocking _cds_wfq_dequeue_blocking > +#define cds_wfq_splice_blocking _cds_wfq_splice_blocking > +#define cds_wfq_first_blocking _cds_wfq_first_blocking > +#define cds_wfq_next_blocking _cds_wfq_next_blocking > + > +/* Locking ensured by caller */ > +#define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking > +#define __cds_wfq_splice_blocking ___cds_wfq_splice_blocking > +#define __cds_wfq_first_blocking ___cds_wfq_first_blocking > +#define __cds_wfq_next_blocking ___cds_wfq_next_blocking > > #else /* !_LGPL_SOURCE */ > > extern void cds_wfq_node_init(struct cds_wfq_node *node); > extern void cds_wfq_init(struct cds_wfq_queue *q); > +extern bool cds_wfq_empty(struct cds_wfq_queue *q); > extern void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node); > -/* __cds_wfq_dequeue_blocking: caller ensures mutual exclusion between dequeues */ > -extern struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); > + > +/* Locking performed within cds_wfq calls. */ > extern struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); > +extern void cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, > + struct cds_wfq_queue *src_q); > +extern struct cds_wfq_node *cds_wfq_first_blocking(struct cds_wfq_queue *q); > +extern struct cds_wfq_node *cds_wfq_next_blocking(struct cds_wfq_queue *q, > + struct cds_wfq_node *node); > + > +/* > + * __cds_wfq_dequeue_blocking: caller ensures mutual exclusion of dequeue > + * and splice operations. > + */ > +extern struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); > + > +/* > + * __cds_wfq_splice_blocking: caller ensures mutual exclusion of dequeue and > + * splice operations on src_q. dest_q must be already initialized. > + */ > +extern void __cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, > + struct cds_wfq_queue *src_q); > + > +/* > + * __cds_wfq_first_blocking: mutual exclusion with "dequeue" and > + * "splice" operations must be ensured by the caller. > + */ > +extern struct cds_wfq_node *__cds_wfq_first_blocking(struct cds_wfq_queue *q); > + > +/* > + * __cds_wfq_next_blocking: mutual exclusion with "dequeue" and "splice" > + * operations must be ensured by the caller. > + */ > +extern struct cds_wfq_node *__cds_wfq_next_blocking(struct cds_wfq_queue *q, > + struct cds_wfq_node *node); > > #endif /* !_LGPL_SOURCE */ > > +/* > + * cds_wfq_for_each_blocking: Iterate over all nodes in a queue, without > + * dequeuing them. > + * > + * cds_wfq_for_each_blocking: mutual exclusion is performed within the > + * cds_wfq calls. > + */ > +#define cds_wfq_for_each_blocking(q, node) \ > + for (node = cds_wfq_first_blocking(q); \ > + node != NULL; \ > + node = cds_wfq_next_blocking(q, node)) > + > +/* > + * __cds_wfq_for_each_blocking: Iterate over all nodes in a queue, > + * without dequeuing them. > + * > + * Mutual exclusion with "dequeue" and "splice" operations must be > + * ensured by the caller. > + */ > + > +#define __cds_wfq_for_each_blocking(q, node) \ > + for (node = __cds_wfq_first_blocking(q); \ > + node != NULL; \ > + node = __cds_wfq_next_blocking(q, node)) > + > #ifdef __cplusplus > } > #endif > diff --git a/wfqueue.c b/wfqueue.c > index 3337171..cf3dae6 100644 > --- a/wfqueue.c > +++ b/wfqueue.c > @@ -3,7 +3,8 @@ > * > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > * > - * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2010-2012 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -38,17 +39,56 @@ void cds_wfq_init(struct cds_wfq_queue *q) > _cds_wfq_init(q); > } > > +bool cds_wfq_empty(struct cds_wfq_queue *q) > +{ > + return _cds_wfq_empty(q); > +} > + > void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node) > { > _cds_wfq_enqueue(q, node); > } > > +struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > +{ > + return _cds_wfq_dequeue_blocking(q); > +} > + > +void cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, > + struct cds_wfq_queue *src_q) > +{ > + _cds_wfq_splice_blocking(dest_q, src_q); > +} > + > +struct cds_wfq_node *cds_wfq_first_blocking(struct cds_wfq_queue *q) > +{ > + return _cds_wfq_first_blocking(q); > +} > + > +struct cds_wfq_node *cds_wfq_next_blocking(struct cds_wfq_queue *q, > + struct cds_wfq_node *node) > +{ > + return _cds_wfq_next_blocking(q, node); > +} > + > struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > { > return ___cds_wfq_dequeue_blocking(q); > } > > -struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > +void __cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, > + struct cds_wfq_queue *src_q) > { > - return _cds_wfq_dequeue_blocking(q); > + ___cds_wfq_splice_blocking(dest_q, src_q); > +} > + > +struct cds_wfq_node *__cds_wfq_first_blocking(struct cds_wfq_queue *q) > +{ > + return ___cds_wfq_first_blocking(q); > +} > + > +struct cds_wfq_node *__cds_wfq_next_blocking(struct cds_wfq_queue *q, > + struct cds_wfq_node *node) > +{ > + return ___cds_wfq_next_blocking(q, node); > } > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com Thanks, Lai From r.han at umiami.edu Mon Aug 13 14:05:33 2012 From: r.han at umiami.edu (Rui Han) Date: Mon, 13 Aug 2012 14:05:33 -0400 Subject: [lttng-dev] Can I use function to retrieve socket information in the SC_TRACE_EVENT macro? Message-ID: Dear all, I am working on retrieving socket information for socket related syscalls by the kernel trace, such as sys_connect() and sys_bind(). What I did is in the .h file "instrumentation/syscalls/headers/x86-64-syscalls-3.0.4_pointers.h", I modify the sys_connect SC_TRACE_EVENT. I add _field(size_t, port) and tp_assign(port, getport() ) to the TP_STRUCT_entry() and TP_fast_assign(), respectively. The getport() function is a function I wrote to retrieve the port number and ip address by the function getpeername(). The lttng module got installed correctly after the modification, (with some #ifndef manipulations), However, when I reboot the machine and try to do the kernel trace. It gives me error messages: Spawning a session daemon FATAL: Error inserting lttng_tracer (/lib/modules/3.2.0-29-generic/extra/lttng-tracer.ko): Unknown symbol in module, or unknown parameter (see dmesg) Error: Unable to load module lttng-tracer Warning: No kernel tracer available Warning: No tracing group detected Session rui-session created. Traces will be written in /root/lttng-traces/rui-session-20120813-133953 FATAL: Error inserting lttng_tracer (/lib/modules/3.2.0-29-generic/extra/lttng-tracer.ko): Unknown symbol in module, or unknown parameter (see dmesg) Error: Unable to load module lttng-tracer Warning: No kernel tracer available and in the syslog file: I got the following error: [ 82.001603] lttng_tracer: Unknown symbol getport (err 0) [ 82.377213] lttng_tracer: Unknown symbol getport (err 0) I am new to the kernel module programming. My question is: should I do some kinds of symbol register before define a function in the kernel space? Any suggestion for solving the problem? Thanks, Rui -------------- next part -------------- An HTML attachment was scrubbed... URL: From mathieu.desnoyers at efficios.com Mon Aug 13 16:12:19 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Mon, 13 Aug 2012 16:12:19 -0400 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost In-Reply-To: References: <20120812145000.GA32747@Krystal> Message-ID: <20120813201219.GA27637@Krystal> Hi Lai, * Lai Jiangshan (eag0628 at gmail.com) wrote: > On Sun, Aug 12, 2012 at 10:50 PM, Mathieu Desnoyers > wrote: > > This work is derived from the patch from Lai Jiangshan submitted as > > "urcu: new wfqueue implementation" > > (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) > > > > Its changelog: > > > >> Some guys would be surprised by this fact: > >> There are already TWO implementations of wfqueue in urcu. > >> > >> The first one is in urcu/static/wfqueue.h: > >> 1) enqueue: exchange the tail and then update previous->next > >> 2) dequeue: wait for first node's next pointer and them shift, a dummy node > >> is introduced to avoid the queue->tail become NULL when shift. > >> > >> The second one shares some code with the first one, and the left code > >> are spreading in urcu-call-rcu-impl.h: > >> 1) enqueue: share with the first one > >> 2) no dequeue operation: and no shift, so it don't need dummy node, > >> Although the dummy node is queued when initialization, but it is removed > >> after the first dequeue_all operation in call_rcu_thread(). > >> call_rcu_data_free() forgets to handle the dummy node if it is not removed. > >> 3)dequeue_all: record the old head and tail, and queue->head become the special > >> tail node.(atomic record the tail and change the tail). > >> > >> The second implementation's code are spreading, bad for review, and it is not > >> tested by tests/test_urcu_wfq. > >> > >> So we need a better implementation avoid the dummy node dancing and can service > >> both generic wfqueue APIs and dequeue_all API for call rcu. > >> > >> The new implementation: > >> 1) enqueue: share with the first one/original implementation. > >> 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. > >> no dummy node, save memory. > >> 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail > >> and return the old head.next. > >> > >> More implementation details are in the code. > >> tests/test_urcu_wfq will be update in future for testing new APIs. > > > > The patch proposed by Lai brings a very interesting simplification to > > the single-node handling (which is kept here), and moves all queue > > handling code away from call_rcu implementation, back into the wfqueue > > code. This has the benefit to allow testing enhancements. > > > > I modified it so the API does not expose implementation details to the > > user (e.g. ___cds_wfq_node_sync_next). I added a "splice" operation and > > a for loop iterator which should allow wfqueue users to use the list > > very efficiently both from LGPL/GPL code and from non-LGPL-compatible > > code. > > > > Benchmarks performed on Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz > > (dual-core, with hyperthreading) > > > > Benchmark invoked: > > test_urcu_wfq 2 2 10 > > > > Only did 2 runs, but a small improvement seems to be clear for the > > dequeue speed: > > > > Before patch: > > > > testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 136251248 nr_dequeues 54694027 successful enqueues 136251248 successful dequeues 54693904 end_dequeues 81557344 nr_ops 190945275 > > testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 137258881 nr_dequeues 54463340 successful enqueues 137258881 successful dequeues 54463238 end_dequeues 82795643 nr_ops 191722221 > > > > After patch: > > > > testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 138589301 nr_dequeues 56911253 successful enqueues 138589301 successful dequeues 56910916 end_dequeues 81678385 nr_ops 195500554 > > testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 139007622 nr_dequeues 57281502 successful enqueues 139007622 successful dequeues 57281348 end_dequeues 81726274 nr_ops 196289124 > > > > Summary: Number of enqueues is slightly lower, > > ?! > I see the nr_enqueues and successful enqueues are both increased after > after patch. Oh, you're right. I wrote that summary in a hurry. Sorry about that, will fix. > > > probably due to higher > > dequeue rate. Number of dequeue increased. Respective rate change is > > within 1% (slowdown) for enqueue, 2% (performance improvement) for > > dequeue. Overall number of operations (dequeue+enqueue) increased with > > the patch. > > > > We can verify that: > > successful enqueues - successful dequeues = end_dequeues > > > > For all runs (ensures correctness: no lost node). > > > > CC: Lai Jiangshan > > CC: Paul McKenney > > Signed-off-by: Mathieu Desnoyers > > --- > > diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h > > index 13b24ff..5363fe0 100644 > > --- a/urcu-call-rcu-impl.h > > +++ b/urcu-call-rcu-impl.h > > @@ -21,6 +21,7 @@ > > */ > > > > #define _GNU_SOURCE > > +#define _LGPL_SOURCE > > #include > > #include > > #include > > @@ -220,10 +221,7 @@ static void call_rcu_wake_up(struct call_rcu_data *crdp) > > static void *call_rcu_thread(void *arg) > > { > > unsigned long cbcount; > > - struct cds_wfq_node *cbs; > > - struct cds_wfq_node **cbs_tail; > > - struct call_rcu_data *crdp = (struct call_rcu_data *)arg; > > - struct rcu_head *rhp; > > + struct call_rcu_data *crdp = (struct call_rcu_data *) arg; > > int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); > > int ret; > > > > @@ -243,35 +241,29 @@ static void *call_rcu_thread(void *arg) > > cmm_smp_mb(); > > } > > for (;;) { > > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > > - poll(NULL, 0, 1); > > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > > - cbs_tail = (struct cds_wfq_node **) > > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > > + struct cds_wfq_queue cbs_tmp; > > + struct cds_wfq_node *cbs; > > + > > + cds_wfq_init(&cbs_tmp); > > + __cds_wfq_splice_blocking(&cbs_tmp, &crdp->cbs); > > + if (!cds_wfq_empty(&cbs_tmp)) { > > synchronize_rcu(); > > cbcount = 0; > > - do { > > - while (cbs->next == NULL && > > - &cbs->next != cbs_tail) > > - poll(NULL, 0, 1); > > - if (cbs == &crdp->cbs.dummy) { > > - cbs = cbs->next; > > - continue; > > - } > > - rhp = (struct rcu_head *)cbs; > > - cbs = cbs->next; > > + __cds_wfq_for_each_blocking(&cbs_tmp, cbs) { > > + struct rcu_head *rhp; > > + > > + rhp = caa_container_of(cbs, > > + struct rcu_head, next); > > rhp->func(rhp); > > > cbs is freed hear, but it will be used in __cds_wfq_next_blocking(). > Introduce __cds_wfq_for_each_blocking_safe() ? Good point! Will do. > > > cbcount++; > > - } while (cbs != NULL); > > + } > > uatomic_sub(&crdp->qlen, cbcount); > > } > > if (uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOP) > > break; > > rcu_thread_offline(); > > if (!rt) { > > - if (&crdp->cbs.head > > - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { > > + if (cds_wfq_empty(&crdp->cbs)) { > > call_rcu_wait(crdp); > > poll(NULL, 0, 10); > > uatomic_dec(&crdp->futex); > > @@ -625,32 +617,32 @@ void call_rcu(struct rcu_head *head, > > */ > > void call_rcu_data_free(struct call_rcu_data *crdp) > > { > > - struct cds_wfq_node *cbs; > > - struct cds_wfq_node **cbs_tail; > > - struct cds_wfq_node **cbs_endprev; > > - > > if (crdp == NULL || crdp == default_call_rcu_data) { > > return; > > } > > + > > if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { > > uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); > > wake_call_rcu_thread(crdp); > > while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) > > poll(NULL, 0, 1); > > } > > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > > - poll(NULL, 0, 1); > > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > > - cbs_tail = (struct cds_wfq_node **) > > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > > + > > + if (!cds_wfq_empty(&crdp->cbs)) { > > + struct cds_wfq_queue cbs_tmp; > > + > > + cds_wfq_init(&cbs_tmp); > > + __cds_wfq_splice_blocking(&cbs_tmp, &crdp->cbs); > > + > > /* Create default call rcu data if need be */ > > (void) get_default_call_rcu_data(); > > - cbs_endprev = (struct cds_wfq_node **) > > - uatomic_xchg(&default_call_rcu_data, cbs_tail); > > - *cbs_endprev = cbs; > > + > > + __cds_wfq_splice_blocking(&default_call_rcu_data->cbs, > > + &cbs_tmp); > > + > > Too much code to me, cbs_tmp is not required here. > > > /* Create default call rcu data if need be */ > (void) get_default_call_rcu_data(); > + __cds_wfq_splice_blocking(&default_call_rcu_data->cbs,&crdp->cbs); You're right. I initially thought that the order with the call to get_default_call_rcu_data() was important, but it rather looks like we call get_default_call_rcu_data() just to ensure that we have somewhere to send the already enqueued callbacks. Indeed, we can merge those as you propose. Will do! > > > > > uatomic_add(&default_call_rcu_data->qlen, > > uatomic_read(&crdp->qlen)); > > + > > wake_call_rcu_thread(default_call_rcu_data); > > } > > > > diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h > > index 636e1af..08d8d52 100644 > > --- a/urcu/static/wfqueue.h > > +++ b/urcu/static/wfqueue.h > > @@ -9,7 +9,8 @@ > > * TO BE INCLUDED ONLY IN LGPL-COMPATIBLE CODE. See wfqueue.h for linking > > * dynamically with the userspace rcu library. > > * > > - * Copyright 2010 - Mathieu Desnoyers > > + * Copyright 2010-2012 - Mathieu Desnoyers > > + * Copyright 2011-2012 - Lai Jiangshan > > * > > * This library is free software; you can redistribute it and/or > > * modify it under the terms of the GNU Lesser General Public > > @@ -29,6 +30,7 @@ > > #include > > #include > > #include > > +#include > > #include > > #include > > > > @@ -38,11 +40,16 @@ extern "C" { > > > > /* > > * Queue with wait-free enqueue/blocking dequeue. > > - * This implementation adds a dummy head node when the queue is empty to ensure > > - * we can always update the queue locklessly. > > * > > * Inspired from half-wait-free/half-blocking queue implementation done by > > * Paul E. McKenney. > > + * > > + * Caller must ensure mutual exclusion of queue update operations > > + * "dequeue" and "splice" source queue. Queue read operations "first" > > + * and "next" need to be protected against concurrent "dequeue" and > > + * "splice" (for source queue) by the caller. "enqueue", "splice" > > + * (destination queue), and "empty" are the only operations that can be > > + * used without any mutual exclusion. > > */ > > > > #define WFQ_ADAPT_ATTEMPTS 10 /* Retry if being set */ > > @@ -57,31 +64,51 @@ static inline void _cds_wfq_init(struct cds_wfq_queue *q) > > { > > int ret; > > > > - _cds_wfq_node_init(&q->dummy); > > /* Set queue head and tail */ > > - q->head = &q->dummy; > > - q->tail = &q->dummy.next; > > + _cds_wfq_node_init(&q->head); > > + q->tail = &q->head; > > ret = pthread_mutex_init(&q->lock, NULL); > > assert(!ret); > > } > > > > -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > > - struct cds_wfq_node *node) > > +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) > > +{ > > + /* > > + * Queue is empty if no node is pointed by q->head.next nor q->tail. > > + */ > > + return CMM_LOAD_SHARED(q->head.next) == NULL > > + && CMM_LOAD_SHARED(q->tail) == &q->head; > > +} > > + > > +static inline void ___cds_wfq_append(struct cds_wfq_queue *q, > > + struct cds_wfq_node *new_head, > > + struct cds_wfq_node *new_tail) > > { > > - struct cds_wfq_node **old_tail; > > + struct cds_wfq_node *old_tail; > > > > /* > > - * uatomic_xchg() implicit memory barrier orders earlier stores to data > > - * structure containing node and setting node->next to NULL before > > - * publication. > > + * Implicit memory barrier before uatomic_xchg() orders earlier > > + * stores to data structure containing node and setting > > + * node->next to NULL before publication. > > */ > > - old_tail = uatomic_xchg(&q->tail, &node->next); > > + old_tail = uatomic_xchg(&q->tail, new_tail); > > + > > /* > > - * At this point, dequeuers see a NULL old_tail->next, which indicates > > - * that the queue is being appended to. The following store will append > > - * "node" to the queue from a dequeuer perspective. > > + * Implicit memory barrier after uatomic_xchg() orders store to > > + * q->tail before store to old_tail->next. > > + * > > + * At this point, dequeuers see a NULL q->tail->next, which > > + * indicates that the queue is being appended to. The following > > + * store will append "node" to the queue from a dequeuer > > + * perspective. > > */ > > - CMM_STORE_SHARED(*old_tail, node); > > + CMM_STORE_SHARED(old_tail->next, new_head); > > +} > > + > > +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > > + struct cds_wfq_node *new_tail) > > +{ > > + ___cds_wfq_append(q, new_tail, new_tail); > > } > > > > /* > > @@ -100,14 +127,45 @@ ___cds_wfq_node_sync_next(struct cds_wfq_node *node) > > if (++attempt >= WFQ_ADAPT_ATTEMPTS) { > > poll(NULL, 0, WFQ_WAIT); /* Wait for 10ms */ > > attempt = 0; > > - } else > > + } else { > > caa_cpu_relax(); > > + } > > } > > > > return next; > > } > > > > /* > > + * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. > > + * > > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured > > + * by the caller. > > + */ > > +static inline struct cds_wfq_node * > > +___cds_wfq_first_blocking(struct cds_wfq_queue *q) > > +{ > > + if (_cds_wfq_empty(q)) > > + return NULL; > > + return ___cds_wfq_node_sync_next(&q->head); > > +} > > + > > +/* > > + * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. > > + * > > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured > > + * by the caller. > > + */ > > +static inline struct cds_wfq_node * > > +___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > > +{ > > + if (CMM_LOAD_SHARED(q->tail) == node) > > + return NULL; > > + return ___cds_wfq_node_sync_next(node); > > +} > > > The same BUG as you told me. > If q has only one node just enqueued by other thread. > but if q->head.next is seen, ___cds_wfq_first_blocking() returns a node, > And the update of q->tail is not seen, it is still &q->head, > ___cds_wfq_node_sync_next(node) will be loop for every if there is no > other enqueue. Good catch ! :-) > > > > static inline struct cds_wfq_node * > ___cds_wfq_first_blocking(struct cds_wfq_queue *q) > { > + struct cds_wfq_node *ret. > if (_cds_wfq_empty(q)) > return NULL; > ret = ___cds_wfq_node_sync_next(&q->head); > + cmm_smp_rmb(); > + return ret; > } However, I think we should add the rmb at the beginning of ___cds_wfq_next_blocking(), so it applies at each "next" call. Otherwise, I think we could end up in a situation where we wait for a NULL next forever in the second of two consecutive ___cds_wfq_next_blocking() calls. I therefore propose: static inline struct cds_wfq_node * ___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) { /* Load previous node->next before q->tail */ cmm_smp_rmb(); if (CMM_LOAD_SHARED(q->tail) == node) return NULL; return ___cds_wfq_node_sync_next(node); } > > > > + > > +/* > > + * ___cds_wfq_dequeue_blocking: dequeue a node from the queue. > > + * > > * It is valid to reuse and free a dequeued node immediately. > > * > > * No need to go on a waitqueue here, as there is no possible state in which the > > @@ -120,42 +178,123 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > > { > > struct cds_wfq_node *node, *next; > > > > - /* > > - * Queue is empty if it only contains the dummy node. > > - */ > > - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) > > + if (_cds_wfq_empty(q)) > > return NULL; > > - node = q->head; > > > > - next = ___cds_wfq_node_sync_next(node); > > + node = ___cds_wfq_node_sync_next(&q->head); > > + > > + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { > > + /* Load node->next before q->tail */ > > + cmm_smp_rmb(); > > + if (CMM_LOAD_SHARED(q->tail) == node) { > > I don't know why I added this "if" since it is likely true. > Could you remove the above 3 lines? > (I remember there is a mb() before uatomic_cmpxchg() which means > this mb() is before the test in uatomic_cmpxchg()) Indeed. I replaced these by a large comment. > > > + /* > > + * @node is the only node in the queue. > > + * Try to move the tail to &q->head > > + */ > > + _cds_wfq_node_init(&q->head); > > + if (uatomic_cmpxchg(&q->tail, node, &q->head) == node) > > + return node; > > + } > > + next = ___cds_wfq_node_sync_next(node); > > + } > > > > /* > > * Move queue head forward. > > */ > > - q->head = next; > > + q->head.next = next; > > + > > + return node; > > +} > > + > > +/* > > + * ___cds_wfq_splice_blocking: enqueue all src_q nodes at the end of dest_q. > > + * > > + * Dequeue all nodes from src_q. > > + * dest_q must be already initialized. > > + * caller ensures mutual exclusion of dequeue and splice operations on > > + * src_q. > > + */ > > +static inline void > > +___cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, > > + struct cds_wfq_queue *src_q) > > +{ > > + struct cds_wfq_node *head, *tail; > > + > > + if (_cds_wfq_empty(src_q)) > > + return; > > + > > + head = ___cds_wfq_node_sync_next(&src_q->head); > > + _cds_wfq_node_init(&src_q->head); > > + > > + /* > > + * Memory barrier implied before uatomic_xchg() orders store to > > + * src_q->head before store to src_q->tail. This is required by > > + * concurrent enqueue on src_q, which exchanges the tail before > > + * updating the previous tail's next pointer. > > + */ > > + tail = uatomic_xchg(&src_q->tail, &src_q->head); > > + > > /* > > - * Requeue dummy node if we just dequeued it. > > + * Append the spliced content of src_q into dest_q. Does not > > + * require mutual exclusion on dest_q (wait-free). > > */ > > - if (node == &q->dummy) { > > - _cds_wfq_node_init(node); > > - _cds_wfq_enqueue(q, node); > > - return ___cds_wfq_dequeue_blocking(q); > > - } > > - return node; > > + ___cds_wfq_append(dest_q, head, tail); > > +} > > + > > +/* Locking performed within cds_wfq calls. */ > > +static inline struct cds_wfq_node * > > +_cds_wfq_first_blocking(struct cds_wfq_queue *q) > > +{ > > + struct cds_wfq_node *retval; > > + int ret; > > + > > + ret = pthread_mutex_lock(&q->lock); > > + assert(!ret); > > + retval = ___cds_wfq_first_blocking(q); > > + ret = pthread_mutex_unlock(&q->lock); > > + assert(!ret); > > + return retval; > > +} > > + > > +static inline struct cds_wfq_node * > > +_cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > > +{ > > + struct cds_wfq_node *retval; > > + int ret; > > + > > + ret = pthread_mutex_lock(&q->lock); > > + assert(!ret); > > + retval = ___cds_wfq_next_blocking(q, node); > > + ret = pthread_mutex_unlock(&q->lock); > > + assert(!ret); > > + return retval; > > } > > I reject these _cds_wfq_first_blocking(), _cds_wfq_next_blocking() > and cds_wfq_for_each_blocking(), because the claimed "Locking" > makes no sense: > 1. It protects nothing in _cds_wfq_next_blocking(). Good point! > 2. There is no "Locking" in the loop body, @node is not dequeued, > it will be invalid if some other dequeue it, > and _cds_wfq_next_blocking() results BUG. Indeed. So do you recommend we just leave locking to the callers then ? What locking rules would you recommend we document in the API ? The only API members that would still have implicit locking are thus: - cds_wfq_dequeue_blocking() - cds_wfq_splice_blocking() The only reason I leave them there is that we already expose a "cds_wfq_dequeue_blocking" to users which provides locking, and I don't want to break the API. > > > > > static inline struct cds_wfq_node * > > _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > > { > > - struct cds_wfq_node *retnode; > > + struct cds_wfq_node *retval; > > int ret; > > > > ret = pthread_mutex_lock(&q->lock); > > assert(!ret); > > - retnode = ___cds_wfq_dequeue_blocking(q); > > + retval = ___cds_wfq_dequeue_blocking(q); > > ret = pthread_mutex_unlock(&q->lock); > > assert(!ret); > > - return retnode; > > + return retval; > > +} > > + > > +static inline void > > +_cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, > > + struct cds_wfq_queue *src_q) > > +{ > > + int ret; > > + > > + ret = pthread_mutex_lock(&src_q->lock); > > + assert(!ret); > > + ___cds_wfq_splice_blocking(dest_q, src_q); > > + ret = pthread_mutex_unlock(&src_q->lock); > > + assert(!ret); > > } > > > > #ifdef __cplusplus > > diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h > > index 03a73f1..d33d47a 100644 > > --- a/urcu/wfqueue.h > > +++ b/urcu/wfqueue.h > > @@ -6,7 +6,8 @@ > > * > > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > > * > > - * Copyright 2010 - Mathieu Desnoyers > > + * Copyright 2010-2012 - Mathieu Desnoyers > > + * Copyright 2011-2012 - Lai Jiangshan > > * > > * This library is free software; you can redistribute it and/or > > * modify it under the terms of the GNU Lesser General Public > > @@ -25,6 +26,7 @@ > > > > #include > > #include > > +#include > > #include > > > > #ifdef __cplusplus > > @@ -33,8 +35,6 @@ extern "C" { > > > > /* > > * Queue with wait-free enqueue/blocking dequeue. > > - * This implementation adds a dummy head node when the queue is empty to ensure > > - * we can always update the queue locklessly. > > * > > * Inspired from half-wait-free/half-blocking queue implementation done by > > * Paul E. McKenney. > > @@ -45,8 +45,8 @@ struct cds_wfq_node { > > }; > > > > struct cds_wfq_queue { > > - struct cds_wfq_node *head, **tail; > > - struct cds_wfq_node dummy; /* Dummy node */ > > + struct cds_wfq_node head, *tail; > > + struct cds_wfq_node padding; /* unused */ > > pthread_mutex_t lock; > > }; > > Why keep the padding? This is mainly to keep compatibility with applications already using wfqueue.h. We could indeed make the size of the cds_wfq_queue smaller, but we cannot enlarge it without breaking the API. Therefore, I think it is safe to keep some unused padding rather than shrink the size, if we ever need to put an extra flag or pointer in the structure. What I can do, though, is to move the padding to the end of the structure, and give it a void * type. Is that OK ? Thanks, Mathieu -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From dgoulet at efficios.com Mon Aug 13 16:51:18 2012 From: dgoulet at efficios.com (David Goulet) Date: Mon, 13 Aug 2012 16:51:18 -0400 Subject: [lttng-dev] LTTng: problem with lttng-sessiond. In-Reply-To: <20120813132231.GA24219@Krystal> References: <20120813132231.GA24219@Krystal> Message-ID: <502968C6.70301@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Oh my! This is a good one haha! So, first of all, I think a gdb backtrace would be useful to know exactly where there is an illegal instruction. Also, on which arch and OS version are you running this. Thanks! David Mathieu Desnoyers: > CCing David Goulet, maintainer of lttng-tools. He will look into > your issue. > > Thanks, > > Mathieu > > * Henrik Hautakoski (henrik at fiktivkod.org) wrote: >> Hi, We have some problems with spawning a session daemon. running >> the command "lttng-sessiond --vvv --no-kernel" and get "Illegal >> instruction" See output: >> >> DEBUG3: Creating LTTng run directory: /var/run/lttng [in >> create_lttng_rundir() at main.c:5323] DEBUG2: Kernel consumer err >> path: /var/run/lttng/kconsumerd/error [in main() at main.c:5571] >> DEBUG2: Kernel consumer cmd path: >> /var/run/lttng/kconsumerd/command [in main() at main.c:5573] >> DEBUG1: Client socket path /var/run/lttng/client-lttng-sessiond >> [in main() at main.c:5626] DEBUG1: Application socket path >> /var/run/lttng/apps-lttng-sessiond [in main() at main.c:5627] >> DEBUG1: LTTng run directory path: /var/run/lttng [in main() at >> main.c:5628] DEBUG2: UST consumer 32 bits err path: >> /var/run/lttng/ustconsumerd32/error [in main() at main.c:5637] >> DEBUG2: UST consumer 32 bits cmd path: >> /var/run/lttng/ustconsumerd32/command [in main() at main.c:5639] >> DEBUG2: UST consumer 64 bits err path: >> /var/run/lttng/ustconsumerd64/error [in main() at main.c:5648] >> DEBUG2: UST consumer 64 bits cmd path: >> /var/run/lttng/ustconsumerd64/command [in main() at main.c:5650] >> DEBUG3: Created hashtable size 4 at 0x10064080 of type 1 [in >> lttng_ht_new() at hashtable.c:96] DEBUG3: Created hashtable size >> 4 at 0x10064168 of type 1 [in lttng_ht_new() at hashtable.c:96] >> DEBUG2: Creating consumer directory: /var/run/lttng/kconsumerd >> [in set_consumer_sockets() at main.c:5365] DEBUG2: Creating >> consumer directory: /var/run/lttng/ustconsumerd64 [in >> set_consumer_sockets() at main.c:5365] DEBUG2: Creating consumer >> directory: /var/run/lttng/ustconsumerd32 [in >> set_consumer_sockets() at main.c:5365] DEBUG1: Signal handler set >> for SIGTERM, SIGPIPE and SIGINT [in set_signal_handler() at >> main.c:5457] DEBUG1: All permissions are set [in >> set_permissions() at main.c:5310] DEBUG1: epoll set max size is >> 180337 [in compat_epoll_set_max_size() at compat-epoll.c:224] >> Illegal instruction >> >> What to do? :P >> >> -- Henrik Hautakoski henrik at fiktivkod.org > -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQKWjDAAoJEELoaioR9I02VdsIAIlyG8BFbTPZDgHpxynWUD6i 0Uh4N4TTQgxRQ3WnbSWMOnx1l2qxG9XbJ9wwvrjJvnH2UCBcQ2g8vl2BQiaMN0BI ZwD2/UDLRioP8BLv5y/5zAAoU6MSVz+ZCbLOI/oTHDcYzA5TdXmzpcW+XS7rKxHd uRyUmGsRdxViOZ3JA7vCEZR/ea02p17PzFrq+/bn/mE2L07ZrKN7TLdpmTlMyPP/ KmaRBcQUZkwrQ+0wqkgj6pkDPjLfLt8/au3v6x+iPvi09bHlHF9nB5wCve3g1UpZ NNvz07XPtgnSa6quM3LXqdIp02Zrvr2xx/eL7i2ZoReWtjnNnYcHCWZQvfdsjkY= =1ekn -----END PGP SIGNATURE----- From henrik at fiktivkod.org Tue Aug 14 09:08:05 2012 From: henrik at fiktivkod.org (Henrik Hautakoski) Date: Tue, 14 Aug 2012 15:08:05 +0200 Subject: [lttng-dev] LTTng: problem with lttng-sessiond. In-Reply-To: <502968C6.70301@efficios.com> References: <20120813132231.GA24219@Krystal> <502968C6.70301@efficios.com> Message-ID: After running a gdb backtrace. the problem was that the function _uatomic_add_return() in urcu/uatomic.h was passed a length of 8. But urcu was compiled for length 4 (32bit). The source of the problem was that the `current` member in `struct state` in lttng-tools/.../lttng-sessiond/health.h the size of this datatype is later passed to _uatomic_add_return(). But the type was uint64_t undependant of arch. Commit 139ac87245fd1ca18d60a0efca32b50e4c1d8730 in lttng-tools fixed this issue. Thanks ;) ( And we started crosscompiling one day before this patch was commited :/ ) On Mon, Aug 13, 2012 at 10:51 PM, David Goulet wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Oh my! This is a good one haha! > > So, first of all, I think a gdb backtrace would be useful to know > exactly where there is an illegal instruction. > > Also, on which arch and OS version are you running this. > > Thanks! > David > > Mathieu Desnoyers: >> CCing David Goulet, maintainer of lttng-tools. He will look into >> your issue. >> >> Thanks, >> >> Mathieu >> >> * Henrik Hautakoski (henrik at fiktivkod.org) wrote: >>> Hi, We have some problems with spawning a session daemon. running >>> the command "lttng-sessiond --vvv --no-kernel" and get "Illegal >>> instruction" See output: >>> >>> DEBUG3: Creating LTTng run directory: /var/run/lttng [in >>> create_lttng_rundir() at main.c:5323] DEBUG2: Kernel consumer err >>> path: /var/run/lttng/kconsumerd/error [in main() at main.c:5571] >>> DEBUG2: Kernel consumer cmd path: >>> /var/run/lttng/kconsumerd/command [in main() at main.c:5573] >>> DEBUG1: Client socket path /var/run/lttng/client-lttng-sessiond >>> [in main() at main.c:5626] DEBUG1: Application socket path >>> /var/run/lttng/apps-lttng-sessiond [in main() at main.c:5627] >>> DEBUG1: LTTng run directory path: /var/run/lttng [in main() at >>> main.c:5628] DEBUG2: UST consumer 32 bits err path: >>> /var/run/lttng/ustconsumerd32/error [in main() at main.c:5637] >>> DEBUG2: UST consumer 32 bits cmd path: >>> /var/run/lttng/ustconsumerd32/command [in main() at main.c:5639] >>> DEBUG2: UST consumer 64 bits err path: >>> /var/run/lttng/ustconsumerd64/error [in main() at main.c:5648] >>> DEBUG2: UST consumer 64 bits cmd path: >>> /var/run/lttng/ustconsumerd64/command [in main() at main.c:5650] >>> DEBUG3: Created hashtable size 4 at 0x10064080 of type 1 [in >>> lttng_ht_new() at hashtable.c:96] DEBUG3: Created hashtable size >>> 4 at 0x10064168 of type 1 [in lttng_ht_new() at hashtable.c:96] >>> DEBUG2: Creating consumer directory: /var/run/lttng/kconsumerd >>> [in set_consumer_sockets() at main.c:5365] DEBUG2: Creating >>> consumer directory: /var/run/lttng/ustconsumerd64 [in >>> set_consumer_sockets() at main.c:5365] DEBUG2: Creating consumer >>> directory: /var/run/lttng/ustconsumerd32 [in >>> set_consumer_sockets() at main.c:5365] DEBUG1: Signal handler set >>> for SIGTERM, SIGPIPE and SIGINT [in set_signal_handler() at >>> main.c:5457] DEBUG1: All permissions are set [in >>> set_permissions() at main.c:5310] DEBUG1: epoll set max size is >>> 180337 [in compat_epoll_set_max_size() at compat-epoll.c:224] >>> Illegal instruction >>> >>> What to do? :P >>> >>> -- Henrik Hautakoski henrik at fiktivkod.org >> > -----BEGIN PGP SIGNATURE----- > > iQEcBAEBCgAGBQJQKWjDAAoJEELoaioR9I02VdsIAIlyG8BFbTPZDgHpxynWUD6i > 0Uh4N4TTQgxRQ3WnbSWMOnx1l2qxG9XbJ9wwvrjJvnH2UCBcQ2g8vl2BQiaMN0BI > ZwD2/UDLRioP8BLv5y/5zAAoU6MSVz+ZCbLOI/oTHDcYzA5TdXmzpcW+XS7rKxHd > uRyUmGsRdxViOZ3JA7vCEZR/ea02p17PzFrq+/bn/mE2L07ZrKN7TLdpmTlMyPP/ > KmaRBcQUZkwrQ+0wqkgj6pkDPjLfLt8/au3v6x+iPvi09bHlHF9nB5wCve3g1UpZ > NNvz07XPtgnSa6quM3LXqdIp02Zrvr2xx/eL7i2ZoReWtjnNnYcHCWZQvfdsjkY= > =1ekn > -----END PGP SIGNATURE----- -- Henrik Hautakoski henrik at fiktivkod.org From eag0628 at gmail.com Tue Aug 14 09:49:42 2012 From: eag0628 at gmail.com (Lai Jiangshan) Date: Tue, 14 Aug 2012 21:49:42 +0800 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost In-Reply-To: <20120813201219.GA27637@Krystal> References: <20120812145000.GA32747@Krystal> <20120813201219.GA27637@Krystal> Message-ID: On Tue, Aug 14, 2012 at 4:12 AM, Mathieu Desnoyers wrote: > Hi Lai, > > * Lai Jiangshan (eag0628 at gmail.com) wrote: >> On Sun, Aug 12, 2012 at 10:50 PM, Mathieu Desnoyers >> wrote: >> > This work is derived from the patch from Lai Jiangshan submitted as >> > "urcu: new wfqueue implementation" >> > (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) >> > >> > Its changelog: >> > >> >> Some guys would be surprised by this fact: >> >> There are already TWO implementations of wfqueue in urcu. >> >> >> >> The first one is in urcu/static/wfqueue.h: >> >> 1) enqueue: exchange the tail and then update previous->next >> >> 2) dequeue: wait for first node's next pointer and them shift, a dummy node >> >> is introduced to avoid the queue->tail become NULL when shift. >> >> >> >> The second one shares some code with the first one, and the left code >> >> are spreading in urcu-call-rcu-impl.h: >> >> 1) enqueue: share with the first one >> >> 2) no dequeue operation: and no shift, so it don't need dummy node, >> >> Although the dummy node is queued when initialization, but it is removed >> >> after the first dequeue_all operation in call_rcu_thread(). >> >> call_rcu_data_free() forgets to handle the dummy node if it is not removed. >> >> 3)dequeue_all: record the old head and tail, and queue->head become the special >> >> tail node.(atomic record the tail and change the tail). >> >> >> >> The second implementation's code are spreading, bad for review, and it is not >> >> tested by tests/test_urcu_wfq. >> >> >> >> So we need a better implementation avoid the dummy node dancing and can service >> >> both generic wfqueue APIs and dequeue_all API for call rcu. >> >> >> >> The new implementation: >> >> 1) enqueue: share with the first one/original implementation. >> >> 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. >> >> no dummy node, save memory. >> >> 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail >> >> and return the old head.next. >> >> >> >> More implementation details are in the code. >> >> tests/test_urcu_wfq will be update in future for testing new APIs. >> > >> > The patch proposed by Lai brings a very interesting simplification to >> > the single-node handling (which is kept here), and moves all queue >> > handling code away from call_rcu implementation, back into the wfqueue >> > code. This has the benefit to allow testing enhancements. >> > >> > I modified it so the API does not expose implementation details to the >> > user (e.g. ___cds_wfq_node_sync_next). I added a "splice" operation and >> > a for loop iterator which should allow wfqueue users to use the list >> > very efficiently both from LGPL/GPL code and from non-LGPL-compatible >> > code. >> > >> > Benchmarks performed on Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz >> > (dual-core, with hyperthreading) >> > >> > Benchmark invoked: >> > test_urcu_wfq 2 2 10 >> > >> > Only did 2 runs, but a small improvement seems to be clear for the >> > dequeue speed: >> > >> > Before patch: >> > >> > testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 136251248 nr_dequeues 54694027 successful enqueues 136251248 successful dequeues 54693904 end_dequeues 81557344 nr_ops 190945275 >> > testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 137258881 nr_dequeues 54463340 successful enqueues 137258881 successful dequeues 54463238 end_dequeues 82795643 nr_ops 191722221 >> > >> > After patch: >> > >> > testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 138589301 nr_dequeues 56911253 successful enqueues 138589301 successful dequeues 56910916 end_dequeues 81678385 nr_ops 195500554 >> > testdur 10 nr_enqueuers 2 wdelay 0 nr_dequeuers 2 rdur 0 nr_enqueues 139007622 nr_dequeues 57281502 successful enqueues 139007622 successful dequeues 57281348 end_dequeues 81726274 nr_ops 196289124 >> > >> > Summary: Number of enqueues is slightly lower, >> >> ?! >> I see the nr_enqueues and successful enqueues are both increased after >> after patch. > > Oh, you're right. I wrote that summary in a hurry. Sorry about that, > will fix. > >> >> > probably due to higher >> > dequeue rate. Number of dequeue increased. Respective rate change is >> > within 1% (slowdown) for enqueue, 2% (performance improvement) for >> > dequeue. Overall number of operations (dequeue+enqueue) increased with >> > the patch. >> > >> > We can verify that: >> > successful enqueues - successful dequeues = end_dequeues >> > >> > For all runs (ensures correctness: no lost node). >> > >> > CC: Lai Jiangshan >> > CC: Paul McKenney >> > Signed-off-by: Mathieu Desnoyers >> > --- >> > diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h >> > index 13b24ff..5363fe0 100644 >> > --- a/urcu-call-rcu-impl.h >> > +++ b/urcu-call-rcu-impl.h >> > @@ -21,6 +21,7 @@ >> > */ >> > >> > #define _GNU_SOURCE >> > +#define _LGPL_SOURCE >> > #include >> > #include >> > #include >> > @@ -220,10 +221,7 @@ static void call_rcu_wake_up(struct call_rcu_data *crdp) >> > static void *call_rcu_thread(void *arg) >> > { >> > unsigned long cbcount; >> > - struct cds_wfq_node *cbs; >> > - struct cds_wfq_node **cbs_tail; >> > - struct call_rcu_data *crdp = (struct call_rcu_data *)arg; >> > - struct rcu_head *rhp; >> > + struct call_rcu_data *crdp = (struct call_rcu_data *) arg; >> > int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); >> > int ret; >> > >> > @@ -243,35 +241,29 @@ static void *call_rcu_thread(void *arg) >> > cmm_smp_mb(); >> > } >> > for (;;) { >> > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { >> > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) >> > - poll(NULL, 0, 1); >> > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); >> > - cbs_tail = (struct cds_wfq_node **) >> > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); >> > + struct cds_wfq_queue cbs_tmp; >> > + struct cds_wfq_node *cbs; >> > + >> > + cds_wfq_init(&cbs_tmp); >> > + __cds_wfq_splice_blocking(&cbs_tmp, &crdp->cbs); >> > + if (!cds_wfq_empty(&cbs_tmp)) { >> > synchronize_rcu(); >> > cbcount = 0; >> > - do { >> > - while (cbs->next == NULL && >> > - &cbs->next != cbs_tail) >> > - poll(NULL, 0, 1); >> > - if (cbs == &crdp->cbs.dummy) { >> > - cbs = cbs->next; >> > - continue; >> > - } >> > - rhp = (struct rcu_head *)cbs; >> > - cbs = cbs->next; >> > + __cds_wfq_for_each_blocking(&cbs_tmp, cbs) { >> > + struct rcu_head *rhp; >> > + >> > + rhp = caa_container_of(cbs, >> > + struct rcu_head, next); >> > rhp->func(rhp); >> >> >> cbs is freed hear, but it will be used in __cds_wfq_next_blocking(). >> Introduce __cds_wfq_for_each_blocking_safe() ? > > Good point! Will do. > >> >> > cbcount++; >> > - } while (cbs != NULL); >> > + } >> > uatomic_sub(&crdp->qlen, cbcount); >> > } >> > if (uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOP) >> > break; >> > rcu_thread_offline(); >> > if (!rt) { >> > - if (&crdp->cbs.head >> > - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { >> > + if (cds_wfq_empty(&crdp->cbs)) { >> > call_rcu_wait(crdp); >> > poll(NULL, 0, 10); >> > uatomic_dec(&crdp->futex); >> > @@ -625,32 +617,32 @@ void call_rcu(struct rcu_head *head, >> > */ >> > void call_rcu_data_free(struct call_rcu_data *crdp) >> > { >> > - struct cds_wfq_node *cbs; >> > - struct cds_wfq_node **cbs_tail; >> > - struct cds_wfq_node **cbs_endprev; >> > - >> > if (crdp == NULL || crdp == default_call_rcu_data) { >> > return; >> > } >> > + >> > if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { >> > uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); >> > wake_call_rcu_thread(crdp); >> > while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) >> > poll(NULL, 0, 1); >> > } >> > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { >> > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) >> > - poll(NULL, 0, 1); >> > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); >> > - cbs_tail = (struct cds_wfq_node **) >> > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); >> > + >> > + if (!cds_wfq_empty(&crdp->cbs)) { >> > + struct cds_wfq_queue cbs_tmp; >> > + >> > + cds_wfq_init(&cbs_tmp); >> > + __cds_wfq_splice_blocking(&cbs_tmp, &crdp->cbs); >> > + >> > /* Create default call rcu data if need be */ >> > (void) get_default_call_rcu_data(); >> > - cbs_endprev = (struct cds_wfq_node **) >> > - uatomic_xchg(&default_call_rcu_data, cbs_tail); >> > - *cbs_endprev = cbs; >> > + >> > + __cds_wfq_splice_blocking(&default_call_rcu_data->cbs, >> > + &cbs_tmp); >> > + >> >> Too much code to me, cbs_tmp is not required here. >> >> >> /* Create default call rcu data if need be */ >> (void) get_default_call_rcu_data(); >> + __cds_wfq_splice_blocking(&default_call_rcu_data->cbs,&crdp->cbs); > > You're right. I initially thought that the order with the call to > get_default_call_rcu_data() was important, but it rather looks like we > call get_default_call_rcu_data() just to ensure that we have somewhere > to send the already enqueued callbacks. Indeed, we can merge those as > you propose. Will do! > >> >> >> >> > uatomic_add(&default_call_rcu_data->qlen, >> > uatomic_read(&crdp->qlen)); >> > + >> > wake_call_rcu_thread(default_call_rcu_data); >> > } >> > >> > diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h >> > index 636e1af..08d8d52 100644 >> > --- a/urcu/static/wfqueue.h >> > +++ b/urcu/static/wfqueue.h >> > @@ -9,7 +9,8 @@ >> > * TO BE INCLUDED ONLY IN LGPL-COMPATIBLE CODE. See wfqueue.h for linking >> > * dynamically with the userspace rcu library. >> > * >> > - * Copyright 2010 - Mathieu Desnoyers >> > + * Copyright 2010-2012 - Mathieu Desnoyers >> > + * Copyright 2011-2012 - Lai Jiangshan >> > * >> > * This library is free software; you can redistribute it and/or >> > * modify it under the terms of the GNU Lesser General Public >> > @@ -29,6 +30,7 @@ >> > #include >> > #include >> > #include >> > +#include >> > #include >> > #include >> > >> > @@ -38,11 +40,16 @@ extern "C" { >> > >> > /* >> > * Queue with wait-free enqueue/blocking dequeue. >> > - * This implementation adds a dummy head node when the queue is empty to ensure >> > - * we can always update the queue locklessly. >> > * >> > * Inspired from half-wait-free/half-blocking queue implementation done by >> > * Paul E. McKenney. >> > + * >> > + * Caller must ensure mutual exclusion of queue update operations >> > + * "dequeue" and "splice" source queue. Queue read operations "first" >> > + * and "next" need to be protected against concurrent "dequeue" and >> > + * "splice" (for source queue) by the caller. "enqueue", "splice" >> > + * (destination queue), and "empty" are the only operations that can be >> > + * used without any mutual exclusion. >> > */ >> > >> > #define WFQ_ADAPT_ATTEMPTS 10 /* Retry if being set */ >> > @@ -57,31 +64,51 @@ static inline void _cds_wfq_init(struct cds_wfq_queue *q) >> > { >> > int ret; >> > >> > - _cds_wfq_node_init(&q->dummy); >> > /* Set queue head and tail */ >> > - q->head = &q->dummy; >> > - q->tail = &q->dummy.next; >> > + _cds_wfq_node_init(&q->head); >> > + q->tail = &q->head; >> > ret = pthread_mutex_init(&q->lock, NULL); >> > assert(!ret); >> > } >> > >> > -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, >> > - struct cds_wfq_node *node) >> > +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) >> > +{ >> > + /* >> > + * Queue is empty if no node is pointed by q->head.next nor q->tail. >> > + */ >> > + return CMM_LOAD_SHARED(q->head.next) == NULL >> > + && CMM_LOAD_SHARED(q->tail) == &q->head; >> > +} >> > + >> > +static inline void ___cds_wfq_append(struct cds_wfq_queue *q, >> > + struct cds_wfq_node *new_head, >> > + struct cds_wfq_node *new_tail) >> > { >> > - struct cds_wfq_node **old_tail; >> > + struct cds_wfq_node *old_tail; >> > >> > /* >> > - * uatomic_xchg() implicit memory barrier orders earlier stores to data >> > - * structure containing node and setting node->next to NULL before >> > - * publication. >> > + * Implicit memory barrier before uatomic_xchg() orders earlier >> > + * stores to data structure containing node and setting >> > + * node->next to NULL before publication. >> > */ >> > - old_tail = uatomic_xchg(&q->tail, &node->next); >> > + old_tail = uatomic_xchg(&q->tail, new_tail); >> > + >> > /* >> > - * At this point, dequeuers see a NULL old_tail->next, which indicates >> > - * that the queue is being appended to. The following store will append >> > - * "node" to the queue from a dequeuer perspective. >> > + * Implicit memory barrier after uatomic_xchg() orders store to >> > + * q->tail before store to old_tail->next. >> > + * >> > + * At this point, dequeuers see a NULL q->tail->next, which >> > + * indicates that the queue is being appended to. The following >> > + * store will append "node" to the queue from a dequeuer >> > + * perspective. >> > */ >> > - CMM_STORE_SHARED(*old_tail, node); >> > + CMM_STORE_SHARED(old_tail->next, new_head); >> > +} >> > + >> > +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, >> > + struct cds_wfq_node *new_tail) >> > +{ >> > + ___cds_wfq_append(q, new_tail, new_tail); >> > } >> > >> > /* >> > @@ -100,14 +127,45 @@ ___cds_wfq_node_sync_next(struct cds_wfq_node *node) >> > if (++attempt >= WFQ_ADAPT_ATTEMPTS) { >> > poll(NULL, 0, WFQ_WAIT); /* Wait for 10ms */ >> > attempt = 0; >> > - } else >> > + } else { >> > caa_cpu_relax(); >> > + } >> > } >> > >> > return next; >> > } >> > >> > /* >> > + * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. >> > + * >> > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured >> > + * by the caller. >> > + */ >> > +static inline struct cds_wfq_node * >> > +___cds_wfq_first_blocking(struct cds_wfq_queue *q) >> > +{ >> > + if (_cds_wfq_empty(q)) >> > + return NULL; >> > + return ___cds_wfq_node_sync_next(&q->head); >> > +} >> > + >> > +/* >> > + * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. >> > + * >> > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured >> > + * by the caller. >> > + */ >> > +static inline struct cds_wfq_node * >> > +___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) >> > +{ >> > + if (CMM_LOAD_SHARED(q->tail) == node) >> > + return NULL; >> > + return ___cds_wfq_node_sync_next(node); >> > +} >> >> >> The same BUG as you told me. >> If q has only one node just enqueued by other thread. >> but if q->head.next is seen, ___cds_wfq_first_blocking() returns a node, >> And the update of q->tail is not seen, it is still &q->head, >> ___cds_wfq_node_sync_next(node) will be loop for every if there is no >> other enqueue. > > Good catch ! :-) > >> >> >> >> static inline struct cds_wfq_node * >> ___cds_wfq_first_blocking(struct cds_wfq_queue *q) >> { >> + struct cds_wfq_node *ret. >> if (_cds_wfq_empty(q)) >> return NULL; >> ret = ___cds_wfq_node_sync_next(&q->head); >> + cmm_smp_rmb(); >> + return ret; >> } > > However, I think we should add the rmb at the beginning of > ___cds_wfq_next_blocking(), so it applies at each "next" call. > Otherwise, I think we could end up in a situation where we wait for a > NULL next forever in the second of two consecutive > ___cds_wfq_next_blocking() calls. I therefore propose: How this can happen(wait for a NULL next forever in the second ...)? A rmb is enough to leave the state(simgle node && q->tail == &q->head). "CMM_LOAD_SHARED(q->tail) == node" will be true in the loop except this state. > > static inline struct cds_wfq_node * > ___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > { > /* Load previous node->next before q->tail */ > cmm_smp_rmb(); > if (CMM_LOAD_SHARED(q->tail) == node) > return NULL; > return ___cds_wfq_node_sync_next(node); > } > >> >> >> > + >> > +/* >> > + * ___cds_wfq_dequeue_blocking: dequeue a node from the queue. >> > + * >> > * It is valid to reuse and free a dequeued node immediately. >> > * >> > * No need to go on a waitqueue here, as there is no possible state in which the >> > @@ -120,42 +178,123 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) >> > { >> > struct cds_wfq_node *node, *next; >> > >> > - /* >> > - * Queue is empty if it only contains the dummy node. >> > - */ >> > - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) >> > + if (_cds_wfq_empty(q)) >> > return NULL; >> > - node = q->head; >> > >> > - next = ___cds_wfq_node_sync_next(node); >> > + node = ___cds_wfq_node_sync_next(&q->head); >> > + >> > + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { >> > + /* Load node->next before q->tail */ >> > + cmm_smp_rmb(); >> > + if (CMM_LOAD_SHARED(q->tail) == node) { >> >> I don't know why I added this "if" since it is likely true. >> Could you remove the above 3 lines? >> (I remember there is a mb() before uatomic_cmpxchg() which means >> this mb() is before the test in uatomic_cmpxchg()) > > Indeed. I replaced these by a large comment. > >> >> > + /* >> > + * @node is the only node in the queue. >> > + * Try to move the tail to &q->head >> > + */ >> > + _cds_wfq_node_init(&q->head); >> > + if (uatomic_cmpxchg(&q->tail, node, &q->head) == node) >> > + return node; >> > + } >> > + next = ___cds_wfq_node_sync_next(node); >> > + } >> > >> > /* >> > * Move queue head forward. >> > */ >> > - q->head = next; >> > + q->head.next = next; >> > + >> > + return node; >> > +} >> > + >> > +/* >> > + * ___cds_wfq_splice_blocking: enqueue all src_q nodes at the end of dest_q. >> > + * >> > + * Dequeue all nodes from src_q. >> > + * dest_q must be already initialized. >> > + * caller ensures mutual exclusion of dequeue and splice operations on >> > + * src_q. >> > + */ >> > +static inline void >> > +___cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, >> > + struct cds_wfq_queue *src_q) >> > +{ >> > + struct cds_wfq_node *head, *tail; >> > + >> > + if (_cds_wfq_empty(src_q)) >> > + return; >> > + >> > + head = ___cds_wfq_node_sync_next(&src_q->head); >> > + _cds_wfq_node_init(&src_q->head); >> > + >> > + /* >> > + * Memory barrier implied before uatomic_xchg() orders store to >> > + * src_q->head before store to src_q->tail. This is required by >> > + * concurrent enqueue on src_q, which exchanges the tail before >> > + * updating the previous tail's next pointer. >> > + */ >> > + tail = uatomic_xchg(&src_q->tail, &src_q->head); >> > + >> > /* >> > - * Requeue dummy node if we just dequeued it. >> > + * Append the spliced content of src_q into dest_q. Does not >> > + * require mutual exclusion on dest_q (wait-free). >> > */ >> > - if (node == &q->dummy) { >> > - _cds_wfq_node_init(node); >> > - _cds_wfq_enqueue(q, node); >> > - return ___cds_wfq_dequeue_blocking(q); >> > - } >> > - return node; >> > + ___cds_wfq_append(dest_q, head, tail); >> > +} >> > + >> > +/* Locking performed within cds_wfq calls. */ >> > +static inline struct cds_wfq_node * >> > +_cds_wfq_first_blocking(struct cds_wfq_queue *q) >> > +{ >> > + struct cds_wfq_node *retval; >> > + int ret; >> > + >> > + ret = pthread_mutex_lock(&q->lock); >> > + assert(!ret); >> > + retval = ___cds_wfq_first_blocking(q); >> > + ret = pthread_mutex_unlock(&q->lock); >> > + assert(!ret); >> > + return retval; >> > +} >> > + >> > +static inline struct cds_wfq_node * >> > +_cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) >> > +{ >> > + struct cds_wfq_node *retval; >> > + int ret; >> > + >> > + ret = pthread_mutex_lock(&q->lock); >> > + assert(!ret); >> > + retval = ___cds_wfq_next_blocking(q, node); >> > + ret = pthread_mutex_unlock(&q->lock); >> > + assert(!ret); >> > + return retval; >> > } >> >> I reject these _cds_wfq_first_blocking(), _cds_wfq_next_blocking() >> and cds_wfq_for_each_blocking(), because the claimed "Locking" >> makes no sense: >> 1. It protects nothing in _cds_wfq_next_blocking(). > > Good point! > >> 2. There is no "Locking" in the loop body, @node is not dequeued, >> it will be invalid if some other dequeue it, >> and _cds_wfq_next_blocking() results BUG. > > Indeed. > > So do you recommend we just leave locking to the callers then ? What > locking rules would you recommend we document in the API ? > > The only API members that would still have implicit locking are thus: > > - cds_wfq_dequeue_blocking() > - cds_wfq_splice_blocking() > > The only reason I leave them there is that we already expose a > "cds_wfq_dequeue_blocking" to users which provides locking, and I don't > want to break the API. > >> >> > >> > static inline struct cds_wfq_node * >> > _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) >> > { >> > - struct cds_wfq_node *retnode; >> > + struct cds_wfq_node *retval; >> > int ret; >> > >> > ret = pthread_mutex_lock(&q->lock); >> > assert(!ret); >> > - retnode = ___cds_wfq_dequeue_blocking(q); >> > + retval = ___cds_wfq_dequeue_blocking(q); >> > ret = pthread_mutex_unlock(&q->lock); >> > assert(!ret); >> > - return retnode; >> > + return retval; >> > +} >> > + >> > +static inline void >> > +_cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, >> > + struct cds_wfq_queue *src_q) >> > +{ >> > + int ret; >> > + >> > + ret = pthread_mutex_lock(&src_q->lock); >> > + assert(!ret); >> > + ___cds_wfq_splice_blocking(dest_q, src_q); >> > + ret = pthread_mutex_unlock(&src_q->lock); >> > + assert(!ret); >> > } >> > >> > #ifdef __cplusplus >> > diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h >> > index 03a73f1..d33d47a 100644 >> > --- a/urcu/wfqueue.h >> > +++ b/urcu/wfqueue.h >> > @@ -6,7 +6,8 @@ >> > * >> > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue >> > * >> > - * Copyright 2010 - Mathieu Desnoyers >> > + * Copyright 2010-2012 - Mathieu Desnoyers >> > + * Copyright 2011-2012 - Lai Jiangshan >> > * >> > * This library is free software; you can redistribute it and/or >> > * modify it under the terms of the GNU Lesser General Public >> > @@ -25,6 +26,7 @@ >> > >> > #include >> > #include >> > +#include >> > #include >> > >> > #ifdef __cplusplus >> > @@ -33,8 +35,6 @@ extern "C" { >> > >> > /* >> > * Queue with wait-free enqueue/blocking dequeue. >> > - * This implementation adds a dummy head node when the queue is empty to ensure >> > - * we can always update the queue locklessly. >> > * >> > * Inspired from half-wait-free/half-blocking queue implementation done by >> > * Paul E. McKenney. >> > @@ -45,8 +45,8 @@ struct cds_wfq_node { >> > }; >> > >> > struct cds_wfq_queue { >> > - struct cds_wfq_node *head, **tail; >> > - struct cds_wfq_node dummy; /* Dummy node */ >> > + struct cds_wfq_node head, *tail; >> > + struct cds_wfq_node padding; /* unused */ >> > pthread_mutex_t lock; >> > }; >> >> Why keep the padding? > > This is mainly to keep compatibility with applications already using > wfqueue.h. We could indeed make the size of the cds_wfq_queue smaller, > but we cannot enlarge it without breaking the API. Therefore, I think it > is safe to keep some unused padding rather than shrink the size, if we > ever need to put an extra flag or pointer in the structure. > > What I can do, though, is to move the padding to the end of the > structure, and give it a void * type. Is that OK ? > > Thanks, > > Mathieu > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com From mathieu.desnoyers at efficios.com Tue Aug 14 16:15:31 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 14 Aug 2012 16:15:31 -0400 Subject: [lttng-dev] [RFC PATCH LTTng-UST] Fix UST SIGPIPE handling Message-ID: <20120814201531.GA8827@Krystal> When the consumerd dies (from a SIGKILL), it may close all of its file descriptors rather abruptly. We ensured that the UST command threads have all signals blocked, and they use MSG_NOSIGNAL when sending messages to the sessiond over sockets. However, the consumer scheme uses a pipe(2) to transport the "wakeup" info from the application tracing site to the consumer daemon. It may send a SIGPIPE to the application in that case, which could kill the application, an unwanted side-effect. Block thread SIGPIPE around write() and wait for the signal to fix this. Signed-off-by: Mathieu Desnoyers --- diff --git a/libringbuffer/frontend_internal.h b/libringbuffer/frontend_internal.h index 6d1a75b..020b785 100644 --- a/libringbuffer/frontend_internal.h +++ b/libringbuffer/frontend_internal.h @@ -32,6 +32,8 @@ */ #include +#include +#include #include #include "backend_types.h" @@ -397,7 +399,9 @@ void lib_ring_buffer_check_deliver(const struct lttng_ust_lib_ring_buffer_config int wakeup_fd = shm_get_wakeup_fd(handle, &buf->self._ref); if (wakeup_fd >= 0) { - int ret; + sigset_t sigpipe_set, pending_set, old_set; + int ret, sigpipe_was_pending = 0; + /* * Wake-up the other end by * writing a null byte in the @@ -416,13 +420,56 @@ void lib_ring_buffer_check_deliver(const struct lttng_ust_lib_ring_buffer_config * 2) check if there is data in * the buffer. * 3) wait on the pipe (poll). + * + * Discard the SIGPIPE from write(), not + * disturbing any SIGPIPE that might be + * already pending. If a bogus SIGPIPE + * is sent to the entire process + * concurrently by a malicious user, it + * may be simply discarded. + */ + ret = sigemptyset(&pending_set); + assert(!ret); + /* + * sigpending returns the mask + * of signals that are _both_ + * blocked for the thread _and_ + * pending for either the thread + * or the entire process. */ + ret = sigpending(&pending_set); + assert(!ret); + sigpipe_was_pending = sigismember(&pending_set, SIGPIPE); + /* + * if the sigpipe was pending, + * it means it was already + * blocked, so no need to block + * it. + */ + if (!sigpipe_was_pending) { + ret = sigemptyset(&sigpipe_set); + assert(!ret); + ret = sigaddset(&sigpipe_set, SIGPIPE); + assert(!ret); + ret = pthread_sigmask(SIG_BLOCK, &sigpipe_set, &old_set); + assert(!ret); + } do { ret = write(wakeup_fd, "", 1); } while (ret == -1L && errno == EINTR); + if (ret == -1L && errno == EPIPE && !sigpipe_was_pending) { + struct timespec timeout = { 0, 0 }; + do { + ret = sigtimedwait(&sigpipe_set, NULL, + &timeout); + } while (ret == -1L && errno == EINTR); + } + if (!sigpipe_was_pending) { + ret = pthread_sigmask(SIG_SETMASK, &old_set, NULL); + assert(!ret); + } } } - } } } -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From christian.babeux at efficios.com Tue Aug 14 17:13:50 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Tue, 14 Aug 2012 17:13:50 -0400 Subject: [lttng-dev] [RFC PATCH LTTng-UST] Fix UST SIGPIPE handling In-Reply-To: <20120814201531.GA8827@Krystal> References: <20120814201531.GA8827@Krystal> Message-ID: Looks good to me! Acked-by: Christian Babeux From mathieu.desnoyers at efficios.com Tue Aug 14 18:27:24 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 14 Aug 2012 18:27:24 -0400 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost In-Reply-To: References: <20120812145000.GA32747@Krystal> <20120813201219.GA27637@Krystal> Message-ID: <20120814222724.GA9751@Krystal> * Lai Jiangshan (eag0628 at gmail.com) wrote: > On Tue, Aug 14, 2012 at 4:12 AM, Mathieu Desnoyers > wrote: [...] > >> > /* > >> > + * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. > >> > + * > >> > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured > >> > + * by the caller. > >> > + */ > >> > +static inline struct cds_wfq_node * > >> > +___cds_wfq_first_blocking(struct cds_wfq_queue *q) > >> > +{ > >> > + if (_cds_wfq_empty(q)) > >> > + return NULL; > >> > + return ___cds_wfq_node_sync_next(&q->head); > >> > +} > >> > + > >> > +/* > >> > + * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. > >> > + * > >> > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured > >> > + * by the caller. > >> > + */ > >> > +static inline struct cds_wfq_node * > >> > +___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > >> > +{ > >> > + if (CMM_LOAD_SHARED(q->tail) == node) > >> > + return NULL; > >> > + return ___cds_wfq_node_sync_next(node); > >> > +} > >> > >> > >> The same BUG as you told me. > >> If q has only one node just enqueued by other thread. > >> but if q->head.next is seen, ___cds_wfq_first_blocking() returns a node, > >> And the update of q->tail is not seen, it is still &q->head, > >> ___cds_wfq_node_sync_next(node) will be loop for every if there is no > >> other enqueue. > > > > Good catch ! :-) > > > >> > >> > >> > >> static inline struct cds_wfq_node * > >> ___cds_wfq_first_blocking(struct cds_wfq_queue *q) > >> { > >> + struct cds_wfq_node *ret. > >> if (_cds_wfq_empty(q)) > >> return NULL; > >> ret = ___cds_wfq_node_sync_next(&q->head); > >> + cmm_smp_rmb(); > >> + return ret; > >> } > > > > However, I think we should add the rmb at the beginning of > > ___cds_wfq_next_blocking(), so it applies at each "next" call. > > Otherwise, I think we could end up in a situation where we wait for a > > NULL next forever in the second of two consecutive > > ___cds_wfq_next_blocking() calls. I therefore propose: > > How this can happen(wait for a NULL next forever in the second ...)? > > A rmb is enough to leave the state(simgle node && q->tail == &q->head). > "CMM_LOAD_SHARED(q->tail) == node" will be true in the loop except this state. You are indeed right, but I now realise there is a small thing we need to change. I tried a couple of scenarios to convince myself of it, and here are my conclusions: - In all cases (except for the first node), comparing q->tail with node ensures that we never go beyond the tail. Except for the first node (q->head), we always perform this comparison, so at no point can we reach the nodes through "next" pointers that would be beyond the q->tail node. - In the case of the first node, we do the comparison with CMM_LOAD_SHARED(q->head.next) == NULL && CMM_LOAD_SHARED(q->tail) == &q->head; in no particular order (checking if empty) before we move to the first node with ___cds_wfq_node_sync_next(&q->head). The reason why the first node is different from all the others is because we don't use "CMM_LOAD_SHARED(q->tail) == &q->head" alone to distinguish between an empty queue and a queue that contains nodes: we first check for NULL q->head.next, AFAIU for performance reasons: whenever possible, we don't want to read the q->tail pointer because it is being heavily updated by the enqueuer. So assuming the goal beyond the CMM_LOAD_SHARED(q->head.next) == NULL check in the empty queue check is for performance of dequeue vs enqueue, I think it would be appropriate to do a similar optimisation for the ___cds_wfq_next_blocking() case: first check the next pointer of the current node to see if it is NULL. If not, then we know we can fetch the next pointer without comparing our position to the tail (performance optimisation). However, if we do that, we'll need a cmm_smp_rmb() at each ___cds_wfq_next_blocking() invocation. Here is the resulting code: /* * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. * * Mutual exclusion with "dequeue" and "splice" operations must be ensured * by the caller. */ static inline struct cds_wfq_node * ___cds_wfq_first_blocking(struct cds_wfq_queue *q) { if (_cds_wfq_empty(q)) return NULL; return ___cds_wfq_node_sync_next(&q->head); } /* * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. * * Mutual exclusion with "dequeue" and "splice" operations must be ensured * by the caller. */ static inline struct cds_wfq_node * ___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) { struct cds_wfq_node *next; if ((next = CMM_LOAD_SHARED(node->next)) != NULL) return next; /* Load node->next before q->tail */ cmm_smp_rmb(); if (CMM_LOAD_SHARED(q->tail) == node) return NULL; return ___cds_wfq_node_sync_next(node); } Is my understanding correct ? Thanks, Mathieu -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Tue Aug 14 19:06:35 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 14 Aug 2012 19:06:35 -0400 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost In-Reply-To: <20120814222724.GA9751@Krystal> References: <20120812145000.GA32747@Krystal> <20120813201219.GA27637@Krystal> <20120814222724.GA9751@Krystal> Message-ID: <20120814230634.GA11255@Krystal> * Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote: > * Lai Jiangshan (eag0628 at gmail.com) wrote: > > On Tue, Aug 14, 2012 at 4:12 AM, Mathieu Desnoyers > > wrote: > [...] > > >> > /* > > >> > + * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. > > >> > + * > > >> > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured > > >> > + * by the caller. > > >> > + */ > > >> > +static inline struct cds_wfq_node * > > >> > +___cds_wfq_first_blocking(struct cds_wfq_queue *q) > > >> > +{ > > >> > + if (_cds_wfq_empty(q)) > > >> > + return NULL; > > >> > + return ___cds_wfq_node_sync_next(&q->head); > > >> > +} > > >> > + > > >> > +/* > > >> > + * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. > > >> > + * > > >> > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured > > >> > + * by the caller. > > >> > + */ > > >> > +static inline struct cds_wfq_node * > > >> > +___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > > >> > +{ > > >> > + if (CMM_LOAD_SHARED(q->tail) == node) > > >> > + return NULL; > > >> > + return ___cds_wfq_node_sync_next(node); > > >> > +} > > >> > > >> > > >> The same BUG as you told me. > > >> If q has only one node just enqueued by other thread. > > >> but if q->head.next is seen, ___cds_wfq_first_blocking() returns a node, > > >> And the update of q->tail is not seen, it is still &q->head, > > >> ___cds_wfq_node_sync_next(node) will be loop for every if there is no > > >> other enqueue. > > > > > > Good catch ! :-) > > > > > >> > > >> > > >> > > >> static inline struct cds_wfq_node * > > >> ___cds_wfq_first_blocking(struct cds_wfq_queue *q) > > >> { > > >> + struct cds_wfq_node *ret. > > >> if (_cds_wfq_empty(q)) > > >> return NULL; > > >> ret = ___cds_wfq_node_sync_next(&q->head); > > >> + cmm_smp_rmb(); > > >> + return ret; > > >> } > > > > > > However, I think we should add the rmb at the beginning of > > > ___cds_wfq_next_blocking(), so it applies at each "next" call. > > > Otherwise, I think we could end up in a situation where we wait for a > > > NULL next forever in the second of two consecutive > > > ___cds_wfq_next_blocking() calls. I therefore propose: > > > > How this can happen(wait for a NULL next forever in the second ...)? > > > > A rmb is enough to leave the state(simgle node && q->tail == &q->head). > > "CMM_LOAD_SHARED(q->tail) == node" will be true in the loop except this state. > > You are indeed right, but I now realise there is a small thing we need > to change. I tried a couple of scenarios to convince myself of it, and > here are my conclusions: > > - In all cases (except for the first node), comparing q->tail with node > ensures that we never go beyond the tail. Except for the first node > (q->head), we always perform this comparison, so at no point can we > reach the nodes through "next" pointers that would be beyond the > q->tail node. > > - In the case of the first node, we do the comparison with > > CMM_LOAD_SHARED(q->head.next) == NULL > && CMM_LOAD_SHARED(q->tail) == &q->head; > > in no particular order (checking if empty) before we move to the > first node with ___cds_wfq_node_sync_next(&q->head). > The reason why the first node is different from all the others is > because we don't use "CMM_LOAD_SHARED(q->tail) == &q->head" alone > to distinguish between an empty queue and a queue that contains > nodes: we first check for NULL q->head.next, AFAIU for performance > reasons: whenever possible, we don't want to read the q->tail pointer > because it is being heavily updated by the enqueuer. > > So assuming the goal beyond the CMM_LOAD_SHARED(q->head.next) == NULL > check in the empty queue check is for performance of dequeue vs enqueue, > I think it would be appropriate to do a similar optimisation for the > ___cds_wfq_next_blocking() case: first check the next pointer of the > current node to see if it is NULL. If not, then we know we can fetch the > next pointer without comparing our position to the tail (performance > optimisation). However, if we do that, we'll need a cmm_smp_rmb() at > each ___cds_wfq_next_blocking() invocation. > > Here is the resulting code: > > /* > * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. > * > * Mutual exclusion with "dequeue" and "splice" operations must be ensured > * by the caller. > */ > static inline struct cds_wfq_node * > ___cds_wfq_first_blocking(struct cds_wfq_queue *q) > { > if (_cds_wfq_empty(q)) > return NULL; > return ___cds_wfq_node_sync_next(&q->head); > } > > /* > * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. > * > * Mutual exclusion with "dequeue" and "splice" operations must be ensured > * by the caller. > */ > static inline struct cds_wfq_node * > ___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > { > struct cds_wfq_node *next; > > if ((next = CMM_LOAD_SHARED(node->next)) != NULL) > return next; > /* Load node->next before q->tail */ > cmm_smp_rmb(); > if (CMM_LOAD_SHARED(q->tail) == node) > return NULL; > return ___cds_wfq_node_sync_next(node); > } > > Is my understanding correct ? One more thing: I think we need to add cmm_smp_read_barrier_depends() before returning the node pointers in dequeue, first, and next operations. This memory barrier would match the implicit barrier in the enqueue ordering write to the prior tail's next pointer with writes to the node content that would have been performed beforehand by the caller. Thoughts ? Thanks, Mathieu > > Thanks, > > Mathieu > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Tue Aug 14 19:34:02 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 14 Aug 2012 19:34:02 -0400 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost In-Reply-To: <20120814230634.GA11255@Krystal> References: <20120812145000.GA32747@Krystal> <20120813201219.GA27637@Krystal> <20120814222724.GA9751@Krystal> <20120814230634.GA11255@Krystal> Message-ID: <20120814233402.GA14033@Krystal> * Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote: [...] > One more thing: I think we need to add cmm_smp_read_barrier_depends() > before returning the node pointers in dequeue, first, and next > operations. This memory barrier would match the implicit barrier in the > enqueue ordering write to the prior tail's next pointer with writes to > the node content that would have been performed beforehand by the > caller. Please note that the only reason why I propose to use "cmm_smp_read_barrier_depends()" before returning nodes in dequeue, first, and next functions rather than a full cmm_smp_mb() is because I fail to see a case where an architecture would reorder a dependent store before the load of the pointer. I very well imagine a use-case where we have node N within the structure X like the scenario below, where we write to X immediately after dequeue: CPU 0 CPU 1 store X content (implicit full memory barrier implied before uatomic_xchg) enqueue X->N dequeue X->N (which barrier here ?) write to X immediately. If we assume that no reordering of a store that depends on loading a pointer can arrive, then I think we should be good with only a read_barrier_depends() on CPU 1. Otherwise, we might want to put a full barrier there to ensure that we don't race with CPU 0 storing content to X. Thoughts ? Thanks, Mathieu -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From eag0628 at gmail.com Tue Aug 14 23:42:34 2012 From: eag0628 at gmail.com (Lai Jiangshan) Date: Wed, 15 Aug 2012 11:42:34 +0800 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost In-Reply-To: <20120814222724.GA9751@Krystal> References: <20120812145000.GA32747@Krystal> <20120813201219.GA27637@Krystal> <20120814222724.GA9751@Krystal> Message-ID: On Wed, Aug 15, 2012 at 6:27 AM, Mathieu Desnoyers wrote: > * Lai Jiangshan (eag0628 at gmail.com) wrote: >> On Tue, Aug 14, 2012 at 4:12 AM, Mathieu Desnoyers >> wrote: > [...] >> >> > /* >> >> > + * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. >> >> > + * >> >> > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured >> >> > + * by the caller. >> >> > + */ >> >> > +static inline struct cds_wfq_node * >> >> > +___cds_wfq_first_blocking(struct cds_wfq_queue *q) >> >> > +{ >> >> > + if (_cds_wfq_empty(q)) >> >> > + return NULL; >> >> > + return ___cds_wfq_node_sync_next(&q->head); >> >> > +} >> >> > + >> >> > +/* >> >> > + * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. >> >> > + * >> >> > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured >> >> > + * by the caller. >> >> > + */ >> >> > +static inline struct cds_wfq_node * >> >> > +___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) >> >> > +{ >> >> > + if (CMM_LOAD_SHARED(q->tail) == node) >> >> > + return NULL; >> >> > + return ___cds_wfq_node_sync_next(node); >> >> > +} >> >> >> >> >> >> The same BUG as you told me. >> >> If q has only one node just enqueued by other thread. >> >> but if q->head.next is seen, ___cds_wfq_first_blocking() returns a node, >> >> And the update of q->tail is not seen, it is still &q->head, >> >> ___cds_wfq_node_sync_next(node) will be loop for every if there is no >> >> other enqueue. >> > >> > Good catch ! :-) >> > >> >> >> >> >> >> >> >> static inline struct cds_wfq_node * >> >> ___cds_wfq_first_blocking(struct cds_wfq_queue *q) >> >> { >> >> + struct cds_wfq_node *ret. >> >> if (_cds_wfq_empty(q)) >> >> return NULL; >> >> ret = ___cds_wfq_node_sync_next(&q->head); >> >> + cmm_smp_rmb(); >> >> + return ret; >> >> } >> > >> > However, I think we should add the rmb at the beginning of >> > ___cds_wfq_next_blocking(), so it applies at each "next" call. >> > Otherwise, I think we could end up in a situation where we wait for a >> > NULL next forever in the second of two consecutive >> > ___cds_wfq_next_blocking() calls. I therefore propose: >> >> How this can happen(wait for a NULL next forever in the second ...)? >> >> A rmb is enough to leave the state(simgle node && q->tail == &q->head). >> "CMM_LOAD_SHARED(q->tail) == node" will be true in the loop except this state. > > You are indeed right, but I now realise there is a small thing we need > to change. I tried a couple of scenarios to convince myself of it, and > here are my conclusions: > > - In all cases (except for the first node), comparing q->tail with node > ensures that we never go beyond the tail. Except for the first node > (q->head), we always perform this comparison, so at no point can we > reach the nodes through "next" pointers that would be beyond the > q->tail node. > > - In the case of the first node, we do the comparison with > > CMM_LOAD_SHARED(q->head.next) == NULL > && CMM_LOAD_SHARED(q->tail) == &q->head; > > in no particular order (checking if empty) before we move to the > first node with ___cds_wfq_node_sync_next(&q->head). > The reason why the first node is different from all the others is > because we don't use "CMM_LOAD_SHARED(q->tail) == &q->head" alone > to distinguish between an empty queue and a queue that contains > nodes: we first check for NULL q->head.next, AFAIU for performance > reasons: whenever possible, we don't want to read the q->tail pointer > because it is being heavily updated by the enqueuer. Is it false sharing? Access to q->head.next and access to q->tail have the same performance because they are in the same cache line. > > So assuming the goal beyond the CMM_LOAD_SHARED(q->head.next) == NULL > check in the empty queue check is for performance of dequeue vs enqueue, > I think it would be appropriate to do a similar optimisation for the > ___cds_wfq_next_blocking() case: first check the next pointer of the > current node to see if it is NULL. If not, then we know we can fetch the > next pointer without comparing our position to the tail (performance > optimisation). However, if we do that, we'll need a cmm_smp_rmb() at > each ___cds_wfq_next_blocking() invocation. > > Here is the resulting code: > > /* > * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. > * > * Mutual exclusion with "dequeue" and "splice" operations must be ensured > * by the caller. > */ > static inline struct cds_wfq_node * > ___cds_wfq_first_blocking(struct cds_wfq_queue *q) > { > if (_cds_wfq_empty(q)) > return NULL; > return ___cds_wfq_node_sync_next(&q->head); > } > > /* > * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. > * > * Mutual exclusion with "dequeue" and "splice" operations must be ensured > * by the caller. > */ > static inline struct cds_wfq_node * > ___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > { > struct cds_wfq_node *next; > > if ((next = CMM_LOAD_SHARED(node->next)) != NULL) > return next; > /* Load node->next before q->tail */ > cmm_smp_rmb(); > if (CMM_LOAD_SHARED(q->tail) == node) > return NULL; > return ___cds_wfq_node_sync_next(node); > } > > Is my understanding correct ? > > Thanks, > > Mathieu > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com From mathieu.desnoyers at efficios.com Wed Aug 15 08:52:10 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Wed, 15 Aug 2012 08:52:10 -0400 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost In-Reply-To: References: <20120812145000.GA32747@Krystal> <20120813201219.GA27637@Krystal> <20120814222724.GA9751@Krystal> Message-ID: <20120815125209.GA11737@Krystal> * Lai Jiangshan (eag0628 at gmail.com) wrote: > On Wed, Aug 15, 2012 at 6:27 AM, Mathieu Desnoyers > wrote: > > * Lai Jiangshan (eag0628 at gmail.com) wrote: > >> On Tue, Aug 14, 2012 at 4:12 AM, Mathieu Desnoyers > >> wrote: > > [...] > >> >> > /* > >> >> > + * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. > >> >> > + * > >> >> > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured > >> >> > + * by the caller. > >> >> > + */ > >> >> > +static inline struct cds_wfq_node * > >> >> > +___cds_wfq_first_blocking(struct cds_wfq_queue *q) > >> >> > +{ > >> >> > + if (_cds_wfq_empty(q)) > >> >> > + return NULL; > >> >> > + return ___cds_wfq_node_sync_next(&q->head); > >> >> > +} > >> >> > + > >> >> > +/* > >> >> > + * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. > >> >> > + * > >> >> > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured > >> >> > + * by the caller. > >> >> > + */ > >> >> > +static inline struct cds_wfq_node * > >> >> > +___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > >> >> > +{ > >> >> > + if (CMM_LOAD_SHARED(q->tail) == node) > >> >> > + return NULL; > >> >> > + return ___cds_wfq_node_sync_next(node); > >> >> > +} > >> >> > >> >> > >> >> The same BUG as you told me. > >> >> If q has only one node just enqueued by other thread. > >> >> but if q->head.next is seen, ___cds_wfq_first_blocking() returns a node, > >> >> And the update of q->tail is not seen, it is still &q->head, > >> >> ___cds_wfq_node_sync_next(node) will be loop for every if there is no > >> >> other enqueue. > >> > > >> > Good catch ! :-) > >> > > >> >> > >> >> > >> >> > >> >> static inline struct cds_wfq_node * > >> >> ___cds_wfq_first_blocking(struct cds_wfq_queue *q) > >> >> { > >> >> + struct cds_wfq_node *ret. > >> >> if (_cds_wfq_empty(q)) > >> >> return NULL; > >> >> ret = ___cds_wfq_node_sync_next(&q->head); > >> >> + cmm_smp_rmb(); > >> >> + return ret; > >> >> } > >> > > >> > However, I think we should add the rmb at the beginning of > >> > ___cds_wfq_next_blocking(), so it applies at each "next" call. > >> > Otherwise, I think we could end up in a situation where we wait for a > >> > NULL next forever in the second of two consecutive > >> > ___cds_wfq_next_blocking() calls. I therefore propose: > >> > >> How this can happen(wait for a NULL next forever in the second ...)? > >> > >> A rmb is enough to leave the state(simgle node && q->tail == &q->head). > >> "CMM_LOAD_SHARED(q->tail) == node" will be true in the loop except this state. > > > > You are indeed right, but I now realise there is a small thing we need > > to change. I tried a couple of scenarios to convince myself of it, and > > here are my conclusions: > > > > - In all cases (except for the first node), comparing q->tail with node > > ensures that we never go beyond the tail. Except for the first node > > (q->head), we always perform this comparison, so at no point can we > > reach the nodes through "next" pointers that would be beyond the > > q->tail node. > > > > - In the case of the first node, we do the comparison with > > > > CMM_LOAD_SHARED(q->head.next) == NULL > > && CMM_LOAD_SHARED(q->tail) == &q->head; > > > > in no particular order (checking if empty) before we move to the > > first node with ___cds_wfq_node_sync_next(&q->head). > > The reason why the first node is different from all the others is > > because we don't use "CMM_LOAD_SHARED(q->tail) == &q->head" alone > > to distinguish between an empty queue and a queue that contains > > nodes: we first check for NULL q->head.next, AFAIU for performance > > reasons: whenever possible, we don't want to read the q->tail pointer > > because it is being heavily updated by the enqueuer. > > Is it false sharing? > Access to q->head.next and access to q->tail have the same performance > because they are in the same cache line. Yes! you are right! And a quick benchmark confirms it: with head and tail on same cache line: SUMMARY /home/compudj/doc/userspace-rcu/tests/.libs/lt-test_urcu_wfq testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 100833595 nr_dequeues 88647134 successful enqueues 100833595 successful dequeues 88646898 end_dequeues 12186697 nr_ops 189480729 with a 256 bytes padding between head and tail, keeping the mutex on the "head" cache line: SUMMARY /home/compudj/doc/userspace-rcu/tests/.libs/lt-test_urcu_wfq testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 228992829 nr_dequeues 228921791 successful enqueues 228992829 successful dequeues 228921367 end_dequeues 71462 nr_ops 457914620 enqueue: 127% speedup dequeue: 158% speedup That is indeed a _really_ huge difference. However, to get this, we would have to increase the size of struct cds_wfq_queue beyond its current size, which would break API compatibility. Any idea on how to best do this without causing incompatibility would be welcome. Thanks! Mathieu > > > > > So assuming the goal beyond the CMM_LOAD_SHARED(q->head.next) == NULL > > check in the empty queue check is for performance of dequeue vs enqueue, > > I think it would be appropriate to do a similar optimisation for the > > ___cds_wfq_next_blocking() case: first check the next pointer of the > > current node to see if it is NULL. If not, then we know we can fetch the > > next pointer without comparing our position to the tail (performance > > optimisation). However, if we do that, we'll need a cmm_smp_rmb() at > > each ___cds_wfq_next_blocking() invocation. > > > > Here is the resulting code: > > > > /* > > * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. > > * > > * Mutual exclusion with "dequeue" and "splice" operations must be ensured > > * by the caller. > > */ > > static inline struct cds_wfq_node * > > ___cds_wfq_first_blocking(struct cds_wfq_queue *q) > > { > > if (_cds_wfq_empty(q)) > > return NULL; > > return ___cds_wfq_node_sync_next(&q->head); > > } > > > > /* > > * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. > > * > > * Mutual exclusion with "dequeue" and "splice" operations must be ensured > > * by the caller. > > */ > > static inline struct cds_wfq_node * > > ___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > > { > > struct cds_wfq_node *next; > > > > if ((next = CMM_LOAD_SHARED(node->next)) != NULL) > > return next; > > /* Load node->next before q->tail */ > > cmm_smp_rmb(); > > if (CMM_LOAD_SHARED(q->tail) == node) > > return NULL; > > return ___cds_wfq_node_sync_next(node); > > } > > > > Is my understanding correct ? > > > > Thanks, > > > > Mathieu > > > > -- > > Mathieu Desnoyers > > Operating System Efficiency R&D Consultant > > EfficiOS Inc. > > http://www.efficios.com -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From christian.babeux at efficios.com Wed Aug 15 12:39:30 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Wed, 15 Aug 2012 12:39:30 -0400 Subject: [lttng-dev] [PATCH lttng-tools 1/2] Fix: Invalid free on session_name when destroying session Message-ID: <1345048771-18936-1-git-send-email-christian.babeux@efficios.com> The session_name should not be free(3) if the user has specified a session name on the command line. Also, the caller is responsible to free the allocated string when calling get_session_name(). Handle both cases gracefully. Signed-off-by: Christian Babeux --- src/bin/lttng/commands/destroy.c | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/src/bin/lttng/commands/destroy.c b/src/bin/lttng/commands/destroy.c index 5b69cb5..7b7ea0e 100644 --- a/src/bin/lttng/commands/destroy.c +++ b/src/bin/lttng/commands/destroy.c @@ -28,6 +28,7 @@ #include +static char *opt_session_name; static int opt_destroy_all; enum { @@ -156,28 +157,32 @@ int cmd_destroy(int argc, const char **argv) goto end; } - session_name = (char *) poptGetArg(pc); - - /* - * ignore session name in case all - * sessions are to be destroyed - */ + /* Ignore session name in case all sessions are to be destroyed */ if (opt_destroy_all) { ret = destroy_all_sessions(); goto end; } - if (session_name == NULL) { - ret = get_default_session_name(&session_name); - if (ret < 0 || session_name == NULL) { + + opt_session_name = (char *) poptGetArg(pc); + + if (opt_session_name == NULL) { + /* No session name specified, lookup default */ + session_name = get_session_name(); + if (session_name == NULL) { + ret = CMD_ERROR; goto end; } + } else { + session_name = opt_session_name; } + ret = destroy_session(session_name); end: - poptFreeContext(pc); - if (session_name != NULL) { + if (opt_session_name == NULL) { free(session_name); } + + poptFreeContext(pc); return ret; } -- 1.7.11.4 From christian.babeux at efficios.com Wed Aug 15 12:39:31 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Wed, 15 Aug 2012 12:39:31 -0400 Subject: [lttng-dev] [PATCH lttng-tools 2/2] Cleanup: Remove unused get_default_session_name() function In-Reply-To: <1345048771-18936-1-git-send-email-christian.babeux@efficios.com> References: <1345048771-18936-1-git-send-email-christian.babeux@efficios.com> Message-ID: <1345048771-18936-2-git-send-email-christian.babeux@efficios.com> Signed-off-by: Christian Babeux --- src/bin/lttng/commands/destroy.c | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/src/bin/lttng/commands/destroy.c b/src/bin/lttng/commands/destroy.c index 7b7ea0e..cdc2c53 100644 --- a/src/bin/lttng/commands/destroy.c +++ b/src/bin/lttng/commands/destroy.c @@ -114,21 +114,6 @@ error: } /* - * get_default_session_name - * - * Returns the default sessions name, if any - */ -static int get_default_session_name(char **name) -{ - char *session_name = get_session_name(); - if (session_name == NULL) { - return CMD_ERROR; - } - *name = session_name; - return CMD_SUCCESS; -} - -/* * The 'destroy ' first level command */ int cmd_destroy(int argc, const char **argv) -- 1.7.11.4 From mathieu.desnoyers at efficios.com Wed Aug 15 17:28:06 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Wed, 15 Aug 2012 17:28:06 -0400 Subject: [lttng-dev] [RFC URCU PATCH] Introduce wfqueue v0 ABI compatibility Message-ID: <20120815212806.GA17224@Krystal> Introduce wfqueue v0 ABI compatibility Preparing for v1 ABI. CC: Lai Jiangshan CC: Paul McKenney Signed-off-by: Mathieu Desnoyers --- diff --git a/wfqueue0.c b/wfqueue0.c new file mode 100644 index 0000000..e928c0a --- /dev/null +++ b/wfqueue0.c @@ -0,0 +1,191 @@ +/* + * wfqueue0.c + * + * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue + * + * Copyright 2010-2012 - Mathieu Desnoyers + * Copyright 2011-2012 - Lai Jiangshan + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/* This file is provided for ABI backward compatibility with wfqueue v0. */ + +#include +#include +#include +#include +#include + +/* + * Queue with wait-free enqueue/blocking dequeue. + * This implementation adds a dummy head node when the queue is empty to ensure + * we can always update the queue locklessly. + * + * Inspired from half-wait-free/half-blocking queue implementation done by + * Paul E. McKenney. + */ + +struct cds_wfq_node { + struct cds_wfq_node *next; +}; + +struct cds_wfq_queue { + struct cds_wfq_node *head, **tail; + struct cds_wfq_node dummy; /* Dummy node */ + pthread_mutex_t lock; +}; + +#define WFQ_ADAPT_ATTEMPTS 10 /* Retry if being set */ +#define WFQ_WAIT 10 /* Wait 10 ms if being set */ + +static inline void _cds_wfq_node_init(struct cds_wfq_node *node) +{ + node->next = NULL; +} + +static inline void _cds_wfq_init(struct cds_wfq_queue *q) +{ + int ret; + + _cds_wfq_node_init(&q->dummy); + /* Set queue head and tail */ + q->head = &q->dummy; + q->tail = &q->dummy.next; + ret = pthread_mutex_init(&q->lock, NULL); + assert(!ret); +} + +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, + struct cds_wfq_node *node) +{ + struct cds_wfq_node **old_tail; + + /* + * uatomic_xchg() implicit memory barrier orders earlier stores to data + * structure containing node and setting node->next to NULL before + * publication. + */ + old_tail = uatomic_xchg(&q->tail, &node->next); + /* + * At this point, dequeuers see a NULL old_tail->next, which indicates + * that the queue is being appended to. The following store will append + * "node" to the queue from a dequeuer perspective. + */ + CMM_STORE_SHARED(*old_tail, node); +} + +/* + * Waiting for enqueuer to complete enqueue and return the next node + */ +static inline struct cds_wfq_node * +___cds_wfq_node_sync_next(struct cds_wfq_node *node) +{ + struct cds_wfq_node *next; + int attempt = 0; + + /* + * Adaptative busy-looping waiting for enqueuer to complete enqueue. + */ + while ((next = CMM_LOAD_SHARED(node->next)) == NULL) { + if (++attempt >= WFQ_ADAPT_ATTEMPTS) { + poll(NULL, 0, WFQ_WAIT); /* Wait for 10ms */ + attempt = 0; + } else + caa_cpu_relax(); + } + + return next; +} + +/* + * It is valid to reuse and free a dequeued node immediately. + * + * No need to go on a waitqueue here, as there is no possible state in which the + * list could cause dequeue to busy-loop needlessly while waiting for another + * thread to be scheduled. The queue appears empty until tail->next is set by + * enqueue. + */ +static inline struct cds_wfq_node * +___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) +{ + struct cds_wfq_node *node, *next; + + /* + * Queue is empty if it only contains the dummy node. + */ + if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) + return NULL; + node = q->head; + + next = ___cds_wfq_node_sync_next(node); + + /* + * Move queue head forward. + */ + q->head = next; + /* + * Requeue dummy node if we just dequeued it. + */ + if (node == &q->dummy) { + _cds_wfq_node_init(node); + _cds_wfq_enqueue(q, node); + return ___cds_wfq_dequeue_blocking(q); + } + return node; +} + +static inline struct cds_wfq_node * +_cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) +{ + struct cds_wfq_node *retnode; + int ret; + + ret = pthread_mutex_lock(&q->lock); + assert(!ret); + retnode = ___cds_wfq_dequeue_blocking(q); + ret = pthread_mutex_unlock(&q->lock); + assert(!ret); + return retnode; +} + +/* + * library wrappers to be used by non-LGPL compatible source code. + */ + +void cds_wfq_node_init(struct cds_wfq_node *node) +{ + _cds_wfq_node_init(node); +} + +void cds_wfq_init(struct cds_wfq_queue *q) +{ + _cds_wfq_init(q); +} + +void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node) +{ + _cds_wfq_enqueue(q, node); +} + +struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) +{ + return _cds_wfq_dequeue_blocking(q); +} + +struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) +{ + return ___cds_wfq_dequeue_blocking(q); +} -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Wed Aug 15 17:31:07 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Wed, 15 Aug 2012 17:31:07 -0400 Subject: [lttng-dev] [RFC URCU PATCH] wfqueue: ABI v1, simplify implementation, 2.3x to 2.6x performance boost Message-ID: <20120815213107.GB17224@Krystal> This work is derived from the patch from Lai Jiangshan submitted as "urcu: new wfqueue implementation" (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) Its changelog: > Some guys would be surprised by this fact: > There are already TWO implementations of wfqueue in urcu. > > The first one is in urcu/static/wfqueue.h: > 1) enqueue: exchange the tail and then update previous->next > 2) dequeue: wait for first node's next pointer and them shift, a dummy node > is introduced to avoid the queue->tail become NULL when shift. > > The second one shares some code with the first one, and the left code > are spreading in urcu-call-rcu-impl.h: > 1) enqueue: share with the first one > 2) no dequeue operation: and no shift, so it don't need dummy node, > Although the dummy node is queued when initialization, but it is removed > after the first dequeue_all operation in call_rcu_thread(). > call_rcu_data_free() forgets to handle the dummy node if it is not removed. > 3)dequeue_all: record the old head and tail, and queue->head become the special > tail node.(atomic record the tail and change the tail). > > The second implementation's code are spreading, bad for review, and it is not > tested by tests/test_urcu_wfq. > > So we need a better implementation avoid the dummy node dancing and can service > both generic wfqueue APIs and dequeue_all API for call rcu. > > The new implementation: > 1) enqueue: share with the first one/original implementation. > 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. > no dummy node, save memory. > 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail > and return the old head.next. > > More implementation details are in the code. > tests/test_urcu_wfq will be update in future for testing new APIs. The patch proposed by Lai brings a very interesting simplification to the single-node handling (which is kept here), and moves all queue handling code away from call_rcu implementation, back into the wfqueue code. This has the benefit to allow testing enhancements. I modified it so the API does not expose implementation details to the user (e.g. ___cds_wfq_node_sync_next). I added a "splice" operation and a for loop iterator which should allow wfqueue users to use the list very efficiently both from LGPL/GPL code and from non-LGPL-compatible code. Benchmarks performed on Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz (dual-core, with hyperthreading) Benchmark invoked: for a in $(seq 1 10); do ./test_urcu_wfq 1 1 10 -a 0 -a 2; done (using cpu number 0 and 2, which should correspond to two cores of my Intel 2-core/hyperthread processor) Before patch: testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 97274297 nr_dequeues 80745742 successful enqueues 97274297 successful dequeues 80745321 end_dequeues 16528976 nr_ops 178020039 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 92300568 nr_dequeues 75019529 successful enqueues 92300568 successful dequeues 74973237 end_dequeues 17327331 nr_ops 167320097 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 93516443 nr_dequeues 75846726 successful enqueues 93516443 successful dequeues 75826578 end_dequeues 17689865 nr_ops 169363169 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 94160362 nr_dequeues 77967638 successful enqueues 94160362 successful dequeues 77967638 end_dequeues 16192724 nr_ops 172128000 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 97491956 nr_dequeues 81001191 successful enqueues 97491956 successful dequeues 81000247 end_dequeues 16491709 nr_ops 178493147 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 94101298 nr_dequeues 75650510 successful enqueues 94101298 successful dequeues 75649318 end_dequeues 18451980 nr_ops 169751808 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 94742803 nr_dequeues 75402105 successful enqueues 94742803 successful dequeues 75341859 end_dequeues 19400944 nr_ops 170144908 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 92198835 nr_dequeues 75037877 successful enqueues 92198835 successful dequeues 75027605 end_dequeues 17171230 nr_ops 167236712 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 94159560 nr_dequeues 77895972 successful enqueues 94159560 successful dequeues 77858442 end_dequeues 16301118 nr_ops 172055532 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 96059399 nr_dequeues 80115442 successful enqueues 96059399 successful dequeues 80066843 end_dequeues 15992556 nr_ops 176174841 After patch: testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 221229322 nr_dequeues 210645491 successful enqueues 221229322 successful dequeues 210645088 end_dequeues 10584234 nr_ops 431874813 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 219803943 nr_dequeues 210377337 successful enqueues 219803943 successful dequeues 210368680 end_dequeues 9435263 nr_ops 430181280 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 237006358 nr_dequeues 237035340 successful enqueues 237006358 successful dequeues 236997050 end_dequeues 9308 nr_ops 474041698 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 235822443 nr_dequeues 235815942 successful enqueues 235822443 successful dequeues 235814020 end_dequeues 8423 nr_ops 471638385 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 235825567 nr_dequeues 235811803 successful enqueues 235825567 successful dequeues 235810526 end_dequeues 15041 nr_ops 471637370 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 221974953 nr_dequeues 210938190 successful enqueues 221974953 successful dequeues 210938190 end_dequeues 11036763 nr_ops 432913143 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 237994492 nr_dequeues 237938119 successful enqueues 237994492 successful dequeues 237930648 end_dequeues 63844 nr_ops 475932611 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 220634365 nr_dequeues 210491382 successful enqueues 220634365 successful dequeues 210490995 end_dequeues 10143370 nr_ops 431125747 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 237388065 nr_dequeues 237401251 successful enqueues 237388065 successful dequeues 237380295 end_dequeues 7770 nr_ops 474789316 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 221201436 nr_dequeues 210831162 successful enqueues 221201436 successful dequeues 210831162 end_dequeues 10370274 nr_ops 432032598 Summary: Both enqueue and dequeue speed increase: around 2.3x speedup for enqueue, and around 2.6x for dequeue. We can verify that: successful enqueues - successful dequeues = end_dequeues For all runs (ensures correctness: no lost node). * Introduce wfqueue ABI v1 (false-sharing fix) wfqueue v0 suffers from false-sharing between head and tail. By cache-aligning head and tail, we get a significant speedup on benchmarks. But in order to do that, we need to break the ABI to enlarge struct cds_wfq_queue. Provide a backward compatibility ABI for the old wfqueue by defining the original symbols to the old implementation (which uses dummy node). Programs compiled against old headers, which are not LGPL_SOURCE, will still use the old implementation. Any code compiled against the new header will directly use the new ABI. This does not require any change in the way users call the API: the new ABI symbols are simply defined with _1 suffix, and wrapped by preprocessor macros. Known limitation: users should *not* link together objects using the v0 and v1 APIs of wfqueue and exchange struct cds_wfq_queue structures between the two. The only way to run into this corner-case would be to combine objects compiled with different versions of the urcu/wfqueue.h header. CC: Lai Jiangshan CC: Paul McKenney Signed-off-by: Mathieu Desnoyers --- diff --git a/Makefile.am b/Makefile.am index 2396fcf..31052ef 100644 --- a/Makefile.am +++ b/Makefile.am @@ -53,7 +53,7 @@ lib_LTLIBRARIES = liburcu-common.la \ # liburcu-common contains wait-free queues (needed by call_rcu) as well # as futex fallbacks. # -liburcu_common_la_SOURCES = wfqueue.c wfstack.c $(COMPAT) +liburcu_common_la_SOURCES = wfqueue.c wfqueue0.c wfstack.c $(COMPAT) liburcu_la_SOURCES = urcu.c urcu-pointer.c $(COMPAT) liburcu_la_LIBADD = liburcu-common.la diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h index 13b24ff..d8537d0 100644 --- a/urcu-call-rcu-impl.h +++ b/urcu-call-rcu-impl.h @@ -21,6 +21,7 @@ */ #define _GNU_SOURCE +#define _LGPL_SOURCE #include #include #include @@ -220,10 +221,7 @@ static void call_rcu_wake_up(struct call_rcu_data *crdp) static void *call_rcu_thread(void *arg) { unsigned long cbcount; - struct cds_wfq_node *cbs; - struct cds_wfq_node **cbs_tail; - struct call_rcu_data *crdp = (struct call_rcu_data *)arg; - struct rcu_head *rhp; + struct call_rcu_data *crdp = (struct call_rcu_data *) arg; int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); int ret; @@ -243,35 +241,30 @@ static void *call_rcu_thread(void *arg) cmm_smp_mb(); } for (;;) { - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) - poll(NULL, 0, 1); - _CMM_STORE_SHARED(crdp->cbs.head, NULL); - cbs_tail = (struct cds_wfq_node **) - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); + struct cds_wfq_queue cbs_tmp; + struct cds_wfq_node *cbs, *tmp_cbs; + + cds_wfq_init(&cbs_tmp); + __cds_wfq_splice_blocking(&cbs_tmp, &crdp->cbs); + if (!cds_wfq_empty(&cbs_tmp)) { synchronize_rcu(); cbcount = 0; - do { - while (cbs->next == NULL && - &cbs->next != cbs_tail) - poll(NULL, 0, 1); - if (cbs == &crdp->cbs.dummy) { - cbs = cbs->next; - continue; - } - rhp = (struct rcu_head *)cbs; - cbs = cbs->next; + __cds_wfq_for_each_blocking_safe(&cbs_tmp, + cbs, tmp_cbs) { + struct rcu_head *rhp; + + rhp = caa_container_of(cbs, + struct rcu_head, next); rhp->func(rhp); cbcount++; - } while (cbs != NULL); + } uatomic_sub(&crdp->qlen, cbcount); } if (uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOP) break; rcu_thread_offline(); if (!rt) { - if (&crdp->cbs.head - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { + if (cds_wfq_empty(&crdp->cbs)) { call_rcu_wait(crdp); poll(NULL, 0, 10); uatomic_dec(&crdp->futex); @@ -625,32 +618,27 @@ void call_rcu(struct rcu_head *head, */ void call_rcu_data_free(struct call_rcu_data *crdp) { - struct cds_wfq_node *cbs; - struct cds_wfq_node **cbs_tail; - struct cds_wfq_node **cbs_endprev; - if (crdp == NULL || crdp == default_call_rcu_data) { return; } + if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); wake_call_rcu_thread(crdp); while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) poll(NULL, 0, 1); } - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) - poll(NULL, 0, 1); - _CMM_STORE_SHARED(crdp->cbs.head, NULL); - cbs_tail = (struct cds_wfq_node **) - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); + + if (!cds_wfq_empty(&crdp->cbs)) { /* Create default call rcu data if need be */ (void) get_default_call_rcu_data(); - cbs_endprev = (struct cds_wfq_node **) - uatomic_xchg(&default_call_rcu_data, cbs_tail); - *cbs_endprev = cbs; + + __cds_wfq_splice_blocking(&default_call_rcu_data->cbs, + &crdp->cbs); + uatomic_add(&default_call_rcu_data->qlen, uatomic_read(&crdp->qlen)); + wake_call_rcu_thread(default_call_rcu_data); } diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h index 636e1af..52e452d 100644 --- a/urcu/static/wfqueue.h +++ b/urcu/static/wfqueue.h @@ -9,7 +9,8 @@ * TO BE INCLUDED ONLY IN LGPL-COMPATIBLE CODE. See wfqueue.h for linking * dynamically with the userspace rcu library. * - * Copyright 2010 - Mathieu Desnoyers + * Copyright 2010-2012 - Mathieu Desnoyers + * Copyright 2011-2012 - Lai Jiangshan * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -29,6 +30,7 @@ #include #include #include +#include #include #include @@ -38,11 +40,16 @@ extern "C" { /* * Queue with wait-free enqueue/blocking dequeue. - * This implementation adds a dummy head node when the queue is empty to ensure - * we can always update the queue locklessly. * * Inspired from half-wait-free/half-blocking queue implementation done by * Paul E. McKenney. + * + * Caller must ensure mutual exclusion of queue update operations + * "dequeue" and "splice" source queue. Queue read operations "first" + * and "next" need to be protected against concurrent "dequeue" and + * "splice" (for source queue) by the caller. "enqueue", "splice" + * (destination queue), and "empty" are the only operations that can be + * used without any mutual exclusion. */ #define WFQ_ADAPT_ATTEMPTS 10 /* Retry if being set */ @@ -57,31 +64,55 @@ static inline void _cds_wfq_init(struct cds_wfq_queue *q) { int ret; - _cds_wfq_node_init(&q->dummy); /* Set queue head and tail */ - q->head = &q->dummy; - q->tail = &q->dummy.next; - ret = pthread_mutex_init(&q->lock, NULL); + _cds_wfq_node_init(&q->dequeue.head); + q->enqueue.tail = &q->dequeue.head; + ret = pthread_mutex_init(&q->dequeue.lock, NULL); assert(!ret); } -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, - struct cds_wfq_node *node) +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) { - struct cds_wfq_node **old_tail; + /* + * Queue is empty if no node is pointed by q->head.next nor + * q->tail. Even though the q->tail check is sufficient to find + * out of the queue is empty, we first check q->head.next as a + * common case to ensure that dequeuers do not frequently access + * enqueuer's q->tail cache line. + */ + return CMM_LOAD_SHARED(q->dequeue.head.next) == NULL + && CMM_LOAD_SHARED(q->enqueue.tail) == &q->dequeue.head; +} + +static inline void ___cds_wfq_append(struct cds_wfq_queue *q, + struct cds_wfq_node *new_head, + struct cds_wfq_node *new_tail) +{ + struct cds_wfq_node *old_tail; /* - * uatomic_xchg() implicit memory barrier orders earlier stores to data - * structure containing node and setting node->next to NULL before - * publication. + * Implicit memory barrier before uatomic_xchg() orders earlier + * stores to data structure containing node and setting + * node->next to NULL before publication. */ - old_tail = uatomic_xchg(&q->tail, &node->next); + old_tail = uatomic_xchg(&q->enqueue.tail, new_tail); + /* - * At this point, dequeuers see a NULL old_tail->next, which indicates - * that the queue is being appended to. The following store will append - * "node" to the queue from a dequeuer perspective. + * Implicit memory barrier after uatomic_xchg() orders store to + * q->tail before store to old_tail->next. + * + * At this point, dequeuers see a NULL q->tail->next, which + * indicates that the queue is being appended to. The following + * store will append "node" to the queue from a dequeuer + * perspective. */ - CMM_STORE_SHARED(*old_tail, node); + CMM_STORE_SHARED(old_tail->next, new_head); +} + +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, + struct cds_wfq_node *new_tail) +{ + ___cds_wfq_append(q, new_tail, new_tail); } /* @@ -100,14 +131,68 @@ ___cds_wfq_node_sync_next(struct cds_wfq_node *node) if (++attempt >= WFQ_ADAPT_ATTEMPTS) { poll(NULL, 0, WFQ_WAIT); /* Wait for 10ms */ attempt = 0; - } else + } else { caa_cpu_relax(); + } } return next; } /* + * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. + * + * Mutual exclusion with "dequeue" and "splice" operations must be ensured + * by the caller. + */ +static inline struct cds_wfq_node * +___cds_wfq_first_blocking(struct cds_wfq_queue *q) +{ + struct cds_wfq_node *node; + + if (_cds_wfq_empty(q)) + return NULL; + node = ___cds_wfq_node_sync_next(&q->dequeue.head); + /* Load q->head.next before loading node's content */ + cmm_smp_read_barrier_depends(); + return node; +} + +/* + * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. + * + * Mutual exclusion with "dequeue" and "splice" operations must be ensured + * by the caller. + */ +static inline struct cds_wfq_node * +___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) +{ + struct cds_wfq_node *next; + + /* + * Even though the following q->tail check is sufficient to find + * out if we reached the end of the queue, we first check + * node->next as a common case to ensure that iteration on nodes + * do not frequently access enqueuer's q->tail cache line. + */ + if ((next = CMM_LOAD_SHARED(node->next)) != NULL) { + /* Load node->next before loading next's content */ + cmm_smp_read_barrier_depends(); + return next; + } + /* Load node->next before q->tail */ + cmm_smp_rmb(); + if (CMM_LOAD_SHARED(q->enqueue.tail) == node) + return NULL; + next = ___cds_wfq_node_sync_next(node); + /* Load node->next before loading next's content */ + cmm_smp_read_barrier_depends(); + return next; +} + +/* + * ___cds_wfq_dequeue_blocking: dequeue a node from the queue. + * * It is valid to reuse and free a dequeued node immediately. * * No need to go on a waitqueue here, as there is no possible state in which the @@ -120,42 +205,104 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) { struct cds_wfq_node *node, *next; - /* - * Queue is empty if it only contains the dummy node. - */ - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) + if (_cds_wfq_empty(q)) return NULL; - node = q->head; - next = ___cds_wfq_node_sync_next(node); + node = ___cds_wfq_node_sync_next(&q->dequeue.head); + + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { + /* + * @node is probably the only node in the queue. + * Try to move the tail to &q->head. + * q->head.next is set to NULL here, and stays + * NULL if the cmpxchg succeeds. Should the + * cmpxchg fail due to a concurrent enqueue, the + * q->head.next will be set to the next node. + * The implicit memory barrier before + * uatomic_cmpxchg() orders load node->next + * before loading q->tail. + * The implicit memory barrier before uatomic_cmpxchg + * orders load q->head.next before loading node's + * content. + */ + _cds_wfq_node_init(&q->dequeue.head); + if (uatomic_cmpxchg(&q->enqueue.tail, node, + &q->dequeue.head) == node) + return node; + next = ___cds_wfq_node_sync_next(node); + } /* * Move queue head forward. */ - q->head = next; + q->dequeue.head.next = next; + + /* Load q->head.next before loading node's content */ + cmm_smp_read_barrier_depends(); + return node; +} + +/* + * ___cds_wfq_splice_blocking: enqueue all src_q nodes at the end of dest_q. + * + * Dequeue all nodes from src_q. + * dest_q must be already initialized. + * caller ensures mutual exclusion of dequeue and splice operations on + * src_q. + */ +static inline void +___cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q) +{ + struct cds_wfq_node *head, *tail; + + if (_cds_wfq_empty(src_q)) + return; + + head = ___cds_wfq_node_sync_next(&src_q->dequeue.head); + _cds_wfq_node_init(&src_q->dequeue.head); + /* - * Requeue dummy node if we just dequeued it. + * Memory barrier implied before uatomic_xchg() orders store to + * src_q->head before store to src_q->tail. This is required by + * concurrent enqueue on src_q, which exchanges the tail before + * updating the previous tail's next pointer. */ - if (node == &q->dummy) { - _cds_wfq_node_init(node); - _cds_wfq_enqueue(q, node); - return ___cds_wfq_dequeue_blocking(q); - } - return node; + tail = uatomic_xchg(&src_q->enqueue.tail, &src_q->dequeue.head); + + /* + * Append the spliced content of src_q into dest_q. Does not + * require mutual exclusion on dest_q (wait-free). + */ + ___cds_wfq_append(dest_q, head, tail); } +/* Locking performed within cds_wfq calls. */ static inline struct cds_wfq_node * _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) { - struct cds_wfq_node *retnode; + struct cds_wfq_node *retval; + int ret; + + ret = pthread_mutex_lock(&q->dequeue.lock); + assert(!ret); + retval = ___cds_wfq_dequeue_blocking(q); + ret = pthread_mutex_unlock(&q->dequeue.lock); + assert(!ret); + return retval; +} + +static inline void +_cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q) +{ int ret; - ret = pthread_mutex_lock(&q->lock); + ret = pthread_mutex_lock(&src_q->dequeue.lock); assert(!ret); - retnode = ___cds_wfq_dequeue_blocking(q); - ret = pthread_mutex_unlock(&q->lock); + ___cds_wfq_splice_blocking(dest_q, src_q); + ret = pthread_mutex_unlock(&src_q->dequeue.lock); assert(!ret); - return retnode; } #ifdef __cplusplus diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h index 03a73f1..446c94c 100644 --- a/urcu/wfqueue.h +++ b/urcu/wfqueue.h @@ -6,7 +6,8 @@ * * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue * - * Copyright 2010 - Mathieu Desnoyers + * Copyright 2010-2012 - Mathieu Desnoyers + * Copyright 2011-2012 - Lai Jiangshan * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -25,7 +26,9 @@ #include #include +#include #include +#include #ifdef __cplusplus extern "C" { @@ -33,8 +36,6 @@ extern "C" { /* * Queue with wait-free enqueue/blocking dequeue. - * This implementation adds a dummy head node when the queue is empty to ensure - * we can always update the queue locklessly. * * Inspired from half-wait-free/half-blocking queue implementation done by * Paul E. McKenney. @@ -45,9 +46,13 @@ struct cds_wfq_node { }; struct cds_wfq_queue { - struct cds_wfq_node *head, **tail; - struct cds_wfq_node dummy; /* Dummy node */ - pthread_mutex_t lock; + struct { + struct cds_wfq_node head; + pthread_mutex_t lock; + } __attribute__((aligned((CAA_CACHE_LINE_SIZE)))) dequeue; + struct { + struct cds_wfq_node *tail; + } __attribute__((aligned((CAA_CACHE_LINE_SIZE)))) enqueue; }; #ifdef _LGPL_SOURCE @@ -55,22 +60,104 @@ struct cds_wfq_queue { #include #define cds_wfq_node_init _cds_wfq_node_init -#define cds_wfq_init _cds_wfq_init -#define cds_wfq_enqueue _cds_wfq_enqueue -#define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking +#define cds_wfq_init _cds_wfq_init +#define cds_wfq_empty _cds_wfq_empty +#define cds_wfq_enqueue _cds_wfq_enqueue + +/* Locking performed within cds_wfq calls. */ #define cds_wfq_dequeue_blocking _cds_wfq_dequeue_blocking +#define cds_wfq_splice_blocking _cds_wfq_splice_blocking +#define cds_wfq_first_blocking _cds_wfq_first_blocking +#define cds_wfq_next_blocking _cds_wfq_next_blocking + +/* Locking ensured by caller */ +#define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking +#define __cds_wfq_splice_blocking ___cds_wfq_splice_blocking +#define __cds_wfq_first_blocking ___cds_wfq_first_blocking +#define __cds_wfq_next_blocking ___cds_wfq_next_blocking #else /* !_LGPL_SOURCE */ -extern void cds_wfq_node_init(struct cds_wfq_node *node); -extern void cds_wfq_init(struct cds_wfq_queue *q); -extern void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node); -/* __cds_wfq_dequeue_blocking: caller ensures mutual exclusion between dequeues */ -extern struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); -extern struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); +extern void cds_wfq_node_init_1(struct cds_wfq_node *node); +extern void cds_wfq_init_1(struct cds_wfq_queue *q); +extern bool cds_wfq_empty_1(struct cds_wfq_queue *q); +extern void cds_wfq_enqueue_1(struct cds_wfq_queue *q, struct cds_wfq_node *node); + +/* Locking performed within cds_wfq calls. */ +extern struct cds_wfq_node *cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q); +extern void cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q); + +/* + * __cds_wfq_dequeue_blocking: caller ensures mutual exclusion of dequeue + * and splice operations. + */ +extern struct cds_wfq_node *__cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q); + +/* + * __cds_wfq_splice_blocking: caller ensures mutual exclusion of dequeue and + * splice operations on src_q. dest_q must be already initialized. + */ +extern void __cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q); + +/* + * __cds_wfq_first_blocking: mutual exclusion with "dequeue" and + * "splice" operations must be ensured by the caller. + */ +extern struct cds_wfq_node *__cds_wfq_first_blocking_1(struct cds_wfq_queue *q); + +/* + * __cds_wfq_next_blocking: mutual exclusion with "dequeue" and "splice" + * operations must be ensured by the caller. + */ +extern struct cds_wfq_node *__cds_wfq_next_blocking_1(struct cds_wfq_queue *q, + struct cds_wfq_node *node); + +#define cds_wfq_node_init cds_wfq_node_init_1 +#define cds_wfq_init cds_wfq_init_1 +#define cds_wfq_empty cds_wfq_empty_1 +#define cds_wfq_enqueue cds_wfq_enqueue_1 + +/* Locking performed within cds_wfq calls. */ +#define cds_wfq_dequeue_blocking cds_wfq_dequeue_blocking_1 +#define cds_wfq_splice_blocking cds_wfq_splice_blocking_1 +#define cds_wfq_first_blocking cds_wfq_first_blocking_1 +#define cds_wfq_next_blocking cds_wfq_next_blocking_1 + +/* Locking ensured by caller */ +#define __cds_wfq_dequeue_blocking __cds_wfq_dequeue_blocking_1 +#define __cds_wfq_splice_blocking __cds_wfq_splice_blocking_1 +#define __cds_wfq_first_blocking __cds_wfq_first_blocking_1 +#define __cds_wfq_next_blocking __cds_wfq_next_blocking_1 #endif /* !_LGPL_SOURCE */ +/* + * __cds_wfq_for_each_blocking: Iterate over all nodes in a queue, + * without dequeuing them. + * + * Mutual exclusion with "dequeue" and "splice" operations must be + * ensured by the caller. + */ +#define __cds_wfq_for_each_blocking(q, node) \ + for (node = __cds_wfq_first_blocking(q); \ + node != NULL; \ + node = __cds_wfq_next_blocking(q, node)) + +/* + * __cds_wfq_for_each_blocking_safe: Iterate over all nodes in a queue, + * without dequeuing them. Safe against deletion. + * + * Mutual exclusion with "dequeue" and "splice" operations must be + * ensured by the caller. + */ +#define __cds_wfq_for_each_blocking_safe(q, node, n) \ + for (node = __cds_wfq_first_blocking(q), \ + n = (node ? __cds_wfq_next_blocking(q, node) : NULL); \ + node != NULL; \ + node = n, n = (node ? __cds_wfq_next_blocking(q, node) : NULL)) + #ifdef __cplusplus } #endif diff --git a/wfqueue.c b/wfqueue.c index 3337171..b5fba7b 100644 --- a/wfqueue.c +++ b/wfqueue.c @@ -3,7 +3,8 @@ * * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue * - * Copyright 2010 - Mathieu Desnoyers + * Copyright 2010-2012 - Mathieu Desnoyers + * Copyright 2011-2012 - Lai Jiangshan * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -28,27 +29,55 @@ * library wrappers to be used by non-LGPL compatible source code. */ -void cds_wfq_node_init(struct cds_wfq_node *node) +void cds_wfq_node_init_1(struct cds_wfq_node *node) { _cds_wfq_node_init(node); } -void cds_wfq_init(struct cds_wfq_queue *q) +void cds_wfq_init_1(struct cds_wfq_queue *q) { _cds_wfq_init(q); } -void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node) +bool cds_wfq_empty_1(struct cds_wfq_queue *q) +{ + return _cds_wfq_empty(q); +} + +void cds_wfq_enqueue_1(struct cds_wfq_queue *q, struct cds_wfq_node *node) { _cds_wfq_enqueue(q, node); } -struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) +struct cds_wfq_node *cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q) +{ + return _cds_wfq_dequeue_blocking(q); +} + +void cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q) +{ + _cds_wfq_splice_blocking(dest_q, src_q); +} + +struct cds_wfq_node *__cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q) { return ___cds_wfq_dequeue_blocking(q); } -struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) +void __cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q) { - return _cds_wfq_dequeue_blocking(q); + ___cds_wfq_splice_blocking(dest_q, src_q); +} + +struct cds_wfq_node *__cds_wfq_first_blocking_1(struct cds_wfq_queue *q) +{ + return ___cds_wfq_first_blocking(q); +} + +struct cds_wfq_node *__cds_wfq_next_blocking_1(struct cds_wfq_queue *q, + struct cds_wfq_node *node) +{ + return ___cds_wfq_next_blocking(q, node); } -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From eag0628 at gmail.com Wed Aug 15 22:08:54 2012 From: eag0628 at gmail.com (Lai Jiangshan) Date: Thu, 16 Aug 2012 10:08:54 +0800 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost In-Reply-To: <20120815125209.GA11737@Krystal> References: <20120812145000.GA32747@Krystal> <20120813201219.GA27637@Krystal> <20120814222724.GA9751@Krystal> <20120815125209.GA11737@Krystal> Message-ID: >> >> Is it false sharing? >> Access to q->head.next and access to q->tail have the same performance >> because they are in the same cache line. > > Yes! you are right! And a quick benchmark confirms it: > > with head and tail on same cache line: > > SUMMARY /home/compudj/doc/userspace-rcu/tests/.libs/lt-test_urcu_wfq testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 100833595 nr_dequeues 88647134 successful enqueues 100833595 successful dequeues 88646898 end_dequeues 12186697 nr_ops 189480729 > > with a 256 bytes padding between head and tail, keeping the mutex on the > "head" cache line: > > SUMMARY /home/compudj/doc/userspace-rcu/tests/.libs/lt-test_urcu_wfq testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 228992829 nr_dequeues 228921791 successful enqueues 228992829 successful dequeues 228921367 end_dequeues 71462 nr_ops 457914620 > > enqueue: 127% speedup > dequeue: 158% speedup > > That is indeed a _really_ huge difference. However, to get this, we > would have to increase the size of struct cds_wfq_queue beyond its > current size, which would break API compatibility. Any idea on how to > best do this without causing incompatibility would be welcome. > choice 1) two set of APIs?(cache-line-opt and none-cache-line-opt), many users don't need the cache-line-opt. choice 2) Just break the compatibility for NONE-LGPL. I think NONE-LGPL-user of it is rare. And current version of urcu <1.0, I don't like too much burden when <1.0. thanks, Lai From eag0628 at gmail.com Wed Aug 15 22:13:39 2012 From: eag0628 at gmail.com (Lai Jiangshan) Date: Thu, 16 Aug 2012 10:13:39 +0800 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost In-Reply-To: References: <20120812145000.GA32747@Krystal> <20120813201219.GA27637@Krystal> <20120814222724.GA9751@Krystal> <20120815125209.GA11737@Krystal> Message-ID: We need the smallest patch at first, all other things left in disscussion. no free_each(), simple changes. thanks, Lai On Thu, Aug 16, 2012 at 10:08 AM, Lai Jiangshan wrote: >>> >>> Is it false sharing? >>> Access to q->head.next and access to q->tail have the same performance >>> because they are in the same cache line. >> >> Yes! you are right! And a quick benchmark confirms it: >> >> with head and tail on same cache line: >> >> SUMMARY /home/compudj/doc/userspace-rcu/tests/.libs/lt-test_urcu_wfq testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 100833595 nr_dequeues 88647134 successful enqueues 100833595 successful dequeues 88646898 end_dequeues 12186697 nr_ops 189480729 >> >> with a 256 bytes padding between head and tail, keeping the mutex on the >> "head" cache line: >> >> SUMMARY /home/compudj/doc/userspace-rcu/tests/.libs/lt-test_urcu_wfq testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 228992829 nr_dequeues 228921791 successful enqueues 228992829 successful dequeues 228921367 end_dequeues 71462 nr_ops 457914620 >> >> enqueue: 127% speedup >> dequeue: 158% speedup >> >> That is indeed a _really_ huge difference. However, to get this, we >> would have to increase the size of struct cds_wfq_queue beyond its >> current size, which would break API compatibility. Any idea on how to >> best do this without causing incompatibility would be welcome. >> > > choice 1) two set of APIs?(cache-line-opt and none-cache-line-opt), > many users don't need the cache-line-opt. > choice 2) Just break the compatibility for NONE-LGPL. I think > NONE-LGPL-user of it is rare. And current version of urcu <1.0, I > don't like too much burden when <1.0. > > > thanks, > Lai From srsamarthyam at gmail.com Thu Aug 16 08:37:07 2012 From: srsamarthyam at gmail.com (Santhosh Samarthyam) Date: Thu, 16 Aug 2012 18:07:07 +0530 Subject: [lttng-dev] LTTng on Android Message-ID: Dear All, I have been closely following the status of LTTng on Android based on the discussions in this mailing list. I would be great if someone could validate the following - No need for LTTng kernel patch if kernel version > 2.6.38 - No dependency on glibc after version 2.0 of lttng. So can I build LTTng on Android (Version details below) Android version: Icecream sandwich (4.0.1) Kernel version: 3.0.1 Can we build the following on an ARM based board running android? (After creating the android make files) LTTng-tools 2.0.4 LTTng-modules 2.0.4 LTTng-UST 2.0.4 liburcu 0.7.3 I am more interested in getting the LTTng-UST to work. Any pointers on how to achieve this would be really helpful. -- Cheers, Santos From mathieu.desnoyers at efficios.com Thu Aug 16 10:18:53 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 16 Aug 2012 10:18:53 -0400 Subject: [lttng-dev] [PATCH lttng-tools 1/2] Fix: Invalid free on session_name when destroying session In-Reply-To: <1345048771-18936-1-git-send-email-christian.babeux@efficios.com> References: <1345048771-18936-1-git-send-email-christian.babeux@efficios.com> Message-ID: <20120816141853.GA29703@Krystal> * Christian Babeux (christian.babeux at efficios.com) wrote: > The session_name should not be free(3) if the user has specified > a session name on the command line. Also, the caller is responsible to > free the allocated string when calling get_session_name(). > Handle both cases gracefully. > > Signed-off-by: Christian Babeux Acked-by: Mathieu Desnoyers > --- > src/bin/lttng/commands/destroy.c | 27 ++++++++++++++++----------- > 1 file changed, 16 insertions(+), 11 deletions(-) > > diff --git a/src/bin/lttng/commands/destroy.c b/src/bin/lttng/commands/destroy.c > index 5b69cb5..7b7ea0e 100644 > --- a/src/bin/lttng/commands/destroy.c > +++ b/src/bin/lttng/commands/destroy.c > @@ -28,6 +28,7 @@ > > #include > > +static char *opt_session_name; > static int opt_destroy_all; > > enum { > @@ -156,28 +157,32 @@ int cmd_destroy(int argc, const char **argv) > goto end; > } > > - session_name = (char *) poptGetArg(pc); > - > - /* > - * ignore session name in case all > - * sessions are to be destroyed > - */ > + /* Ignore session name in case all sessions are to be destroyed */ > if (opt_destroy_all) { > ret = destroy_all_sessions(); > goto end; > } > - if (session_name == NULL) { > - ret = get_default_session_name(&session_name); > - if (ret < 0 || session_name == NULL) { > + > + opt_session_name = (char *) poptGetArg(pc); > + > + if (opt_session_name == NULL) { > + /* No session name specified, lookup default */ > + session_name = get_session_name(); > + if (session_name == NULL) { > + ret = CMD_ERROR; > goto end; > } > + } else { > + session_name = opt_session_name; > } > + > ret = destroy_session(session_name); > > end: > - poptFreeContext(pc); > - if (session_name != NULL) { > + if (opt_session_name == NULL) { > free(session_name); > } > + > + poptFreeContext(pc); > return ret; > } > -- > 1.7.11.4 > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 16 10:18:59 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 16 Aug 2012 10:18:59 -0400 Subject: [lttng-dev] [PATCH lttng-tools 2/2] Cleanup: Remove unused get_default_session_name() function In-Reply-To: <1345048771-18936-2-git-send-email-christian.babeux@efficios.com> References: <1345048771-18936-1-git-send-email-christian.babeux@efficios.com> <1345048771-18936-2-git-send-email-christian.babeux@efficios.com> Message-ID: <20120816141859.GB29703@Krystal> * Christian Babeux (christian.babeux at efficios.com) wrote: > > Signed-off-by: Christian Babeux Acked-by: Mathieu Desnoyers > --- > src/bin/lttng/commands/destroy.c | 15 --------------- > 1 file changed, 15 deletions(-) > > diff --git a/src/bin/lttng/commands/destroy.c b/src/bin/lttng/commands/destroy.c > index 7b7ea0e..cdc2c53 100644 > --- a/src/bin/lttng/commands/destroy.c > +++ b/src/bin/lttng/commands/destroy.c > @@ -114,21 +114,6 @@ error: > } > > /* > - * get_default_session_name > - * > - * Returns the default sessions name, if any > - */ > -static int get_default_session_name(char **name) > -{ > - char *session_name = get_session_name(); > - if (session_name == NULL) { > - return CMD_ERROR; > - } > - *name = session_name; > - return CMD_SUCCESS; > -} > - > -/* > * The 'destroy ' first level command > */ > int cmd_destroy(int argc, const char **argv) > -- > 1.7.11.4 > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From francis.deslauriers at polymtl.ca Thu Aug 16 14:14:02 2012 From: francis.deslauriers at polymtl.ca (Francis Deslauriers) Date: Thu, 16 Aug 2012 14:14:02 -0400 Subject: [lttng-dev] [Babeltrace patch] Add BT_SEEK_LAST type to bt_iter_pos. Message-ID: <1345140842-21819-1-git-send-email-francis.deslauriers@polymtl.ca> Signed-off-by: Francis Deslauriers --- include/babeltrace/iterator.h | 1 + lib/iterator.c | 190 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 191 insertions(+) diff --git a/include/babeltrace/iterator.h b/include/babeltrace/iterator.h index aa6470e..c13055d 100644 --- a/include/babeltrace/iterator.h +++ b/include/babeltrace/iterator.h @@ -48,6 +48,7 @@ struct bt_iter_pos { BT_SEEK_CUR, BT_SEEK_BEGIN, BT_SEEK_END, + BT_SEEK_LAST, } type; union { uint64_t seek_time; diff --git a/lib/iterator.c b/lib/iterator.c index bcb77d8..7029a54 100644 --- a/lib/iterator.c +++ b/lib/iterator.c @@ -187,6 +187,177 @@ static int seek_ctf_trace_by_timestamp(struct ctf_trace *tin, return found ? 0 : EOF; } +static int seek_last_element_ctf_file_stream(struct trace_collection *tc, + struct ctf_file_stream **cfs, int max_packet, + int max_stream_id, int max_stream_class_id, + int max_trace_descriptor_id, uint64_t max_timestamp) +{ + int ret; + struct trace_descriptor *max_td_read; + struct ctf_trace *max_tin; + struct ctf_stream_declaration *max_stream_class; + struct ctf_stream_definition *max_stream; + struct ctf_stream_pos *max_stream_pos; + + max_td_read = g_ptr_array_index(tc->array, + max_trace_descriptor_id); + max_tin = container_of(max_td_read, + struct ctf_trace, parent); + max_stream_class = g_ptr_array_index(max_tin->streams, + max_stream_class_id); + max_stream = g_ptr_array_index(max_stream_class->streams, + max_stream_id); + *cfs = container_of(max_stream, + struct ctf_file_stream, parent); + max_stream_pos = &(*cfs)->pos; + /* we seek to the last packet of the stream.*/ + max_stream_pos->packet_seek(&max_stream_pos->parent, + max_packet, + SEEK_SET); + /* + * iterate over every event until we reach on an event that + * its timestamp correspond with the max saved previously. + */ + do { + ret = stream_read_event(*cfs); + } while ((*cfs)->parent.real_timestamp != max_timestamp && ret == 0); + + /* Return > 0 if error, 0 if ok or EOF */ + return ret; +} + +static int find_last_event(struct ctf_file_stream *cfs, + uint64_t *timestamp_end, + int *packet) +{ + int ret = 0, count = 0, event_read = 0, i; + uint64_t tmp = 0; + struct ctf_stream_pos *stream_pos; + stream_pos = &cfs->pos; + + /* + * we start by the last packet as the current one. + * If the current one is empty we go back one packet if possible. + * Check if we have iterated on all the packets. + * If we are not short on packets, we check what + * made us leave the reading event loop. + */ + for (i = stream_pos->packet_real_index->len - 1; i >= 0 && count == 0; i--) { + stream_pos->packet_seek(&stream_pos->parent, i, SEEK_SET); + count = 0; + /* read each event until we reach the end of the packet */ + do { + tmp = cfs->parent.real_timestamp; + ret = stream_read_event(cfs); + if (ret == 0) { + count++; + } + } while (ret == 0); + + /* Error */ + if (ret > 0) { + goto end; + } + event_read += count; + if (!count) { + break; + } + } + *packet = i; + *timestamp_end = tmp; + + /* Check if we read one or less element and return error if so */ + if (event_read <= 1) { + ret = 1; + goto end; + } + ret = 0; + +end: + return ret; +} + +static int find_max_timestamp_ctf_file_stream( + struct ctf_stream_declaration *stream_class, int *max_stream_id, + int *packet, uint64_t *max_timestamp, int *new_max) +{ + struct ctf_file_stream *cfs; + int max_packet = 0, ret = 0, error = 1, i; + uint64_t savedtime; + + for (i = 0; i < stream_class->streams->len; i++) { + struct ctf_stream_definition *stream; + stream = g_ptr_array_index( + stream_class->streams, i); + if (!stream) + continue; + cfs = container_of(stream, struct ctf_file_stream, parent); + ret = find_last_event(cfs, &savedtime, &max_packet); + /* Can return either EOF, 0, or error (> 0). */ + if (ret == 0 && savedtime >= *max_timestamp) { + *new_max = 1; + *max_stream_id = i; + *packet = max_packet; + *max_timestamp = savedtime; + } + /* if one of the return values is equal to 0 than + * error equals 0 + */ + error = error & ret; + } + return error; +} + +static int seek_last_ctf_file_stream(struct trace_collection *tc, + struct ctf_file_stream **cfs) +{ + int i, j, ret, new_max_found, max_packet, max_stream_id, + max_stream_class_id, max_trace_descriptor_id; + int found = 0; + uint64_t max_timestamp = 0; + + /* For each trace in the trace_collection*/ + for (i = 0; i < tc->array->len; i++) { + struct ctf_trace *tin; + struct trace_descriptor *td_read; + td_read = g_ptr_array_index(tc->array, i); + if (!td_read) + continue; + tin = container_of(td_read, struct ctf_trace, parent); + /* For each stream_class in the trace*/ + for (j = 0; j < tin->streams->len; j++) { + struct ctf_stream_declaration *stream_class; + + stream_class = g_ptr_array_index(tin->streams, j); + if (!stream_class) + continue; + /* For each file_stream in the stream_class */ + new_max_found = 0; + ret = find_max_timestamp_ctf_file_stream(stream_class, + &max_stream_id, &max_packet, + &max_timestamp, &new_max_found); + if (!ret && new_max_found) { + found = 1; + max_trace_descriptor_id = i; + max_stream_class_id = j; + } + } + } + /* + * Now we know in which trace, stream_class and stream is the + * last event of the trace_collection. + * We can seek to this targeted event. + */ + if (!found) { + ret = -1; + } else { + ret = seek_last_element_ctf_file_stream(tc, cfs, max_packet, + max_stream_id, max_stream_class_id, + max_trace_descriptor_id, max_timestamp); + } + return ret; +} + int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) { struct trace_collection *tc; @@ -326,6 +497,25 @@ int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) } } break; + case BT_SEEK_LAST: + { + struct ctf_file_stream *cfs; + tc = iter->ctx->tc; + + ret = seek_last_ctf_file_stream(tc, &cfs); + if (ret < 0) + goto error; + /* remove all stream from the heap*/ + heap_free(iter->stream_heap); + /* Create a new empty heap*/ + ret = heap_init(iter->stream_heap, 0, stream_compare); + if (ret < 0) + goto error; + /* Insert the stream that contains the last event. */ + heap_insert(iter->stream_heap, cfs); + + return 0; + } default: /* not implemented */ return -EINVAL; -- 1.7.9.5 From dgoulet at efficios.com Thu Aug 16 16:12:46 2012 From: dgoulet at efficios.com (David Goulet) Date: Thu, 16 Aug 2012 16:12:46 -0400 Subject: [lttng-dev] [PATCH lttng-tools 1/2] Fix: Invalid free on session_name when destroying session In-Reply-To: <1345048771-18936-1-git-send-email-christian.babeux@efficios.com> References: <1345048771-18936-1-git-send-email-christian.babeux@efficios.com> Message-ID: <502D543E.8080209@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Everything is merged and upstream! Thanks! Christian Babeux: > The session_name should not be free(3) if the user has specified a > session name on the command line. Also, the caller is responsible > to free the allocated string when calling get_session_name(). > Handle both cases gracefully. > > Signed-off-by: Christian Babeux > --- src/bin/lttng/commands/destroy.c | 27 > ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 > deletions(-) > > diff --git a/src/bin/lttng/commands/destroy.c > b/src/bin/lttng/commands/destroy.c index 5b69cb5..7b7ea0e 100644 > --- a/src/bin/lttng/commands/destroy.c +++ > b/src/bin/lttng/commands/destroy.c @@ -28,6 +28,7 @@ > > #include > > +static char *opt_session_name; static int opt_destroy_all; > > enum { @@ -156,28 +157,32 @@ int cmd_destroy(int argc, const char > **argv) goto end; } > > - session_name = (char *) poptGetArg(pc); - - /* - * ignore > session name in case all - * sessions are to be destroyed - */ + > /* Ignore session name in case all sessions are to be destroyed */ > if (opt_destroy_all) { ret = destroy_all_sessions(); goto end; } - > if (session_name == NULL) { - ret = > get_default_session_name(&session_name); - if (ret < 0 || > session_name == NULL) { + + opt_session_name = (char *) > poptGetArg(pc); + + if (opt_session_name == NULL) { + /* No > session name specified, lookup default */ + session_name = > get_session_name(); + if (session_name == NULL) { + ret = > CMD_ERROR; goto end; } + } else { + session_name = > opt_session_name; } + ret = destroy_session(session_name); > > end: - poptFreeContext(pc); - if (session_name != NULL) { + if > (opt_session_name == NULL) { free(session_name); } + + > poptFreeContext(pc); return ret; } -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQLVQ7AAoJEELoaioR9I02GWcIAJPV/bC2X+P3UaCRi/4Xf8yC KJ7vbGRBFPX9iFZP2bgwBfBKY9xLOqK3spOPotirKTUhuLaJooEqq42t4xsnw4nk GzQrQ8G9vzfr+ZLQvECMvYSyJF524RdK7xowiB8lmTKaYTcH3+HEFpmCWG7UzuWX Q/Rcs8VTxj34zgL1s1nczO/Vpn9TehXi6rEvy9iluOrkYecwql/zpI4zEd9lE5bb FTdbrAyFvDgFSJw3YudqalgC7IIMaaoLZj3MraLl0IRI31m2po1ofK3RtUfjWquT Igmm0Hr3by5fRwp8h6I8t7tC03WlalMX1DpC5x0LhgxwdMuzB6f5Fx4afNA811c= =EmBV -----END PGP SIGNATURE----- From mathieu.desnoyers at efficios.com Thu Aug 16 17:02:57 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 16 Aug 2012 17:02:57 -0400 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost In-Reply-To: References: <20120812145000.GA32747@Krystal> <20120813201219.GA27637@Krystal> <20120814222724.GA9751@Krystal> <20120815125209.GA11737@Krystal> Message-ID: <20120816210257.GA3331@Krystal> * Lai Jiangshan (eag0628 at gmail.com) wrote: > >> > >> Is it false sharing? > >> Access to q->head.next and access to q->tail have the same performance > >> because they are in the same cache line. > > > > Yes! you are right! And a quick benchmark confirms it: > > > > with head and tail on same cache line: > > > > SUMMARY /home/compudj/doc/userspace-rcu/tests/.libs/lt-test_urcu_wfq testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 100833595 nr_dequeues 88647134 successful enqueues 100833595 successful dequeues 88646898 end_dequeues 12186697 nr_ops 189480729 > > > > with a 256 bytes padding between head and tail, keeping the mutex on the > > "head" cache line: > > > > SUMMARY /home/compudj/doc/userspace-rcu/tests/.libs/lt-test_urcu_wfq testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 228992829 nr_dequeues 228921791 successful enqueues 228992829 successful dequeues 228921367 end_dequeues 71462 nr_ops 457914620 > > > > enqueue: 127% speedup > > dequeue: 158% speedup > > > > That is indeed a _really_ huge difference. However, to get this, we > > would have to increase the size of struct cds_wfq_queue beyond its > > current size, which would break API compatibility. Any idea on how to > > best do this without causing incompatibility would be welcome. > > > > choice 1) two set of APIs?(cache-line-opt and none-cache-line-opt), > many users don't need the cache-line-opt. > choice 2) Just break the compatibility for NONE-LGPL. I think > NONE-LGPL-user of it is rare. And current version of urcu <1.0, I > don't like too much burden when <1.0. Hi Lai, Keeping backward compatibility for a while, and gradually deprecating APIs, is very important for userspace RCU in my opinion: failure to provide this is the first thing that will discourage users from using the library. It is also the first thing that will turn away package maintainers, and distributions, from this project. I sent 2 patches that implement a compatibility ABI and the new ABI in a way that will let us continue development on wfqueue without having to have legacy code in our way: I plan to move all the legacy ABI to wfqueue0.c, and have a fresh start in terms of ABI for the new wfqueue. That way, we can eventually remove the wfqueue v0 ABI when we decide to bump the library version number (and break compatibility). Feedback is welcome, Thanks! Mathieu -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 16 17:11:55 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 16 Aug 2012 17:11:55 -0400 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost In-Reply-To: References: <20120812145000.GA32747@Krystal> <20120813201219.GA27637@Krystal> <20120814222724.GA9751@Krystal> <20120815125209.GA11737@Krystal> Message-ID: <20120816211155.GA3487@Krystal> * Lai Jiangshan (eag0628 at gmail.com) wrote: > We need the smallest patch at first, all other things left in disscussion. > no free_each(), simple changes. I guess you mean for_each(). If we limit ourself to the versions where the user is doing the locking, I don't think there were any issues left. In the version I provided (in the last series of 2 patches), I took care of all issues that were raised in our email discussion. I prefer to introduce these new API members all in one go, mainly because I really don't want to add new API members (exposed through the public API) and then remove them afterward when we notice that they expose too many details. I think the current splice, dequeue, next, and first API members allow any user to do the kind of use-case that call_rcu is doing: this lets us achieve your original goal of not duplicating the code everywhere. If you still notice issues with __cds_wfq_for_each_blocking() and __cds_wfq_for_each_blocking_safe() in the last patch, please let me know, Thanks ! Mathieu > > thanks, > Lai > > On Thu, Aug 16, 2012 at 10:08 AM, Lai Jiangshan wrote: > >>> > >>> Is it false sharing? > >>> Access to q->head.next and access to q->tail have the same performance > >>> because they are in the same cache line. > >> > >> Yes! you are right! And a quick benchmark confirms it: > >> > >> with head and tail on same cache line: > >> > >> SUMMARY /home/compudj/doc/userspace-rcu/tests/.libs/lt-test_urcu_wfq testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 100833595 nr_dequeues 88647134 successful enqueues 100833595 successful dequeues 88646898 end_dequeues 12186697 nr_ops 189480729 > >> > >> with a 256 bytes padding between head and tail, keeping the mutex on the > >> "head" cache line: > >> > >> SUMMARY /home/compudj/doc/userspace-rcu/tests/.libs/lt-test_urcu_wfq testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 228992829 nr_dequeues 228921791 successful enqueues 228992829 successful dequeues 228921367 end_dequeues 71462 nr_ops 457914620 > >> > >> enqueue: 127% speedup > >> dequeue: 158% speedup > >> > >> That is indeed a _really_ huge difference. However, to get this, we > >> would have to increase the size of struct cds_wfq_queue beyond its > >> current size, which would break API compatibility. Any idea on how to > >> best do this without causing incompatibility would be welcome. > >> > > > > choice 1) two set of APIs?(cache-line-opt and none-cache-line-opt), > > many users don't need the cache-line-opt. > > choice 2) Just break the compatibility for NONE-LGPL. I think > > NONE-LGPL-user of it is rare. And current version of urcu <1.0, I > > don't like too much burden when <1.0. > > > > > > thanks, > > Lai -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From jdesfossez at efficios.com Thu Aug 16 17:48:07 2012 From: jdesfossez at efficios.com (Julien Desfossez) Date: Thu, 16 Aug 2012 17:48:07 -0400 Subject: [lttng-dev] [BABELTRACE PATCH v2] Add BT_SEEK_LAST type to bt_iter_pos In-Reply-To: <1345140842-21819-1-git-send-email-francis.deslauriers@polymtl.ca> References: <1345140842-21819-1-git-send-email-francis.deslauriers@polymtl.ca> Message-ID: <1345153687-15832-1-git-send-email-jdesfossez@efficios.com> From: Francis Deslauriers Signed-off-by: Francis Deslauriers Signed-off-by: Julien Desfossez --- include/babeltrace/iterator.h | 1 + lib/iterator.c | 207 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 208 insertions(+) diff --git a/include/babeltrace/iterator.h b/include/babeltrace/iterator.h index aa6470e..c8edbd1 100644 --- a/include/babeltrace/iterator.h +++ b/include/babeltrace/iterator.h @@ -48,6 +48,7 @@ struct bt_iter_pos { BT_SEEK_CUR, BT_SEEK_BEGIN, BT_SEEK_END, + BT_SEEK_LAST } type; union { uint64_t seek_time; diff --git a/lib/iterator.c b/lib/iterator.c index f5f413e..16b54bc 100644 --- a/lib/iterator.c +++ b/lib/iterator.c @@ -187,6 +187,193 @@ static int seek_ctf_trace_by_timestamp(struct ctf_trace *tin, return found ? 0 : EOF; } +int seek_last_element_ctf_file_stream(struct trace_collection *tc, + struct ctf_file_stream **cfs, int max_packet, + int max_stream_id, int max_stream_class_id, + int max_trace_descriptor_id, uint64_t max_timestamp) +{ + struct trace_descriptor *max_td_read; + struct ctf_trace *max_tin; + struct ctf_stream_declaration *max_stream_class; + struct ctf_stream_definition *max_stream; + struct ctf_stream_pos *max_stream_pos; + int ret; + + if (!tc) + return -EINVAL; + + /* get the targeted trace descriptor */ + max_td_read = g_ptr_array_index(tc->array, max_trace_descriptor_id); + max_tin = container_of(max_td_read, struct ctf_trace, parent); + + /* get targeted stream class */ + max_stream_class = g_ptr_array_index(max_tin->streams, + max_stream_class_id); + /* get targeted stream */ + max_stream = g_ptr_array_index(max_stream_class->streams, + max_stream_id); + *cfs = container_of(max_stream, struct ctf_file_stream, parent); + max_stream_pos = &(*cfs)->pos; + + /* we seek to the last packet of the stream. */ + max_stream_pos->packet_seek(&max_stream_pos->parent, + max_packet, SEEK_SET); + /* + * iterate over all the event until we reach on that + * his timestamp correspond with the max saved previously. + */ + do { + ret = stream_read_event(*cfs); + } while ((*cfs)->parent.real_timestamp != max_timestamp && ret == 0); + + /* insert the stream in the heap */ + return ret; +} + +int find_last_event(struct ctf_file_stream *cfs, uint64_t *timestamp_end, + int *packet) +{ + struct ctf_stream_pos *stream_pos; + uint64_t tmp = 0; + int ret = 0; + int count = 0; + int event_read = 0; + int i; + + if (!cfs) + return -EINVAL; + + stream_pos = &cfs->pos; + /* + * we start by the last packet as the current one. + * If the current one is empty we go back one packet if possible. + */ + for (i = stream_pos->packet_real_index->len - 1; i >= 0; i--) { + stream_pos->packet_seek(&stream_pos->parent, i, SEEK_SET); + count = 0; + /* read each event until we reach the end of the packet */ + do { + tmp = cfs->parent.real_timestamp; + ret = stream_read_event(cfs); + count++; + } while (ret == 0); + + if (count > 1) + event_read = 1; + /* + * Check if we have read at least one event before + * reaching the end of file. + */ + if (ret == EOF && count > 1) { + *timestamp_end = tmp; + *packet = i; + break; + } + /* Error */ + else if (ret > 0) { + return ret; + } + } + + /* Check if we read at least one event on the stream */ + if (!event_read) { + return 1; + } + return 0; +} + +int find_max_timestamp_ctf_file_stream( + struct ctf_stream_declaration *stream_class, int *max_stream_id, + int *packet, uint64_t *max_timestamp, int *new_max) +{ + struct ctf_file_stream *cfs; + uint64_t savedtime; + int i; + int max_packet = 0; + int ret = 0; + int error = 1; + + for (i = 0; i < stream_class->streams->len; i++) { + struct ctf_stream_definition *stream; + + stream = g_ptr_array_index(stream_class->streams, i); + if (!stream) + continue; + cfs = container_of(stream, struct ctf_file_stream, parent); + ret = find_last_event(cfs, &savedtime, &max_packet); + /* Can return either EOF, 0, or error (> 0). */ + if (ret == 0 && savedtime >= *max_timestamp) { + *new_max = 1; + *max_stream_id = i; + *packet = max_packet; + *max_timestamp = savedtime; + } + error = error & ret; + } + + return error; +} + +int seek_last_ctf_file_stream(struct trace_collection *tc, + struct ctf_file_stream **cfs) +{ + uint64_t max_timestamp = 0; + int i, j; + int ret; + int found = 0; + int new_max_found; + int max_packet; + int max_stream_id; + int max_stream_class_id; + int max_trace_descriptor_id; + + if (!tc) + return -EINVAL; + + /* For each trace in the trace_collection */ + for (i = 0; i < tc->array->len; i++) { + struct ctf_trace *tin; + struct trace_descriptor *td_read; + + td_read = g_ptr_array_index(tc->array, i); + if (!td_read) + continue; + tin = container_of(td_read, struct ctf_trace, parent); + /* For each stream_class in the trace */ + for (j = 0; j < tin->streams->len; j++) { + struct ctf_stream_declaration *stream_class; + + stream_class = g_ptr_array_index(tin->streams, j); + if (!stream_class) + continue; + /* For each file_stream in the stream_class */ + new_max_found = 0; + ret = find_max_timestamp_ctf_file_stream(stream_class, + &max_stream_id, &max_packet, + &max_timestamp, &new_max_found); + if (!ret && new_max_found) { + found = 1; + max_trace_descriptor_id = i; + max_stream_class_id = j; + } + } + } + /* + * Now we know in which trace, stream_class and stream is the + * last event of the trace_collection. + * We can seek to this targeted event. + */ + if (!found) { + ret = -1; + } else { + ret = seek_last_element_ctf_file_stream(tc, cfs, max_packet, + max_stream_id, max_stream_class_id, + max_trace_descriptor_id, max_timestamp); + } + + return ret; +} + int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) { struct trace_collection *tc; @@ -329,6 +516,26 @@ int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) } } break; + case BT_SEEK_LAST: + { + struct ctf_file_stream *cfs; + + tc = iter->ctx->tc; + ret = seek_last_ctf_file_stream(tc, &cfs); + if (ret < 0) + goto error; + + /* remove all stream from the heap*/ + heap_free(iter->stream_heap); + /* Create a new empty heap*/ + ret = heap_init(iter->stream_heap, 0, stream_compare); + if (ret < 0) + goto error; + /*Insert the stream that contains the last event.*/ + heap_insert(iter->stream_heap, cfs); + + return 0; + } default: /* not implemented */ return -EINVAL; -- 1.7.10.4 From mathieu.desnoyers at efficios.com Thu Aug 16 17:53:29 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 16 Aug 2012 17:53:29 -0400 Subject: [lttng-dev] [BABELTRACE PATCH v2] Add BT_SEEK_LAST type to bt_iter_pos In-Reply-To: <1345153687-15832-1-git-send-email-jdesfossez@efficios.com> References: <1345140842-21819-1-git-send-email-francis.deslauriers@polymtl.ca> <1345153687-15832-1-git-send-email-jdesfossez@efficios.com> Message-ID: <20120816215328.GA4864@Krystal> * Julien Desfossez (jdesfossez at efficios.com) wrote: > From: Francis Deslauriers > something tells me it does not start from the last patch sent by Francis, am I correct ? A mixup maybe ? > Signed-off-by: Francis Deslauriers > Signed-off-by: Julien Desfossez > --- > include/babeltrace/iterator.h | 1 + > lib/iterator.c | 207 +++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 208 insertions(+) > > diff --git a/include/babeltrace/iterator.h b/include/babeltrace/iterator.h > index aa6470e..c8edbd1 100644 > --- a/include/babeltrace/iterator.h > +++ b/include/babeltrace/iterator.h > @@ -48,6 +48,7 @@ struct bt_iter_pos { > BT_SEEK_CUR, > BT_SEEK_BEGIN, > BT_SEEK_END, > + BT_SEEK_LAST > } type; > union { > uint64_t seek_time; > diff --git a/lib/iterator.c b/lib/iterator.c > index f5f413e..16b54bc 100644 > --- a/lib/iterator.c > +++ b/lib/iterator.c > @@ -187,6 +187,193 @@ static int seek_ctf_trace_by_timestamp(struct ctf_trace *tin, > return found ? 0 : EOF; > } > > +int seek_last_element_ctf_file_stream(struct trace_collection *tc, > + struct ctf_file_stream **cfs, int max_packet, > + int max_stream_id, int max_stream_class_id, > + int max_trace_descriptor_id, uint64_t max_timestamp) > +{ > + struct trace_descriptor *max_td_read; > + struct ctf_trace *max_tin; > + struct ctf_stream_declaration *max_stream_class; > + struct ctf_stream_definition *max_stream; > + struct ctf_stream_pos *max_stream_pos; > + int ret; > + > + if (!tc) > + return -EINVAL; > + > + /* get the targeted trace descriptor */ > + max_td_read = g_ptr_array_index(tc->array, max_trace_descriptor_id); > + max_tin = container_of(max_td_read, struct ctf_trace, parent); > + > + /* get targeted stream class */ > + max_stream_class = g_ptr_array_index(max_tin->streams, > + max_stream_class_id); > + /* get targeted stream */ > + max_stream = g_ptr_array_index(max_stream_class->streams, > + max_stream_id); > + *cfs = container_of(max_stream, struct ctf_file_stream, parent); > + max_stream_pos = &(*cfs)->pos; > + > + /* we seek to the last packet of the stream. */ > + max_stream_pos->packet_seek(&max_stream_pos->parent, > + max_packet, SEEK_SET); > + /* > + * iterate over all the event until we reach on that > + * his timestamp correspond with the max saved previously. > + */ > + do { > + ret = stream_read_event(*cfs); > + } while ((*cfs)->parent.real_timestamp != max_timestamp && ret == 0); > + > + /* insert the stream in the heap */ > + return ret; > +} > + > +int find_last_event(struct ctf_file_stream *cfs, uint64_t *timestamp_end, > + int *packet) > +{ > + struct ctf_stream_pos *stream_pos; > + uint64_t tmp = 0; > + int ret = 0; > + int count = 0; > + int event_read = 0; > + int i; > + > + if (!cfs) > + return -EINVAL; > + > + stream_pos = &cfs->pos; > + /* > + * we start by the last packet as the current one. > + * If the current one is empty we go back one packet if possible. > + */ > + for (i = stream_pos->packet_real_index->len - 1; i >= 0; i--) { > + stream_pos->packet_seek(&stream_pos->parent, i, SEEK_SET); > + count = 0; > + /* read each event until we reach the end of the packet */ > + do { > + tmp = cfs->parent.real_timestamp; > + ret = stream_read_event(cfs); > + count++; > + } while (ret == 0); > + > + if (count > 1) > + event_read = 1; > + /* > + * Check if we have read at least one event before > + * reaching the end of file. > + */ > + if (ret == EOF && count > 1) { > + *timestamp_end = tmp; > + *packet = i; > + break; > + } > + /* Error */ > + else if (ret > 0) { > + return ret; > + } > + } > + > + /* Check if we read at least one event on the stream */ > + if (!event_read) { > + return 1; > + } > + return 0; > +} > + > +int find_max_timestamp_ctf_file_stream( > + struct ctf_stream_declaration *stream_class, int *max_stream_id, > + int *packet, uint64_t *max_timestamp, int *new_max) > +{ > + struct ctf_file_stream *cfs; > + uint64_t savedtime; > + int i; > + int max_packet = 0; > + int ret = 0; > + int error = 1; > + > + for (i = 0; i < stream_class->streams->len; i++) { > + struct ctf_stream_definition *stream; > + > + stream = g_ptr_array_index(stream_class->streams, i); > + if (!stream) > + continue; > + cfs = container_of(stream, struct ctf_file_stream, parent); > + ret = find_last_event(cfs, &savedtime, &max_packet); > + /* Can return either EOF, 0, or error (> 0). */ > + if (ret == 0 && savedtime >= *max_timestamp) { > + *new_max = 1; > + *max_stream_id = i; > + *packet = max_packet; > + *max_timestamp = savedtime; > + } > + error = error & ret; > + } > + > + return error; > +} > + > +int seek_last_ctf_file_stream(struct trace_collection *tc, > + struct ctf_file_stream **cfs) > +{ > + uint64_t max_timestamp = 0; > + int i, j; > + int ret; > + int found = 0; > + int new_max_found; > + int max_packet; > + int max_stream_id; > + int max_stream_class_id; > + int max_trace_descriptor_id; > + > + if (!tc) > + return -EINVAL; > + > + /* For each trace in the trace_collection */ > + for (i = 0; i < tc->array->len; i++) { > + struct ctf_trace *tin; > + struct trace_descriptor *td_read; > + > + td_read = g_ptr_array_index(tc->array, i); > + if (!td_read) > + continue; > + tin = container_of(td_read, struct ctf_trace, parent); > + /* For each stream_class in the trace */ > + for (j = 0; j < tin->streams->len; j++) { > + struct ctf_stream_declaration *stream_class; > + > + stream_class = g_ptr_array_index(tin->streams, j); > + if (!stream_class) > + continue; > + /* For each file_stream in the stream_class */ > + new_max_found = 0; > + ret = find_max_timestamp_ctf_file_stream(stream_class, > + &max_stream_id, &max_packet, > + &max_timestamp, &new_max_found); > + if (!ret && new_max_found) { > + found = 1; > + max_trace_descriptor_id = i; > + max_stream_class_id = j; > + } > + } > + } > + /* > + * Now we know in which trace, stream_class and stream is the > + * last event of the trace_collection. > + * We can seek to this targeted event. > + */ > + if (!found) { > + ret = -1; > + } else { > + ret = seek_last_element_ctf_file_stream(tc, cfs, max_packet, > + max_stream_id, max_stream_class_id, > + max_trace_descriptor_id, max_timestamp); > + } > + > + return ret; > +} > + > int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) > { > struct trace_collection *tc; > @@ -329,6 +516,26 @@ int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) > } > } > break; > + case BT_SEEK_LAST: > + { > + struct ctf_file_stream *cfs; > + > + tc = iter->ctx->tc; > + ret = seek_last_ctf_file_stream(tc, &cfs); > + if (ret < 0) > + goto error; > + > + /* remove all stream from the heap*/ > + heap_free(iter->stream_heap); > + /* Create a new empty heap*/ > + ret = heap_init(iter->stream_heap, 0, stream_compare); > + if (ret < 0) > + goto error; > + /*Insert the stream that contains the last event.*/ > + heap_insert(iter->stream_heap, cfs); > + > + return 0; > + } > default: > /* not implemented */ > return -EINVAL; > -- > 1.7.10.4 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From jdesfossez at efficios.com Thu Aug 16 18:25:11 2012 From: jdesfossez at efficios.com (Julien Desfossez) Date: Thu, 16 Aug 2012 18:25:11 -0400 Subject: [lttng-dev] [BABELTRACE PATCH v2] Add BT_SEEK_LAST type to bt_iter_pos In-Reply-To: <20120816215328.GA4864@Krystal> References: <1345140842-21819-1-git-send-email-francis.deslauriers@polymtl.ca> <1345153687-15832-1-git-send-email-jdesfossez@efficios.com> <20120816215328.GA4864@Krystal> Message-ID: <502D7347.4040000@efficios.com> On 16/08/12 05:53 PM, Mathieu Desnoyers wrote: > * Julien Desfossez (jdesfossez at efficios.com) wrote: >> From: Francis Deslauriers >> > > something tells me it does not start from the last patch sent by > Francis, am I correct ? A mixup maybe ? No it's just a v2. I fixed some formatting issues, the timestamp manipulation (was not adapted to the new real/cycles timestamps) and an bug if the first stream read contained the max timestamp. From jdesfossez at efficios.com Thu Aug 16 19:31:26 2012 From: jdesfossez at efficios.com (Julien Desfossez) Date: Thu, 16 Aug 2012 19:31:26 -0400 Subject: [lttng-dev] [BABELTRACE PATCH v3] Add BT_SEEK_LAST type to bt_iter_pos In-Reply-To: <1345153687-15832-1-git-send-email-jdesfossez@efficios.com> References: <1345153687-15832-1-git-send-email-jdesfossez@efficios.com> Message-ID: <1345159886-21597-1-git-send-email-jdesfossez@efficios.com> From: Francis Deslauriers Signed-off-by: Francis Deslauriers Signed-off-by: Julien Desfossez --- include/babeltrace/iterator.h | 1 + lib/iterator.c | 192 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 193 insertions(+) diff --git a/include/babeltrace/iterator.h b/include/babeltrace/iterator.h index aa6470e..c13055d 100644 --- a/include/babeltrace/iterator.h +++ b/include/babeltrace/iterator.h @@ -48,6 +48,7 @@ struct bt_iter_pos { BT_SEEK_CUR, BT_SEEK_BEGIN, BT_SEEK_END, + BT_SEEK_LAST, } type; union { uint64_t seek_time; diff --git a/lib/iterator.c b/lib/iterator.c index f5f413e..b7ab2b5 100644 --- a/lib/iterator.c +++ b/lib/iterator.c @@ -187,6 +187,179 @@ static int seek_ctf_trace_by_timestamp(struct ctf_trace *tin, return found ? 0 : EOF; } +static int seek_last_element_ctf_file_stream(struct trace_collection *tc, + struct ctf_file_stream **cfs, int max_packet, + int max_stream_id, int max_stream_class_id, + int max_trace_descriptor_id, uint64_t max_timestamp) +{ + int ret; + struct trace_descriptor *max_td_read; + struct ctf_trace *max_tin; + struct ctf_stream_declaration *max_stream_class; + struct ctf_stream_definition *max_stream; + struct ctf_stream_pos *max_stream_pos; + + max_td_read = g_ptr_array_index(tc->array, + max_trace_descriptor_id); + max_tin = container_of(max_td_read, + struct ctf_trace, parent); + max_stream_class = g_ptr_array_index(max_tin->streams, + max_stream_class_id); + max_stream = g_ptr_array_index(max_stream_class->streams, + max_stream_id); + *cfs = container_of(max_stream, + struct ctf_file_stream, parent); + max_stream_pos = &(*cfs)->pos; + /* we seek to the last packet of the stream. */ + max_stream_pos->packet_seek(&max_stream_pos->parent, + max_packet, SEEK_SET); + /* + * iterate over every event until we reach on an event that + * its timestamp correspond with the max saved previously. + */ + do { + ret = stream_read_event(*cfs); + } while ((*cfs)->parent.real_timestamp != max_timestamp && ret == 0); + + /* Return > 0 if error, 0 if ok or EOF */ + return ret; +} + +static int find_last_event(struct ctf_file_stream *cfs, + uint64_t *timestamp_end, int *packet) +{ + int ret = 0, count = 0, event_read = 0, i; + uint64_t tmp = 0; + struct ctf_stream_pos *stream_pos; + + stream_pos = &cfs->pos; + /* + * we start by the last packet as the current one. + * If the current one is empty we go back one packet if possible. + * Check if we have iterated on all the packets. + * If we are not short on packets, we check what + * made us leave the reading event loop. + */ + for (i = stream_pos->packet_real_index->len - 1; i >= 0 && count == 0; i--) { + stream_pos->packet_seek(&stream_pos->parent, i, SEEK_SET); + count = 0; + /* read each event until we reach the end of the packet */ + do { + tmp = cfs->parent.real_timestamp; + ret = stream_read_event(cfs); + if (ret == 0) { + count++; + } + } while (ret == 0); + + /* Error */ + if (ret > 0) { + goto end; + } + event_read += count; + if (!count) { + break; + } + } + *packet = i; + *timestamp_end = tmp; + + /* Check if we read one or less element and return error if so */ + if (event_read <= 1) { + ret = 1; + goto end; + } + ret = 0; + +end: + return ret; +} + +static int find_max_timestamp_ctf_file_stream( + struct ctf_stream_declaration *stream_class, int *max_stream_id, + int *packet, uint64_t *max_timestamp, int *new_max) +{ + struct ctf_file_stream *cfs; + int max_packet = 0, ret = 0, error = 1, i; + uint64_t savedtime; + + for (i = 0; i < stream_class->streams->len; i++) { + struct ctf_stream_definition *stream; + + stream = g_ptr_array_index(stream_class->streams, i); + if (!stream) + continue; + cfs = container_of(stream, struct ctf_file_stream, parent); + ret = find_last_event(cfs, &savedtime, &max_packet); + /* Can return either EOF, 0, or error (> 0). */ + if (ret == 0 && savedtime >= *max_timestamp) { + *new_max = 1; + *max_stream_id = i; + *packet = max_packet; + *max_timestamp = savedtime; + } + /* + * if one of the return values is equal to 0 than + * error equals 0 + */ + error = error & ret; + } + return error; +} + +static int seek_last_ctf_file_stream(struct trace_collection *tc, + struct ctf_file_stream **cfs) +{ + int i, j, ret, new_max_found, max_packet, max_stream_id, + max_stream_class_id, max_trace_descriptor_id; + int found = 0; + uint64_t max_timestamp = 0; + + if (!tc) + return -EINVAL; + + /* For each trace in the trace_collection */ + for (i = 0; i < tc->array->len; i++) { + struct ctf_trace *tin; + struct trace_descriptor *td_read; + td_read = g_ptr_array_index(tc->array, i); + if (!td_read) + continue; + tin = container_of(td_read, struct ctf_trace, parent); + /* For each stream_class in the trace */ + for (j = 0; j < tin->streams->len; j++) { + struct ctf_stream_declaration *stream_class; + + stream_class = g_ptr_array_index(tin->streams, j); + if (!stream_class) + continue; + /* For each file_stream in the stream_class */ + new_max_found = 0; + ret = find_max_timestamp_ctf_file_stream(stream_class, + &max_stream_id, &max_packet, + &max_timestamp, &new_max_found); + if (!ret && new_max_found) { + found = 1; + max_trace_descriptor_id = i; + max_stream_class_id = j; + } + } + } + /* + * Now we know in which trace, stream_class and stream is the + * last event of the trace_collection. + * We can seek to this targeted event. + */ + if (!found) { + ret = -1; + } else { + ret = seek_last_element_ctf_file_stream(tc, cfs, max_packet, + max_stream_id, max_stream_class_id, + max_trace_descriptor_id, max_timestamp); + } + return ret; +} + int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) { struct trace_collection *tc; @@ -329,6 +502,25 @@ int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) } } break; + case BT_SEEK_LAST: + { + struct ctf_file_stream *cfs; + + tc = iter->ctx->tc; + ret = seek_last_ctf_file_stream(tc, &cfs); + if (ret < 0) + goto error; + /* remove all stream from the heap */ + heap_free(iter->stream_heap); + /* Create a new empty heap */ + ret = heap_init(iter->stream_heap, 0, stream_compare); + if (ret < 0) + goto error; + /* Insert the stream that contains the last event */ + heap_insert(iter->stream_heap, cfs); + + return 0; + } default: /* not implemented */ return -EINVAL; -- 1.7.10.4 From mathieu.desnoyers at efficios.com Thu Aug 16 21:30:36 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 16 Aug 2012 21:30:36 -0400 Subject: [lttng-dev] [BABELTRACE PATCH v4] Add BT_SEEK_LAST type to bt_iter_pos Message-ID: <20120817013036.GB6294@Krystal> Inspired from patch by Francis Deslauriers and Julien Desfossez. Comments are welcome. I left a FIXME as question. Not tested. Not-Signed-off-by: Mathieu Desnoyers --- diff --git a/include/babeltrace/iterator.h b/include/babeltrace/iterator.h index aa6470e..98504a5 100644 --- a/include/babeltrace/iterator.h +++ b/include/babeltrace/iterator.h @@ -40,6 +40,15 @@ struct bt_saved_pos; * is expressed in nanoseconds * - restore is a position saved with bt_iter_get_pos, it is used with * BT_SEEK_RESTORE. + * + * Note about BT_SEEK_LAST: if many events happen to be at the last + * timestamp, it is implementation-defined which event will be the last, + * and the order of events with the same timestamp may not be the same + * as normal iteration on the trace. Therefore, it is recommended to + * only use BT_SEEK_LAST to get the timestamp of the last event(s) in + * the trace. + * FIXME: should we name it BT_SEEK_LAST_TIMESTAMP instead, if we ever + * want to keep BT_SEEK_LAST for the last event in iteration order ? */ struct bt_iter_pos { enum { @@ -48,6 +57,7 @@ struct bt_iter_pos { BT_SEEK_CUR, BT_SEEK_BEGIN, BT_SEEK_END, + BT_SEEK_LAST, } type; union { uint64_t seek_time; diff --git a/lib/iterator.c b/lib/iterator.c index 2dbd77c..b305f16 100644 --- a/lib/iterator.c +++ b/lib/iterator.c @@ -103,6 +103,10 @@ void bt_iter_free_pos(struct bt_iter_pos *iter_pos) * * Return 0 if the seek succeded, EOF if we didn't find any packet * containing the timestamp, or a positive integer for error. + * + * TODO: this should be turned into a binary search! It is currently + * doing a linear search in the packets. This is a O(n) operation on a + * very frequent code path. */ static int seek_file_stream_by_timestamp(struct ctf_file_stream *cfs, uint64_t timestamp) @@ -187,6 +191,148 @@ static int seek_ctf_trace_by_timestamp(struct ctf_trace *tin, return found ? 0 : EOF; } +/* + * Find timestamp of last event in the stream. + * + * Return value: 0 if OK, positive error value on error, EOF if no + * events were found. + */ +static int find_max_timestamp_ctf_file_stream(struct ctf_file_stream *cfs, + uint64_t *timestamp_end) +{ + int ret, count = 0, i; + uint64_t timestamp = 0; + struct ctf_stream_pos *stream_pos; + + stream_pos = &cfs->pos; + /* + * We start by the last packet, and iterate backwards until we + * either find at least one event, or we reach the first packet + * (some packets can be empty). + */ + for (i = stream_pos->packet_real_index->len - 1; i >= 0; i--) { + stream_pos->packet_seek(&stream_pos->parent, i, SEEK_SET); + count = 0; + /* read each event until we reach the end of the stream */ + do { + ret = stream_read_event(cfs); + if (ret == 0) { + count++; + timestamp = cfs->parent.real_timestamp; + } + } while (ret == 0); + + /* Error */ + if (ret > 0) + goto end; + assert(ret == EOF); + if (count) + break; + } + + if (count) { + *timestamp_end = timestamp; + ret = 0; + } else { + /* Return EOF if no events were found */ + ret = EOF; + } +end: + return ret; +} + +/* + * Find the stream within a stream class that contains the event with + * the largest timestamp, and save that timestamp. + * + * Return 0 if OK, EOF if no events were found in the streams, or + * positive value on error. + */ +static int find_max_timestamp_ctf_stream_class( + struct ctf_stream_declaration *stream_class, + struct ctf_file_stream **cfsp, + uint64_t *max_timestamp) +{ + int ret = EOF, i; + + for (i = 0; i < stream_class->streams->len; i++) { + struct ctf_stream_definition *stream; + struct ctf_file_stream *cfs; + uint64_t current_max_ts = 0; + + stream = g_ptr_array_index(stream_class->streams, i); + if (!stream) + continue; + cfs = container_of(stream, struct ctf_file_stream, parent); + ret = find_max_timestamp_ctf_file_stream(cfs, ¤t_max_ts); + if (ret == EOF) + continue; + if (ret != 0) + break; + if (current_max_ts >= *max_timestamp) { + *max_timestamp = current_max_ts; + *cfsp = cfs; + } + } + assert(ret >= 0 || ret == EOF); + return ret; +} + +/* + * seek_last_ctf_trace_collection: seek trace collection to last event. + * + * Return 0 if OK, EOF if no events were found, or positive error value + * on error. + */ +static int seek_last_ctf_trace_collection(struct trace_collection *tc, + struct ctf_file_stream **cfsp) +{ + int i, j, ret; + int found = 0; + uint64_t max_timestamp = 0; + + if (!tc) + return 1; + + /* For each trace in the trace_collection */ + for (i = 0; i < tc->array->len; i++) { + struct ctf_trace *tin; + struct trace_descriptor *td_read; + + td_read = g_ptr_array_index(tc->array, i); + if (!td_read) + continue; + tin = container_of(td_read, struct ctf_trace, parent); + /* For each stream_class in the trace */ + for (j = 0; j < tin->streams->len; j++) { + struct ctf_stream_declaration *stream_class; + + stream_class = g_ptr_array_index(tin->streams, j); + if (!stream_class) + continue; + ret = find_max_timestamp_ctf_stream_class(stream_class, + cfsp, &max_timestamp); + if (ret > 0) + goto end; + if (ret == 0) + found = 1; + assert(ret == EOF || ret == 0); + } + } + /* + * Now we know in which file stream the last event is located, + * and we know its timestamp. + */ + if (!found) { + ret = EOF; + } else { + ret = seek_file_stream_by_timestamp(*cfsp, max_timestamp); + assert(ret == 0); + } +end: + return ret; +} + int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) { struct trace_collection *tc; @@ -332,6 +478,25 @@ int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) } } break; + case BT_SEEK_LAST: + { + struct ctf_file_stream *cfs; + + tc = iter->ctx->tc; + ret = seek_last_ctf_trace_collection(tc, &cfs); + if (ret != 0) + goto error; + /* remove all streams from the heap */ + heap_free(iter->stream_heap); + /* Create a new empty heap */ + ret = heap_init(iter->stream_heap, 0, stream_compare); + if (ret < 0) + goto error; + /* Insert the stream that contains the last event */ + ret = heap_insert(iter->stream_heap, cfs); + if (ret) + goto error; + } default: /* not implemented */ return -EINVAL; -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From eag0628 at gmail.com Thu Aug 16 21:57:34 2012 From: eag0628 at gmail.com (Lai Jiangshan) Date: Fri, 17 Aug 2012 09:57:34 +0800 Subject: [lttng-dev] [RFC PATCH] wfqueue: expand API, simplify implementation, small performance boost In-Reply-To: <20120816211155.GA3487@Krystal> References: <20120812145000.GA32747@Krystal> <20120813201219.GA27637@Krystal> <20120814222724.GA9751@Krystal> <20120815125209.GA11737@Krystal> <20120816211155.GA3487@Krystal> Message-ID: Patches are sent? I'm in vacation, I can't recieve them. __cds_wfq_for_each_blocking_safe() sounds like we can delete any node in for_each(), it breaks the mantics of QUEUE. and the users and we don't know how to delete a node. for_each_safe() is just a special things for call_rcu_thread() which destroy all node in a tmp queue. so I prefer to use sync_next() in the call_rcu_thread() as my original patch now, it is also very simple. the for_each_safe() sugar is not required in call_rcu_thread(). ===== I think we need dequeue_all() API. splice() = dequeue_all() + enqueue(list). dequeue_all(): 1 xchg() enqueue(): 1 xchg() splice(): 2 xchg() And if no dequeue_all(), the users have to use splice()+tmp instead, not straightly. So it is worth to have dequeue_all(). ===== the first smallest patch can be: equeue(),sync_next(),dequeue(),dequeue_all(),splice(). Leave all the other things in future patches. Thanks, Lai. "other things" list in my view: What memroy barrier should be add to dequeue()? for_each() APIs and mantics. cache-line-opt queue or current(none-cache-line-opt) or both and related compatibility thing. On Fri, Aug 17, 2012 at 5:11 AM, Mathieu Desnoyers wrote: > * Lai Jiangshan (eag0628 at gmail.com) wrote: >> We need the smallest patch at first, all other things left in disscussion. >> no free_each(), simple changes. > > I guess you mean for_each(). > > If we limit ourself to the versions where the user is doing the locking, > I don't think there were any issues left. > > In the version I provided (in the last series of 2 patches), I took care > of all issues that were raised in our email discussion. > > I prefer to introduce these new API members all in one go, mainly because > I really don't want to add new API members (exposed through the public API) > and then remove them afterward when we notice that they expose too many > details. I think the current splice, dequeue, next, and first API > members allow any user to do the kind of use-case that call_rcu is > doing: this lets us achieve your original goal of not duplicating the > code everywhere. > > If you still notice issues with __cds_wfq_for_each_blocking() and > __cds_wfq_for_each_blocking_safe() in the last patch, please let me > know, > > Thanks ! > > Mathieu > >> >> thanks, >> Lai >> >> On Thu, Aug 16, 2012 at 10:08 AM, Lai Jiangshan wrote: >> >>> >> >>> Is it false sharing? >> >>> Access to q->head.next and access to q->tail have the same performance >> >>> because they are in the same cache line. >> >> >> >> Yes! you are right! And a quick benchmark confirms it: >> >> >> >> with head and tail on same cache line: >> >> >> >> SUMMARY /home/compudj/doc/userspace-rcu/tests/.libs/lt-test_urcu_wfq testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 100833595 nr_dequeues 88647134 successful enqueues 100833595 successful dequeues 88646898 end_dequeues 12186697 nr_ops 189480729 >> >> >> >> with a 256 bytes padding between head and tail, keeping the mutex on the >> >> "head" cache line: >> >> >> >> SUMMARY /home/compudj/doc/userspace-rcu/tests/.libs/lt-test_urcu_wfq testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 228992829 nr_dequeues 228921791 successful enqueues 228992829 successful dequeues 228921367 end_dequeues 71462 nr_ops 457914620 >> >> >> >> enqueue: 127% speedup >> >> dequeue: 158% speedup >> >> >> >> That is indeed a _really_ huge difference. However, to get this, we >> >> would have to increase the size of struct cds_wfq_queue beyond its >> >> current size, which would break API compatibility. Any idea on how to >> >> best do this without causing incompatibility would be welcome. >> >> >> > >> > choice 1) two set of APIs?(cache-line-opt and none-cache-line-opt), >> > many users don't need the cache-line-opt. >> > choice 2) Just break the compatibility for NONE-LGPL. I think >> > NONE-LGPL-user of it is rare. And current version of urcu <1.0, I >> > don't like too much burden when <1.0. >> > >> > >> > thanks, >> > Lai > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com From jdesfossez at efficios.com Thu Aug 16 22:36:10 2012 From: jdesfossez at efficios.com (Julien Desfossez) Date: Thu, 16 Aug 2012 22:36:10 -0400 Subject: [lttng-dev] [BABELTRACE PATCH v4] Add BT_SEEK_LAST type to bt_iter_pos In-Reply-To: <20120817013036.GB6294@Krystal> References: <20120817013036.GB6294@Krystal> Message-ID: <502DAE1A.8020303@efficios.com> On 16/08/12 09:30 PM, Mathieu Desnoyers wrote: > Inspired from patch by Francis Deslauriers and Julien Desfossez. > > Comments are welcome. I left a FIXME as question. > > Not tested. I tested it, it is working, just need to add the "break" missing I added in comments below. > > Not-Signed-off-by: Mathieu Desnoyers > --- > diff --git a/include/babeltrace/iterator.h b/include/babeltrace/iterator.h > index aa6470e..98504a5 100644 > --- a/include/babeltrace/iterator.h > +++ b/include/babeltrace/iterator.h > @@ -40,6 +40,15 @@ struct bt_saved_pos; > * is expressed in nanoseconds > * - restore is a position saved with bt_iter_get_pos, it is used with > * BT_SEEK_RESTORE. > + * > + * Note about BT_SEEK_LAST: if many events happen to be at the last > + * timestamp, it is implementation-defined which event will be the last, > + * and the order of events with the same timestamp may not be the same > + * as normal iteration on the trace. Therefore, it is recommended to > + * only use BT_SEEK_LAST to get the timestamp of the last event(s) in > + * the trace. > + * FIXME: should we name it BT_SEEK_LAST_TIMESTAMP instead, if we ever > + * want to keep BT_SEEK_LAST for the last event in iteration order ? > */ > struct bt_iter_pos { > enum { > @@ -48,6 +57,7 @@ struct bt_iter_pos { > BT_SEEK_CUR, > BT_SEEK_BEGIN, > BT_SEEK_END, > + BT_SEEK_LAST, > } type; > union { > uint64_t seek_time; > diff --git a/lib/iterator.c b/lib/iterator.c > index 2dbd77c..b305f16 100644 > --- a/lib/iterator.c > +++ b/lib/iterator.c > @@ -103,6 +103,10 @@ void bt_iter_free_pos(struct bt_iter_pos *iter_pos) > * > * Return 0 if the seek succeded, EOF if we didn't find any packet > * containing the timestamp, or a positive integer for error. > + * > + * TODO: this should be turned into a binary search! It is currently > + * doing a linear search in the packets. This is a O(n) operation on a > + * very frequent code path. > */ > static int seek_file_stream_by_timestamp(struct ctf_file_stream *cfs, > uint64_t timestamp) > @@ -187,6 +191,148 @@ static int seek_ctf_trace_by_timestamp(struct ctf_trace *tin, > return found ? 0 : EOF; > } > > +/* > + * Find timestamp of last event in the stream. > + * > + * Return value: 0 if OK, positive error value on error, EOF if no > + * events were found. > + */ > +static int find_max_timestamp_ctf_file_stream(struct ctf_file_stream *cfs, > + uint64_t *timestamp_end) > +{ > + int ret, count = 0, i; > + uint64_t timestamp = 0; > + struct ctf_stream_pos *stream_pos; > + > + stream_pos = &cfs->pos; > + /* > + * We start by the last packet, and iterate backwards until we > + * either find at least one event, or we reach the first packet > + * (some packets can be empty). > + */ > + for (i = stream_pos->packet_real_index->len - 1; i >= 0; i--) { > + stream_pos->packet_seek(&stream_pos->parent, i, SEEK_SET); > + count = 0; > + /* read each event until we reach the end of the stream */ > + do { > + ret = stream_read_event(cfs); > + if (ret == 0) { > + count++; > + timestamp = cfs->parent.real_timestamp; > + } > + } while (ret == 0); > + > + /* Error */ > + if (ret > 0) > + goto end; > + assert(ret == EOF); > + if (count) > + break; > + } > + > + if (count) { > + *timestamp_end = timestamp; > + ret = 0; > + } else { > + /* Return EOF if no events were found */ > + ret = EOF; > + } > +end: > + return ret; > +} > + > +/* > + * Find the stream within a stream class that contains the event with > + * the largest timestamp, and save that timestamp. > + * > + * Return 0 if OK, EOF if no events were found in the streams, or > + * positive value on error. > + */ > +static int find_max_timestamp_ctf_stream_class( > + struct ctf_stream_declaration *stream_class, > + struct ctf_file_stream **cfsp, > + uint64_t *max_timestamp) > +{ > + int ret = EOF, i; > + > + for (i = 0; i < stream_class->streams->len; i++) { > + struct ctf_stream_definition *stream; > + struct ctf_file_stream *cfs; > + uint64_t current_max_ts = 0; > + > + stream = g_ptr_array_index(stream_class->streams, i); > + if (!stream) > + continue; > + cfs = container_of(stream, struct ctf_file_stream, parent); > + ret = find_max_timestamp_ctf_file_stream(cfs, ¤t_max_ts); > + if (ret == EOF) > + continue; > + if (ret != 0) > + break; > + if (current_max_ts >= *max_timestamp) { > + *max_timestamp = current_max_ts; > + *cfsp = cfs; > + } > + } > + assert(ret >= 0 || ret == EOF); > + return ret; > +} > + > +/* > + * seek_last_ctf_trace_collection: seek trace collection to last event. > + * > + * Return 0 if OK, EOF if no events were found, or positive error value > + * on error. > + */ > +static int seek_last_ctf_trace_collection(struct trace_collection *tc, > + struct ctf_file_stream **cfsp) > +{ > + int i, j, ret; > + int found = 0; > + uint64_t max_timestamp = 0; > + > + if (!tc) > + return 1; > + > + /* For each trace in the trace_collection */ > + for (i = 0; i < tc->array->len; i++) { > + struct ctf_trace *tin; > + struct trace_descriptor *td_read; > + > + td_read = g_ptr_array_index(tc->array, i); > + if (!td_read) > + continue; > + tin = container_of(td_read, struct ctf_trace, parent); > + /* For each stream_class in the trace */ > + for (j = 0; j < tin->streams->len; j++) { > + struct ctf_stream_declaration *stream_class; > + > + stream_class = g_ptr_array_index(tin->streams, j); > + if (!stream_class) > + continue; > + ret = find_max_timestamp_ctf_stream_class(stream_class, > + cfsp, &max_timestamp); > + if (ret > 0) > + goto end; > + if (ret == 0) > + found = 1; > + assert(ret == EOF || ret == 0); > + } > + } > + /* > + * Now we know in which file stream the last event is located, > + * and we know its timestamp. > + */ > + if (!found) { > + ret = EOF; > + } else { > + ret = seek_file_stream_by_timestamp(*cfsp, max_timestamp); > + assert(ret == 0); > + } > +end: > + return ret; > +} > + > int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) > { > struct trace_collection *tc; > @@ -332,6 +478,25 @@ int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) > } > } > break; > + case BT_SEEK_LAST: > + { > + struct ctf_file_stream *cfs; > + > + tc = iter->ctx->tc; > + ret = seek_last_ctf_trace_collection(tc, &cfs); > + if (ret != 0) > + goto error; > + /* remove all streams from the heap */ > + heap_free(iter->stream_heap); > + /* Create a new empty heap */ > + ret = heap_init(iter->stream_heap, 0, stream_compare); > + if (ret < 0) > + goto error; > + /* Insert the stream that contains the last event */ > + ret = heap_insert(iter->stream_heap, cfs); > + if (ret) > + goto error; break or return missing here. > + } > default: > /* not implemented */ > return -EINVAL; From mathieu.desnoyers at efficios.com Fri Aug 17 00:03:44 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 17 Aug 2012 00:03:44 -0400 Subject: [lttng-dev] [BABELTRACE PATCH v4] Add BT_SEEK_LAST type to bt_iter_pos In-Reply-To: <502DAE1A.8020303@efficios.com> References: <20120817013036.GB6294@Krystal> <502DAE1A.8020303@efficios.com> Message-ID: <20120817040344.GB8145@Krystal> * Julien Desfossez (jdesfossez at efficios.com) wrote: > > > On 16/08/12 09:30 PM, Mathieu Desnoyers wrote: > > Inspired from patch by Francis Deslauriers and Julien Desfossez. > > > > Comments are welcome. I left a FIXME as question. > > > > Not tested. > I tested it, it is working, just need to add the "break" missing I added > in comments below. OK, fixed. I pushed the following: commit f7ed656326285831256061407c2c533c16c50f36 Author: Mathieu Desnoyers Date: Fri Aug 17 00:01:54 2012 -0400 Add BT_SEEK_LAST type to bt_iter_pos Inspired from patch by Francis Deslauriers and Julien Desfossez. Acked-by: Julien Desfossez Signed-off-by: Mathieu Desnoyers Thanks! Mathieu > > > > > Not-Signed-off-by: Mathieu Desnoyers > > --- > > diff --git a/include/babeltrace/iterator.h b/include/babeltrace/iterator.h > > index aa6470e..98504a5 100644 > > --- a/include/babeltrace/iterator.h > > +++ b/include/babeltrace/iterator.h > > @@ -40,6 +40,15 @@ struct bt_saved_pos; > > * is expressed in nanoseconds > > * - restore is a position saved with bt_iter_get_pos, it is used with > > * BT_SEEK_RESTORE. > > + * > > + * Note about BT_SEEK_LAST: if many events happen to be at the last > > + * timestamp, it is implementation-defined which event will be the last, > > + * and the order of events with the same timestamp may not be the same > > + * as normal iteration on the trace. Therefore, it is recommended to > > + * only use BT_SEEK_LAST to get the timestamp of the last event(s) in > > + * the trace. > > + * FIXME: should we name it BT_SEEK_LAST_TIMESTAMP instead, if we ever > > + * want to keep BT_SEEK_LAST for the last event in iteration order ? > > */ > > struct bt_iter_pos { > > enum { > > @@ -48,6 +57,7 @@ struct bt_iter_pos { > > BT_SEEK_CUR, > > BT_SEEK_BEGIN, > > BT_SEEK_END, > > + BT_SEEK_LAST, > > } type; > > union { > > uint64_t seek_time; > > diff --git a/lib/iterator.c b/lib/iterator.c > > index 2dbd77c..b305f16 100644 > > --- a/lib/iterator.c > > +++ b/lib/iterator.c > > @@ -103,6 +103,10 @@ void bt_iter_free_pos(struct bt_iter_pos *iter_pos) > > * > > * Return 0 if the seek succeded, EOF if we didn't find any packet > > * containing the timestamp, or a positive integer for error. > > + * > > + * TODO: this should be turned into a binary search! It is currently > > + * doing a linear search in the packets. This is a O(n) operation on a > > + * very frequent code path. > > */ > > static int seek_file_stream_by_timestamp(struct ctf_file_stream *cfs, > > uint64_t timestamp) > > @@ -187,6 +191,148 @@ static int seek_ctf_trace_by_timestamp(struct ctf_trace *tin, > > return found ? 0 : EOF; > > } > > > > +/* > > + * Find timestamp of last event in the stream. > > + * > > + * Return value: 0 if OK, positive error value on error, EOF if no > > + * events were found. > > + */ > > +static int find_max_timestamp_ctf_file_stream(struct ctf_file_stream *cfs, > > + uint64_t *timestamp_end) > > +{ > > + int ret, count = 0, i; > > + uint64_t timestamp = 0; > > + struct ctf_stream_pos *stream_pos; > > + > > + stream_pos = &cfs->pos; > > + /* > > + * We start by the last packet, and iterate backwards until we > > + * either find at least one event, or we reach the first packet > > + * (some packets can be empty). > > + */ > > + for (i = stream_pos->packet_real_index->len - 1; i >= 0; i--) { > > + stream_pos->packet_seek(&stream_pos->parent, i, SEEK_SET); > > + count = 0; > > + /* read each event until we reach the end of the stream */ > > + do { > > + ret = stream_read_event(cfs); > > + if (ret == 0) { > > + count++; > > + timestamp = cfs->parent.real_timestamp; > > + } > > + } while (ret == 0); > > + > > + /* Error */ > > + if (ret > 0) > > + goto end; > > + assert(ret == EOF); > > + if (count) > > + break; > > + } > > + > > + if (count) { > > + *timestamp_end = timestamp; > > + ret = 0; > > + } else { > > + /* Return EOF if no events were found */ > > + ret = EOF; > > + } > > +end: > > + return ret; > > +} > > + > > +/* > > + * Find the stream within a stream class that contains the event with > > + * the largest timestamp, and save that timestamp. > > + * > > + * Return 0 if OK, EOF if no events were found in the streams, or > > + * positive value on error. > > + */ > > +static int find_max_timestamp_ctf_stream_class( > > + struct ctf_stream_declaration *stream_class, > > + struct ctf_file_stream **cfsp, > > + uint64_t *max_timestamp) > > +{ > > + int ret = EOF, i; > > + > > + for (i = 0; i < stream_class->streams->len; i++) { > > + struct ctf_stream_definition *stream; > > + struct ctf_file_stream *cfs; > > + uint64_t current_max_ts = 0; > > + > > + stream = g_ptr_array_index(stream_class->streams, i); > > + if (!stream) > > + continue; > > + cfs = container_of(stream, struct ctf_file_stream, parent); > > + ret = find_max_timestamp_ctf_file_stream(cfs, ¤t_max_ts); > > + if (ret == EOF) > > + continue; > > + if (ret != 0) > > + break; > > + if (current_max_ts >= *max_timestamp) { > > + *max_timestamp = current_max_ts; > > + *cfsp = cfs; > > + } > > + } > > + assert(ret >= 0 || ret == EOF); > > + return ret; > > +} > > + > > +/* > > + * seek_last_ctf_trace_collection: seek trace collection to last event. > > + * > > + * Return 0 if OK, EOF if no events were found, or positive error value > > + * on error. > > + */ > > +static int seek_last_ctf_trace_collection(struct trace_collection *tc, > > + struct ctf_file_stream **cfsp) > > +{ > > + int i, j, ret; > > + int found = 0; > > + uint64_t max_timestamp = 0; > > + > > + if (!tc) > > + return 1; > > + > > + /* For each trace in the trace_collection */ > > + for (i = 0; i < tc->array->len; i++) { > > + struct ctf_trace *tin; > > + struct trace_descriptor *td_read; > > + > > + td_read = g_ptr_array_index(tc->array, i); > > + if (!td_read) > > + continue; > > + tin = container_of(td_read, struct ctf_trace, parent); > > + /* For each stream_class in the trace */ > > + for (j = 0; j < tin->streams->len; j++) { > > + struct ctf_stream_declaration *stream_class; > > + > > + stream_class = g_ptr_array_index(tin->streams, j); > > + if (!stream_class) > > + continue; > > + ret = find_max_timestamp_ctf_stream_class(stream_class, > > + cfsp, &max_timestamp); > > + if (ret > 0) > > + goto end; > > + if (ret == 0) > > + found = 1; > > + assert(ret == EOF || ret == 0); > > + } > > + } > > + /* > > + * Now we know in which file stream the last event is located, > > + * and we know its timestamp. > > + */ > > + if (!found) { > > + ret = EOF; > > + } else { > > + ret = seek_file_stream_by_timestamp(*cfsp, max_timestamp); > > + assert(ret == 0); > > + } > > +end: > > + return ret; > > +} > > + > > int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) > > { > > struct trace_collection *tc; > > @@ -332,6 +478,25 @@ int bt_iter_set_pos(struct bt_iter *iter, const struct bt_iter_pos *iter_pos) > > } > > } > > break; > > + case BT_SEEK_LAST: > > + { > > + struct ctf_file_stream *cfs; > > + > > + tc = iter->ctx->tc; > > + ret = seek_last_ctf_trace_collection(tc, &cfs); > > + if (ret != 0) > > + goto error; > > + /* remove all streams from the heap */ > > + heap_free(iter->stream_heap); > > + /* Create a new empty heap */ > > + ret = heap_init(iter->stream_heap, 0, stream_compare); > > + if (ret < 0) > > + goto error; > > + /* Insert the stream that contains the last event */ > > + ret = heap_insert(iter->stream_heap, cfs); > > + if (ret) > > + goto error; > break or return missing here. > > + } > > default: > > /* not implemented */ > > return -EINVAL; -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Fri Aug 17 00:09:47 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 17 Aug 2012 00:09:47 -0400 Subject: [lttng-dev] [RFC URCU PATCH] Introduce wfqueue v0 ABI compatibility In-Reply-To: <20120815212806.GA17224@Krystal> References: <20120815212806.GA17224@Krystal> Message-ID: <20120817040946.GA8382@Krystal> Hi Lai, Here is a copy of the first patch I sent you (compatibility ABI). Comments are welcome, but don't forget to take vacation! ;-) Thanks, Mathieu * Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote: > Introduce wfqueue v0 ABI compatibility > > Preparing for v1 ABI. > > CC: Lai Jiangshan > CC: Paul McKenney > Signed-off-by: Mathieu Desnoyers > --- > diff --git a/wfqueue0.c b/wfqueue0.c > new file mode 100644 > index 0000000..e928c0a > --- /dev/null > +++ b/wfqueue0.c > @@ -0,0 +1,191 @@ > +/* > + * wfqueue0.c > + * > + * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > + * > + * Copyright 2010-2012 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > + * > + * This library is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * This library is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with this library; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +/* This file is provided for ABI backward compatibility with wfqueue v0. */ > + > +#include > +#include > +#include > +#include > +#include > + > +/* > + * Queue with wait-free enqueue/blocking dequeue. > + * This implementation adds a dummy head node when the queue is empty to ensure > + * we can always update the queue locklessly. > + * > + * Inspired from half-wait-free/half-blocking queue implementation done by > + * Paul E. McKenney. > + */ > + > +struct cds_wfq_node { > + struct cds_wfq_node *next; > +}; > + > +struct cds_wfq_queue { > + struct cds_wfq_node *head, **tail; > + struct cds_wfq_node dummy; /* Dummy node */ > + pthread_mutex_t lock; > +}; > + > +#define WFQ_ADAPT_ATTEMPTS 10 /* Retry if being set */ > +#define WFQ_WAIT 10 /* Wait 10 ms if being set */ > + > +static inline void _cds_wfq_node_init(struct cds_wfq_node *node) > +{ > + node->next = NULL; > +} > + > +static inline void _cds_wfq_init(struct cds_wfq_queue *q) > +{ > + int ret; > + > + _cds_wfq_node_init(&q->dummy); > + /* Set queue head and tail */ > + q->head = &q->dummy; > + q->tail = &q->dummy.next; > + ret = pthread_mutex_init(&q->lock, NULL); > + assert(!ret); > +} > + > +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > + struct cds_wfq_node *node) > +{ > + struct cds_wfq_node **old_tail; > + > + /* > + * uatomic_xchg() implicit memory barrier orders earlier stores to data > + * structure containing node and setting node->next to NULL before > + * publication. > + */ > + old_tail = uatomic_xchg(&q->tail, &node->next); > + /* > + * At this point, dequeuers see a NULL old_tail->next, which indicates > + * that the queue is being appended to. The following store will append > + * "node" to the queue from a dequeuer perspective. > + */ > + CMM_STORE_SHARED(*old_tail, node); > +} > + > +/* > + * Waiting for enqueuer to complete enqueue and return the next node > + */ > +static inline struct cds_wfq_node * > +___cds_wfq_node_sync_next(struct cds_wfq_node *node) > +{ > + struct cds_wfq_node *next; > + int attempt = 0; > + > + /* > + * Adaptative busy-looping waiting for enqueuer to complete enqueue. > + */ > + while ((next = CMM_LOAD_SHARED(node->next)) == NULL) { > + if (++attempt >= WFQ_ADAPT_ATTEMPTS) { > + poll(NULL, 0, WFQ_WAIT); /* Wait for 10ms */ > + attempt = 0; > + } else > + caa_cpu_relax(); > + } > + > + return next; > +} > + > +/* > + * It is valid to reuse and free a dequeued node immediately. > + * > + * No need to go on a waitqueue here, as there is no possible state in which the > + * list could cause dequeue to busy-loop needlessly while waiting for another > + * thread to be scheduled. The queue appears empty until tail->next is set by > + * enqueue. > + */ > +static inline struct cds_wfq_node * > +___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > +{ > + struct cds_wfq_node *node, *next; > + > + /* > + * Queue is empty if it only contains the dummy node. > + */ > + if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) > + return NULL; > + node = q->head; > + > + next = ___cds_wfq_node_sync_next(node); > + > + /* > + * Move queue head forward. > + */ > + q->head = next; > + /* > + * Requeue dummy node if we just dequeued it. > + */ > + if (node == &q->dummy) { > + _cds_wfq_node_init(node); > + _cds_wfq_enqueue(q, node); > + return ___cds_wfq_dequeue_blocking(q); > + } > + return node; > +} > + > +static inline struct cds_wfq_node * > +_cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > +{ > + struct cds_wfq_node *retnode; > + int ret; > + > + ret = pthread_mutex_lock(&q->lock); > + assert(!ret); > + retnode = ___cds_wfq_dequeue_blocking(q); > + ret = pthread_mutex_unlock(&q->lock); > + assert(!ret); > + return retnode; > +} > + > +/* > + * library wrappers to be used by non-LGPL compatible source code. > + */ > + > +void cds_wfq_node_init(struct cds_wfq_node *node) > +{ > + _cds_wfq_node_init(node); > +} > + > +void cds_wfq_init(struct cds_wfq_queue *q) > +{ > + _cds_wfq_init(q); > +} > + > +void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node) > +{ > + _cds_wfq_enqueue(q, node); > +} > + > +struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > +{ > + return _cds_wfq_dequeue_blocking(q); > +} > + > +struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > +{ > + return ___cds_wfq_dequeue_blocking(q); > +} > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Fri Aug 17 00:10:40 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 17 Aug 2012 00:10:40 -0400 Subject: [lttng-dev] [RFC URCU PATCH] wfqueue: ABI v1, simplify implementation, 2.3x to 2.6x performance boost In-Reply-To: <20120815213107.GB17224@Krystal> References: <20120815213107.GB17224@Krystal> Message-ID: <20120817041040.GB8382@Krystal> And here is the new ABI I propose. Thanks, Mathieu * Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote: > This work is derived from the patch from Lai Jiangshan submitted as > "urcu: new wfqueue implementation" > (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) > > Its changelog: > > > Some guys would be surprised by this fact: > > There are already TWO implementations of wfqueue in urcu. > > > > The first one is in urcu/static/wfqueue.h: > > 1) enqueue: exchange the tail and then update previous->next > > 2) dequeue: wait for first node's next pointer and them shift, a dummy node > > is introduced to avoid the queue->tail become NULL when shift. > > > > The second one shares some code with the first one, and the left code > > are spreading in urcu-call-rcu-impl.h: > > 1) enqueue: share with the first one > > 2) no dequeue operation: and no shift, so it don't need dummy node, > > Although the dummy node is queued when initialization, but it is removed > > after the first dequeue_all operation in call_rcu_thread(). > > call_rcu_data_free() forgets to handle the dummy node if it is not removed. > > 3)dequeue_all: record the old head and tail, and queue->head become the special > > tail node.(atomic record the tail and change the tail). > > > > The second implementation's code are spreading, bad for review, and it is not > > tested by tests/test_urcu_wfq. > > > > So we need a better implementation avoid the dummy node dancing and can service > > both generic wfqueue APIs and dequeue_all API for call rcu. > > > > The new implementation: > > 1) enqueue: share with the first one/original implementation. > > 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. > > no dummy node, save memory. > > 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail > > and return the old head.next. > > > > More implementation details are in the code. > > tests/test_urcu_wfq will be update in future for testing new APIs. > > The patch proposed by Lai brings a very interesting simplification to > the single-node handling (which is kept here), and moves all queue > handling code away from call_rcu implementation, back into the wfqueue > code. This has the benefit to allow testing enhancements. > > I modified it so the API does not expose implementation details to the > user (e.g. ___cds_wfq_node_sync_next). I added a "splice" operation and > a for loop iterator which should allow wfqueue users to use the list > very efficiently both from LGPL/GPL code and from non-LGPL-compatible > code. > > Benchmarks performed on Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz > (dual-core, with hyperthreading) > > Benchmark invoked: > for a in $(seq 1 10); do ./test_urcu_wfq 1 1 10 -a 0 -a 2; done > > (using cpu number 0 and 2, which should correspond to two cores of my > Intel 2-core/hyperthread processor) > > Before patch: > > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 97274297 nr_dequeues 80745742 successful enqueues 97274297 successful dequeues 80745321 end_dequeues 16528976 nr_ops 178020039 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 92300568 nr_dequeues 75019529 successful enqueues 92300568 successful dequeues 74973237 end_dequeues 17327331 nr_ops 167320097 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 93516443 nr_dequeues 75846726 successful enqueues 93516443 successful dequeues 75826578 end_dequeues 17689865 nr_ops 169363169 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 94160362 nr_dequeues 77967638 successful enqueues 94160362 successful dequeues 77967638 end_dequeues 16192724 nr_ops 172128000 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 97491956 nr_dequeues 81001191 successful enqueues 97491956 successful dequeues 81000247 end_dequeues 16491709 nr_ops 178493147 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 94101298 nr_dequeues 75650510 successful enqueues 94101298 successful dequeues 75649318 end_dequeues 18451980 nr_ops 169751808 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 94742803 nr_dequeues 75402105 successful enqueues 94742803 successful dequeues 75341859 end_dequeues 19400944 nr_ops 170144908 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 92198835 nr_dequeues 75037877 successful enqueues 92198835 successful dequeues 75027605 end_dequeues 17171230 nr_ops 167236712 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 94159560 nr_dequeues 77895972 successful enqueues 94159560 successful dequeues 77858442 end_dequeues 16301118 nr_ops 172055532 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 96059399 nr_dequeues 80115442 successful enqueues 96059399 successful dequeues 80066843 end_dequeues 15992556 nr_ops 176174841 > > After patch: > > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 221229322 nr_dequeues 210645491 successful enqueues 221229322 successful dequeues 210645088 end_dequeues 10584234 nr_ops 431874813 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 219803943 nr_dequeues 210377337 successful enqueues 219803943 successful dequeues 210368680 end_dequeues 9435263 nr_ops 430181280 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 237006358 nr_dequeues 237035340 successful enqueues 237006358 successful dequeues 236997050 end_dequeues 9308 nr_ops 474041698 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 235822443 nr_dequeues 235815942 successful enqueues 235822443 successful dequeues 235814020 end_dequeues 8423 nr_ops 471638385 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 235825567 nr_dequeues 235811803 successful enqueues 235825567 successful dequeues 235810526 end_dequeues 15041 nr_ops 471637370 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 221974953 nr_dequeues 210938190 successful enqueues 221974953 successful dequeues 210938190 end_dequeues 11036763 nr_ops 432913143 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 237994492 nr_dequeues 237938119 successful enqueues 237994492 successful dequeues 237930648 end_dequeues 63844 nr_ops 475932611 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 220634365 nr_dequeues 210491382 successful enqueues 220634365 successful dequeues 210490995 end_dequeues 10143370 nr_ops 431125747 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 237388065 nr_dequeues 237401251 successful enqueues 237388065 successful dequeues 237380295 end_dequeues 7770 nr_ops 474789316 > testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 221201436 nr_dequeues 210831162 successful enqueues 221201436 successful dequeues 210831162 end_dequeues 10370274 nr_ops 432032598 > > Summary: Both enqueue and dequeue speed increase: around 2.3x speedup > for enqueue, and around 2.6x for dequeue. > > We can verify that: > successful enqueues - successful dequeues = end_dequeues > > For all runs (ensures correctness: no lost node). > > * Introduce wfqueue ABI v1 (false-sharing fix) > > wfqueue v0 suffers from false-sharing between head and tail. By > cache-aligning head and tail, we get a significant speedup on > benchmarks. But in order to do that, we need to break the ABI to enlarge > struct cds_wfq_queue. > > Provide a backward compatibility ABI for the old wfqueue by defining the > original symbols to the old implementation (which uses dummy node). > Programs compiled against old headers, which are not LGPL_SOURCE, will > still use the old implementation. > > Any code compiled against the new header will directly use the new ABI. > > This does not require any change in the way users call the API: the new > ABI symbols are simply defined with _1 suffix, and wrapped by > preprocessor macros. > > Known limitation: users should *not* link together objects using the v0 > and v1 APIs of wfqueue and exchange struct cds_wfq_queue structures > between the two. The only way to run into this corner-case would be to > combine objects compiled with different versions of the urcu/wfqueue.h > header. > > CC: Lai Jiangshan > CC: Paul McKenney > Signed-off-by: Mathieu Desnoyers > --- > diff --git a/Makefile.am b/Makefile.am > index 2396fcf..31052ef 100644 > --- a/Makefile.am > +++ b/Makefile.am > @@ -53,7 +53,7 @@ lib_LTLIBRARIES = liburcu-common.la \ > # liburcu-common contains wait-free queues (needed by call_rcu) as well > # as futex fallbacks. > # > -liburcu_common_la_SOURCES = wfqueue.c wfstack.c $(COMPAT) > +liburcu_common_la_SOURCES = wfqueue.c wfqueue0.c wfstack.c $(COMPAT) > > liburcu_la_SOURCES = urcu.c urcu-pointer.c $(COMPAT) > liburcu_la_LIBADD = liburcu-common.la > diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h > index 13b24ff..d8537d0 100644 > --- a/urcu-call-rcu-impl.h > +++ b/urcu-call-rcu-impl.h > @@ -21,6 +21,7 @@ > */ > > #define _GNU_SOURCE > +#define _LGPL_SOURCE > #include > #include > #include > @@ -220,10 +221,7 @@ static void call_rcu_wake_up(struct call_rcu_data *crdp) > static void *call_rcu_thread(void *arg) > { > unsigned long cbcount; > - struct cds_wfq_node *cbs; > - struct cds_wfq_node **cbs_tail; > - struct call_rcu_data *crdp = (struct call_rcu_data *)arg; > - struct rcu_head *rhp; > + struct call_rcu_data *crdp = (struct call_rcu_data *) arg; > int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); > int ret; > > @@ -243,35 +241,30 @@ static void *call_rcu_thread(void *arg) > cmm_smp_mb(); > } > for (;;) { > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > - poll(NULL, 0, 1); > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > - cbs_tail = (struct cds_wfq_node **) > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > + struct cds_wfq_queue cbs_tmp; > + struct cds_wfq_node *cbs, *tmp_cbs; > + > + cds_wfq_init(&cbs_tmp); > + __cds_wfq_splice_blocking(&cbs_tmp, &crdp->cbs); > + if (!cds_wfq_empty(&cbs_tmp)) { > synchronize_rcu(); > cbcount = 0; > - do { > - while (cbs->next == NULL && > - &cbs->next != cbs_tail) > - poll(NULL, 0, 1); > - if (cbs == &crdp->cbs.dummy) { > - cbs = cbs->next; > - continue; > - } > - rhp = (struct rcu_head *)cbs; > - cbs = cbs->next; > + __cds_wfq_for_each_blocking_safe(&cbs_tmp, > + cbs, tmp_cbs) { > + struct rcu_head *rhp; > + > + rhp = caa_container_of(cbs, > + struct rcu_head, next); > rhp->func(rhp); > cbcount++; > - } while (cbs != NULL); > + } > uatomic_sub(&crdp->qlen, cbcount); > } > if (uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOP) > break; > rcu_thread_offline(); > if (!rt) { > - if (&crdp->cbs.head > - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { > + if (cds_wfq_empty(&crdp->cbs)) { > call_rcu_wait(crdp); > poll(NULL, 0, 10); > uatomic_dec(&crdp->futex); > @@ -625,32 +618,27 @@ void call_rcu(struct rcu_head *head, > */ > void call_rcu_data_free(struct call_rcu_data *crdp) > { > - struct cds_wfq_node *cbs; > - struct cds_wfq_node **cbs_tail; > - struct cds_wfq_node **cbs_endprev; > - > if (crdp == NULL || crdp == default_call_rcu_data) { > return; > } > + > if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { > uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); > wake_call_rcu_thread(crdp); > while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) > poll(NULL, 0, 1); > } > - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { > - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) > - poll(NULL, 0, 1); > - _CMM_STORE_SHARED(crdp->cbs.head, NULL); > - cbs_tail = (struct cds_wfq_node **) > - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); > + > + if (!cds_wfq_empty(&crdp->cbs)) { > /* Create default call rcu data if need be */ > (void) get_default_call_rcu_data(); > - cbs_endprev = (struct cds_wfq_node **) > - uatomic_xchg(&default_call_rcu_data, cbs_tail); > - *cbs_endprev = cbs; > + > + __cds_wfq_splice_blocking(&default_call_rcu_data->cbs, > + &crdp->cbs); > + > uatomic_add(&default_call_rcu_data->qlen, > uatomic_read(&crdp->qlen)); > + > wake_call_rcu_thread(default_call_rcu_data); > } > > diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h > index 636e1af..52e452d 100644 > --- a/urcu/static/wfqueue.h > +++ b/urcu/static/wfqueue.h > @@ -9,7 +9,8 @@ > * TO BE INCLUDED ONLY IN LGPL-COMPATIBLE CODE. See wfqueue.h for linking > * dynamically with the userspace rcu library. > * > - * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2010-2012 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -29,6 +30,7 @@ > #include > #include > #include > +#include > #include > #include > > @@ -38,11 +40,16 @@ extern "C" { > > /* > * Queue with wait-free enqueue/blocking dequeue. > - * This implementation adds a dummy head node when the queue is empty to ensure > - * we can always update the queue locklessly. > * > * Inspired from half-wait-free/half-blocking queue implementation done by > * Paul E. McKenney. > + * > + * Caller must ensure mutual exclusion of queue update operations > + * "dequeue" and "splice" source queue. Queue read operations "first" > + * and "next" need to be protected against concurrent "dequeue" and > + * "splice" (for source queue) by the caller. "enqueue", "splice" > + * (destination queue), and "empty" are the only operations that can be > + * used without any mutual exclusion. > */ > > #define WFQ_ADAPT_ATTEMPTS 10 /* Retry if being set */ > @@ -57,31 +64,55 @@ static inline void _cds_wfq_init(struct cds_wfq_queue *q) > { > int ret; > > - _cds_wfq_node_init(&q->dummy); > /* Set queue head and tail */ > - q->head = &q->dummy; > - q->tail = &q->dummy.next; > - ret = pthread_mutex_init(&q->lock, NULL); > + _cds_wfq_node_init(&q->dequeue.head); > + q->enqueue.tail = &q->dequeue.head; > + ret = pthread_mutex_init(&q->dequeue.lock, NULL); > assert(!ret); > } > > -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > - struct cds_wfq_node *node) > +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) > { > - struct cds_wfq_node **old_tail; > + /* > + * Queue is empty if no node is pointed by q->head.next nor > + * q->tail. Even though the q->tail check is sufficient to find > + * out of the queue is empty, we first check q->head.next as a > + * common case to ensure that dequeuers do not frequently access > + * enqueuer's q->tail cache line. > + */ > + return CMM_LOAD_SHARED(q->dequeue.head.next) == NULL > + && CMM_LOAD_SHARED(q->enqueue.tail) == &q->dequeue.head; > +} > + > +static inline void ___cds_wfq_append(struct cds_wfq_queue *q, > + struct cds_wfq_node *new_head, > + struct cds_wfq_node *new_tail) > +{ > + struct cds_wfq_node *old_tail; > > /* > - * uatomic_xchg() implicit memory barrier orders earlier stores to data > - * structure containing node and setting node->next to NULL before > - * publication. > + * Implicit memory barrier before uatomic_xchg() orders earlier > + * stores to data structure containing node and setting > + * node->next to NULL before publication. > */ > - old_tail = uatomic_xchg(&q->tail, &node->next); > + old_tail = uatomic_xchg(&q->enqueue.tail, new_tail); > + > /* > - * At this point, dequeuers see a NULL old_tail->next, which indicates > - * that the queue is being appended to. The following store will append > - * "node" to the queue from a dequeuer perspective. > + * Implicit memory barrier after uatomic_xchg() orders store to > + * q->tail before store to old_tail->next. > + * > + * At this point, dequeuers see a NULL q->tail->next, which > + * indicates that the queue is being appended to. The following > + * store will append "node" to the queue from a dequeuer > + * perspective. > */ > - CMM_STORE_SHARED(*old_tail, node); > + CMM_STORE_SHARED(old_tail->next, new_head); > +} > + > +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, > + struct cds_wfq_node *new_tail) > +{ > + ___cds_wfq_append(q, new_tail, new_tail); > } > > /* > @@ -100,14 +131,68 @@ ___cds_wfq_node_sync_next(struct cds_wfq_node *node) > if (++attempt >= WFQ_ADAPT_ATTEMPTS) { > poll(NULL, 0, WFQ_WAIT); /* Wait for 10ms */ > attempt = 0; > - } else > + } else { > caa_cpu_relax(); > + } > } > > return next; > } > > /* > + * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. > + * > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured > + * by the caller. > + */ > +static inline struct cds_wfq_node * > +___cds_wfq_first_blocking(struct cds_wfq_queue *q) > +{ > + struct cds_wfq_node *node; > + > + if (_cds_wfq_empty(q)) > + return NULL; > + node = ___cds_wfq_node_sync_next(&q->dequeue.head); > + /* Load q->head.next before loading node's content */ > + cmm_smp_read_barrier_depends(); > + return node; > +} > + > +/* > + * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. > + * > + * Mutual exclusion with "dequeue" and "splice" operations must be ensured > + * by the caller. > + */ > +static inline struct cds_wfq_node * > +___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > +{ > + struct cds_wfq_node *next; > + > + /* > + * Even though the following q->tail check is sufficient to find > + * out if we reached the end of the queue, we first check > + * node->next as a common case to ensure that iteration on nodes > + * do not frequently access enqueuer's q->tail cache line. > + */ > + if ((next = CMM_LOAD_SHARED(node->next)) != NULL) { > + /* Load node->next before loading next's content */ > + cmm_smp_read_barrier_depends(); > + return next; > + } > + /* Load node->next before q->tail */ > + cmm_smp_rmb(); > + if (CMM_LOAD_SHARED(q->enqueue.tail) == node) > + return NULL; > + next = ___cds_wfq_node_sync_next(node); > + /* Load node->next before loading next's content */ > + cmm_smp_read_barrier_depends(); > + return next; > +} > + > +/* > + * ___cds_wfq_dequeue_blocking: dequeue a node from the queue. > + * > * It is valid to reuse and free a dequeued node immediately. > * > * No need to go on a waitqueue here, as there is no possible state in which the > @@ -120,42 +205,104 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > { > struct cds_wfq_node *node, *next; > > - /* > - * Queue is empty if it only contains the dummy node. > - */ > - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) > + if (_cds_wfq_empty(q)) > return NULL; > - node = q->head; > > - next = ___cds_wfq_node_sync_next(node); > + node = ___cds_wfq_node_sync_next(&q->dequeue.head); > + > + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { > + /* > + * @node is probably the only node in the queue. > + * Try to move the tail to &q->head. > + * q->head.next is set to NULL here, and stays > + * NULL if the cmpxchg succeeds. Should the > + * cmpxchg fail due to a concurrent enqueue, the > + * q->head.next will be set to the next node. > + * The implicit memory barrier before > + * uatomic_cmpxchg() orders load node->next > + * before loading q->tail. > + * The implicit memory barrier before uatomic_cmpxchg > + * orders load q->head.next before loading node's > + * content. > + */ > + _cds_wfq_node_init(&q->dequeue.head); > + if (uatomic_cmpxchg(&q->enqueue.tail, node, > + &q->dequeue.head) == node) > + return node; > + next = ___cds_wfq_node_sync_next(node); > + } > > /* > * Move queue head forward. > */ > - q->head = next; > + q->dequeue.head.next = next; > + > + /* Load q->head.next before loading node's content */ > + cmm_smp_read_barrier_depends(); > + return node; > +} > + > +/* > + * ___cds_wfq_splice_blocking: enqueue all src_q nodes at the end of dest_q. > + * > + * Dequeue all nodes from src_q. > + * dest_q must be already initialized. > + * caller ensures mutual exclusion of dequeue and splice operations on > + * src_q. > + */ > +static inline void > +___cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, > + struct cds_wfq_queue *src_q) > +{ > + struct cds_wfq_node *head, *tail; > + > + if (_cds_wfq_empty(src_q)) > + return; > + > + head = ___cds_wfq_node_sync_next(&src_q->dequeue.head); > + _cds_wfq_node_init(&src_q->dequeue.head); > + > /* > - * Requeue dummy node if we just dequeued it. > + * Memory barrier implied before uatomic_xchg() orders store to > + * src_q->head before store to src_q->tail. This is required by > + * concurrent enqueue on src_q, which exchanges the tail before > + * updating the previous tail's next pointer. > */ > - if (node == &q->dummy) { > - _cds_wfq_node_init(node); > - _cds_wfq_enqueue(q, node); > - return ___cds_wfq_dequeue_blocking(q); > - } > - return node; > + tail = uatomic_xchg(&src_q->enqueue.tail, &src_q->dequeue.head); > + > + /* > + * Append the spliced content of src_q into dest_q. Does not > + * require mutual exclusion on dest_q (wait-free). > + */ > + ___cds_wfq_append(dest_q, head, tail); > } > > +/* Locking performed within cds_wfq calls. */ > static inline struct cds_wfq_node * > _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > { > - struct cds_wfq_node *retnode; > + struct cds_wfq_node *retval; > + int ret; > + > + ret = pthread_mutex_lock(&q->dequeue.lock); > + assert(!ret); > + retval = ___cds_wfq_dequeue_blocking(q); > + ret = pthread_mutex_unlock(&q->dequeue.lock); > + assert(!ret); > + return retval; > +} > + > +static inline void > +_cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, > + struct cds_wfq_queue *src_q) > +{ > int ret; > > - ret = pthread_mutex_lock(&q->lock); > + ret = pthread_mutex_lock(&src_q->dequeue.lock); > assert(!ret); > - retnode = ___cds_wfq_dequeue_blocking(q); > - ret = pthread_mutex_unlock(&q->lock); > + ___cds_wfq_splice_blocking(dest_q, src_q); > + ret = pthread_mutex_unlock(&src_q->dequeue.lock); > assert(!ret); > - return retnode; > } > > #ifdef __cplusplus > diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h > index 03a73f1..446c94c 100644 > --- a/urcu/wfqueue.h > +++ b/urcu/wfqueue.h > @@ -6,7 +6,8 @@ > * > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > * > - * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2010-2012 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -25,7 +26,9 @@ > > #include > #include > +#include > #include > +#include > > #ifdef __cplusplus > extern "C" { > @@ -33,8 +36,6 @@ extern "C" { > > /* > * Queue with wait-free enqueue/blocking dequeue. > - * This implementation adds a dummy head node when the queue is empty to ensure > - * we can always update the queue locklessly. > * > * Inspired from half-wait-free/half-blocking queue implementation done by > * Paul E. McKenney. > @@ -45,9 +46,13 @@ struct cds_wfq_node { > }; > > struct cds_wfq_queue { > - struct cds_wfq_node *head, **tail; > - struct cds_wfq_node dummy; /* Dummy node */ > - pthread_mutex_t lock; > + struct { > + struct cds_wfq_node head; > + pthread_mutex_t lock; > + } __attribute__((aligned((CAA_CACHE_LINE_SIZE)))) dequeue; > + struct { > + struct cds_wfq_node *tail; > + } __attribute__((aligned((CAA_CACHE_LINE_SIZE)))) enqueue; > }; > > #ifdef _LGPL_SOURCE > @@ -55,22 +60,104 @@ struct cds_wfq_queue { > #include > > #define cds_wfq_node_init _cds_wfq_node_init > -#define cds_wfq_init _cds_wfq_init > -#define cds_wfq_enqueue _cds_wfq_enqueue > -#define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking > +#define cds_wfq_init _cds_wfq_init > +#define cds_wfq_empty _cds_wfq_empty > +#define cds_wfq_enqueue _cds_wfq_enqueue > + > +/* Locking performed within cds_wfq calls. */ > #define cds_wfq_dequeue_blocking _cds_wfq_dequeue_blocking > +#define cds_wfq_splice_blocking _cds_wfq_splice_blocking > +#define cds_wfq_first_blocking _cds_wfq_first_blocking > +#define cds_wfq_next_blocking _cds_wfq_next_blocking > + > +/* Locking ensured by caller */ > +#define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking > +#define __cds_wfq_splice_blocking ___cds_wfq_splice_blocking > +#define __cds_wfq_first_blocking ___cds_wfq_first_blocking > +#define __cds_wfq_next_blocking ___cds_wfq_next_blocking > > #else /* !_LGPL_SOURCE */ > > -extern void cds_wfq_node_init(struct cds_wfq_node *node); > -extern void cds_wfq_init(struct cds_wfq_queue *q); > -extern void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node); > -/* __cds_wfq_dequeue_blocking: caller ensures mutual exclusion between dequeues */ > -extern struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); > -extern struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); > +extern void cds_wfq_node_init_1(struct cds_wfq_node *node); > +extern void cds_wfq_init_1(struct cds_wfq_queue *q); > +extern bool cds_wfq_empty_1(struct cds_wfq_queue *q); > +extern void cds_wfq_enqueue_1(struct cds_wfq_queue *q, struct cds_wfq_node *node); > + > +/* Locking performed within cds_wfq calls. */ > +extern struct cds_wfq_node *cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q); > +extern void cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, > + struct cds_wfq_queue *src_q); > + > +/* > + * __cds_wfq_dequeue_blocking: caller ensures mutual exclusion of dequeue > + * and splice operations. > + */ > +extern struct cds_wfq_node *__cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q); > + > +/* > + * __cds_wfq_splice_blocking: caller ensures mutual exclusion of dequeue and > + * splice operations on src_q. dest_q must be already initialized. > + */ > +extern void __cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, > + struct cds_wfq_queue *src_q); > + > +/* > + * __cds_wfq_first_blocking: mutual exclusion with "dequeue" and > + * "splice" operations must be ensured by the caller. > + */ > +extern struct cds_wfq_node *__cds_wfq_first_blocking_1(struct cds_wfq_queue *q); > + > +/* > + * __cds_wfq_next_blocking: mutual exclusion with "dequeue" and "splice" > + * operations must be ensured by the caller. > + */ > +extern struct cds_wfq_node *__cds_wfq_next_blocking_1(struct cds_wfq_queue *q, > + struct cds_wfq_node *node); > + > +#define cds_wfq_node_init cds_wfq_node_init_1 > +#define cds_wfq_init cds_wfq_init_1 > +#define cds_wfq_empty cds_wfq_empty_1 > +#define cds_wfq_enqueue cds_wfq_enqueue_1 > + > +/* Locking performed within cds_wfq calls. */ > +#define cds_wfq_dequeue_blocking cds_wfq_dequeue_blocking_1 > +#define cds_wfq_splice_blocking cds_wfq_splice_blocking_1 > +#define cds_wfq_first_blocking cds_wfq_first_blocking_1 > +#define cds_wfq_next_blocking cds_wfq_next_blocking_1 > + > +/* Locking ensured by caller */ > +#define __cds_wfq_dequeue_blocking __cds_wfq_dequeue_blocking_1 > +#define __cds_wfq_splice_blocking __cds_wfq_splice_blocking_1 > +#define __cds_wfq_first_blocking __cds_wfq_first_blocking_1 > +#define __cds_wfq_next_blocking __cds_wfq_next_blocking_1 > > #endif /* !_LGPL_SOURCE */ > > +/* > + * __cds_wfq_for_each_blocking: Iterate over all nodes in a queue, > + * without dequeuing them. > + * > + * Mutual exclusion with "dequeue" and "splice" operations must be > + * ensured by the caller. > + */ > +#define __cds_wfq_for_each_blocking(q, node) \ > + for (node = __cds_wfq_first_blocking(q); \ > + node != NULL; \ > + node = __cds_wfq_next_blocking(q, node)) > + > +/* > + * __cds_wfq_for_each_blocking_safe: Iterate over all nodes in a queue, > + * without dequeuing them. Safe against deletion. > + * > + * Mutual exclusion with "dequeue" and "splice" operations must be > + * ensured by the caller. > + */ > +#define __cds_wfq_for_each_blocking_safe(q, node, n) \ > + for (node = __cds_wfq_first_blocking(q), \ > + n = (node ? __cds_wfq_next_blocking(q, node) : NULL); \ > + node != NULL; \ > + node = n, n = (node ? __cds_wfq_next_blocking(q, node) : NULL)) > + > #ifdef __cplusplus > } > #endif > diff --git a/wfqueue.c b/wfqueue.c > index 3337171..b5fba7b 100644 > --- a/wfqueue.c > +++ b/wfqueue.c > @@ -3,7 +3,8 @@ > * > * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue > * > - * Copyright 2010 - Mathieu Desnoyers > + * Copyright 2010-2012 - Mathieu Desnoyers > + * Copyright 2011-2012 - Lai Jiangshan > * > * This library is free software; you can redistribute it and/or > * modify it under the terms of the GNU Lesser General Public > @@ -28,27 +29,55 @@ > * library wrappers to be used by non-LGPL compatible source code. > */ > > -void cds_wfq_node_init(struct cds_wfq_node *node) > +void cds_wfq_node_init_1(struct cds_wfq_node *node) > { > _cds_wfq_node_init(node); > } > > -void cds_wfq_init(struct cds_wfq_queue *q) > +void cds_wfq_init_1(struct cds_wfq_queue *q) > { > _cds_wfq_init(q); > } > > -void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node) > +bool cds_wfq_empty_1(struct cds_wfq_queue *q) > +{ > + return _cds_wfq_empty(q); > +} > + > +void cds_wfq_enqueue_1(struct cds_wfq_queue *q, struct cds_wfq_node *node) > { > _cds_wfq_enqueue(q, node); > } > > -struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > +struct cds_wfq_node *cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q) > +{ > + return _cds_wfq_dequeue_blocking(q); > +} > + > +void cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, > + struct cds_wfq_queue *src_q) > +{ > + _cds_wfq_splice_blocking(dest_q, src_q); > +} > + > +struct cds_wfq_node *__cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q) > { > return ___cds_wfq_dequeue_blocking(q); > } > > -struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > +void __cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, > + struct cds_wfq_queue *src_q) > { > - return _cds_wfq_dequeue_blocking(q); > + ___cds_wfq_splice_blocking(dest_q, src_q); > +} > + > +struct cds_wfq_node *__cds_wfq_first_blocking_1(struct cds_wfq_queue *q) > +{ > + return ___cds_wfq_first_blocking(q); > +} > + > +struct cds_wfq_node *__cds_wfq_next_blocking_1(struct cds_wfq_queue *q, > + struct cds_wfq_node *node) > +{ > + return ___cds_wfq_next_blocking(q, node); > } > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Fri Aug 17 09:37:13 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 17 Aug 2012 09:37:13 -0400 Subject: [lttng-dev] [RFC URCU PATCH] wfqueue: ABI v1, simplify implementation, 2.3x to 2.6x performance boost (v2) Message-ID: <20120817133713.GA15217@Krystal> This work is derived from the patch from Lai Jiangshan submitted as "urcu: new wfqueue implementation" (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) Its changelog: > Some guys would be surprised by this fact: > There are already TWO implementations of wfqueue in urcu. > > The first one is in urcu/static/wfqueue.h: > 1) enqueue: exchange the tail and then update previous->next > 2) dequeue: wait for first node's next pointer and them shift, a dummy node > is introduced to avoid the queue->tail become NULL when shift. > > The second one shares some code with the first one, and the left code > are spreading in urcu-call-rcu-impl.h: > 1) enqueue: share with the first one > 2) no dequeue operation: and no shift, so it don't need dummy node, > Although the dummy node is queued when initialization, but it is removed > after the first dequeue_all operation in call_rcu_thread(). > call_rcu_data_free() forgets to handle the dummy node if it is not removed. > 3)dequeue_all: record the old head and tail, and queue->head become the special > tail node.(atomic record the tail and change the tail). > > The second implementation's code are spreading, bad for review, and it is not > tested by tests/test_urcu_wfq. > > So we need a better implementation avoid the dummy node dancing and can service > both generic wfqueue APIs and dequeue_all API for call rcu. > > The new implementation: > 1) enqueue: share with the first one/original implementation. > 2) dequeue: shift when node count >= 2, cmpxchg when node count = 1. > no dummy node, save memory. > 3) dequeue_all: simply set queue->head.next to NULL, xchg the tail > and return the old head.next. > > More implementation details are in the code. > tests/test_urcu_wfq will be update in future for testing new APIs. The patch proposed by Lai brings a very interesting simplification to the single-node handling (which is kept here), and moves all queue handling code away from call_rcu implementation, back into the wfqueue code. This has the benefit to allow testing enhancements. I modified it so the API does not expose implementation details to the user (e.g. ___cds_wfq_node_sync_next). I added a "splice" operation and a for loop iterator which should allow wfqueue users to use the list very efficiently both from LGPL/GPL code and from non-LGPL-compatible code. Benchmarks performed on Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz (dual-core, with hyperthreading) Benchmark invoked: for a in $(seq 1 10); do ./test_urcu_wfq 1 1 10 -a 0 -a 2; done (using cpu number 0 and 2, which should correspond to two cores of my Intel 2-core/hyperthread processor) Before patch: testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 97274297 nr_dequeues 80745742 successful enqueues 97274297 successful dequeues 80745321 end_dequeues 16528976 nr_ops 178020039 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 92300568 nr_dequeues 75019529 successful enqueues 92300568 successful dequeues 74973237 end_dequeues 17327331 nr_ops 167320097 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 93516443 nr_dequeues 75846726 successful enqueues 93516443 successful dequeues 75826578 end_dequeues 17689865 nr_ops 169363169 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 94160362 nr_dequeues 77967638 successful enqueues 94160362 successful dequeues 77967638 end_dequeues 16192724 nr_ops 172128000 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 97491956 nr_dequeues 81001191 successful enqueues 97491956 successful dequeues 81000247 end_dequeues 16491709 nr_ops 178493147 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 94101298 nr_dequeues 75650510 successful enqueues 94101298 successful dequeues 75649318 end_dequeues 18451980 nr_ops 169751808 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 94742803 nr_dequeues 75402105 successful enqueues 94742803 successful dequeues 75341859 end_dequeues 19400944 nr_ops 170144908 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 92198835 nr_dequeues 75037877 successful enqueues 92198835 successful dequeues 75027605 end_dequeues 17171230 nr_ops 167236712 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 94159560 nr_dequeues 77895972 successful enqueues 94159560 successful dequeues 77858442 end_dequeues 16301118 nr_ops 172055532 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 96059399 nr_dequeues 80115442 successful enqueues 96059399 successful dequeues 80066843 end_dequeues 15992556 nr_ops 176174841 After patch: testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 221229322 nr_dequeues 210645491 successful enqueues 221229322 successful dequeues 210645088 end_dequeues 10584234 nr_ops 431874813 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 219803943 nr_dequeues 210377337 successful enqueues 219803943 successful dequeues 210368680 end_dequeues 9435263 nr_ops 430181280 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 237006358 nr_dequeues 237035340 successful enqueues 237006358 successful dequeues 236997050 end_dequeues 9308 nr_ops 474041698 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 235822443 nr_dequeues 235815942 successful enqueues 235822443 successful dequeues 235814020 end_dequeues 8423 nr_ops 471638385 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 235825567 nr_dequeues 235811803 successful enqueues 235825567 successful dequeues 235810526 end_dequeues 15041 nr_ops 471637370 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 221974953 nr_dequeues 210938190 successful enqueues 221974953 successful dequeues 210938190 end_dequeues 11036763 nr_ops 432913143 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 237994492 nr_dequeues 237938119 successful enqueues 237994492 successful dequeues 237930648 end_dequeues 63844 nr_ops 475932611 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 220634365 nr_dequeues 210491382 successful enqueues 220634365 successful dequeues 210490995 end_dequeues 10143370 nr_ops 431125747 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 237388065 nr_dequeues 237401251 successful enqueues 237388065 successful dequeues 237380295 end_dequeues 7770 nr_ops 474789316 testdur 10 nr_enqueuers 1 wdelay 0 nr_dequeuers 1 rdur 0 nr_enqueues 221201436 nr_dequeues 210831162 successful enqueues 221201436 successful dequeues 210831162 end_dequeues 10370274 nr_ops 432032598 Summary: Both enqueue and dequeue speed increase: around 2.3x speedup for enqueue, and around 2.6x for dequeue. We can verify that: successful enqueues - successful dequeues = end_dequeues For all runs (ensures correctness: no lost node). * Introduce wfqueue ABI v1 (false-sharing fix) wfqueue v0 suffers from false-sharing between head and tail. By cache-aligning head and tail, we get a significant speedup on benchmarks. But in order to do that, we need to break the ABI to enlarge struct cds_wfq_queue. Provide a backward compatibility ABI for the old wfqueue by defining the original symbols to the old implementation (which uses dummy node). Programs compiled against old headers, which are not LGPL_SOURCE, will still use the old implementation. Any code compiled against the new header will directly use the new ABI. This does not require any change in the way users call the API: the new ABI symbols are simply defined with _1 suffix, and wrapped by preprocessor macros. Known limitation: users should *not* link together objects using the v0 and v1 APIs of wfqueue and exchange struct cds_wfq_queue structures between the two. The only way to run into this corner-case would be to combine objects compiled with different versions of the urcu/wfqueue.h header. * Changes since v1 - Add documentation to each API member. - Document locking rules. - Document memory ordering guarantees. CC: Lai Jiangshan CC: Paul McKenney Signed-off-by: Mathieu Desnoyers --- Makefile.am | 2 urcu-call-rcu-impl.h | 60 ++++------ urcu/static/wfqueue.h | 273 ++++++++++++++++++++++++++++++++++++++++++-------- urcu/wfqueue.h | 188 +++++++++++++++++++++++++++++++--- wfqueue.c | 43 ++++++- 5 files changed, 467 insertions(+), 99 deletions(-) diff --git a/Makefile.am b/Makefile.am index 2396fcf..31052ef 100644 --- a/Makefile.am +++ b/Makefile.am @@ -53,7 +53,7 @@ lib_LTLIBRARIES = liburcu-common.la \ # liburcu-common contains wait-free queues (needed by call_rcu) as well # as futex fallbacks. # -liburcu_common_la_SOURCES = wfqueue.c wfstack.c $(COMPAT) +liburcu_common_la_SOURCES = wfqueue.c wfqueue0.c wfstack.c $(COMPAT) liburcu_la_SOURCES = urcu.c urcu-pointer.c $(COMPAT) liburcu_la_LIBADD = liburcu-common.la diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h index 13b24ff..d8537d0 100644 --- a/urcu-call-rcu-impl.h +++ b/urcu-call-rcu-impl.h @@ -21,6 +21,7 @@ */ #define _GNU_SOURCE +#define _LGPL_SOURCE #include #include #include @@ -220,10 +221,7 @@ static void call_rcu_wake_up(struct call_rcu_data *crdp) static void *call_rcu_thread(void *arg) { unsigned long cbcount; - struct cds_wfq_node *cbs; - struct cds_wfq_node **cbs_tail; - struct call_rcu_data *crdp = (struct call_rcu_data *)arg; - struct rcu_head *rhp; + struct call_rcu_data *crdp = (struct call_rcu_data *) arg; int rt = !!(uatomic_read(&crdp->flags) & URCU_CALL_RCU_RT); int ret; @@ -243,35 +241,30 @@ static void *call_rcu_thread(void *arg) cmm_smp_mb(); } for (;;) { - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) - poll(NULL, 0, 1); - _CMM_STORE_SHARED(crdp->cbs.head, NULL); - cbs_tail = (struct cds_wfq_node **) - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); + struct cds_wfq_queue cbs_tmp; + struct cds_wfq_node *cbs, *tmp_cbs; + + cds_wfq_init(&cbs_tmp); + __cds_wfq_splice_blocking(&cbs_tmp, &crdp->cbs); + if (!cds_wfq_empty(&cbs_tmp)) { synchronize_rcu(); cbcount = 0; - do { - while (cbs->next == NULL && - &cbs->next != cbs_tail) - poll(NULL, 0, 1); - if (cbs == &crdp->cbs.dummy) { - cbs = cbs->next; - continue; - } - rhp = (struct rcu_head *)cbs; - cbs = cbs->next; + __cds_wfq_for_each_blocking_safe(&cbs_tmp, + cbs, tmp_cbs) { + struct rcu_head *rhp; + + rhp = caa_container_of(cbs, + struct rcu_head, next); rhp->func(rhp); cbcount++; - } while (cbs != NULL); + } uatomic_sub(&crdp->qlen, cbcount); } if (uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOP) break; rcu_thread_offline(); if (!rt) { - if (&crdp->cbs.head - == _CMM_LOAD_SHARED(crdp->cbs.tail)) { + if (cds_wfq_empty(&crdp->cbs)) { call_rcu_wait(crdp); poll(NULL, 0, 10); uatomic_dec(&crdp->futex); @@ -625,32 +618,27 @@ void call_rcu(struct rcu_head *head, */ void call_rcu_data_free(struct call_rcu_data *crdp) { - struct cds_wfq_node *cbs; - struct cds_wfq_node **cbs_tail; - struct cds_wfq_node **cbs_endprev; - if (crdp == NULL || crdp == default_call_rcu_data) { return; } + if ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) { uatomic_or(&crdp->flags, URCU_CALL_RCU_STOP); wake_call_rcu_thread(crdp); while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) poll(NULL, 0, 1); } - if (&crdp->cbs.head != _CMM_LOAD_SHARED(crdp->cbs.tail)) { - while ((cbs = _CMM_LOAD_SHARED(crdp->cbs.head)) == NULL) - poll(NULL, 0, 1); - _CMM_STORE_SHARED(crdp->cbs.head, NULL); - cbs_tail = (struct cds_wfq_node **) - uatomic_xchg(&crdp->cbs.tail, &crdp->cbs.head); + + if (!cds_wfq_empty(&crdp->cbs)) { /* Create default call rcu data if need be */ (void) get_default_call_rcu_data(); - cbs_endprev = (struct cds_wfq_node **) - uatomic_xchg(&default_call_rcu_data, cbs_tail); - *cbs_endprev = cbs; + + __cds_wfq_splice_blocking(&default_call_rcu_data->cbs, + &crdp->cbs); + uatomic_add(&default_call_rcu_data->qlen, uatomic_read(&crdp->qlen)); + wake_call_rcu_thread(default_call_rcu_data); } diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h index 636e1af..7cafec8 100644 --- a/urcu/static/wfqueue.h +++ b/urcu/static/wfqueue.h @@ -9,7 +9,8 @@ * TO BE INCLUDED ONLY IN LGPL-COMPATIBLE CODE. See wfqueue.h for linking * dynamically with the userspace rcu library. * - * Copyright 2010 - Mathieu Desnoyers + * Copyright 2010-2012 - Mathieu Desnoyers + * Copyright 2011-2012 - Lai Jiangshan * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -29,6 +30,7 @@ #include #include #include +#include #include #include @@ -38,54 +40,106 @@ extern "C" { /* * Queue with wait-free enqueue/blocking dequeue. - * This implementation adds a dummy head node when the queue is empty to ensure - * we can always update the queue locklessly. * * Inspired from half-wait-free/half-blocking queue implementation done by * Paul E. McKenney. + * + * Mutual exclusion of __cds_wfq_* API + * + * Caller must ensure mutual exclusion of queue update operations + * "dequeue" and "splice" (for source queue). Queue read operations + * "first" and "next" need to be protected against concurrent "dequeue" + * and "splice" (for source queue) by the caller. "enqueue", "splice" + * (for destination queue), and "empty" are the only operations that can + * be used without any mutual exclusion. + * + * For convenience, cds_wfq_dequeue_blocking() and + * cds_wfq_splice_blocking() provide their own locking, but they must + * not be used concurrently with __cds_wfq_* API members. */ #define WFQ_ADAPT_ATTEMPTS 10 /* Retry if being set */ #define WFQ_WAIT 10 /* Wait 10 ms if being set */ +/* + * cds_wfq_node_init: initialize wait-free queue node. + */ static inline void _cds_wfq_node_init(struct cds_wfq_node *node) { node->next = NULL; } +/* + * cds_wfq_init: initialize wait-free queue. + */ static inline void _cds_wfq_init(struct cds_wfq_queue *q) { int ret; - _cds_wfq_node_init(&q->dummy); /* Set queue head and tail */ - q->head = &q->dummy; - q->tail = &q->dummy.next; - ret = pthread_mutex_init(&q->lock, NULL); + _cds_wfq_node_init(&q->dequeue.head); + q->enqueue.tail = &q->dequeue.head; + ret = pthread_mutex_init(&q->dequeue.lock, NULL); assert(!ret); } -static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, - struct cds_wfq_node *node) +/* + * cds_wfq_empty: return whether wait-free queue is empty. + * + * No memory barrier is issued. No mutual exclusion is required. + */ +static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) { - struct cds_wfq_node **old_tail; + /* + * Queue is empty if no node is pointed by q->head.next nor + * q->tail. Even though the q->tail check is sufficient to find + * out of the queue is empty, we first check q->head.next as a + * common case to ensure that dequeuers do not frequently access + * enqueuer's q->tail cache line. + */ + return CMM_LOAD_SHARED(q->dequeue.head.next) == NULL + && CMM_LOAD_SHARED(q->enqueue.tail) == &q->dequeue.head; +} + +static inline void ___cds_wfq_append(struct cds_wfq_queue *q, + struct cds_wfq_node *new_head, + struct cds_wfq_node *new_tail) +{ + struct cds_wfq_node *old_tail; /* - * uatomic_xchg() implicit memory barrier orders earlier stores to data - * structure containing node and setting node->next to NULL before - * publication. + * Implicit memory barrier before uatomic_xchg() orders earlier + * stores to data structure containing node and setting + * node->next to NULL before publication. */ - old_tail = uatomic_xchg(&q->tail, &node->next); + old_tail = uatomic_xchg(&q->enqueue.tail, new_tail); + /* - * At this point, dequeuers see a NULL old_tail->next, which indicates - * that the queue is being appended to. The following store will append - * "node" to the queue from a dequeuer perspective. + * Implicit memory barrier after uatomic_xchg() orders store to + * q->tail before store to old_tail->next. + * + * At this point, dequeuers see a NULL q->tail->next, which + * indicates that the queue is being appended to. The following + * store will append "node" to the queue from a dequeuer + * perspective. */ - CMM_STORE_SHARED(*old_tail, node); + CMM_STORE_SHARED(old_tail->next, new_head); +} + +/* + * cds_wfq_enqueue: enqueue a node into a wait-free queue. + * + * Issues a full memory barrier before enqueue. No mutual exclusion is + * required. + */ +static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q, + struct cds_wfq_node *new_tail) +{ + ___cds_wfq_append(q, new_tail, new_tail); } /* - * Waiting for enqueuer to complete enqueue and return the next node + * Waiting for enqueuer to complete enqueue and return the next node. */ static inline struct cds_wfq_node * ___cds_wfq_node_sync_next(struct cds_wfq_node *node) @@ -100,62 +154,201 @@ ___cds_wfq_node_sync_next(struct cds_wfq_node *node) if (++attempt >= WFQ_ADAPT_ATTEMPTS) { poll(NULL, 0, WFQ_WAIT); /* Wait for 10ms */ attempt = 0; - } else + } else { caa_cpu_relax(); + } } return next; } /* - * It is valid to reuse and free a dequeued node immediately. + * __cds_wfq_first_blocking: get first node of a queue, without dequeuing. + * + * Content written into the node before enqueue is guaranteed to be + * consistent, but no other memory ordering is ensured. + * Mutual exclusion must be ensured by the caller. + */ +static inline struct cds_wfq_node * +___cds_wfq_first_blocking(struct cds_wfq_queue *q) +{ + struct cds_wfq_node *node; + + if (_cds_wfq_empty(q)) + return NULL; + node = ___cds_wfq_node_sync_next(&q->dequeue.head); + /* Load q->head.next before loading node's content */ + cmm_smp_read_barrier_depends(); + return node; +} + +/* + * __cds_wfq_next_blocking: get next node of a queue, without dequeuing. + * + * Content written into the node before enqueue is guaranteed to be + * consistent, but no other memory ordering is ensured. + * Mutual exclusion must be ensured by the caller. + */ +static inline struct cds_wfq_node * +___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) +{ + struct cds_wfq_node *next; + + /* + * Even though the following q->tail check is sufficient to find + * out if we reached the end of the queue, we first check + * node->next as a common case to ensure that iteration on nodes + * do not frequently access enqueuer's q->tail cache line. + */ + if ((next = CMM_LOAD_SHARED(node->next)) != NULL) { + /* Load node->next before loading next's content */ + cmm_smp_read_barrier_depends(); + return next; + } + /* Load node->next before q->tail */ + cmm_smp_rmb(); + if (CMM_LOAD_SHARED(q->enqueue.tail) == node) + return NULL; + next = ___cds_wfq_node_sync_next(node); + /* Load node->next before loading next's content */ + cmm_smp_read_barrier_depends(); + return next; +} + +/* + * __cds_wfq_dequeue_blocking: dequeue a node from the queue. * * No need to go on a waitqueue here, as there is no possible state in which the * list could cause dequeue to busy-loop needlessly while waiting for another * thread to be scheduled. The queue appears empty until tail->next is set by * enqueue. + * + * Content written into the node before enqueue is guaranteed to be + * consistent, but no other memory ordering is ensured. + * Mutual exclusion must be ensured by the caller. + * It is valid to reuse and free a dequeued node immediately. */ static inline struct cds_wfq_node * ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) { struct cds_wfq_node *node, *next; - /* - * Queue is empty if it only contains the dummy node. - */ - if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next) + if (_cds_wfq_empty(q)) return NULL; - node = q->head; - next = ___cds_wfq_node_sync_next(node); + node = ___cds_wfq_node_sync_next(&q->dequeue.head); + + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { + /* + * @node is probably the only node in the queue. + * Try to move the tail to &q->head. + * q->head.next is set to NULL here, and stays + * NULL if the cmpxchg succeeds. Should the + * cmpxchg fail due to a concurrent enqueue, the + * q->head.next will be set to the next node. + * The implicit memory barrier before + * uatomic_cmpxchg() orders load node->next + * before loading q->tail. + * The implicit memory barrier before uatomic_cmpxchg + * orders load q->head.next before loading node's + * content. + */ + _cds_wfq_node_init(&q->dequeue.head); + if (uatomic_cmpxchg(&q->enqueue.tail, node, + &q->dequeue.head) == node) + return node; + next = ___cds_wfq_node_sync_next(node); + } /* * Move queue head forward. */ - q->head = next; + q->dequeue.head.next = next; + + /* Load q->head.next before loading node's content */ + cmm_smp_read_barrier_depends(); + return node; +} + +/* + * __cds_wfq_splice_blocking: enqueue all src_q nodes at the end of dest_q. + * + * Dequeue all nodes from src_q. + * dest_q must be already initialized. + * caller ensures mutual exclusion of dequeue and splice operations on + * src_q. + */ +static inline void +___cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q) +{ + struct cds_wfq_node *head, *tail; + + if (_cds_wfq_empty(src_q)) + return; + + head = ___cds_wfq_node_sync_next(&src_q->dequeue.head); + _cds_wfq_node_init(&src_q->dequeue.head); + /* - * Requeue dummy node if we just dequeued it. + * Memory barrier implied before uatomic_xchg() orders store to + * src_q->head before store to src_q->tail. This is required by + * concurrent enqueue on src_q, which exchanges the tail before + * updating the previous tail's next pointer. */ - if (node == &q->dummy) { - _cds_wfq_node_init(node); - _cds_wfq_enqueue(q, node); - return ___cds_wfq_dequeue_blocking(q); - } - return node; + tail = uatomic_xchg(&src_q->enqueue.tail, &src_q->dequeue.head); + + /* + * Append the spliced content of src_q into dest_q. Does not + * require mutual exclusion on dest_q (wait-free). + */ + ___cds_wfq_append(dest_q, head, tail); } +/* + * cds_wfq_dequeue_blocking: dequeue a node from a wait-free queue. + * + * Content written into the node before enqueue is guaranteed to be + * consistent, but no other memory ordering is ensured. + * Mutual exlusion with (and only with) cds_wfq_splice_blocking is + * ensured. + * It is valid to reuse and free a dequeued node immediately. + */ static inline struct cds_wfq_node * _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) { - struct cds_wfq_node *retnode; + struct cds_wfq_node *retval; + int ret; + + ret = pthread_mutex_lock(&q->dequeue.lock); + assert(!ret); + retval = ___cds_wfq_dequeue_blocking(q); + ret = pthread_mutex_unlock(&q->dequeue.lock); + assert(!ret); + return retval; +} + +/* + * cds_wfq_splice_blocking: enqueue all src_q nodes at the end of dest_q. + * + * Dequeue all nodes from src_q. + * dest_q must be already initialized. + * Content written into the node before enqueue is guaranteed to be + * consistent, but no other memory ordering is ensured. + * Mutual exlusion with (and only with) cds_wfq_dequeue_blocking is + * ensured. + */ +static inline void +_cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q) +{ int ret; - ret = pthread_mutex_lock(&q->lock); + ret = pthread_mutex_lock(&src_q->dequeue.lock); assert(!ret); - retnode = ___cds_wfq_dequeue_blocking(q); - ret = pthread_mutex_unlock(&q->lock); + ___cds_wfq_splice_blocking(dest_q, src_q); + ret = pthread_mutex_unlock(&src_q->dequeue.lock); assert(!ret); - return retnode; } #ifdef __cplusplus diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h index 03a73f1..b9f7eb6 100644 --- a/urcu/wfqueue.h +++ b/urcu/wfqueue.h @@ -6,7 +6,8 @@ * * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue * - * Copyright 2010 - Mathieu Desnoyers + * Copyright 2010-2012 - Mathieu Desnoyers + * Copyright 2011-2012 - Lai Jiangshan * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -25,7 +26,9 @@ #include #include +#include #include +#include #ifdef __cplusplus extern "C" { @@ -33,8 +36,6 @@ extern "C" { /* * Queue with wait-free enqueue/blocking dequeue. - * This implementation adds a dummy head node when the queue is empty to ensure - * we can always update the queue locklessly. * * Inspired from half-wait-free/half-blocking queue implementation done by * Paul E. McKenney. @@ -45,9 +46,13 @@ struct cds_wfq_node { }; struct cds_wfq_queue { - struct cds_wfq_node *head, **tail; - struct cds_wfq_node dummy; /* Dummy node */ - pthread_mutex_t lock; + struct { + struct cds_wfq_node head; + pthread_mutex_t lock; + } __attribute__((aligned((CAA_CACHE_LINE_SIZE)))) dequeue; + struct { + struct cds_wfq_node *tail; + } __attribute__((aligned((CAA_CACHE_LINE_SIZE)))) enqueue; }; #ifdef _LGPL_SOURCE @@ -55,22 +60,175 @@ struct cds_wfq_queue { #include #define cds_wfq_node_init _cds_wfq_node_init -#define cds_wfq_init _cds_wfq_init -#define cds_wfq_enqueue _cds_wfq_enqueue -#define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking +#define cds_wfq_init _cds_wfq_init +#define cds_wfq_empty _cds_wfq_empty +#define cds_wfq_enqueue _cds_wfq_enqueue + +/* Locking performed within cds_wfq calls. */ #define cds_wfq_dequeue_blocking _cds_wfq_dequeue_blocking +#define cds_wfq_splice_blocking _cds_wfq_splice_blocking +#define cds_wfq_first_blocking _cds_wfq_first_blocking +#define cds_wfq_next_blocking _cds_wfq_next_blocking + +/* Locking ensured by caller */ +#define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking +#define __cds_wfq_splice_blocking ___cds_wfq_splice_blocking +#define __cds_wfq_first_blocking ___cds_wfq_first_blocking +#define __cds_wfq_next_blocking ___cds_wfq_next_blocking #else /* !_LGPL_SOURCE */ -extern void cds_wfq_node_init(struct cds_wfq_node *node); -extern void cds_wfq_init(struct cds_wfq_queue *q); -extern void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node); -/* __cds_wfq_dequeue_blocking: caller ensures mutual exclusion between dequeues */ -extern struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); -extern struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q); +/* + * Mutual exclusion of cds_wfq_* / __cds_wfq_* API + * + * Caller must ensure mutual exclusion of queue update operations + * "dequeue" and "splice" (for source queue). Queue read operations + * "first" and "next" need to be protected against concurrent "dequeue" + * and "splice" (for source queue) by the caller. "enqueue", "splice" + * (for destination queue), and "empty" are the only operations that can + * be used without any mutual exclusion. + * + * For convenience, cds_wfq_dequeue_blocking() and + * cds_wfq_splice_blocking() provide their own locking, but they must + * not be used concurrently with __cds_wfq_* API members. + */ + +/* + * cds_wfq_node_init: initialize wait-free queue node. + */ +extern void cds_wfq_node_init_1(struct cds_wfq_node *node); + +/* + * cds_wfq_init: initialize wait-free queue. + */ +extern void cds_wfq_init_1(struct cds_wfq_queue *q); + +/* + * cds_wfq_empty: return whether wait-free queue is empty. + * + * No memory barrier is issued. No mutual exclusion is required. + */ +extern bool cds_wfq_empty_1(struct cds_wfq_queue *q); + +/* + * cds_wfq_enqueue: enqueue a node into a wait-free queue. + * + * Issues a full memory barrier before enqueue. No mutual exclusion is + * required. + */ +extern void cds_wfq_enqueue_1(struct cds_wfq_queue *q, struct cds_wfq_node *node); + +/* + * cds_wfq_dequeue_blocking: dequeue a node from a wait-free queue. + * + * Content written into the node before enqueue is guaranteed to be + * consistent, but no other memory ordering is ensured. + * Mutual exlusion with (and only with) cds_wfq_splice_blocking is + * ensured. + * It is valid to reuse and free a dequeued node immediately. + */ +extern struct cds_wfq_node *cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q); + +/* + * cds_wfq_splice_blocking: enqueue all src_q nodes at the end of dest_q. + * + * Dequeue all nodes from src_q. + * dest_q must be already initialized. + * Content written into the node before enqueue is guaranteed to be + * consistent, but no other memory ordering is ensured. + * Mutual exlusion with (and only with) cds_wfq_dequeue_blocking is + * ensured. + */ +extern void cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q); + +/* + * __cds_wfq_dequeue_blocking: + * + * Content written into the node before enqueue is guaranteed to be + * consistent, but no other memory ordering is ensured. + * Mutual exclusion must be ensured by the caller. + * It is valid to reuse and free a dequeued node immediately. + */ +extern struct cds_wfq_node *__cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q); + +/* + * __cds_wfq_splice_blocking: enqueue all src_q nodes at the end of dest_q. + * + * Dequeue all nodes from src_q. + * dest_q must be already initialized. + * Content written into the node before enqueue is guaranteed to be + * consistent, but no other memory ordering is ensured. + * Mutual exclusion must be ensured by the caller. + */ +extern void __cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q); + +/* + * __cds_wfq_first_blocking: get first node of a queue, without dequeuing. + * + * Content written into the node before enqueue is guaranteed to be + * consistent, but no other memory ordering is ensured. + * Mutual exclusion must be ensured by the caller. + */ +extern struct cds_wfq_node *__cds_wfq_first_blocking_1(struct cds_wfq_queue *q); + +/* + * __cds_wfq_next_blocking: get next node of a queue, without dequeuing. + * + * Content written into the node before enqueue is guaranteed to be + * consistent, but no other memory ordering is ensured. + * Mutual exclusion must be ensured by the caller. + */ +extern struct cds_wfq_node *__cds_wfq_next_blocking_1(struct cds_wfq_queue *q, + struct cds_wfq_node *node); + +#define cds_wfq_node_init cds_wfq_node_init_1 +#define cds_wfq_init cds_wfq_init_1 +#define cds_wfq_empty cds_wfq_empty_1 +#define cds_wfq_enqueue cds_wfq_enqueue_1 + +/* Locking performed within cds_wfq calls. */ +#define cds_wfq_dequeue_blocking cds_wfq_dequeue_blocking_1 +#define cds_wfq_splice_blocking cds_wfq_splice_blocking_1 +#define cds_wfq_first_blocking cds_wfq_first_blocking_1 +#define cds_wfq_next_blocking cds_wfq_next_blocking_1 + +/* Locking ensured by caller */ +#define __cds_wfq_dequeue_blocking __cds_wfq_dequeue_blocking_1 +#define __cds_wfq_splice_blocking __cds_wfq_splice_blocking_1 +#define __cds_wfq_first_blocking __cds_wfq_first_blocking_1 +#define __cds_wfq_next_blocking __cds_wfq_next_blocking_1 #endif /* !_LGPL_SOURCE */ +/* + * __cds_wfq_for_each_blocking: Iterate over all nodes in a queue, + * without dequeuing them. + * + * Content written into each node before enqueue is guaranteed to be + * consistent, but no other memory ordering is ensured. + * Mutual exclusion must be ensured by the caller. + */ +#define __cds_wfq_for_each_blocking(q, node) \ + for (node = __cds_wfq_first_blocking(q); \ + node != NULL; \ + node = __cds_wfq_next_blocking(q, node)) + +/* + * __cds_wfq_for_each_blocking_safe: Iterate over all nodes in a queue, + * without dequeuing them. Safe against deletion. + * + * Content written into each node before enqueue is guaranteed to be + * consistent, but no other memory ordering is ensured. + * Mutual exclusion must be ensured by the caller. + */ +#define __cds_wfq_for_each_blocking_safe(q, node, n) \ + for (node = __cds_wfq_first_blocking(q), \ + n = (node ? __cds_wfq_next_blocking(q, node) : NULL); \ + node != NULL; \ + node = n, n = (node ? __cds_wfq_next_blocking(q, node) : NULL)) + #ifdef __cplusplus } #endif diff --git a/wfqueue.c b/wfqueue.c index 3337171..b5fba7b 100644 --- a/wfqueue.c +++ b/wfqueue.c @@ -3,7 +3,8 @@ * * Userspace RCU library - Queue with Wait-Free Enqueue/Blocking Dequeue * - * Copyright 2010 - Mathieu Desnoyers + * Copyright 2010-2012 - Mathieu Desnoyers + * Copyright 2011-2012 - Lai Jiangshan * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -28,27 +29,55 @@ * library wrappers to be used by non-LGPL compatible source code. */ -void cds_wfq_node_init(struct cds_wfq_node *node) +void cds_wfq_node_init_1(struct cds_wfq_node *node) { _cds_wfq_node_init(node); } -void cds_wfq_init(struct cds_wfq_queue *q) +void cds_wfq_init_1(struct cds_wfq_queue *q) { _cds_wfq_init(q); } -void cds_wfq_enqueue(struct cds_wfq_queue *q, struct cds_wfq_node *node) +bool cds_wfq_empty_1(struct cds_wfq_queue *q) +{ + return _cds_wfq_empty(q); +} + +void cds_wfq_enqueue_1(struct cds_wfq_queue *q, struct cds_wfq_node *node) { _cds_wfq_enqueue(q, node); } -struct cds_wfq_node *__cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) +struct cds_wfq_node *cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q) +{ + return _cds_wfq_dequeue_blocking(q); +} + +void cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q) +{ + _cds_wfq_splice_blocking(dest_q, src_q); +} + +struct cds_wfq_node *__cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q) { return ___cds_wfq_dequeue_blocking(q); } -struct cds_wfq_node *cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) +void __cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, + struct cds_wfq_queue *src_q) { - return _cds_wfq_dequeue_blocking(q); + ___cds_wfq_splice_blocking(dest_q, src_q); +} + +struct cds_wfq_node *__cds_wfq_first_blocking_1(struct cds_wfq_queue *q) +{ + return ___cds_wfq_first_blocking(q); +} + +struct cds_wfq_node *__cds_wfq_next_blocking_1(struct cds_wfq_queue *q, + struct cds_wfq_node *node) +{ + return ___cds_wfq_next_blocking(q, node); } -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Fri Aug 17 11:08:06 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 17 Aug 2012 11:08:06 -0400 Subject: [lttng-dev] More LTTng problems In-Reply-To: References: Message-ID: <20120817150806.GA15919@Krystal> It looks like you did not build a 32-bit consumer. I'm ccing David Goulet, maintainer of lttng-tools. Thanks, Mathieu * Henrik Hautakoski (henrik at fiktivkod.org) wrote: > Hi. got lttng working now. but when given the following command, we > get an error. > > $ lttng enable-event -a -u > Error: Events: 32-bit UST consumer start failed (channel channel0, > session mysession) > > Could not find any information about this error massage. do you have any ideas? > > -- > Henrik Hautakoski > henrik at fiktivkod.org -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From dgoulet at efficios.com Fri Aug 17 11:15:46 2012 From: dgoulet at efficios.com (David Goulet) Date: Fri, 17 Aug 2012 11:15:46 -0400 Subject: [lttng-dev] More LTTng problems In-Reply-To: <20120817150806.GA15919@Krystal> References: <20120817150806.GA15919@Krystal> Message-ID: <502E6022.60208@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, Actually, this error means that a problem occurred *during* the 32 bit consumer spawning process... It seems the binary was found but failed to exec it. Running the session daemon (lttng-sessiond) with options "-vvv - --verbose-consumer" will help us pin down the problem. Send me back the output, I'll be able to tell you what's going on (and also the series of command you guys did before the enable-event). Last thing, make sure the lttng-consumerd did not segfault (dmesg) or any other lttng-tools component. Thanks! David Mathieu Desnoyers: > It looks like you did not build a 32-bit consumer. I'm ccing David > Goulet, maintainer of lttng-tools. > > Thanks, > > Mathieu > > * Henrik Hautakoski (henrik at fiktivkod.org) wrote: >> Hi. got lttng working now. but when given the following command, >> we get an error. >> >> $ lttng enable-event -a -u Error: Events: 32-bit UST consumer >> start failed (channel channel0, session mysession) >> >> Could not find any information about this error massage. do you >> have any ideas? >> >> -- Henrik Hautakoski henrik at fiktivkod.org > -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQLmAfAAoJEELoaioR9I02rqEH/2w2QXXmpLE0iEYnl2FMrfYq 5e6p0TH1Qb4NSKvyAuSOfFJYWpNXjAKKDC/ajUuEH5PZiFtPZqiNK/jQQcTwns9t dwZb1WSLm4Ue5z7jRV3yk8Gae/IaFQqbS5l3tTdKkE7rFOVO9Iew5+BPORHtBYA3 Bvmx9CcwxeOVizKMab5Y0JK9RbuFcnurUV33tUI9DAisloutrItm84GdKyE5wZ92 mXaneN+a0PhI68GsAmBq7dJMdjo+XABurr8oVL/kdWvUo8T1Sn298b5ki+se4zLL kIfAXv7x+5K8tp6jnXznG6COsjbxf7Zt3nXouyled+oKr7muhGBDi5ul1r3W9IM= =p8bq -----END PGP SIGNATURE----- From dgoulet at efficios.com Fri Aug 17 16:11:58 2012 From: dgoulet at efficios.com (David Goulet) Date: Fri, 17 Aug 2012 16:11:58 -0400 Subject: [lttng-dev] [RELEASE] LTTng-tools 2.1.0-rc1 Message-ID: <502EA58E.2060003@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Greetings everyone (including LTTng elves), The lttng-tools project provides a session daemon (lttng-sessiond) that acts as a tracing registry, the "lttng" command line for tracing control, a lttng-ctl library for tracing control and a lttng-relayd for network streaming. This is the first release candidate for lttng-tools 2.1 which brings two exciting new features, network streaming and filtering support. It's the combination of a lot of work from the LTTng team so please, to help us improve our tools, report any bugs or misbehaving feature(s) that you may encounter through this mailing list or the bug tracker (https://bugs.lttng.org). - From now on, lttng-tools is in feature freeze mode unless an *IMMENSE* show stopper is found. 2012-08-17 lttng-tools 2.1.0-rc1 * Feature: Network Streaming * Add the lttng-relayd binary for network streaming * Support user space tracer filtering * Multiple fixes Project website: http://lttng.org/lttng2.0 Download link: http://lttng.org/files/lttng-tools/lttng-tools-2.1.0-rc1.tar.bz2 Cheers! David -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQLqWLAAoJEELoaioR9I023WUH+gNuU+NL0gZInlFbFiqGG5n8 DyvtZaUOtHr7ZoSU2cmEcNfgxzOj0jTt5lGPm7EaU7X5OUiQCiu8gz5kKsL0h5PW UPZ7DbGHxv8JQQCt/Ac6cydlOjbRT1fB/5GXGT+OX+g/Xzp+Utb6vrTCWCC0RvSO 01U2NK2+IWTuRTol7ykK1WWqTStQSKPZ6ncdYjgq1FJu2gakQA4RnhblMbXWCuP9 EZPxafSaH19mHlhDiDmtdTl7hcWTPAd0u+qNoq7kmh+stcNSn2zjv/bNldbNfaHI m0kfl8NferiyRpyeD4CgNtFE5W4jm29NId7IRI14/Ow8psZegF8SNlAL76Puqo8= =PU4p -----END PGP SIGNATURE----- From y_trottier at hotmail.ca Fri Aug 17 16:17:18 2012 From: y_trottier at hotmail.ca (Yann Trottier) Date: Fri, 17 Aug 2012 16:17:18 -0400 Subject: [lttng-dev] LTTng video tutorials Message-ID: The LTTng youtube channel has been updated with several video tutorials providing step-by-step instructions on how to set up the latest versions of LTTng and of its components. The tutorials cover the following material: - Installing LTTng on Ubuntu (youtu.be/K31ORI1pvXw) - Installing LTTng from its source code (youtu.be/xKaTXahSW5A) - Installing the stable release of TMF (youtu.be/OInQHc4HeF8) - Installing the development snapshot of TMF (youtu.be/uzZ7Jb6piUs) - Creating a tracing session using the command-line (youtu.be/lQnin_xNKmA) - Brief overview of TMF and of its features (youtu.be/AdffSOmSTQY) All videos can be watched on the LTTng youtube channel. (http://youtube.com/lttng) -------------- next part -------------- An HTML attachment was scrubbed... URL: From laijs at cn.fujitsu.com Mon Aug 20 05:37:17 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Mon, 20 Aug 2012 17:37:17 +0800 Subject: [lttng-dev] [RFC URCU PATCH] wfqueue: ABI v1, simplify implementation, 2.3x to 2.6x performance boost In-Reply-To: <20120815213107.GB17224@Krystal> References: <20120815213107.GB17224@Krystal> Message-ID: <5032054D.7050508@cn.fujitsu.com> On 08/16/2012 05:31 AM, Mathieu Desnoyers wrote: > This work is derived from the patch from Lai Jiangshan submitted as > "urcu: new wfqueue implementation" > (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) > Hi, Mathieu please add this part to your patch. Changes(item5 has alternative): 1) Reorder the head and the tail in struct cds_wfq_queue Reason: wfq is enqueue-preference 2) Reorder the code of ___cds_wfq_next_blocking() Reason: short code, better readability 3) Add cds_wfq_dequeue_[un]lock Reason: the fields of struct cds_wfq_queue are not exposed to users of lib, but some APIs needs to have this lock held. Add this locking function to allow the users use such APIs 4) Add cds_wfq_node_sync_next(), cds_wfq_dequeue_all_blocking() Reason: helper for for_each_* 5) Add more for_each_*, which default semantic is dequeuing Reason: __cds_wfq_for_each_blocking_undequeue is enough for undequeuing loops don't need to add more. looping and dequeuing are the most probable use-cases. dequeuing for_each_* make the queue like a pipe. make it act as a channel.(GO language) Alternative: rename these dequeuing __cds_wfq_for_each*() to __cds_wfq_dequeue_for_each* 6) Remove __cds_wfq_for_each_blocking_safe Reason: its semantic is not clear, hard to define a well-defined semantic to it. when we use it, we have to delete none or delete all nodes, even when we delete all nodes, struct cds_wfq_queue still becomes stale while looping or after loop. 7) Rename old __cds_wfq_for_each_blocking() to __cds_wfq_for_each_blocking_undequeue() Reason: the default semantic of the other for_each_* is dequeuing. --- urcu-call-rcu-impl.h | 13 +--- urcu/static/wfqueue.h | 131 ++++++++++++++++++++++++++++-------------- urcu/wfqueue.h | 154 ++++++++++++++++++++++++++++++++++++-------------- wfqueue.c | 27 ++++++++ 4 files changed, 233 insertions(+), 92 deletions(-) diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h index d8537d0..061c2d4 100644 --- a/urcu-call-rcu-impl.h +++ b/urcu-call-rcu-impl.h @@ -241,16 +241,15 @@ static void *call_rcu_thread(void *arg) cmm_smp_mb(); } for (;;) { - struct cds_wfq_queue cbs_tmp; - struct cds_wfq_node *cbs, *tmp_cbs; + while (!cds_wfq_empty(&crdp->cbs)) { + struct cds_wfq_node *cbs, *next, *tail; - cds_wfq_init(&cbs_tmp); - __cds_wfq_splice_blocking(&cbs_tmp, &crdp->cbs); - if (!cds_wfq_empty(&cbs_tmp)) { + next = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &tail); synchronize_rcu(); cbcount = 0; - __cds_wfq_for_each_blocking_safe(&cbs_tmp, - cbs, tmp_cbs) { + + __cds_wfq_for_each_blocking_nobreak_internal( + next, cbs, next, tail) { struct rcu_head *rhp; rhp = caa_container_of(cbs, diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h index 52e452d..e5a01ec 100644 --- a/urcu/static/wfqueue.h +++ b/urcu/static/wfqueue.h @@ -84,6 +84,22 @@ static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) && CMM_LOAD_SHARED(q->enqueue.tail) == &q->dequeue.head; } +static inline void _cds_wfq_dequeue_lock(struct cds_wfq_queue *q) +{ + int ret; + + ret = pthread_mutex_lock(&q->dequeue.lock); + assert(!ret); +} + +static inline void _cds_wfq_dequeue_unlock(struct cds_wfq_queue *q) +{ + int ret; + + ret = pthread_mutex_unlock(&q->dequeue.lock); + assert(!ret); +} + static inline void ___cds_wfq_append(struct cds_wfq_queue *q, struct cds_wfq_node *new_head, struct cds_wfq_node *new_tail) @@ -139,30 +155,33 @@ ___cds_wfq_node_sync_next(struct cds_wfq_node *node) return next; } +static inline struct cds_wfq_node * +_cds_wfq_node_sync_next(struct cds_wfq_node *node) +{ + struct cds_wfq_node *next = ___cds_wfq_node_sync_next(node); + + /* Load node->next before loading next's content */ + cmm_smp_read_barrier_depends(); + return next; +} + /* * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. * - * Mutual exclusion with "dequeue" and "splice" operations must be ensured - * by the caller. + * Should be called with cds_wfq_dequeue_lock() held. */ static inline struct cds_wfq_node * ___cds_wfq_first_blocking(struct cds_wfq_queue *q) { - struct cds_wfq_node *node; - if (_cds_wfq_empty(q)) return NULL; - node = ___cds_wfq_node_sync_next(&q->dequeue.head); - /* Load q->head.next before loading node's content */ - cmm_smp_read_barrier_depends(); - return node; + return _cds_wfq_node_sync_next(&q->dequeue.head); } /* * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. * - * Mutual exclusion with "dequeue" and "splice" operations must be ensured - * by the caller. + * Should be called with cds_wfq_dequeue_lock() held. */ static inline struct cds_wfq_node * ___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) @@ -175,16 +194,13 @@ ___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) * node->next as a common case to ensure that iteration on nodes * do not frequently access enqueuer's q->tail cache line. */ - if ((next = CMM_LOAD_SHARED(node->next)) != NULL) { - /* Load node->next before loading next's content */ - cmm_smp_read_barrier_depends(); - return next; + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { + /* Load node->next before q->tail */ + cmm_smp_rmb(); + if (CMM_LOAD_SHARED(q->enqueue.tail) == node) + return NULL; + next = ___cds_wfq_node_sync_next(node); } - /* Load node->next before q->tail */ - cmm_smp_rmb(); - if (CMM_LOAD_SHARED(q->enqueue.tail) == node) - return NULL; - next = ___cds_wfq_node_sync_next(node); /* Load node->next before loading next's content */ cmm_smp_read_barrier_depends(); return next; @@ -199,6 +215,8 @@ ___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) * list could cause dequeue to busy-loop needlessly while waiting for another * thread to be scheduled. The queue appears empty until tail->next is set by * enqueue. + * + * Should be called with cds_wfq_dequeue_lock() held. */ static inline struct cds_wfq_node * ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) @@ -243,12 +261,41 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) } /* + * ___cds_wfq_dequeue_all_blocking: dequeue all nodes + * + * Dequeue all nodes from @q. + * Returns the head of the dequeued nodes which tail is saved in @ret_tail. + * Should be called with cds_wfq_dequeue_lock() held. + */ +static inline struct cds_wfq_node * +___cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, + struct cds_wfq_node **ret_tail) +{ + struct cds_wfq_node *head; + + if (_cds_wfq_empty(q)) + return NULL; + + head = ___cds_wfq_node_sync_next(&q->dequeue.head); + _cds_wfq_node_init(&q->dequeue.head); + + /* + * Memory barrier implied before uatomic_xchg() orders store to + * q->dequeue.head before store to q->enqueue.tail. This is required + * by concurrent enqueue on q, which exchanges the tail before + * updating the previous tail's next pointer. + */ + *ret_tail = uatomic_xchg(&q->enqueue.tail, &q->dequeue.head); + + return head; +} + +/* * ___cds_wfq_splice_blocking: enqueue all src_q nodes at the end of dest_q. * * Dequeue all nodes from src_q. * dest_q must be already initialized. - * caller ensures mutual exclusion of dequeue and splice operations on - * src_q. + * Should be called with cds_wfq_dequeue_lock() held. */ static inline void ___cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, @@ -256,20 +303,9 @@ ___cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, { struct cds_wfq_node *head, *tail; - if (_cds_wfq_empty(src_q)) + if ((head = ___cds_wfq_dequeue_all_blocking(src_q, &tail)) == NULL) return; - head = ___cds_wfq_node_sync_next(&src_q->dequeue.head); - _cds_wfq_node_init(&src_q->dequeue.head); - - /* - * Memory barrier implied before uatomic_xchg() orders store to - * src_q->head before store to src_q->tail. This is required by - * concurrent enqueue on src_q, which exchanges the tail before - * updating the previous tail's next pointer. - */ - tail = uatomic_xchg(&src_q->enqueue.tail, &src_q->dequeue.head); - /* * Append the spliced content of src_q into dest_q. Does not * require mutual exclusion on dest_q (wait-free). @@ -282,27 +318,34 @@ static inline struct cds_wfq_node * _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) { struct cds_wfq_node *retval; - int ret; - ret = pthread_mutex_lock(&q->dequeue.lock); - assert(!ret); + _cds_wfq_dequeue_lock(q); retval = ___cds_wfq_dequeue_blocking(q); - ret = pthread_mutex_unlock(&q->dequeue.lock); - assert(!ret); + _cds_wfq_dequeue_unlock(q); + return retval; } +static inline struct cds_wfq_node * +_cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, + struct cds_wfq_node **ret_tail) +{ + struct cds_wfq_node *head; + + _cds_wfq_dequeue_lock(q); + head = _cds_wfq_dequeue_all_blocking(q, ret_tail); + _cds_wfq_dequeue_unlock(q); + + return head; +} + static inline void _cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, struct cds_wfq_queue *src_q) { - int ret; - - ret = pthread_mutex_lock(&src_q->dequeue.lock); - assert(!ret); + _cds_wfq_dequeue_lock(src_q); ___cds_wfq_splice_blocking(dest_q, src_q); - ret = pthread_mutex_unlock(&src_q->dequeue.lock); - assert(!ret); + _cds_wfq_dequeue_unlock(src_q); } #ifdef __cplusplus diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h index 446c94c..a3627ac 100644 --- a/urcu/wfqueue.h +++ b/urcu/wfqueue.h @@ -47,12 +47,12 @@ struct cds_wfq_node { struct cds_wfq_queue { struct { + struct cds_wfq_node *tail; + } __attribute__((aligned((CAA_CACHE_LINE_SIZE)))) enqueue; + struct { struct cds_wfq_node head; pthread_mutex_t lock; } __attribute__((aligned((CAA_CACHE_LINE_SIZE)))) dequeue; - struct { - struct cds_wfq_node *tail; - } __attribute__((aligned((CAA_CACHE_LINE_SIZE)))) enqueue; }; #ifdef _LGPL_SOURCE @@ -62,71 +62,97 @@ struct cds_wfq_queue { #define cds_wfq_node_init _cds_wfq_node_init #define cds_wfq_init _cds_wfq_init #define cds_wfq_empty _cds_wfq_empty + +/* Locking */ +#define cds_wfq_dequeue_lock _cds_wfq_dequeue_lock +#define cds_wfq_dequeue_unlock _cds_wfq_dequeue_unlock + #define cds_wfq_enqueue _cds_wfq_enqueue +/* + * Helpers for for_each_*: + * cds_wfq_node_sync_next() + * cds_wfq_dequeue_all_blocking() + * __cds_wfq_dequeue_all_blocking() + * __cds_wfq_first_blocking() + * __cds_wfq_next_blocking() + */ +#define cds_wfq_node_sync_next _cds_wfq_node_sync_next + /* Locking performed within cds_wfq calls. */ #define cds_wfq_dequeue_blocking _cds_wfq_dequeue_blocking +#define cds_wfq_dequeue_all_blocking _cds_wfq_dequeue_all_blocking #define cds_wfq_splice_blocking _cds_wfq_splice_blocking -#define cds_wfq_first_blocking _cds_wfq_first_blocking -#define cds_wfq_next_blocking _cds_wfq_next_blocking -/* Locking ensured by caller */ +/* Locking ensured by caller(called with cds_wfq_dequeue_lock() held) */ #define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking +#define __cds_wfq_dequeue_all_blocking ___cds_wfq_dequeue_all_blocking #define __cds_wfq_splice_blocking ___cds_wfq_splice_blocking #define __cds_wfq_first_blocking ___cds_wfq_first_blocking #define __cds_wfq_next_blocking ___cds_wfq_next_blocking #else /* !_LGPL_SOURCE */ +/* !_LGPL_SOURCE wrappers */ + extern void cds_wfq_node_init_1(struct cds_wfq_node *node); extern void cds_wfq_init_1(struct cds_wfq_queue *q); extern bool cds_wfq_empty_1(struct cds_wfq_queue *q); + +extern void cds_wfq_dequeue_lock_1(struct cds_wfq_queue *q); +extern void cds_wfq_dequeue_unlock_1(struct cds_wfq_queue *q); + extern void cds_wfq_enqueue_1(struct cds_wfq_queue *q, struct cds_wfq_node *node); +extern struct cds_wfq_node *cds_wfq_node_sync_next_1(struct cds_wfq_node *node); + /* Locking performed within cds_wfq calls. */ extern struct cds_wfq_node *cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q); +extern struct cds_wfq_node *cds_wfq_dequeue_all_blocking_1(struct cds_wfq_queue *q, + struct cds_wfq_node **ret_tail); extern void cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, struct cds_wfq_queue *src_q); -/* - * __cds_wfq_dequeue_blocking: caller ensures mutual exclusion of dequeue - * and splice operations. - */ +/* The following functions shouled be called with cds_wfq_dequeue_lock() held */ extern struct cds_wfq_node *__cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q); - -/* - * __cds_wfq_splice_blocking: caller ensures mutual exclusion of dequeue and - * splice operations on src_q. dest_q must be already initialized. - */ +extern struct cds_wfq_node *__cds_wfq_dequeue_all_blocking_1(struct cds_wfq_queue *q, + struct cds_wfq_node **ret_tail); extern void __cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, struct cds_wfq_queue *src_q); - -/* - * __cds_wfq_first_blocking: mutual exclusion with "dequeue" and - * "splice" operations must be ensured by the caller. - */ extern struct cds_wfq_node *__cds_wfq_first_blocking_1(struct cds_wfq_queue *q); - -/* - * __cds_wfq_next_blocking: mutual exclusion with "dequeue" and "splice" - * operations must be ensured by the caller. - */ extern struct cds_wfq_node *__cds_wfq_next_blocking_1(struct cds_wfq_queue *q, struct cds_wfq_node *node); +/* !_LGPL_SOURCE APIs */ + #define cds_wfq_node_init cds_wfq_node_init_1 #define cds_wfq_init cds_wfq_init_1 #define cds_wfq_empty cds_wfq_empty_1 + +/* Locking */ +#define cds_wfq_dequeue_lock cds_wfq_dequeue_lock_1 +#define cds_wfq_dequeue_unlock cds_wfq_dequeue_unlock_1 + #define cds_wfq_enqueue cds_wfq_enqueue_1 +/* + * Helpers for for_each_*: + * cds_wfq_node_sync_next() + * cds_wfq_dequeue_all_blocking() + * __cds_wfq_dequeue_all_blocking() + * __cds_wfq_first_blocking() + * __cds_wfq_next_blocking() + */ +#define cds_wfq_node_sync_next cds_wfq_node_sync_next_1 + /* Locking performed within cds_wfq calls. */ #define cds_wfq_dequeue_blocking cds_wfq_dequeue_blocking_1 +#define cds_wfq_dequeue_all_blocking cds_wfq_dequeue_all_blocking_1 #define cds_wfq_splice_blocking cds_wfq_splice_blocking_1 -#define cds_wfq_first_blocking cds_wfq_first_blocking_1 -#define cds_wfq_next_blocking cds_wfq_next_blocking_1 -/* Locking ensured by caller */ +/* Locking ensured by caller(called with cds_wfq_dequeue_lock() held) */ #define __cds_wfq_dequeue_blocking __cds_wfq_dequeue_blocking_1 +#define __cds_wfq_dequeue_all_blocking __cds_wfq_dequeue_all_blocking_1 #define __cds_wfq_splice_blocking __cds_wfq_splice_blocking_1 #define __cds_wfq_first_blocking __cds_wfq_first_blocking_1 #define __cds_wfq_next_blocking __cds_wfq_next_blocking_1 @@ -134,29 +160,75 @@ extern struct cds_wfq_node *__cds_wfq_next_blocking_1(struct cds_wfq_queue *q, #endif /* !_LGPL_SOURCE */ /* - * __cds_wfq_for_each_blocking: Iterate over all nodes in a queue, + * for_each_* routines: Iterate over all nodes in a queue and dequeue + * the travlled nodes(except __cds_wfq_for_each_blocking_undequeue). + * + * All nodes which are un-dequeued && enqueued before the for_each_* + * will be travelled. + * + * Nodes which are enqueued after the for_each_* started MAY or may NOT + * be travelled. + */ + +/* + * __cds_wfq_for_each_blocking_undequeue: Iterate over all nodes in a queue, * without dequeuing them. * - * Mutual exclusion with "dequeue" and "splice" operations must be - * ensured by the caller. + * Should be called with cds_wfq_dequeue_lock() held. */ -#define __cds_wfq_for_each_blocking(q, node) \ +#define __cds_wfq_for_each_blocking_undequeue(q, node) \ for (node = __cds_wfq_first_blocking(q); \ node != NULL; \ node = __cds_wfq_next_blocking(q, node)) +#define __cds_wfq_for_each_blocking_nobreak_internal(all, node, next, tail) \ + for (node = all, \ + next = ((node && node != tail) ? cds_wfq_node_sync_next(node) : NULL); \ + node != NULL; \ + node = next, \ + next = ((node && node != tail) ? cds_wfq_node_sync_next(node) : NULL)) + /* - * __cds_wfq_for_each_blocking_safe: Iterate over all nodes in a queue, - * without dequeuing them. Safe against deletion. + * __cds_wfq_for_each_blocking_nobreak: Iterate over all nodes in a queue, + * and dequeue them. * - * Mutual exclusion with "dequeue" and "splice" operations must be - * ensured by the caller. + * Should not use "break;" or other statement to leave the control flow + * from the loop body, otherwise it causes un-travelled nodes leak. + * + * Should be called with cds_wfq_dequeue_lock() held. + */ +#define __cds_wfq_for_each_blocking_nobreak(q, node, next, tail) \ + __cds_wfq_for_each_blocking_nobreak_internal( \ + __cds_wfq_dequeue_all_blocking(q, &tail), node, next, tail) + +/* + * cds_wfq_for_each_blocking_nobreak: Iterate over all nodes in a queue, + * and dequeue them. + * + * Should not use "break;" or other statement to leave the control flow + * from the loop body, otherwise it causes un-travelled nodes leak. + */ +#define cds_wfq_for_each_blocking_nobreak(q, node, next, tail) \ + __cds_wfq_for_each_blocking_nobreak_internal( \ + cds_wfq_dequeue_all_blocking(q, &tail), node, next, tail) + +/* + * __cds_wfq_for_each_blocking: Iterate over all nodes in a queue, + * and dequeue them. + * + * Should be called with cds_wfq_dequeue_lock() held. + */ +#define __cds_wfq_for_each_blocking(q, node) \ + for (node = __cds_wfq_dequeue_blocking(q); node; \ + node = __cds_wfq_dequeue_blocking(q)) + +/* + * cds_wfq_for_each_blocking: Iterate over all nodes in a queue, + * and dequeue them. */ -#define __cds_wfq_for_each_blocking_safe(q, node, n) \ - for (node = __cds_wfq_first_blocking(q), \ - n = (node ? __cds_wfq_next_blocking(q, node) : NULL); \ - node != NULL; \ - node = n, n = (node ? __cds_wfq_next_blocking(q, node) : NULL)) +#define cds_wfq_for_each_blocking(q, node) \ + for (node = cds_wfq_dequeue_blocking(q); node; \ + node = cds_wfq_dequeue_blocking(q)) #ifdef __cplusplus } diff --git a/wfqueue.c b/wfqueue.c index b5fba7b..eee1fd8 100644 --- a/wfqueue.c +++ b/wfqueue.c @@ -44,16 +44,37 @@ bool cds_wfq_empty_1(struct cds_wfq_queue *q) return _cds_wfq_empty(q); } +void cds_wfq_dequeue_lock_1(struct cds_wfq_queue *q) +{ + cds_wfq_dequeue_lock(q); +} + +void cds_wfq_dequeue_unlock_1(struct cds_wfq_queue *q) +{ + cds_wfq_dequeue_unlock(q); +} + void cds_wfq_enqueue_1(struct cds_wfq_queue *q, struct cds_wfq_node *node) { _cds_wfq_enqueue(q, node); } +struct cds_wfq_node *cds_wfq_node_sync_next_1(struct cds_wfq_node *node) +{ + return _cds_wfq_node_sync_next(node); +} + struct cds_wfq_node *cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q) { return _cds_wfq_dequeue_blocking(q); } +struct cds_wfq_node *cds_wfq_dequeue_all_blocking_1(struct cds_wfq_queue *q, + struct cds_wfq_node **ret_tail) +{ + return _cds_wfq_dequeue_all_blocking(q, ret_tail); +} + void cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, struct cds_wfq_queue *src_q) { @@ -65,6 +86,12 @@ struct cds_wfq_node *__cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q) return ___cds_wfq_dequeue_blocking(q); } +struct cds_wfq_node *__cds_wfq_dequeue_all_blocking_1(struct cds_wfq_queue *q, + struct cds_wfq_node **ret_tail) +{ + return ___cds_wfq_dequeue_all_blocking(q, ret_tail); +} + void __cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, struct cds_wfq_queue *src_q) { From henrik at fiktivkod.org Mon Aug 20 09:14:56 2012 From: henrik at fiktivkod.org (Henrik Hautakoski) Date: Mon, 20 Aug 2012 15:14:56 +0200 Subject: [lttng-dev] More LTTng problems In-Reply-To: <502E6022.60208@efficios.com> References: <20120817150806.GA15919@Krystal> <502E6022.60208@efficios.com> Message-ID: Ok so I attached the output from lttng-sessiond (it's large). the commands was: # lttng-sessiond --vvv --no-kernel --verbose-consumer # lttng list -u # lttng create mysession # lttng enable-event -a -u Nothing shows up in dmesg. On Fri, Aug 17, 2012 at 5:15 PM, David Goulet wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Hi, > > Actually, this error means that a problem occurred *during* the 32 bit > consumer spawning process... It seems the binary was found but failed > to exec it. > > Running the session daemon (lttng-sessiond) with options "-vvv > - --verbose-consumer" will help us pin down the problem. > > Send me back the output, I'll be able to tell you what's going on (and > also the series of command you guys did before the enable-event). > > Last thing, make sure the lttng-consumerd did not segfault (dmesg) or > any other lttng-tools component. > > Thanks! > David > > Mathieu Desnoyers: >> It looks like you did not build a 32-bit consumer. I'm ccing David >> Goulet, maintainer of lttng-tools. >> >> Thanks, >> >> Mathieu >> >> * Henrik Hautakoski (henrik at fiktivkod.org) wrote: >>> Hi. got lttng working now. but when given the following command, >>> we get an error. >>> >>> $ lttng enable-event -a -u Error: Events: 32-bit UST consumer >>> start failed (channel channel0, session mysession) >>> >>> Could not find any information about this error massage. do you >>> have any ideas? >>> >>> -- Henrik Hautakoski henrik at fiktivkod.org >> > -----BEGIN PGP SIGNATURE----- > > iQEcBAEBCgAGBQJQLmAfAAoJEELoaioR9I02rqEH/2w2QXXmpLE0iEYnl2FMrfYq > 5e6p0TH1Qb4NSKvyAuSOfFJYWpNXjAKKDC/ajUuEH5PZiFtPZqiNK/jQQcTwns9t > dwZb1WSLm4Ue5z7jRV3yk8Gae/IaFQqbS5l3tTdKkE7rFOVO9Iew5+BPORHtBYA3 > Bvmx9CcwxeOVizKMab5Y0JK9RbuFcnurUV33tUI9DAisloutrItm84GdKyE5wZ92 > mXaneN+a0PhI68GsAmBq7dJMdjo+XABurr8oVL/kdWvUo8T1Sn298b5ki+se4zLL > kIfAXv7x+5K8tp6jnXznG6COsjbxf7Zt3nXouyled+oKr7muhGBDi5ul1r3W9IM= > =p8bq > -----END PGP SIGNATURE----- -- Henrik Hautakoski henrik at fiktivkod.org -------------- next part -------------- A non-text attachment was scrubbed... Name: lttng-consumerd-dump Type: application/octet-stream Size: 12169 bytes Desc: not available URL: From mathieu.desnoyers at efficios.com Mon Aug 20 09:16:42 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Mon, 20 Aug 2012 09:16:42 -0400 Subject: [lttng-dev] [RFC URCU PATCH] wfqueue: ABI v1, simplify implementation, 2.3x to 2.6x performance boost In-Reply-To: <5032054D.7050508@cn.fujitsu.com> References: <20120815213107.GB17224@Krystal> <5032054D.7050508@cn.fujitsu.com> Message-ID: <20120820131642.GA551@Krystal> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > On 08/16/2012 05:31 AM, Mathieu Desnoyers wrote: > > This work is derived from the patch from Lai Jiangshan submitted as > > "urcu: new wfqueue implementation" > > (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) > > > Hi Lai, > Hi, Mathieu > > please add this part to your patch. > > Changes(item5 has alternative): > > 1) Reorder the head and the tail in struct cds_wfq_queue > Reason: wfq is enqueue-preference Is this a technical improvement over the original order ? Both cache lines are aligned, so it should not matter which one comes first. So I'm not sure why we should favor one order vs the other. > 2) Reorder the code of ___cds_wfq_next_blocking() > Reason: short code, better readability Yes, I agree with this change. Can you extract it into a separate patch ? > > 3) Add cds_wfq_dequeue_[un]lock > Reason: the fields of struct cds_wfq_queue are not exposed to users of lib, > but some APIs needs to have this lock held. Add this locking function > to allow the users use such APIs I agree with this change too. You can put it into the same patch as (2). > > 4) Add cds_wfq_node_sync_next(), cds_wfq_dequeue_all_blocking() > Reason: helper for for_each_* Points 4-5-6-7 will need more discussion. I don't like exposing "cds_wfq_node_sync_next()", which should remain an internal implementation detail not exposed to queue users. The "get_first"/"get_next" API, as well as the dequeue, are providing an abstraction that hide the node synchronization, which makes it harder for the user to make mistakes. Given that the fast-path of cds_wfq_node_sync_next is just a pointer load and test, I don't think that skipping this the second time we iterate on the same queue would bring any significant performance improvement, but exposing sync_next adds in API complexity, which I would like to avoid. About cds_wfq_dequeue_all_blocking(): the "cds_wfq_splice()" I introduced performs the same action as "cds_wfq_dequeue_all_blocking()", but ensures that we can splice lists multiple times (rather than only once), which is a very neat side-effect. Let's imagine a use-case like Tree RCU, where each CPU produce a queue of call_rcu callbacks, and we need to merge pairs of queues together recursively until we reach the tree root: by doing splice() at each level of the tree, we can combine the queues into a single queue without ever needing to do an iteration on each node: no node synchronization is needed until we actually need to traverse the queue at the root node level. The cost of exposing dequeue_all is that we need to expose sync_next (more complexity for the user). As you exposed in your earlier email, the gain we get by exposing dequeue_all separately from splice is that dequeue_all is 1 xchg() operation, while splice is 2 xchg() operations. However, given that this operation is typically amortized over many nodes makes performance optimization a less compelling argument than simplicity of use. > > 5) Add more for_each_*, which default semantic is dequeuing > Reason: __cds_wfq_for_each_blocking_undequeue is enough for undequeuing loops > don't need to add more. > looping and dequeuing are the most probable use-cases. > dequeuing for_each_* make the queue like a pipe. make it act as a channel.(GO language) > Alternative: rename these dequeuing __cds_wfq_for_each*() to __cds_wfq_dequeue_for_each* I notice, in the iterators you added, the presence of "cds_wfq_for_each_blocking_nobreak", and "cds_wfq_for_each_blocking_nobreak_internal", which document that this special flavor is needed for loops that do not have a "break" statement. That seems to be very much error-prone and counter-intuitive for users, and I would like to avoid that. The "internal" flavor, along with "sync_next" are then used in the call_rcu code, since they are needed to get the best performance possible. I don't like to use special, internal APIs for call_rcu: the original motivation for refactoring the wfqueue code was testability, and I would really like to make sure that the APIs we use internally are also exposed to the outside world, mainly for testability, and also because if we need to directly access internal interfaces to have good performance, it means we did a poor job at exporting APIs that allow to do the same kind of highly-efficient use of the queue. Exposing an API with the "internal" keyword in it clearly discourages users from using it. > > 6) Remove __cds_wfq_for_each_blocking_safe > Reason: its semantic is not clear, hard to define a well-defined semantic to it. > when we use it, we have to delete none or delete all nodes, even when we delete all nodes, > struct cds_wfq_queue still becomes stale while looping or after loop. The typical use-case I envision is: - enqueue many nodes to queue A - then, a dequeuer thread splice the content of queue A into its temporary queue T (which head is local on its stack). - now, the thread is free to iterate on queue T, as many times as it likes, and it does not need to change the structure of the queue while iterating on it (read-only operation, without side-effect). The last time it iterates on queue T, it needs to use the _safe iteration and free the nodes. - after the nodes have been deleted, the queue T can be simply discarded (if on stack, function return will do so; if dynamically allocated, can be simply reinitialized to an empty queue, or freed). As you certainly noticed, we can iterate both on a queue being concurrently enqueued to, and on a queue that has been spliced into. Being able to use iterators and dequeue on any queue (initial enqueue, or queue spliced to) is a feature: we can therefore postpone the "sync_next" synchronization to the moment where loop iteration is really needed (lazy synchronization), which keeps optimal locality of reference of synchronization vs node data access. > > 7) Rename old __cds_wfq_for_each_blocking() to __cds_wfq_for_each_blocking_undequeue() > Reason: the default semantic of the other for_each_* is dequeuing. I'm not convinced that "__cds_wfq_for_each_blocking_nobreak", "__cds_wfq_for_each_blocking_nobreak_internal", and sync_next APIs are improvements over the API I propose. For call_rcu, this turns cds_wfq_enqueue() cds_wfq_splice_blocking() __cds_wfq_for_each_blocking_safe() for iteration into: cds_wfq_enqueue() __cds_wfq_dequeue_all_blocking() __cds_wfq_for_each_blocking_nobreak_internal() for iteration Maybe we could do some documentation improvement to "__cds_wfq_for_each_blocking(_safe)", __cds_wfq_first_blocking(), __cds_wfq_next_blocking() API members I proposed. Maybe I did not document them well enough ? Thanks, Mathieu > > > --- > urcu-call-rcu-impl.h | 13 +--- > urcu/static/wfqueue.h | 131 ++++++++++++++++++++++++++++-------------- > urcu/wfqueue.h | 154 ++++++++++++++++++++++++++++++++++++-------------- > wfqueue.c | 27 ++++++++ > 4 files changed, 233 insertions(+), 92 deletions(-) > > diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h > index d8537d0..061c2d4 100644 > --- a/urcu-call-rcu-impl.h > +++ b/urcu-call-rcu-impl.h > @@ -241,16 +241,15 @@ static void *call_rcu_thread(void *arg) > cmm_smp_mb(); > } > for (;;) { > - struct cds_wfq_queue cbs_tmp; > - struct cds_wfq_node *cbs, *tmp_cbs; > + while (!cds_wfq_empty(&crdp->cbs)) { > + struct cds_wfq_node *cbs, *next, *tail; > > - cds_wfq_init(&cbs_tmp); > - __cds_wfq_splice_blocking(&cbs_tmp, &crdp->cbs); > - if (!cds_wfq_empty(&cbs_tmp)) { > + next = __cds_wfq_dequeue_all_blocking(&crdp->cbs, &tail); > synchronize_rcu(); > cbcount = 0; > - __cds_wfq_for_each_blocking_safe(&cbs_tmp, > - cbs, tmp_cbs) { > + > + __cds_wfq_for_each_blocking_nobreak_internal( > + next, cbs, next, tail) { > struct rcu_head *rhp; > > rhp = caa_container_of(cbs, > diff --git a/urcu/static/wfqueue.h b/urcu/static/wfqueue.h > index 52e452d..e5a01ec 100644 > --- a/urcu/static/wfqueue.h > +++ b/urcu/static/wfqueue.h > @@ -84,6 +84,22 @@ static inline bool _cds_wfq_empty(struct cds_wfq_queue *q) > && CMM_LOAD_SHARED(q->enqueue.tail) == &q->dequeue.head; > } > > +static inline void _cds_wfq_dequeue_lock(struct cds_wfq_queue *q) > +{ > + int ret; > + > + ret = pthread_mutex_lock(&q->dequeue.lock); > + assert(!ret); > +} > + > +static inline void _cds_wfq_dequeue_unlock(struct cds_wfq_queue *q) > +{ > + int ret; > + > + ret = pthread_mutex_unlock(&q->dequeue.lock); > + assert(!ret); > +} > + > static inline void ___cds_wfq_append(struct cds_wfq_queue *q, > struct cds_wfq_node *new_head, > struct cds_wfq_node *new_tail) > @@ -139,30 +155,33 @@ ___cds_wfq_node_sync_next(struct cds_wfq_node *node) > return next; > } > > +static inline struct cds_wfq_node * > +_cds_wfq_node_sync_next(struct cds_wfq_node *node) > +{ > + struct cds_wfq_node *next = ___cds_wfq_node_sync_next(node); > + > + /* Load node->next before loading next's content */ > + cmm_smp_read_barrier_depends(); > + return next; > +} > + > /* > * ___cds_wfq_first_blocking: get first node of a queue, without dequeuing. > * > - * Mutual exclusion with "dequeue" and "splice" operations must be ensured > - * by the caller. > + * Should be called with cds_wfq_dequeue_lock() held. > */ > static inline struct cds_wfq_node * > ___cds_wfq_first_blocking(struct cds_wfq_queue *q) > { > - struct cds_wfq_node *node; > - > if (_cds_wfq_empty(q)) > return NULL; > - node = ___cds_wfq_node_sync_next(&q->dequeue.head); > - /* Load q->head.next before loading node's content */ > - cmm_smp_read_barrier_depends(); > - return node; > + return _cds_wfq_node_sync_next(&q->dequeue.head); > } > > /* > * ___cds_wfq_next_blocking: get next node of a queue, without dequeuing. > * > - * Mutual exclusion with "dequeue" and "splice" operations must be ensured > - * by the caller. > + * Should be called with cds_wfq_dequeue_lock() held. > */ > static inline struct cds_wfq_node * > ___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > @@ -175,16 +194,13 @@ ___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > * node->next as a common case to ensure that iteration on nodes > * do not frequently access enqueuer's q->tail cache line. > */ > - if ((next = CMM_LOAD_SHARED(node->next)) != NULL) { > - /* Load node->next before loading next's content */ > - cmm_smp_read_barrier_depends(); > - return next; > + if ((next = CMM_LOAD_SHARED(node->next)) == NULL) { > + /* Load node->next before q->tail */ > + cmm_smp_rmb(); > + if (CMM_LOAD_SHARED(q->enqueue.tail) == node) > + return NULL; > + next = ___cds_wfq_node_sync_next(node); > } > - /* Load node->next before q->tail */ > - cmm_smp_rmb(); > - if (CMM_LOAD_SHARED(q->enqueue.tail) == node) > - return NULL; > - next = ___cds_wfq_node_sync_next(node); > /* Load node->next before loading next's content */ > cmm_smp_read_barrier_depends(); > return next; > @@ -199,6 +215,8 @@ ___cds_wfq_next_blocking(struct cds_wfq_queue *q, struct cds_wfq_node *node) > * list could cause dequeue to busy-loop needlessly while waiting for another > * thread to be scheduled. The queue appears empty until tail->next is set by > * enqueue. > + * > + * Should be called with cds_wfq_dequeue_lock() held. > */ > static inline struct cds_wfq_node * > ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > @@ -243,12 +261,41 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > } > > /* > + * ___cds_wfq_dequeue_all_blocking: dequeue all nodes > + * > + * Dequeue all nodes from @q. > + * Returns the head of the dequeued nodes which tail is saved in @ret_tail. > + * Should be called with cds_wfq_dequeue_lock() held. > + */ > +static inline struct cds_wfq_node * > +___cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, > + struct cds_wfq_node **ret_tail) > +{ > + struct cds_wfq_node *head; > + > + if (_cds_wfq_empty(q)) > + return NULL; > + > + head = ___cds_wfq_node_sync_next(&q->dequeue.head); > + _cds_wfq_node_init(&q->dequeue.head); > + > + /* > + * Memory barrier implied before uatomic_xchg() orders store to > + * q->dequeue.head before store to q->enqueue.tail. This is required > + * by concurrent enqueue on q, which exchanges the tail before > + * updating the previous tail's next pointer. > + */ > + *ret_tail = uatomic_xchg(&q->enqueue.tail, &q->dequeue.head); > + > + return head; > +} > + > +/* > * ___cds_wfq_splice_blocking: enqueue all src_q nodes at the end of dest_q. > * > * Dequeue all nodes from src_q. > * dest_q must be already initialized. > - * caller ensures mutual exclusion of dequeue and splice operations on > - * src_q. > + * Should be called with cds_wfq_dequeue_lock() held. > */ > static inline void > ___cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, > @@ -256,20 +303,9 @@ ___cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, > { > struct cds_wfq_node *head, *tail; > > - if (_cds_wfq_empty(src_q)) > + if ((head = ___cds_wfq_dequeue_all_blocking(src_q, &tail)) == NULL) > return; > > - head = ___cds_wfq_node_sync_next(&src_q->dequeue.head); > - _cds_wfq_node_init(&src_q->dequeue.head); > - > - /* > - * Memory barrier implied before uatomic_xchg() orders store to > - * src_q->head before store to src_q->tail. This is required by > - * concurrent enqueue on src_q, which exchanges the tail before > - * updating the previous tail's next pointer. > - */ > - tail = uatomic_xchg(&src_q->enqueue.tail, &src_q->dequeue.head); > - > /* > * Append the spliced content of src_q into dest_q. Does not > * require mutual exclusion on dest_q (wait-free). > @@ -282,27 +318,34 @@ static inline struct cds_wfq_node * > _cds_wfq_dequeue_blocking(struct cds_wfq_queue *q) > { > struct cds_wfq_node *retval; > - int ret; > > - ret = pthread_mutex_lock(&q->dequeue.lock); > - assert(!ret); > + _cds_wfq_dequeue_lock(q); > retval = ___cds_wfq_dequeue_blocking(q); > - ret = pthread_mutex_unlock(&q->dequeue.lock); > - assert(!ret); > + _cds_wfq_dequeue_unlock(q); > + > return retval; > } > > +static inline struct cds_wfq_node * > +_cds_wfq_dequeue_all_blocking(struct cds_wfq_queue *q, > + struct cds_wfq_node **ret_tail) > +{ > + struct cds_wfq_node *head; > + > + _cds_wfq_dequeue_lock(q); > + head = _cds_wfq_dequeue_all_blocking(q, ret_tail); > + _cds_wfq_dequeue_unlock(q); > + > + return head; > +} > + > static inline void > _cds_wfq_splice_blocking(struct cds_wfq_queue *dest_q, > struct cds_wfq_queue *src_q) > { > - int ret; > - > - ret = pthread_mutex_lock(&src_q->dequeue.lock); > - assert(!ret); > + _cds_wfq_dequeue_lock(src_q); > ___cds_wfq_splice_blocking(dest_q, src_q); > - ret = pthread_mutex_unlock(&src_q->dequeue.lock); > - assert(!ret); > + _cds_wfq_dequeue_unlock(src_q); > } > > #ifdef __cplusplus > diff --git a/urcu/wfqueue.h b/urcu/wfqueue.h > index 446c94c..a3627ac 100644 > --- a/urcu/wfqueue.h > +++ b/urcu/wfqueue.h > @@ -47,12 +47,12 @@ struct cds_wfq_node { > > struct cds_wfq_queue { > struct { > + struct cds_wfq_node *tail; > + } __attribute__((aligned((CAA_CACHE_LINE_SIZE)))) enqueue; > + struct { > struct cds_wfq_node head; > pthread_mutex_t lock; > } __attribute__((aligned((CAA_CACHE_LINE_SIZE)))) dequeue; > - struct { > - struct cds_wfq_node *tail; > - } __attribute__((aligned((CAA_CACHE_LINE_SIZE)))) enqueue; > }; > > #ifdef _LGPL_SOURCE > @@ -62,71 +62,97 @@ struct cds_wfq_queue { > #define cds_wfq_node_init _cds_wfq_node_init > #define cds_wfq_init _cds_wfq_init > #define cds_wfq_empty _cds_wfq_empty > + > +/* Locking */ > +#define cds_wfq_dequeue_lock _cds_wfq_dequeue_lock > +#define cds_wfq_dequeue_unlock _cds_wfq_dequeue_unlock > + > #define cds_wfq_enqueue _cds_wfq_enqueue > > +/* > + * Helpers for for_each_*: > + * cds_wfq_node_sync_next() > + * cds_wfq_dequeue_all_blocking() > + * __cds_wfq_dequeue_all_blocking() > + * __cds_wfq_first_blocking() > + * __cds_wfq_next_blocking() > + */ > +#define cds_wfq_node_sync_next _cds_wfq_node_sync_next > + > /* Locking performed within cds_wfq calls. */ > #define cds_wfq_dequeue_blocking _cds_wfq_dequeue_blocking > +#define cds_wfq_dequeue_all_blocking _cds_wfq_dequeue_all_blocking > #define cds_wfq_splice_blocking _cds_wfq_splice_blocking > -#define cds_wfq_first_blocking _cds_wfq_first_blocking > -#define cds_wfq_next_blocking _cds_wfq_next_blocking > > -/* Locking ensured by caller */ > +/* Locking ensured by caller(called with cds_wfq_dequeue_lock() held) */ > #define __cds_wfq_dequeue_blocking ___cds_wfq_dequeue_blocking > +#define __cds_wfq_dequeue_all_blocking ___cds_wfq_dequeue_all_blocking > #define __cds_wfq_splice_blocking ___cds_wfq_splice_blocking > #define __cds_wfq_first_blocking ___cds_wfq_first_blocking > #define __cds_wfq_next_blocking ___cds_wfq_next_blocking > > #else /* !_LGPL_SOURCE */ > > +/* !_LGPL_SOURCE wrappers */ > + > extern void cds_wfq_node_init_1(struct cds_wfq_node *node); > extern void cds_wfq_init_1(struct cds_wfq_queue *q); > extern bool cds_wfq_empty_1(struct cds_wfq_queue *q); > + > +extern void cds_wfq_dequeue_lock_1(struct cds_wfq_queue *q); > +extern void cds_wfq_dequeue_unlock_1(struct cds_wfq_queue *q); > + > extern void cds_wfq_enqueue_1(struct cds_wfq_queue *q, struct cds_wfq_node *node); > > +extern struct cds_wfq_node *cds_wfq_node_sync_next_1(struct cds_wfq_node *node); > + > /* Locking performed within cds_wfq calls. */ > extern struct cds_wfq_node *cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q); > +extern struct cds_wfq_node *cds_wfq_dequeue_all_blocking_1(struct cds_wfq_queue *q, > + struct cds_wfq_node **ret_tail); > extern void cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, > struct cds_wfq_queue *src_q); > > -/* > - * __cds_wfq_dequeue_blocking: caller ensures mutual exclusion of dequeue > - * and splice operations. > - */ > +/* The following functions shouled be called with cds_wfq_dequeue_lock() held */ > extern struct cds_wfq_node *__cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q); > - > -/* > - * __cds_wfq_splice_blocking: caller ensures mutual exclusion of dequeue and > - * splice operations on src_q. dest_q must be already initialized. > - */ > +extern struct cds_wfq_node *__cds_wfq_dequeue_all_blocking_1(struct cds_wfq_queue *q, > + struct cds_wfq_node **ret_tail); > extern void __cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, > struct cds_wfq_queue *src_q); > - > -/* > - * __cds_wfq_first_blocking: mutual exclusion with "dequeue" and > - * "splice" operations must be ensured by the caller. > - */ > extern struct cds_wfq_node *__cds_wfq_first_blocking_1(struct cds_wfq_queue *q); > - > -/* > - * __cds_wfq_next_blocking: mutual exclusion with "dequeue" and "splice" > - * operations must be ensured by the caller. > - */ > extern struct cds_wfq_node *__cds_wfq_next_blocking_1(struct cds_wfq_queue *q, > struct cds_wfq_node *node); > > +/* !_LGPL_SOURCE APIs */ > + > #define cds_wfq_node_init cds_wfq_node_init_1 > #define cds_wfq_init cds_wfq_init_1 > #define cds_wfq_empty cds_wfq_empty_1 > + > +/* Locking */ > +#define cds_wfq_dequeue_lock cds_wfq_dequeue_lock_1 > +#define cds_wfq_dequeue_unlock cds_wfq_dequeue_unlock_1 > + > #define cds_wfq_enqueue cds_wfq_enqueue_1 > > +/* > + * Helpers for for_each_*: > + * cds_wfq_node_sync_next() > + * cds_wfq_dequeue_all_blocking() > + * __cds_wfq_dequeue_all_blocking() > + * __cds_wfq_first_blocking() > + * __cds_wfq_next_blocking() > + */ > +#define cds_wfq_node_sync_next cds_wfq_node_sync_next_1 > + > /* Locking performed within cds_wfq calls. */ > #define cds_wfq_dequeue_blocking cds_wfq_dequeue_blocking_1 > +#define cds_wfq_dequeue_all_blocking cds_wfq_dequeue_all_blocking_1 > #define cds_wfq_splice_blocking cds_wfq_splice_blocking_1 > -#define cds_wfq_first_blocking cds_wfq_first_blocking_1 > -#define cds_wfq_next_blocking cds_wfq_next_blocking_1 > > -/* Locking ensured by caller */ > +/* Locking ensured by caller(called with cds_wfq_dequeue_lock() held) */ > #define __cds_wfq_dequeue_blocking __cds_wfq_dequeue_blocking_1 > +#define __cds_wfq_dequeue_all_blocking __cds_wfq_dequeue_all_blocking_1 > #define __cds_wfq_splice_blocking __cds_wfq_splice_blocking_1 > #define __cds_wfq_first_blocking __cds_wfq_first_blocking_1 > #define __cds_wfq_next_blocking __cds_wfq_next_blocking_1 > @@ -134,29 +160,75 @@ extern struct cds_wfq_node *__cds_wfq_next_blocking_1(struct cds_wfq_queue *q, > #endif /* !_LGPL_SOURCE */ > > /* > - * __cds_wfq_for_each_blocking: Iterate over all nodes in a queue, > + * for_each_* routines: Iterate over all nodes in a queue and dequeue > + * the travlled nodes(except __cds_wfq_for_each_blocking_undequeue). > + * > + * All nodes which are un-dequeued && enqueued before the for_each_* > + * will be travelled. > + * > + * Nodes which are enqueued after the for_each_* started MAY or may NOT > + * be travelled. > + */ > + > +/* > + * __cds_wfq_for_each_blocking_undequeue: Iterate over all nodes in a queue, > * without dequeuing them. > * > - * Mutual exclusion with "dequeue" and "splice" operations must be > - * ensured by the caller. > + * Should be called with cds_wfq_dequeue_lock() held. > */ > -#define __cds_wfq_for_each_blocking(q, node) \ > +#define __cds_wfq_for_each_blocking_undequeue(q, node) \ > for (node = __cds_wfq_first_blocking(q); \ > node != NULL; \ > node = __cds_wfq_next_blocking(q, node)) > > +#define __cds_wfq_for_each_blocking_nobreak_internal(all, node, next, tail) \ > + for (node = all, \ > + next = ((node && node != tail) ? cds_wfq_node_sync_next(node) : NULL); \ > + node != NULL; \ > + node = next, \ > + next = ((node && node != tail) ? cds_wfq_node_sync_next(node) : NULL)) > + > /* > - * __cds_wfq_for_each_blocking_safe: Iterate over all nodes in a queue, > - * without dequeuing them. Safe against deletion. > + * __cds_wfq_for_each_blocking_nobreak: Iterate over all nodes in a queue, > + * and dequeue them. > * > - * Mutual exclusion with "dequeue" and "splice" operations must be > - * ensured by the caller. > + * Should not use "break;" or other statement to leave the control flow > + * from the loop body, otherwise it causes un-travelled nodes leak. > + * > + * Should be called with cds_wfq_dequeue_lock() held. > + */ > +#define __cds_wfq_for_each_blocking_nobreak(q, node, next, tail) \ > + __cds_wfq_for_each_blocking_nobreak_internal( \ > + __cds_wfq_dequeue_all_blocking(q, &tail), node, next, tail) > + > +/* > + * cds_wfq_for_each_blocking_nobreak: Iterate over all nodes in a queue, > + * and dequeue them. > + * > + * Should not use "break;" or other statement to leave the control flow > + * from the loop body, otherwise it causes un-travelled nodes leak. > + */ > +#define cds_wfq_for_each_blocking_nobreak(q, node, next, tail) \ > + __cds_wfq_for_each_blocking_nobreak_internal( \ > + cds_wfq_dequeue_all_blocking(q, &tail), node, next, tail) > + > +/* > + * __cds_wfq_for_each_blocking: Iterate over all nodes in a queue, > + * and dequeue them. > + * > + * Should be called with cds_wfq_dequeue_lock() held. > + */ > +#define __cds_wfq_for_each_blocking(q, node) \ > + for (node = __cds_wfq_dequeue_blocking(q); node; \ > + node = __cds_wfq_dequeue_blocking(q)) > + > +/* > + * cds_wfq_for_each_blocking: Iterate over all nodes in a queue, > + * and dequeue them. > */ > -#define __cds_wfq_for_each_blocking_safe(q, node, n) \ > - for (node = __cds_wfq_first_blocking(q), \ > - n = (node ? __cds_wfq_next_blocking(q, node) : NULL); \ > - node != NULL; \ > - node = n, n = (node ? __cds_wfq_next_blocking(q, node) : NULL)) > +#define cds_wfq_for_each_blocking(q, node) \ > + for (node = cds_wfq_dequeue_blocking(q); node; \ > + node = cds_wfq_dequeue_blocking(q)) > > #ifdef __cplusplus > } > diff --git a/wfqueue.c b/wfqueue.c > index b5fba7b..eee1fd8 100644 > --- a/wfqueue.c > +++ b/wfqueue.c > @@ -44,16 +44,37 @@ bool cds_wfq_empty_1(struct cds_wfq_queue *q) > return _cds_wfq_empty(q); > } > > +void cds_wfq_dequeue_lock_1(struct cds_wfq_queue *q) > +{ > + cds_wfq_dequeue_lock(q); > +} > + > +void cds_wfq_dequeue_unlock_1(struct cds_wfq_queue *q) > +{ > + cds_wfq_dequeue_unlock(q); > +} > + > void cds_wfq_enqueue_1(struct cds_wfq_queue *q, struct cds_wfq_node *node) > { > _cds_wfq_enqueue(q, node); > } > > +struct cds_wfq_node *cds_wfq_node_sync_next_1(struct cds_wfq_node *node) > +{ > + return _cds_wfq_node_sync_next(node); > +} > + > struct cds_wfq_node *cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q) > { > return _cds_wfq_dequeue_blocking(q); > } > > +struct cds_wfq_node *cds_wfq_dequeue_all_blocking_1(struct cds_wfq_queue *q, > + struct cds_wfq_node **ret_tail) > +{ > + return _cds_wfq_dequeue_all_blocking(q, ret_tail); > +} > + > void cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, > struct cds_wfq_queue *src_q) > { > @@ -65,6 +86,12 @@ struct cds_wfq_node *__cds_wfq_dequeue_blocking_1(struct cds_wfq_queue *q) > return ___cds_wfq_dequeue_blocking(q); > } > > +struct cds_wfq_node *__cds_wfq_dequeue_all_blocking_1(struct cds_wfq_queue *q, > + struct cds_wfq_node **ret_tail) > +{ > + return ___cds_wfq_dequeue_all_blocking(q, ret_tail); > +} > + > void __cds_wfq_splice_blocking_1(struct cds_wfq_queue *dest_q, > struct cds_wfq_queue *src_q) > { -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From henrik at fiktivkod.org Mon Aug 20 10:13:51 2012 From: henrik at fiktivkod.org (Henrik Hautakoski) Date: Mon, 20 Aug 2012 16:13:51 +0200 Subject: [lttng-dev] More LTTng problems In-Reply-To: References: <20120817150806.GA15919@Krystal> <502E6022.60208@efficios.com> Message-ID: Oh yeah. for the record. DEBUG1: Using 32-bit UST consumer at: /opt/xcompile/powerpc-e500-linux-gnu-lttng/lib/lttng/libexec/lttng-consumerd [in spawn_consumerd() at main.c:1776] Error: Spawning consumerd failed I tried to change the patch with --consumerd32-path flag after. but it did not change the result. On Mon, Aug 20, 2012 at 3:14 PM, Henrik Hautakoski wrote: > Ok so I attached the output from lttng-sessiond (it's large). > > the commands was: > > # lttng-sessiond --vvv --no-kernel --verbose-consumer > > # lttng list -u > > # lttng create mysession > > # lttng enable-event -a -u > > Nothing shows up in dmesg. > > On Fri, Aug 17, 2012 at 5:15 PM, David Goulet wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA512 >> >> Hi, >> >> Actually, this error means that a problem occurred *during* the 32 bit >> consumer spawning process... It seems the binary was found but failed >> to exec it. >> >> Running the session daemon (lttng-sessiond) with options "-vvv >> - --verbose-consumer" will help us pin down the problem. >> >> Send me back the output, I'll be able to tell you what's going on (and >> also the series of command you guys did before the enable-event). >> >> Last thing, make sure the lttng-consumerd did not segfault (dmesg) or >> any other lttng-tools component. >> >> Thanks! >> David >> >> Mathieu Desnoyers: >>> It looks like you did not build a 32-bit consumer. I'm ccing David >>> Goulet, maintainer of lttng-tools. >>> >>> Thanks, >>> >>> Mathieu >>> >>> * Henrik Hautakoski (henrik at fiktivkod.org) wrote: >>>> Hi. got lttng working now. but when given the following command, >>>> we get an error. >>>> >>>> $ lttng enable-event -a -u Error: Events: 32-bit UST consumer >>>> start failed (channel channel0, session mysession) >>>> >>>> Could not find any information about this error massage. do you >>>> have any ideas? >>>> >>>> -- Henrik Hautakoski henrik at fiktivkod.org >>> >> -----BEGIN PGP SIGNATURE----- >> >> iQEcBAEBCgAGBQJQLmAfAAoJEELoaioR9I02rqEH/2w2QXXmpLE0iEYnl2FMrfYq >> 5e6p0TH1Qb4NSKvyAuSOfFJYWpNXjAKKDC/ajUuEH5PZiFtPZqiNK/jQQcTwns9t >> dwZb1WSLm4Ue5z7jRV3yk8Gae/IaFQqbS5l3tTdKkE7rFOVO9Iew5+BPORHtBYA3 >> Bvmx9CcwxeOVizKMab5Y0JK9RbuFcnurUV33tUI9DAisloutrItm84GdKyE5wZ92 >> mXaneN+a0PhI68GsAmBq7dJMdjo+XABurr8oVL/kdWvUo8T1Sn298b5ki+se4zLL >> kIfAXv7x+5K8tp6jnXznG6COsjbxf7Zt3nXouyled+oKr7muhGBDi5ul1r3W9IM= >> =p8bq >> -----END PGP SIGNATURE----- > > > > -- > Henrik Hautakoski > henrik at fiktivkod.org -- Henrik Hautakoski henrik at fiktivkod.org From dgoulet at efficios.com Mon Aug 20 10:42:28 2012 From: dgoulet at efficios.com (David Goulet) Date: Mon, 20 Aug 2012 10:42:28 -0400 Subject: [lttng-dev] More LTTng problems In-Reply-To: References: <20120817150806.GA15919@Krystal> <502E6022.60208@efficios.com> Message-ID: <50324CD4.5050309@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, This is pretty odd... it seems the execl() of the consumer failed... We'll have to confirm this with the perror code I guess. Can you apply this patch and re-run again. diff --git a/src/bin/lttng-sessiond/main.c b/src/bin/lttng-sessiond/main.c index c952fc0..d3c378b 100644 - --- a/src/bin/lttng-sessiond/main.c +++ b/src/bin/lttng-sessiond/main.c @@ -1782,6 +1782,7 @@ static pid_t spawn_consumerd(struct consumer_data *consumer_data) free(tmpnew); } if (ret) { + PERROR("execl 32-bit consumer"); goto error; } break; If you want, to speed things up, you can come on OFTC irc server on #lttng and we'll be able to assist you in real time to find this problem. Cheers David Henrik Hautakoski: > Ok so I attached the output from lttng-sessiond (it's large). > > the commands was: > > # lttng-sessiond --vvv --no-kernel --verbose-consumer > > # lttng list -u > > # lttng create mysession > > # lttng enable-event -a -u > > Nothing shows up in dmesg. > > On Fri, Aug 17, 2012 at 5:15 PM, David Goulet > wrote: Hi, > > Actually, this error means that a problem occurred *during* the 32 > bit consumer spawning process... It seems the binary was found but > failed to exec it. > > Running the session daemon (lttng-sessiond) with options "-vvv > --verbose-consumer" will help us pin down the problem. > > Send me back the output, I'll be able to tell you what's going on > (and also the series of command you guys did before the > enable-event). > > Last thing, make sure the lttng-consumerd did not segfault (dmesg) > or any other lttng-tools component. > > Thanks! David > > Mathieu Desnoyers: >>>> It looks like you did not build a 32-bit consumer. I'm ccing >>>> David Goulet, maintainer of lttng-tools. >>>> >>>> Thanks, >>>> >>>> Mathieu >>>> >>>> * Henrik Hautakoski (henrik at fiktivkod.org) wrote: >>>>> Hi. got lttng working now. but when given the following >>>>> command, we get an error. >>>>> >>>>> $ lttng enable-event -a -u Error: Events: 32-bit UST >>>>> consumer start failed (channel channel0, session >>>>> mysession) >>>>> >>>>> Could not find any information about this error massage. do >>>>> you have any ideas? >>>>> >>>>> -- Henrik Hautakoski henrik at fiktivkod.org >>>> > > > -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQMkzRAAoJEELoaioR9I02cTIIAIxjeagCL1uvHB4WT/SGAdOz 5eIDXhRyjto+7af4VkjibAozT28XZpnJweWJKkeGJjtRLD/1KFMq06QWREUO6sSz NqkU+i+5FHTSCAK4cRS80e0AcqmOCCAo0zfeLpurNCfeu654rHgzk+03XIj+Zn3a QhY7Uw9S9SuJ0NPPP6f50bmlsdh6VGLHmJQMi2nNJQZPFiuwLHF9V7mUOO66ZWqF ikqN5NGDMsmjOoY7XPwcalF7BK1vq9bMv+f9zyWImFV/ky8iHw0d1Cemy9O1cWw1 cbM1wyEqzfDV5QIoFqa0ZKIsbuUzuwy4/bkk7pTrqOaTDLuwOOoqYs108yeJgUg= =mDFB -----END PGP SIGNATURE----- From christian.babeux at efficios.com Mon Aug 20 13:34:36 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Mon, 20 Aug 2012 13:34:36 -0400 Subject: [lttng-dev] [PATCH lttng-tools] Fix: Possible buffer overflows in strncat() usage Message-ID: <1345484076-31111-1-git-send-email-christian.babeux@efficios.com> When using strncat, the size_t n argument must indicate the left over space remaining in the buffer, *not* the total buffer size. Also, proper care must be taken for the case where src contains n or more bytes and thus allow space for the null terminating byte appended to dest (e.g. strncat() will write n+1 bytes). Signed-off-by: Christian Babeux --- src/bin/lttng-sessiond/consumer.c | 5 +++-- src/bin/lttng-sessiond/main.c | 12 +++++++----- src/common/utils.c | 2 +- 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/src/bin/lttng-sessiond/consumer.c b/src/bin/lttng-sessiond/consumer.c index 3503e04..fe2d45a 100644 --- a/src/bin/lttng-sessiond/consumer.c +++ b/src/bin/lttng-sessiond/consumer.c @@ -480,9 +480,10 @@ int consumer_send_stream(int sock, struct consumer_output *dst, break; case CONSUMER_DST_LOCAL: /* Add stream file name to stream path */ - strncat(msg->u.stream.path_name, "/", sizeof(msg->u.stream.path_name)); + strncat(msg->u.stream.path_name, "/", + sizeof(msg->u.stream.path_name) - strlen(msg->u.stream.path_name) - 1); strncat(msg->u.stream.path_name, msg->u.stream.name, - sizeof(msg->u.stream.path_name)); + sizeof(msg->u.stream.path_name) - strlen(msg->u.stream.path_name) - 1); msg->u.stream.path_name[sizeof(msg->u.stream.path_name) - 1] = '\0'; /* Indicate that the stream is NOT network */ msg->u.stream.net_index = -1; diff --git a/src/bin/lttng-sessiond/main.c b/src/bin/lttng-sessiond/main.c index c952fc0..4ca031f 100644 --- a/src/bin/lttng-sessiond/main.c +++ b/src/bin/lttng-sessiond/main.c @@ -2278,7 +2278,8 @@ static int copy_session_consumer(int domain, struct ltt_session *session) } /* Append correct directory to subdir */ - strncat(consumer->subdir, dir_name, sizeof(consumer->subdir)); + strncat(consumer->subdir, dir_name, + sizeof(consumer->subdir) - strlen(consumer->subdir) - 1); DBG3("Copy session consumer subdir %s", consumer->subdir); ret = LTTCOMM_OK; @@ -2809,7 +2810,8 @@ static int add_uri_to_consumer(struct consumer_output *consumer, if (uri->stype == LTTNG_STREAM_CONTROL) { /* On a new subdir, reappend the default trace dir. */ - strncat(consumer->subdir, default_trace_dir, sizeof(consumer->subdir)); + strncat(consumer->subdir, default_trace_dir, + sizeof(consumer->subdir) - strlen(consumer->subdir) - 1); DBG3("Append domain trace name to subdir %s", consumer->subdir); } @@ -2822,7 +2824,7 @@ static int add_uri_to_consumer(struct consumer_output *consumer, sizeof(consumer->dst.trace_path)); /* Append default trace dir */ strncat(consumer->dst.trace_path, default_trace_dir, - sizeof(consumer->dst.trace_path)); + sizeof(consumer->dst.trace_path) - strlen(consumer->dst.trace_path) - 1); /* Flag consumer as local. */ consumer->type = CONSUMER_DST_LOCAL; break; @@ -4257,7 +4259,7 @@ static int cmd_enable_consumer(int domain, struct ltt_session *session) /* Append default kernel trace dir to subdir */ strncat(ksess->consumer->subdir, DEFAULT_KERNEL_TRACE_DIR, - sizeof(ksess->consumer->subdir)); + sizeof(ksess->consumer->subdir) - strlen(ksess->consumer->subdir) - 1); /* * @session-lock @@ -4342,7 +4344,7 @@ static int cmd_enable_consumer(int domain, struct ltt_session *session) /* Append default kernel trace dir to subdir */ strncat(usess->consumer->subdir, DEFAULT_UST_TRACE_DIR, - sizeof(usess->consumer->subdir)); + sizeof(usess->consumer->subdir) - strlen(usess->consumer->subdir) - 1); /* * @session-lock diff --git a/src/common/utils.c b/src/common/utils.c index 0494b23..729aa76 100644 --- a/src/common/utils.c +++ b/src/common/utils.c @@ -70,7 +70,7 @@ char *utils_expand_path(const char *path) } /* Add end part to expanded path */ - strncat(expanded_path, end_path, PATH_MAX); + strncat(expanded_path, end_path, PATH_MAX - strlen(expanded_path) - 1); free(cut_path); return expanded_path; -- 1.7.11.4 From dgoulet at efficios.com Mon Aug 20 16:13:02 2012 From: dgoulet at efficios.com (David Goulet) Date: Mon, 20 Aug 2012 16:13:02 -0400 Subject: [lttng-dev] [PATCH lttng-tools] Fix: Possible buffer overflows in strncat() usage In-Reply-To: <1345484076-31111-1-git-send-email-christian.babeux@efficios.com> References: <1345484076-31111-1-git-send-email-christian.babeux@efficios.com> Message-ID: <50329A4E.1080801@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Merged. Thanks Christian Babeux: > Signed-off-by: Christian Babeux -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQMppKAAoJEELoaioR9I02TREH/RQD3Z5UQinTHl7nDKCb3gDX dViQgYc1XaTL8E3psSt12CcrZA07qQTdWZ0p1S9+IQ3w2WXQqosGDamcZNANC3wj V2QqSnVkpmzXdsfCgx9ZtESqczXJATAtryZNfyeqfp083Wr2n/d+Ee7WJ6km+PsF wZF1BSwS5FoL/Nuryf0cr2oOuM5mkpL4mtxRktEC18xviEQKL3YlLBbIFN5HghGW ONbjrHDHFjfkON+4EsJbfoWbtHUgid1tKZWSzRhgo+ux95wZm6HuLZajPSwiX9oJ nVATRbUvpbSoGMNczlmneIZpdrzi/QzdkdaHj6lXLdZ2Wa02dTSc2fqhg9GGJ0U= =FIxJ -----END PGP SIGNATURE----- From laijs at cn.fujitsu.com Mon Aug 20 20:12:04 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Tue, 21 Aug 2012 08:12:04 +0800 Subject: [lttng-dev] [RFC URCU PATCH] wfqueue: ABI v1, simplify implementation, 2.3x to 2.6x performance boost In-Reply-To: <20120820131642.GA551@Krystal> References: <20120815213107.GB17224@Krystal> <5032054D.7050508@cn.fujitsu.com> <20120820131642.GA551@Krystal> Message-ID: <5032D254.7040207@cn.fujitsu.com> On 08/20/2012 09:16 PM, Mathieu Desnoyers wrote: > * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: >> On 08/16/2012 05:31 AM, Mathieu Desnoyers wrote: >>> This work is derived from the patch from Lai Jiangshan submitted as >>> "urcu: new wfqueue implementation" >>> (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) >>> >> > > Hi Lai, > >> Hi, Mathieu >> >> please add this part to your patch. >> >> Changes(item5 has alternative): >> >> 1) Reorder the head and the tail in struct cds_wfq_queue >> Reason: wfq is enqueue-preference > > Is this a technical improvement over the original order ? Both cache > lines are aligned, so it should not matter which one comes first. So I'm > not sure why we should favor one order vs the other. struct astruct { int foo; /* field offset = 0 */ int bar; /* field offset = 4 */ }; Access to p->foo and access to p->bar are different in CPU instruction. Access to p->bar will need to add a offset to p. (the addition may cost a very small time) instructions will be probably longer than the instructions access to p->foo. If the code of access to p->bar is significant more than the code of access to p->foo, we should swap p->bar and p->foo field's order to shorten the total instruction size to reduce footprint. Our enqueue is inline function, when this inline function is expanded in every call site, the code of access to the tail will much significant, so we need to put the tail at first. > >> 2) Reorder the code of ___cds_wfq_next_blocking() >> Reason: short code, better readability > > Yes, I agree with this change. Can you extract it into a separate patch ? You merge it into your patch which ___cds_wfq_next_blocking() is introduced. > >> >> 3) Add cds_wfq_dequeue_[un]lock >> Reason: the fields of struct cds_wfq_queue are not exposed to users of lib, >> but some APIs needs to have this lock held. Add this locking function >> to allow the users use such APIs > > I agree with this change too. You can put it into the same patch as (2). You merge it into your patch. From mathieu.desnoyers at efficios.com Mon Aug 20 21:03:51 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Mon, 20 Aug 2012 21:03:51 -0400 Subject: [lttng-dev] [RFC URCU PATCH] wfqueue: ABI v1, simplify implementation, 2.3x to 2.6x performance boost In-Reply-To: <5032D254.7040207@cn.fujitsu.com> References: <20120815213107.GB17224@Krystal> <5032054D.7050508@cn.fujitsu.com> <20120820131642.GA551@Krystal> <5032D254.7040207@cn.fujitsu.com> Message-ID: <20120821010351.GA7472@Krystal> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > On 08/20/2012 09:16 PM, Mathieu Desnoyers wrote: > > * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > >> On 08/16/2012 05:31 AM, Mathieu Desnoyers wrote: > >>> This work is derived from the patch from Lai Jiangshan submitted as > >>> "urcu: new wfqueue implementation" > >>> (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) > >>> > >> > > > > Hi Lai, > > > >> Hi, Mathieu > >> > >> please add this part to your patch. > >> > >> Changes(item5 has alternative): > >> > >> 1) Reorder the head and the tail in struct cds_wfq_queue > >> Reason: wfq is enqueue-preference > > > > Is this a technical improvement over the original order ? Both cache > > lines are aligned, so it should not matter which one comes first. So I'm > > not sure why we should favor one order vs the other. > > struct astruct { > int foo; /* field offset = 0 */ > int bar; /* field offset = 4 */ > }; > > Access to p->foo and access to p->bar are different in CPU instruction. > Access to p->bar will need to add a offset to p. (the addition may cost a very small time) > instructions will be probably longer than the instructions access to p->foo. > > If the code of access to p->bar is significant more than the code of access to p->foo, > we should swap p->bar and p->foo field's order to shorten the total instruction size > to reduce footprint. > > Our enqueue is inline function, when this inline function is expanded in every call site, > the code of access to the tail will much significant, so we need to put the tail at first. > Very good point! I'm pulling this change too. > > > > > >> 2) Reorder the code of ___cds_wfq_next_blocking() > >> Reason: short code, better readability > > > > Yes, I agree with this change. Can you extract it into a separate patch ? > > You merge it into your patch which ___cds_wfq_next_blocking() is introduced. > OK. done, > > > >> > >> 3) Add cds_wfq_dequeue_[un]lock > >> Reason: the fields of struct cds_wfq_queue are not exposed to users of lib, > >> but some APIs needs to have this lock held. Add this locking function > >> to allow the users use such APIs > > > > I agree with this change too. You can put it into the same patch as (2). > > You merge it into your patch. OK, done too. The current version (volatile branch) is at git://git.dorsal.polymtl.ca/~compudj/userspace-rcu branch: urcu/wfqueue-volatile Thanks! Mathieu -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From teawater at gmail.com Mon Aug 20 22:17:58 2012 From: teawater at gmail.com (Hui Zhu) Date: Tue, 21 Aug 2012 10:17:58 +0800 Subject: [lttng-dev] [Question] the return of babeltrace's bt_context_create Message-ID: Hi, The comments of this function said: Returns an allocated context on success and NULL on error But when babeltrace use this function, it didn't check the return value of bt_context_create: ctx = bt_context_create(); ret = bt_context_add_traces_recursive(ctx, opt_input_path, opt_input_format, NULL); Could you tell me why? Thanks a lot. Best, Hui From laijs at cn.fujitsu.com Mon Aug 20 23:41:37 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Tue, 21 Aug 2012 11:41:37 +0800 Subject: [lttng-dev] [RFC URCU PATCH] wfqueue: ABI v1, simplify implementation, 2.3x to 2.6x performance boost In-Reply-To: <20120820131642.GA551@Krystal> References: <20120815213107.GB17224@Krystal> <5032054D.7050508@cn.fujitsu.com> <20120820131642.GA551@Krystal> Message-ID: <50330371.6060401@cn.fujitsu.com> On 08/20/2012 09:16 PM, Mathieu Desnoyers wrote: > * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: >> On 08/16/2012 05:31 AM, Mathieu Desnoyers wrote: >>> This work is derived from the patch from Lai Jiangshan submitted as >>> "urcu: new wfqueue implementation" >>> (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) >>> >> > > Hi Lai, > >> Hi, Mathieu >> >> please add this part to your patch. >> >> Changes(item5 has alternative): >> >> 1) Reorder the head and the tail in struct cds_wfq_queue >> Reason: wfq is enqueue-preference > > Is this a technical improvement over the original order ? Both cache > lines are aligned, so it should not matter which one comes first. So I'm > not sure why we should favor one order vs the other. > >> 2) Reorder the code of ___cds_wfq_next_blocking() >> Reason: short code, better readability > > Yes, I agree with this change. Can you extract it into a separate patch ? > >> >> 3) Add cds_wfq_dequeue_[un]lock >> Reason: the fields of struct cds_wfq_queue are not exposed to users of lib, >> but some APIs needs to have this lock held. Add this locking function >> to allow the users use such APIs > > I agree with this change too. You can put it into the same patch as (2). > >> >> 4) Add cds_wfq_node_sync_next(), cds_wfq_dequeue_all_blocking() >> Reason: helper for for_each_* > > Points 4-5-6-7 will need more discussion. > > I don't like exposing "cds_wfq_node_sync_next()", which should remain an > internal implementation detail not exposed to queue users. The > "get_first"/"get_next" API, as well as the dequeue, are providing an > abstraction that hide the node synchronization, which makes it harder > for the user to make mistakes. Given that the fast-path of > cds_wfq_node_sync_next is just a pointer load and test, I don't think > that skipping this the second time we iterate on the same queue would > bring any significant performance improvement, but exposing sync_next > adds in API complexity, which I would like to avoid. If the name "cds_wfq_node_sync_next()" is bad and exposes the implementation, we can rename it to "__cds_wfq_next_blocking_nocheck()" to hide the implementation. "get_next_nocheck": get next node of a queue, the caller ensure that the next node is existed. "get_next": get next node of a queue, it also finds out that whether the next node is existed or not. Or just remove "cds_wfq_node_sync_next()", use "get_next" instead. > > About cds_wfq_dequeue_all_blocking(): the "cds_wfq_splice()" I > introduced performs the same action as "cds_wfq_dequeue_all_blocking()", > but ensures that we can splice lists multiple times (rather than only > once), which is a very neat side-effect. Let's imagine a use-case like > Tree RCU, where each CPU produce a queue of call_rcu callbacks, and we > need to merge pairs of queues together recursively until we reach the > tree root: by doing splice() at each level of the tree, we can combine > the queues into a single queue without ever needing to do an iteration > on each node: no node synchronization is needed until we actually need > to traverse the queue at the root node level. > > The cost of exposing dequeue_all is that we need to expose sync_next > (more complexity for the user). As you exposed in your earlier email, > the gain we get by exposing dequeue_all separately from splice is that > dequeue_all is 1 xchg() operation, while splice is 2 xchg() operations. > However, given that this operation is typically amortized over many > nodes makes performance optimization a less compelling argument than > simplicity of use. Reducing the temporary big struct is also my tempt. --- Related to "internal", see the bottom. > >> >> 5) Add more for_each_*, which default semantic is dequeuing >> Reason: __cds_wfq_for_each_blocking_undequeue is enough for undequeuing loops >> don't need to add more. >> looping and dequeuing are the most probable use-cases. >> dequeuing for_each_* make the queue like a pipe. make it act as a channel.(GO language) >> Alternative: rename these dequeuing __cds_wfq_for_each*() to __cds_wfq_dequeue_for_each* > > I notice, in the iterators you added, the presence of > "cds_wfq_for_each_blocking_nobreak", and > "cds_wfq_for_each_blocking_nobreak_internal", which document that this > special flavor is needed for loops that do not have a "break" statement. > That seems to be very much error-prone and counter-intuitive for users, > and I would like to avoid that. __cds_wfq_for_each_blocking_safe(remove-all) is also "no-break"(*), is it error-prone and ...? So this is not a good reason to reject cds_wfq_for_each_blocking_nobreak(), if the user don't know how to use it, he should use cds_wfq_for_each_blocking() (the version in my patch which is dequeue()-based and we can break from the loop body) __cds_wfq_for_each_blocking_safe() and cds_wfq_for_each_blocking_nobreak() are exactly the same semantic in "remove-all" and "no-break". but cds_wfq_for_each_blocking_nobreak() does NOT CORRUPT the queue. So cds_wfq_for_each_blocking_nobreak() is yet another __cds_wfq_for_each_blocking_safe() and it don't corrupt the queue, it is really safer than __cds_wfq_for_each_blocking_safe(). """"footnote: (*) __cds_wfq_for_each_blocking_safe() should be used for "remove-none" or "remove-all" We should use __cds_wfq_for_each_blocking_undequeue() instead of __cds_wfq_for_each_blocking_safe(remove-none) """ > > The "internal" flavor, along with "sync_next" are then used in the > call_rcu code, since they are needed to get the best performance > possible. I don't like to use special, internal APIs for call_rcu: the > original motivation for refactoring the wfqueue code was testability, > and I would really like to make sure that the APIs we use internally are > also exposed to the outside world, mainly for testability, and also > because if we need to directly access internal interfaces to have good > performance, it means we did a poor job at exporting APIs that allow to > do the same kind of highly-efficient use of the queue. Exposing an API > with the "internal" keyword in it clearly discourages users from using > it. Related to "internal" see the bottom. > > >> >> 6) Remove __cds_wfq_for_each_blocking_safe >> Reason: its semantic is not clear, hard to define a well-defined semantic to it. >> when we use it, we have to delete none or delete all nodes, even when we delete all nodes, >> struct cds_wfq_queue still becomes stale while looping or after loop. > > The typical use-case I envision is: > > - enqueue many nodes to queue A > > - then, a dequeuer thread splice the content of queue A into its temporary > queue T (which head is local on its stack). > > - now, the thread is free to iterate on queue T, as many times as it > likes, and it does not need to change the structure of the queue while > iterating on it (read-only operation, without side-effect). The last > time it iterates on queue T, it needs to use the _safe iteration and > free the nodes. > > - after the nodes have been deleted, the queue T can be simply discarded > (if on stack, function return will do so; if dynamically allocated, > can be simply reinitialized to an empty queue, or freed). My for_each_*() can do exactly the same thing in this use-case since we still have splice() and cds_wfq_for_each_blocking_nobreak()(safer **_safe()). Doesn't it? The memory represented a struct queue is big.(two cache line), something we need to avoid to use it in the stack.(shorten the stack size or avoid to cost 2 or 3 additional cache line). -----------a use case All these local field are hot in the stack(and for some reason, it is not in the registers) int count; struct cds_wfq_queue tmp; struct cds_wfq_node *node, *next; It will use 4 cache line. Should we always recommend the users use splice()+tmp? > > As you certainly noticed, we can iterate both on a queue being > concurrently enqueued to, and on a queue that has been spliced into. > Being able to use iterators and dequeue on any queue (initial enqueue, > or queue spliced to) is a feature: we can therefore postpone the > "sync_next" synchronization to the moment where loop iteration is really > needed (lazy synchronization), which keeps optimal locality of reference > of synchronization vs node data access. > >> >> 7) Rename old __cds_wfq_for_each_blocking() to __cds_wfq_for_each_blocking_undequeue() >> Reason: the default semantic of the other for_each_* is dequeuing. > > I'm not convinced that "__cds_wfq_for_each_blocking_nobreak", > "__cds_wfq_for_each_blocking_nobreak_internal", and sync_next APIs are > improvements over the API I propose. For call_rcu, this turns > > cds_wfq_enqueue() > cds_wfq_splice_blocking() > __cds_wfq_for_each_blocking_safe() for iteration > > into: > > cds_wfq_enqueue() > __cds_wfq_dequeue_all_blocking() > __cds_wfq_for_each_blocking_nobreak_internal() for iteration > > Maybe we could do some documentation improvement to > "__cds_wfq_for_each_blocking(_safe)", __cds_wfq_first_blocking(), > __cds_wfq_next_blocking() API members I proposed. Maybe I did not > document them well enough ? > It does be not enough for __cds_wfq_for_each_blocking_safe(). I *STRONGLY OPPOSE* __cds_wfq_for_each_blocking_safe()(although it is suggested by me). If you oppose the dequeue_all(), I agreed with this list: Remove old __cds_wfq_for_each_blocking_safe() Use manual splice()+get_first()+get_next() for call_rcu_thread(). Remove my proposed cds_wfq_for_each_blocking_nobreak() Remove my proposed sync_next(), dequeue_all(). Keep dequeue-based for_each() (alternative: introduce it in future) If you also agreed this list, could you please partially merge my patch to your patch and send it for review to reduce the review cycle. (don't forget the other minimal fixes in my patch) ========== About internals: I want to expose the for_each_*, I don't want to expose these helpers: cds_wfq_node_sync_next() cds_wfq_dequeue_all_blocking() __cds_wfq_dequeue_all_blocking() __cds_wfq_for_each_blocking_nobreak_internal() (and I want to disallow the users use these helpers directly). but the for_each_* use these helpers, so I have to declare it in the urcu/wfqueue.h, and they become a part of ABI. So these helpers become internal things, how to handle internal things in a exposed *.h? I found "__cds_list_del()" and "__tls_access_ ## name ()" are also internals, and they are exposed. Are they exposed correctly? In my view, exposed internals is OK, what we can do is disallowing the users use it directly and ensuring the compatibility. And we don't need to test exposed internals directly via testcases, because they are internals, just like we don't need to test so much un-exposed internals. When we test the exposed API, the internals will be also tested. The call_rcu_thread() use the internal directly and can't be tested as you mentioned. It is not totally true. call_rcu_thread() need to inject a "synchronize_rcu()" to __cds_wfq_for_each_blocking_nobreak(), so I have to use __cds_wfq_for_each_blocking_nobreak_internal(). Tests for __cds_wfq_for_each_blocking_nobreak() are enough for call_rcu_thread(). In one word, I don't agreed full of your disgust to the exposed internals, but I agree to remove the ones introduced in my patch. Thanks, Lai From laijs at cn.fujitsu.com Mon Aug 20 23:59:24 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Tue, 21 Aug 2012 11:59:24 +0800 Subject: [lttng-dev] [RFC URCU PATCH] wfqueue: ABI v1, simplify implementation, 2.3x to 2.6x performance boost In-Reply-To: <20120821010351.GA7472@Krystal> References: <20120815213107.GB17224@Krystal> <5032054D.7050508@cn.fujitsu.com> <20120820131642.GA551@Krystal> <5032D254.7040207@cn.fujitsu.com> <20120821010351.GA7472@Krystal> Message-ID: <5033079C.4080301@cn.fujitsu.com> On 08/21/2012 09:03 AM, Mathieu Desnoyers wrote: > * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: >> On 08/20/2012 09:16 PM, Mathieu Desnoyers wrote: >>> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: >>>> On 08/16/2012 05:31 AM, Mathieu Desnoyers wrote: >>>>> This work is derived from the patch from Lai Jiangshan submitted as >>>>> "urcu: new wfqueue implementation" >>>>> (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) >>>>> >>>> >>> >>> Hi Lai, >>> >>>> Hi, Mathieu >>>> >>>> please add this part to your patch. >>>> >>>> Changes(item5 has alternative): >>>> >>>> 1) Reorder the head and the tail in struct cds_wfq_queue >>>> Reason: wfq is enqueue-preference >>> >>> Is this a technical improvement over the original order ? Both cache >>> lines are aligned, so it should not matter which one comes first. So I'm >>> not sure why we should favor one order vs the other. >> >> struct astruct { >> int foo; /* field offset = 0 */ >> int bar; /* field offset = 4 */ >> }; >> >> Access to p->foo and access to p->bar are different in CPU instruction. >> Access to p->bar will need to add a offset to p. (the addition may cost a very small time) >> instructions will be probably longer than the instructions access to p->foo. >> >> If the code of access to p->bar is significant more than the code of access to p->foo, >> we should swap p->bar and p->foo field's order to shorten the total instruction size >> to reduce footprint. >> >> Our enqueue is inline function, when this inline function is expanded in every call site, >> the code of access to the tail will much significant, so we need to put the tail at first. >> > > Very good point! I'm pulling this change too. > >> >> >>> >>>> 2) Reorder the code of ___cds_wfq_next_blocking() >>>> Reason: short code, better readability >>> >>> Yes, I agree with this change. Can you extract it into a separate patch ? >> >> You merge it into your patch which ___cds_wfq_next_blocking() is introduced. >> > > OK. done, > >>> >>>> >>>> 3) Add cds_wfq_dequeue_[un]lock >>>> Reason: the fields of struct cds_wfq_queue are not exposed to users of lib, >>>> but some APIs needs to have this lock held. Add this locking function >>>> to allow the users use such APIs >>> >>> I agree with this change too. You can put it into the same patch as (2). >> >> You merge it into your patch. > > OK, done too. > > The current version (volatile branch) is at > > git://git.dorsal.polymtl.ca/~compudj/userspace-rcu > branch: urcu/wfqueue-volatile > > Thanks! > We can merge all things into a single patch until the discussion are finished. I don't want to have several commits in the git-tree that every commit fix some thing of the previous commit. When the big single patch is done, we can separate the patch properly. It will help others to understand the code from git-log. I will be very appreciate if one or two small patch(s) is committed with my name. Thanks, Lai. From kxhdyx at gmail.com Tue Aug 21 02:42:13 2012 From: kxhdyx at gmail.com (=?GB2312?B?v9zP/urN?=) Date: Tue, 21 Aug 2012 14:42:13 +0800 Subject: [lttng-dev] hi,I have some question about lttng Message-ID: Hi all: now my work need lttng ,and why lttng use macro like this:TRACEPOINT_EVENT(tracepoint_provider,event_name,?)? Except I can use any primitive C-type i want,is there any other adavantage about this macro type?And how lttng improve performance compare other log system(specially efficiency ).I need detail information about this mechanism.Can anyone help me? Best regard. -------------- next part -------------- An HTML attachment was scrubbed... URL: From francis.giraldeau at gmail.com Tue Aug 21 04:54:05 2012 From: francis.giraldeau at gmail.com (Francis Giraldeau) Date: Tue, 21 Aug 2012 10:54:05 +0200 Subject: [lttng-dev] hi,I have some question about lttng In-Reply-To: References: Message-ID: <50334CAD.70605@gmail.com> Le 2012-08-21 08:42, ??? a ?crit : > Hi all: > now my work need lttng ,and why lttng use macro like > this:TRACEPOINT_EVENT(tracepoint_provider,event_name,?)? > Except I can use any primitive C-type i want,is there any other > adavantage about this macro type? The macro is used as a way to generate the tracepoint code and data structure. Once this is done once, you can insert as many tracepoint call in your code. Have a look into lttng-ust/doc/examples. > And how lttng improve performance > compare other log system(specially efficiency ).I need detail > information about this mechanism.Can anyone help me? On my system (i5-540M) it takes about 250 nano second to write a simple event. In comparison, the logging framework Poco::Logger for a similar tracepoint requires about 12 micro second per event, roughtly 50x slower. The performance of such a loggin framework is likely to drop while increasing the number of concurrent threads, because the file is locked to prevent race at each event, while LTTng uses per-CPU buffers and has a lockless data structure and will scale perfectly. So, if you want to log few error messages once in a while, a logging library is fine. If you want trace requests or other high frequency events in the system and have precise timestamps, then LTTng is better suited. BR, Francis -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4489 bytes Desc: Signature cryptographique S/MIME URL: From mathieu.desnoyers at efficios.com Tue Aug 21 08:05:32 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 21 Aug 2012 08:05:32 -0400 Subject: [lttng-dev] [Question] the return of babeltrace's bt_context_create In-Reply-To: References: Message-ID: <20120821120531.GA13784@Krystal> I will let Julien look into this, Thanks, Mathieu * Hui Zhu (teawater at gmail.com) wrote: > Hi, > > The comments of this function said: > Returns an allocated context on success and NULL on error > > But when babeltrace use this function, it didn't check the return > value of bt_context_create: > ctx = bt_context_create(); > > ret = bt_context_add_traces_recursive(ctx, opt_input_path, > opt_input_format, NULL); > > Could you tell me why? Thanks a lot. > > Best, > Hui > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Tue Aug 21 08:12:26 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 21 Aug 2012 08:12:26 -0400 Subject: [lttng-dev] [RFC URCU PATCH] wfqueue: ABI v1, simplify implementation, 2.3x to 2.6x performance boost In-Reply-To: <5033079C.4080301@cn.fujitsu.com> References: <20120815213107.GB17224@Krystal> <5032054D.7050508@cn.fujitsu.com> <20120820131642.GA551@Krystal> <5032D254.7040207@cn.fujitsu.com> <20120821010351.GA7472@Krystal> <5033079C.4080301@cn.fujitsu.com> Message-ID: <20120821121226.GA13854@Krystal> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > On 08/21/2012 09:03 AM, Mathieu Desnoyers wrote: > > * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > >> On 08/20/2012 09:16 PM, Mathieu Desnoyers wrote: > >>> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: > >>>> On 08/16/2012 05:31 AM, Mathieu Desnoyers wrote: > >>>>> This work is derived from the patch from Lai Jiangshan submitted as > >>>>> "urcu: new wfqueue implementation" > >>>>> (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) > >>>>> > >>>> > >>> > >>> Hi Lai, > >>> > >>>> Hi, Mathieu > >>>> > >>>> please add this part to your patch. > >>>> > >>>> Changes(item5 has alternative): > >>>> > >>>> 1) Reorder the head and the tail in struct cds_wfq_queue > >>>> Reason: wfq is enqueue-preference > >>> > >>> Is this a technical improvement over the original order ? Both cache > >>> lines are aligned, so it should not matter which one comes first. So I'm > >>> not sure why we should favor one order vs the other. > >> > >> struct astruct { > >> int foo; /* field offset = 0 */ > >> int bar; /* field offset = 4 */ > >> }; > >> > >> Access to p->foo and access to p->bar are different in CPU instruction. > >> Access to p->bar will need to add a offset to p. (the addition may cost a very small time) > >> instructions will be probably longer than the instructions access to p->foo. > >> > >> If the code of access to p->bar is significant more than the code of access to p->foo, > >> we should swap p->bar and p->foo field's order to shorten the total instruction size > >> to reduce footprint. > >> > >> Our enqueue is inline function, when this inline function is expanded in every call site, > >> the code of access to the tail will much significant, so we need to put the tail at first. > >> > > > > Very good point! I'm pulling this change too. > > > >> > >> > >>> > >>>> 2) Reorder the code of ___cds_wfq_next_blocking() > >>>> Reason: short code, better readability > >>> > >>> Yes, I agree with this change. Can you extract it into a separate patch ? > >> > >> You merge it into your patch which ___cds_wfq_next_blocking() is introduced. > >> > > > > OK. done, > > > >>> > >>>> > >>>> 3) Add cds_wfq_dequeue_[un]lock > >>>> Reason: the fields of struct cds_wfq_queue are not exposed to users of lib, > >>>> but some APIs needs to have this lock held. Add this locking function > >>>> to allow the users use such APIs > >>> > >>> I agree with this change too. You can put it into the same patch as (2). > >> > >> You merge it into your patch. > > > > OK, done too. > > > > The current version (volatile branch) is at > > > > git://git.dorsal.polymtl.ca/~compudj/userspace-rcu > > branch: urcu/wfqueue-volatile > > > > Thanks! > > > > > We can merge all things into a single patch until the discussion are finished. > I don't want to have several commits in the git-tree that every > commit fix some thing of the previous commit. Yes, I agree. > > When the big single patch is done, we can separate the patch properly. > It will help others to understand the code from git-log. Yes, that can be done. Hopefully we'll manage to find the time to do this. > > I will be very appreciate if one or two small patch(s) is committed > with my name. Of course! The only reason why I set the From on the current patch to my own name is because I did not want to give you the blame for my own errors (although I kept the credits to you in the changelog). Splitting the work in sub-patches will make it easier to credit each of us for specific portions of the code. Thanks! Mathieu > > Thanks, > Lai. -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From jdesfossez at efficios.com Tue Aug 21 09:43:35 2012 From: jdesfossez at efficios.com (Julien Desfossez) Date: Tue, 21 Aug 2012 09:43:35 -0400 Subject: [lttng-dev] [BABELTRACE PATCH] Fix: check return value of bt_context_create In-Reply-To: <20120821120531.GA13784@Krystal> References: <20120821120531.GA13784@Krystal> Message-ID: <1345556615-8109-1-git-send-email-jdesfossez@efficios.com> Signed-off-by: Julien Desfossez --- converter/babeltrace.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/converter/babeltrace.c b/converter/babeltrace.c index f5541d2..c9007c3 100644 --- a/converter/babeltrace.c +++ b/converter/babeltrace.c @@ -549,6 +549,9 @@ int main(int argc, char **argv) } ctx = bt_context_create(); + if (!ctx) { + goto error_td_read; + } ret = bt_context_add_traces_recursive(ctx, opt_input_path, opt_input_format, NULL); -- 1.7.10.4 From mathieu.desnoyers at efficios.com Tue Aug 21 10:45:33 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 21 Aug 2012 10:45:33 -0400 Subject: [lttng-dev] [BABELTRACE PATCH] Fix: check return value of bt_context_create In-Reply-To: <1345556615-8109-1-git-send-email-jdesfossez@efficios.com> References: <20120821120531.GA13784@Krystal> <1345556615-8109-1-git-send-email-jdesfossez@efficios.com> Message-ID: <20120821144533.GB16246@Krystal> * Julien Desfossez (jdesfossez at efficios.com) wrote: > Signed-off-by: Julien Desfossez Merged, thanks! Mathieu > --- > converter/babeltrace.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/converter/babeltrace.c b/converter/babeltrace.c > index f5541d2..c9007c3 100644 > --- a/converter/babeltrace.c > +++ b/converter/babeltrace.c > @@ -549,6 +549,9 @@ int main(int argc, char **argv) > } > > ctx = bt_context_create(); > + if (!ctx) { > + goto error_td_read; > + } > > ret = bt_context_add_traces_recursive(ctx, opt_input_path, > opt_input_format, NULL); > -- > 1.7.10.4 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From francis.giraldeau at gmail.com Wed Aug 22 07:12:01 2012 From: francis.giraldeau at gmail.com (Francis Giraldeau) Date: Wed, 22 Aug 2012 13:12:01 +0200 Subject: [lttng-dev] hi,I have some question about lttng In-Reply-To: References: <50334CAD.70605@gmail.com> Message-ID: <5034BE81.30701@gmail.com> Le 2012-08-22 09:56, ??? a ?crit : > 1.I already use this macro TRACEPOINT_EVENT(I have finished some > program)in my code,but except these usage your mentioned before,but > When i need a new tracepoint() formation,i must define a corresponding > macro TRACEPOINT_EVENT in header file.I need to change Argument Listing: > TP_ARGS(...),and Fields Listing TP_FIELDS(...).Compare to Variadic > Function like printf(...),it's too complicated and inconvenience.Is > there any reason lttng chose this method? Yes. First, it's typesafe. The compiler will tell you if arguments do not match the signature. Also, the macro is used as a way to generates metadata in the trace that describe the format and length of each fields. It's less convenient than having varargs, but it's the most performant and reliable. > 2.Now we have a clusters environment,So company chose lttng as default > log system.Is this sentence (while LTTng uses per-CPU buffers and has a > lockless data structure and will scale perfectly.) your mentioned > primary cause?If it is ,would you please explain me the detail about > these.And if not,how lttng improve the performance.I want to know > why,not comparison results.Thanks again for resolving my questions. The details are very complicated, but to make a summary, mention that you have a shared resource (like a buffer or a file) to write into, then you need to protect it by locks. To avoid that, LTTng uses per-CPU buffers. Let's say on a two cores system there are at most 2 threads running concurrently at any time. Then, if they got each their own buffer, they will not fight because they have their own buffer. The other major aspect is about RCU data structure used, that is itself lockless, is capable to allow concurrent read and write access to a data structure and produces much less cache-line replication and misses than normal lock. Please look at the LWN article on this for more details. http://lwn.net/Articles/263130/ > And how did lttng consumerd work? In a sentence, it's using splice system call to write blocks of memory to disk. > Why we need a macro > TRACEPOINT_CREATE_PROBES in a new .c file. It's just a way to compile the tracepoint code. It's boilerplate code and the macro is taking care of generating this code. Cheers, Francis -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4489 bytes Desc: Signature cryptographique S/MIME URL: From mathieu.desnoyers at efficios.com Wed Aug 22 07:53:08 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Wed, 22 Aug 2012 07:53:08 -0400 Subject: [lttng-dev] hi,I have some question about lttng In-Reply-To: <5034BE81.30701@gmail.com> References: <50334CAD.70605@gmail.com> <5034BE81.30701@gmail.com> Message-ID: <20120822115308.GA32691@Krystal> * Francis Giraldeau (francis.giraldeau at gmail.com) wrote: > Le 2012-08-22 09:56, ??? a ?crit : > > 1.I already use this macro TRACEPOINT_EVENT(I have finished some > > program)in my code,but except these usage your mentioned before,but > > When i need a new tracepoint() formation,i must define a corresponding > > macro TRACEPOINT_EVENT in header file.I need to change Argument Listing: > > TP_ARGS(...),and Fields Listing TP_FIELDS(...).Compare to Variadic > > Function like printf(...),it's too complicated and inconvenience.Is > > there any reason lttng chose this method? > > Yes. First, it's typesafe. The compiler will tell you if arguments do > not match the signature. Also, the macro is used as a way to generates > metadata in the trace that describe the format and length of each > fields. It's less convenient than having varargs, but it's the most > performant and reliable. > > > 2.Now we have a clusters environment,So company chose lttng as default > > log system.Is this sentence (while LTTng uses per-CPU buffers and has a > > lockless data structure and will scale perfectly.) your mentioned > > primary cause?If it is ,would you please explain me the detail about > > these.And if not,how lttng improve the performance.I want to know > > why,not comparison results.Thanks again for resolving my questions. > > The details are very complicated, but to make a summary, mention that > you have a shared resource (like a buffer or a file) to write into, then > you need to protect it by locks. To avoid that, LTTng uses per-CPU > buffers. Let's say on a two cores system there are at most 2 threads > running concurrently at any time. Then, if they got each their own > buffer, they will not fight because they have their own buffer. The > other major aspect is about RCU data structure used, that is itself > lockless, is capable to allow concurrent read and write access to a data > structure and produces much less cache-line replication and misses than > normal lock. Please look at the LWN article on this for more details. > > http://lwn.net/Articles/263130/ > > > And how did lttng consumerd work? > > In a sentence, it's using splice system call to write blocks of memory > to disk. FYI, lttng-modules (the kernel tracer) uses splice by default to transport data from the buffers to file or to the network. lttng-ust (userspace tracer) writes from memory mapped buffers to a file/network socket. Thanks, Mathieu > > > Why we need a macro > > TRACEPOINT_CREATE_PROBES in a new .c file. > > It's just a way to compile the tracepoint code. It's boilerplate code > and the macro is taking care of generating this code. > > Cheers, > > Francis > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From Bernd.Hufmann at ericsson.com Wed Aug 22 08:46:54 2012 From: Bernd.Hufmann at ericsson.com (Bernd Hufmann) Date: Wed, 22 Aug 2012 08:46:54 -0400 Subject: [lttng-dev] [RELEASE] LTTng-tools 2.1.0-rc1 In-Reply-To: <502EA58E.2060003@efficios.com> References: <502EA58E.2060003@efficios.com> Message-ID: <5034D4BE.7060400@ericsson.com> Hi David I've just installed LTTng-tools 2.1.0-rc1 and I wanted to use it with the Eclipse LTTng tracer control. First, I added 2.1.x as supported version to Eclipse and then I tried to used it with 2.0.x features. I noticed a few differences in the output strings of command outputs [1] as well as the directory structure of generated traces [2]. The Eclipse LTTng tracer control parses the String output of the commands to extract relevant information. Generally, it would be much easier to maintain the Eclipse LTTng tracer control if for existing features the string outputs and the directory structure are the same as before. Please note, if someone uses scripts to generate traces the output format is also important. [1] Command output of lttng create mysession: In v2.0.x the output was: Session mysession created. Traces will be written in /home/user/lttng-traces/mysession-20120314-132824 in v2.1.x the output is: Trace(s) output set to /home/user/lttng-traces/mysession-20120822-082243 Session mysession created. So, the first and second line are swapped. Also the content of the line with the path is changed. Would it be possible, to have the same order of the lines and same output as in v2.0.x? [2] Trace directory structure: I noticed that the UST traces are moved under another sub-directory. In v2.0.x the directory structure looked as follows: /home/user/lttng-traces/ |-mysession-20120314-132824 |-kernel |--- |-ust |----> |----------- In v2.1.x the directory structure looked as follows: /home/user/lttng-traces/ |-mysession-20120314-132824 |-kernel |--- |-ust |--- mysession-20120314-132824 |----> |----------- So as you see, under ust there is another sub directory with trace session name and date/time. I don't see a practical reason to have this sub directory, because the session name and date/time is already known from the top-level directory. Is it possible to revert this back? In Eclipse I would have to implement a special case for different LTTng-tools versions (in the import dialog). Thank you very much in advance. Best Regards Bernd On 08/17/2012 04:11 PM, David Goulet wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Greetings everyone (including LTTng elves), > > The lttng-tools project provides a session daemon (lttng-sessiond) > that acts as a tracing registry, the "lttng" command line for tracing > control, a lttng-ctl library for tracing control and a lttng-relayd > for network streaming. > > This is the first release candidate for lttng-tools 2.1 which brings > two exciting new features, network streaming and filtering support. > > It's the combination of a lot of work from the LTTng team so please, > to help us improve our tools, report any bugs or misbehaving > feature(s) that you may encounter through this mailing list or the bug > tracker (https://bugs.lttng.org). > > - From now on, lttng-tools is in feature freeze mode unless an *IMMENSE* > show stopper is found. > > 2012-08-17 lttng-tools 2.1.0-rc1 > * Feature: Network Streaming > * Add the lttng-relayd binary for network streaming > * Support user space tracer filtering > * Multiple fixes > > Project website: http://lttng.org/lttng2.0 > Download link: > http://lttng.org/files/lttng-tools/lttng-tools-2.1.0-rc1.tar.bz2 > > Cheers! > David > -----BEGIN PGP SIGNATURE----- > > iQEcBAEBCgAGBQJQLqWLAAoJEELoaioR9I023WUH+gNuU+NL0gZInlFbFiqGG5n8 > DyvtZaUOtHr7ZoSU2cmEcNfgxzOj0jTt5lGPm7EaU7X5OUiQCiu8gz5kKsL0h5PW > UPZ7DbGHxv8JQQCt/Ac6cydlOjbRT1fB/5GXGT+OX+g/Xzp+Utb6vrTCWCC0RvSO > 01U2NK2+IWTuRTol7ykK1WWqTStQSKPZ6ncdYjgq1FJu2gakQA4RnhblMbXWCuP9 > EZPxafSaH19mHlhDiDmtdTl7hcWTPAd0u+qNoq7kmh+stcNSn2zjv/bNldbNfaHI > m0kfl8NferiyRpyeD4CgNtFE5W4jm29NId7IRI14/Ow8psZegF8SNlAL76Puqo8= > =PU4p > -----END PGP SIGNATURE----- > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev From dgoulet at efficios.com Wed Aug 22 09:22:58 2012 From: dgoulet at efficios.com (David Goulet) Date: Wed, 22 Aug 2012 09:22:58 -0400 Subject: [lttng-dev] [RELEASE] LTTng-tools 2.1.0-rc1 In-Reply-To: <5034D4BE.7060400@ericsson.com> References: <502EA58E.2060003@efficios.com> <5034D4BE.7060400@ericsson.com> Message-ID: <5034DD32.1010309@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi Bernd, Comments below. Bernd Hufmann: > Hi David > > I've just installed LTTng-tools 2.1.0-rc1 and I wanted to use it > with the Eclipse LTTng tracer control. First, I added 2.1.x as > supported version to Eclipse and then I tried to used it with 2.0.x > features. I noticed a few differences in the output strings of > command outputs [1] as well as the directory structure of generated > traces [2]. The Eclipse LTTng tracer control parses the String > output of the commands to extract relevant information. > > Generally, it would be much easier to maintain the Eclipse LTTng > tracer control if for existing features the string outputs and the > directory structure are the same as before. Please note, if someone > uses scripts to generate traces the output format is also > important. > > [1] Command output of lttng create mysession: In v2.0.x the output > was: Session mysession created. Traces will be written in > /home/user/lttng-traces/mysession-20120314-132824 > > in v2.1.x the output is: Trace(s) output set to > /home/user/lttng-traces/mysession-20120822-082243 Session mysession > created. > > So, the first and second line are swapped. Also the content of the > line with the path is changed. Would it be possible, to have the > same order of the lines and same output as in v2.0.x? Hmmmm, that's a fair point. Normally I would *strongly* recommand you use "lttng list mysession" to get the output path but still, we could make the argument that the output of the "create" command should not changed between minor version. > > [2] Trace directory structure: I noticed that the UST traces are > moved under another sub-directory. > > In v2.0.x the directory structure looked as follows: > /home/user/lttng-traces/ |-mysession-20120314-132824 |-kernel > |--- |-ust |----> > |----------- > > In v2.1.x the directory structure looked as follows: > /home/user/lttng-traces/ |-mysession-20120314-132824 |-kernel > |--- |-ust |--- mysession-20120314-132824 > |----> |----------- files> This is absolutely not suppose to be like that... I'm unable to reproduce this behavior having the "mysession-**" created two times. Can you give me the exact series of command you do to get this directory structure? The normal case and wht 2.1.x should to is provide you the same directory structure as 2.0.x. Ex: ~/lttng-traces/test-20120822-091412/ust/lt-hello-4542-20120822-091415 Thanks! David > > So as you see, under ust there is another sub directory with trace > session name and date/time. I don't see a practical reason to have > this sub directory, because the session name and date/time is > already known from the top-level directory. Is it possible to > revert this back? In Eclipse I would have to implement a special > case for different LTTng-tools versions (in the import dialog). > > Thank you very much in advance. > > Best Regards Bernd > > > On 08/17/2012 04:11 PM, David Goulet wrote: Greetings everyone > (including LTTng elves), > > The lttng-tools project provides a session daemon (lttng-sessiond) > that acts as a tracing registry, the "lttng" command line for > tracing control, a lttng-ctl library for tracing control and a > lttng-relayd for network streaming. > > This is the first release candidate for lttng-tools 2.1 which > brings two exciting new features, network streaming and filtering > support. > > It's the combination of a lot of work from the LTTng team so > please, to help us improve our tools, report any bugs or > misbehaving feature(s) that you may encounter through this mailing > list or the bug tracker (https://bugs.lttng.org). > > - From now on, lttng-tools is in feature freeze mode unless an > *IMMENSE* show stopper is found. > > 2012-08-17 lttng-tools 2.1.0-rc1 * Feature: Network Streaming * Add > the lttng-relayd binary for network streaming * Support user space > tracer filtering * Multiple fixes > > Project website: http://lttng.org/lttng2.0 Download link: > http://lttng.org/files/lttng-tools/lttng-tools-2.1.0-rc1.tar.bz2 > > Cheers! David >> >> _______________________________________________ lttng-dev mailing >> list lttng-dev at lists.lttng.org >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > > _______________________________________________ lttng-dev mailing > list lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQNN0vAAoJEELoaioR9I02VxMH/jQkvukB85gPaAiirzf5P2Zo 0cudVwdvCuK+43mFUsuN9l7FQh0MKjplWtpqDHDqc1uxgQ7MAGnD+ZzHY347qGVx Qdcrn+DBUIeawExfqrqRNZZpiQnKGf4dp0TkQNBHiq72ymVHd3lXCT8NPbrBn8T2 9v0FRgLZW29Z/M87I71n5JfPMrg5iU8e8bcNnl1gEL+rCe26Sxb+1MPtmQ3s7XSa DYCwYxrWQsCJUxqAfIP4CJQasPiYwPFb3uCyTw5zf6YO+8jpVQOEN5XF6hAsLoWK 2XLgDPCFqJAOAXjswykK5SGVw8QuawQQnsQYeG37mxB2lhe29m3JnX4gl1kKwHE= =8KxL -----END PGP SIGNATURE----- From christian.babeux at efficios.com Wed Aug 22 09:34:03 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Wed, 22 Aug 2012 09:34:03 -0400 Subject: [lttng-dev] [PATCH lttng-tools] Filter: Handle the unary bitwise not operator (~) with an unsupported op error Message-ID: <1345642443-30197-1-git-send-email-christian.babeux@efficios.com> Signed-off-by: Christian Babeux --- src/lib/lttng-ctl/filter-ast.h | 1 + src/lib/lttng-ctl/filter-parser.y | 5 +++++ src/lib/lttng-ctl/filter-visitor-generate-ir.c | 12 ++++++++++++ 3 files changed, 18 insertions(+) diff --git a/src/lib/lttng-ctl/filter-ast.h b/src/lib/lttng-ctl/filter-ast.h index 97793c0..1767164 100644 --- a/src/lib/lttng-ctl/filter-ast.h +++ b/src/lib/lttng-ctl/filter-ast.h @@ -91,6 +91,7 @@ enum unary_op_type { AST_UNARY_PLUS, AST_UNARY_MINUS, AST_UNARY_NOT, + AST_UNARY_BIN_NOT, }; enum ast_link_type { diff --git a/src/lib/lttng-ctl/filter-parser.y b/src/lib/lttng-ctl/filter-parser.y index 4ee1d9a..d3be4be 100644 --- a/src/lib/lttng-ctl/filter-parser.y +++ b/src/lib/lttng-ctl/filter-parser.y @@ -481,6 +481,11 @@ unary_operator $$ = make_node(parser_ctx, NODE_UNARY_OP); $$->u.unary_op.type = AST_UNARY_NOT; } + | NOT_BIN + { + $$ = make_node(parser_ctx, NODE_UNARY_OP); + $$->u.unary_op.type = AST_UNARY_BIN_NOT; + } ; multiplicative_expression diff --git a/src/lib/lttng-ctl/filter-visitor-generate-ir.c b/src/lib/lttng-ctl/filter-visitor-generate-ir.c index 899713e..d23372f 100644 --- a/src/lib/lttng-ctl/filter-visitor-generate-ir.c +++ b/src/lib/lttng-ctl/filter-visitor-generate-ir.c @@ -652,6 +652,8 @@ static struct ir_op *make_unary_op(struct filter_parser_ctx *ctx, struct filter_node *node, enum ir_side side) { + const char *op_str = "?"; + switch (node->u.unary_op.type) { case AST_UNARY_UNKNOWN: default: @@ -703,7 +705,17 @@ struct ir_op *make_unary_op(struct filter_parser_ctx *ctx, } return op; } + case AST_UNARY_BIN_NOT: + { + op_str = "~"; + goto error_not_supported; + } } + +error_not_supported: + fprintf(stderr, "[error] %s: unary operation '%s' not supported\n", + __func__, op_str); + return NULL; } static -- 1.7.11.4 From mathieu.desnoyers at efficios.com Wed Aug 22 09:55:43 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Wed, 22 Aug 2012 09:55:43 -0400 Subject: [lttng-dev] [PATCH lttng-tools] Filter: Handle the unary bitwise not operator (~) with an unsupported op error In-Reply-To: <1345642443-30197-1-git-send-email-christian.babeux@efficios.com> References: <1345642443-30197-1-git-send-email-christian.babeux@efficios.com> Message-ID: <20120822135543.GB3499@Krystal> * Christian Babeux (christian.babeux at efficios.com) wrote: Acked-by: Mathieu Desnoyers > > Signed-off-by: Christian Babeux > --- > src/lib/lttng-ctl/filter-ast.h | 1 + > src/lib/lttng-ctl/filter-parser.y | 5 +++++ > src/lib/lttng-ctl/filter-visitor-generate-ir.c | 12 ++++++++++++ > 3 files changed, 18 insertions(+) > > diff --git a/src/lib/lttng-ctl/filter-ast.h b/src/lib/lttng-ctl/filter-ast.h > index 97793c0..1767164 100644 > --- a/src/lib/lttng-ctl/filter-ast.h > +++ b/src/lib/lttng-ctl/filter-ast.h > @@ -91,6 +91,7 @@ enum unary_op_type { > AST_UNARY_PLUS, > AST_UNARY_MINUS, > AST_UNARY_NOT, > + AST_UNARY_BIN_NOT, > }; > > enum ast_link_type { > diff --git a/src/lib/lttng-ctl/filter-parser.y b/src/lib/lttng-ctl/filter-parser.y > index 4ee1d9a..d3be4be 100644 > --- a/src/lib/lttng-ctl/filter-parser.y > +++ b/src/lib/lttng-ctl/filter-parser.y > @@ -481,6 +481,11 @@ unary_operator > $$ = make_node(parser_ctx, NODE_UNARY_OP); > $$->u.unary_op.type = AST_UNARY_NOT; > } > + | NOT_BIN > + { > + $$ = make_node(parser_ctx, NODE_UNARY_OP); > + $$->u.unary_op.type = AST_UNARY_BIN_NOT; > + } > ; > > multiplicative_expression > diff --git a/src/lib/lttng-ctl/filter-visitor-generate-ir.c b/src/lib/lttng-ctl/filter-visitor-generate-ir.c > index 899713e..d23372f 100644 > --- a/src/lib/lttng-ctl/filter-visitor-generate-ir.c > +++ b/src/lib/lttng-ctl/filter-visitor-generate-ir.c > @@ -652,6 +652,8 @@ static > struct ir_op *make_unary_op(struct filter_parser_ctx *ctx, > struct filter_node *node, enum ir_side side) > { > + const char *op_str = "?"; > + > switch (node->u.unary_op.type) { > case AST_UNARY_UNKNOWN: > default: > @@ -703,7 +705,17 @@ struct ir_op *make_unary_op(struct filter_parser_ctx *ctx, > } > return op; > } > + case AST_UNARY_BIN_NOT: > + { > + op_str = "~"; > + goto error_not_supported; > + } > } > + > +error_not_supported: > + fprintf(stderr, "[error] %s: unary operation '%s' not supported\n", > + __func__, op_str); > + return NULL; > } > > static > -- > 1.7.11.4 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From Bernd.Hufmann at ericsson.com Wed Aug 22 10:27:55 2012 From: Bernd.Hufmann at ericsson.com (Bernd Hufmann) Date: Wed, 22 Aug 2012 10:27:55 -0400 Subject: [lttng-dev] [RELEASE] LTTng-tools 2.1.0-rc1 In-Reply-To: <5034DD32.1010309@efficios.com> References: <502EA58E.2060003@efficios.com> <5034D4BE.7060400@ericsson.com> <5034DD32.1010309@efficios.com> Message-ID: <5034EC6B.7010206@ericsson.com> Hi David thanks for your answers. See below for more information. \Bernd On 08/22/2012 09:22 AM, David Goulet wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Hi Bernd, > > Comments below. > > Bernd Hufmann: >> Hi David >> >> I've just installed LTTng-tools 2.1.0-rc1 and I wanted to use it >> with the Eclipse LTTng tracer control. First, I added 2.1.x as >> supported version to Eclipse and then I tried to used it with 2.0.x >> features. I noticed a few differences in the output strings of >> command outputs [1] as well as the directory structure of generated >> traces [2]. The Eclipse LTTng tracer control parses the String >> output of the commands to extract relevant information. >> >> Generally, it would be much easier to maintain the Eclipse LTTng >> tracer control if for existing features the string outputs and the >> directory structure are the same as before. Please note, if someone >> uses scripts to generate traces the output format is also >> important. >> >> [1] Command output of lttng create mysession: In v2.0.x the output >> was: Session mysession created. Traces will be written in >> /home/user/lttng-traces/mysession-20120314-132824 >> >> in v2.1.x the output is: Trace(s) output set to >> /home/user/lttng-traces/mysession-20120822-082243 Session mysession >> created. >> >> So, the first and second line are swapped. Also the content of the >> line with the path is changed. Would it be possible, to have the >> same order of the lines and same output as in v2.0.x? > Hmmmm, that's a fair point. Normally I would *strongly* recommand you > use "lttng list mysession" to get the output path but still, we could > make the argument that the output of the "create" command should not > changed between minor version. I use the list command to get the directory of a session and the session name. However, there are some validation done after "lttng create session" (e.g. check for correct session name or path). So the order of lines and string content is important. I already use regular expressions to problems with minor changes (e.g. more whitespaces), but we cannot foresee all cases. As I understand the numbering scheme, the first digit is for major changes (non-backwards compatible, API breaking changes). The second digit is for non-API breaking changes including feature and API additions. And the last digit (minor) is for bugfixes. Since there is no remote protocol defined for using LTTng tools remotely, the Eclipse LTTng control relies on the string output of the command line tool. So, I consider the output as part of the API :-). >> [2] Trace directory structure: I noticed that the UST traces are >> moved under another sub-directory. >> >> In v2.0.x the directory structure looked as follows: >> /home/user/lttng-traces/ |-mysession-20120314-132824 |-kernel >> |--- |-ust |----> >> |----------- >> >> In v2.1.x the directory structure looked as follows: >> /home/user/lttng-traces/ |-mysession-20120314-132824 |-kernel >> |--- |-ust |--- mysession-20120314-132824 >> |----> |-----------> files> > This is absolutely not suppose to be like that... I'm unable to > reproduce this behavior having the "mysession-**" created two times. > > Can you give me the exact series of command you do to get this > directory structure? See below the log of commands I executed. I also added the output of command "find ." under lttng-traces directory. In there you see the additional sub-directory. By the way, I installed lttng-tools, lttng-ust and userspace-rcu from source code. Here are the corresponding SHA numbers: lttng-tools: 68264071f9d1b789de1350cbec479b52a9b54acf lttng-ust: 0f4eaec3e738fa0f33296a46fe08266a60787c23 userspace-rcu: 768fba83676f49eb73fd1d8ad452016a84c5ec2a > lttng create mysession Trace(s) output set to /home/bernd/lttng-traces/mysession-20120822-094418 Session mysession created. > lttng enable-event -a -k All kernel events are enabled in channel channel0 > lttng enable-event -a -u All UST events are enabled in channel channel0 > lttng start Tracing started for session mysession > lttng stop Tracing stopped for session mysession > lttng destroy Session mysession destroyed > find . . ./mysession-20120822-094418 ./mysession-20120822-094418/mysession-20120822-094542 ./mysession-20120822-094418/mysession-20120822-094542/ust ./mysession-20120822-094418/mysession-20120822-094542/ust/lt-hello-2039-20120822-094551 ./mysession-20120822-094418/mysession-20120822-094542/ust/lt-hello-2039-20120822-094551/channel0_0 ./mysession-20120822-094418/mysession-20120822-094542/ust/lt-hello-2039-20120822-094551/metadata ./mysession-20120822-094418/mysession-20120822-094542/ust/lt-hello-2056-20120822-094551 ./mysession-20120822-094418/mysession-20120822-094542/ust/lt-hello-2056-20120822-094551/channel0_0 ./mysession-20120822-094418/mysession-20120822-094542/ust/lt-hello-2056-20120822-094551/metadata ./mysession-20120822-094418/kernel ./mysession-20120822-094418/kernel/channel0_0 ./mysession-20120822-094418/kernel/metadata > The normal case and wht 2.1.x should to is provide you the same > directory structure as 2.0.x. Ex: > > ~/lttng-traces/test-20120822-091412/ust/lt-hello-4542-20120822-091415 > > Thanks! > David > >> So as you see, under ust there is another sub directory with trace >> session name and date/time. I don't see a practical reason to have >> this sub directory, because the session name and date/time is >> already known from the top-level directory. Is it possible to >> revert this back? In Eclipse I would have to implement a special >> case for different LTTng-tools versions (in the import dialog). >> >> Thank you very much in advance. >> >> Best Regards Bernd >> >> >> On 08/17/2012 04:11 PM, David Goulet wrote: Greetings everyone >> (including LTTng elves), >> >> The lttng-tools project provides a session daemon (lttng-sessiond) >> that acts as a tracing registry, the "lttng" command line for >> tracing control, a lttng-ctl library for tracing control and a >> lttng-relayd for network streaming. >> >> This is the first release candidate for lttng-tools 2.1 which >> brings two exciting new features, network streaming and filtering >> support. >> >> It's the combination of a lot of work from the LTTng team so >> please, to help us improve our tools, report any bugs or >> misbehaving feature(s) that you may encounter through this mailing >> list or the bug tracker (https://bugs.lttng.org). >> >> - From now on, lttng-tools is in feature freeze mode unless an >> *IMMENSE* show stopper is found. >> >> 2012-08-17 lttng-tools 2.1.0-rc1 * Feature: Network Streaming * Add >> the lttng-relayd binary for network streaming * Support user space >> tracer filtering * Multiple fixes >> >> Project website: http://lttng.org/lttng2.0 Download link: >> http://lttng.org/files/lttng-tools/lttng-tools-2.1.0-rc1.tar.bz2 >> >> Cheers! David >>> _______________________________________________ lttng-dev mailing >>> list lttng-dev at lists.lttng.org >>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >> >> _______________________________________________ lttng-dev mailing >> list lttng-dev at lists.lttng.org >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > -----BEGIN PGP SIGNATURE----- > > iQEcBAEBCgAGBQJQNN0vAAoJEELoaioR9I02VxMH/jQkvukB85gPaAiirzf5P2Zo > 0cudVwdvCuK+43mFUsuN9l7FQh0MKjplWtpqDHDqc1uxgQ7MAGnD+ZzHY347qGVx > Qdcrn+DBUIeawExfqrqRNZZpiQnKGf4dp0TkQNBHiq72ymVHd3lXCT8NPbrBn8T2 > 9v0FRgLZW29Z/M87I71n5JfPMrg5iU8e8bcNnl1gEL+rCe26Sxb+1MPtmQ3s7XSa > DYCwYxrWQsCJUxqAfIP4CJQasPiYwPFb3uCyTw5zf6YO+8jpVQOEN5XF6hAsLoWK > 2XLgDPCFqJAOAXjswykK5SGVw8QuawQQnsQYeG37mxB2lhe29m3JnX4gl1kKwHE= > =8KxL > -----END PGP SIGNATURE----- From david.goulet at polymtl.ca Wed Aug 22 11:04:41 2012 From: david.goulet at polymtl.ca (David Goulet) Date: Wed, 22 Aug 2012 11:04:41 -0400 Subject: [lttng-dev] [PATCH lttng-tools] Filter: Handle the unary bitwise not operator (~) with an unsupported op error In-Reply-To: <1345642443-30197-1-git-send-email-christian.babeux@efficios.com> References: <1345642443-30197-1-git-send-email-christian.babeux@efficios.com> Message-ID: <5034F509.5020503@polymtl.ca> Merged! Thanks! On 22/08/12 09:34 AM, Christian Babeux wrote: > > Signed-off-by: Christian Babeux --- > src/lib/lttng-ctl/filter-ast.h | 1 + > src/lib/lttng-ctl/filter-parser.y | 5 +++++ > src/lib/lttng-ctl/filter-visitor-generate-ir.c | 12 ++++++++++++ 3 files > changed, 18 insertions(+) > > diff --git a/src/lib/lttng-ctl/filter-ast.h > b/src/lib/lttng-ctl/filter-ast.h index 97793c0..1767164 100644 --- > a/src/lib/lttng-ctl/filter-ast.h +++ b/src/lib/lttng-ctl/filter-ast.h @@ > -91,6 +91,7 @@ enum unary_op_type { AST_UNARY_PLUS, AST_UNARY_MINUS, > AST_UNARY_NOT, + AST_UNARY_BIN_NOT, }; > > enum ast_link_type { diff --git a/src/lib/lttng-ctl/filter-parser.y > b/src/lib/lttng-ctl/filter-parser.y index 4ee1d9a..d3be4be 100644 --- > a/src/lib/lttng-ctl/filter-parser.y +++ > b/src/lib/lttng-ctl/filter-parser.y @@ -481,6 +481,11 @@ unary_operator $$ > = make_node(parser_ctx, NODE_UNARY_OP); $$->u.unary_op.type = > AST_UNARY_NOT; } + | NOT_BIN + { + $$ = make_node(parser_ctx, > NODE_UNARY_OP); + $$->u.unary_op.type = AST_UNARY_BIN_NOT; + } ; > > multiplicative_expression diff --git > a/src/lib/lttng-ctl/filter-visitor-generate-ir.c > b/src/lib/lttng-ctl/filter-visitor-generate-ir.c index 899713e..d23372f > 100644 --- a/src/lib/lttng-ctl/filter-visitor-generate-ir.c +++ > b/src/lib/lttng-ctl/filter-visitor-generate-ir.c @@ -652,6 +652,8 @@ > static struct ir_op *make_unary_op(struct filter_parser_ctx *ctx, struct > filter_node *node, enum ir_side side) { + const char *op_str = "?"; + > switch (node->u.unary_op.type) { case AST_UNARY_UNKNOWN: default: @@ -703,7 > +705,17 @@ struct ir_op *make_unary_op(struct filter_parser_ctx *ctx, } > return op; } + case AST_UNARY_BIN_NOT: + { + op_str = "~"; + goto > error_not_supported; + } } + +error_not_supported: + fprintf(stderr, > "[error] %s: unary operation '%s' not supported\n", + __func__, op_str); + > return NULL; } > > static From david.goulet at polymtl.ca Wed Aug 22 11:57:01 2012 From: david.goulet at polymtl.ca (David Goulet) Date: Wed, 22 Aug 2012 11:57:01 -0400 Subject: [lttng-dev] [RELEASE] LTTng-tools 2.1.0-rc1 In-Reply-To: <5034EC6B.7010206@ericsson.com> References: <502EA58E.2060003@efficios.com> <5034D4BE.7060400@ericsson.com> <5034DD32.1010309@efficios.com> <5034EC6B.7010206@ericsson.com> Message-ID: <5035014D.1080701@polymtl.ca> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 22/08/12 10:27 AM, Bernd Hufmann wrote: > Hi David > > thanks for your answers. See below for more information. > > \Bernd > > On 08/22/2012 09:22 AM, David Goulet wrote: Hi Bernd, > > Comments below. > > Bernd Hufmann: >>>> Hi David >>>> >>>> I've just installed LTTng-tools 2.1.0-rc1 and I wanted to use it with >>>> the Eclipse LTTng tracer control. First, I added 2.1.x as supported >>>> version to Eclipse and then I tried to used it with 2.0.x features. I >>>> noticed a few differences in the output strings of command outputs >>>> [1] as well as the directory structure of generated traces [2]. The >>>> Eclipse LTTng tracer control parses the String output of the commands >>>> to extract relevant information. >>>> >>>> Generally, it would be much easier to maintain the Eclipse LTTng >>>> tracer control if for existing features the string outputs and the >>>> directory structure are the same as before. Please note, if someone >>>> uses scripts to generate traces the output format is also important. >>>> >>>> [1] Command output of lttng create mysession: In v2.0.x the output >>>> was: Session mysession created. Traces will be written in >>>> /home/user/lttng-traces/mysession-20120314-132824 >>>> >>>> in v2.1.x the output is: Trace(s) output set to >>>> /home/user/lttng-traces/mysession-20120822-082243 Session mysession >>>> created. >>>> >>>> So, the first and second line are swapped. Also the content of the >>>> line with the path is changed. Would it be possible, to have the same >>>> order of the lines and same output as in v2.0.x? > Hmmmm, that's a fair point. Normally I would *strongly* recommand you use > "lttng list mysession" to get the output path but still, we could make the > argument that the output of the "create" command should not changed between > minor version. >> I use the list command to get the directory of a session and the session >> name. However, there are some validation done after "lttng create >> session" (e.g. check for correct session name or path). So the order of >> lines and string content is important. I already use regular expressions >> to problems with minor changes (e.g. more whitespaces), but we cannot >> foresee all cases. > >> As I understand the numbering scheme, the first digit is for major >> changes (non-backwards compatible, API breaking changes). The second >> digit is for non-API breaking changes including feature and API >> additions. And the last digit (minor) is for bugfixes. Since there is no >> remote protocol defined for using LTTng tools remotely, the Eclipse LTTng >> control relies on the string output of the command line tool. So, I >> consider the output as part of the API :-). I will have to agree on that and will change back the output. I'll make a RC2 after that. > >>>> [2] Trace directory structure: I noticed that the UST traces are >>>> moved under another sub-directory. >>>> >>>> In v2.0.x the directory structure looked as follows: >>>> /home/user/lttng-traces/ |-mysession-20120314-132824 |-kernel >>>> |--- |-ust |----> >>>> |----------- >>>> >>>> In v2.1.x the directory structure looked as follows: >>>> /home/user/lttng-traces/ |-mysession-20120314-132824 |-kernel >>>> |--- |-ust |--- mysession-20120314-132824 >>>> |----> |----------->>> files> > This is absolutely not suppose to be like that... I'm unable to reproduce > this behavior having the "mysession-**" created two times. > > Can you give me the exact series of command you do to get this directory > structure? >> See below the log of commands I executed. I also added the output of >> command "find ." under lttng-traces directory. In there you see the >> additional sub-directory. By the way, I installed lttng-tools, lttng-ust >> and userspace-rcu from source code. Here are the corresponding SHA >> numbers: lttng-tools: 68264071f9d1b789de1350cbec479b52a9b54acf lttng-ust: >> 0f4eaec3e738fa0f33296a46fe08266a60787c23 userspace-rcu: >> 768fba83676f49eb73fd1d8ad452016a84c5ec2a > > > lttng create mysession >> Trace(s) output set to >> /home/bernd/lttng-traces/mysession-20120822-094418 Session mysession >> created. > > lttng enable-event -a -k >> All kernel events are enabled in channel channel0 > > lttng enable-event -a -u >> All UST events are enabled in channel channel0 > > lttng start >> Tracing started for session mysession > > lttng stop >> Tracing stopped for session mysession > > lttng destroy >> Session mysession destroyed > > find . >> . ./mysession-20120822-094418 >> ./mysession-20120822-094418/mysession-20120822-094542 >> ./mysession-20120822-094418/mysession-20120822-094542/ust >> ./mysession-20120822-094418/mysession-20120822-094542/ust/lt-hello-2039-20120822-094551 > >> >> ./mysession-20120822-094418/mysession-20120822-094542/ust/lt-hello-2039-20120822-094551/channel0_0 > >> >> ./mysession-20120822-094418/mysession-20120822-094542/ust/lt-hello-2039-20120822-094551/metadata > >> >> ./mysession-20120822-094418/mysession-20120822-094542/ust/lt-hello-2056-20120822-094551 > >> >> ./mysession-20120822-094418/mysession-20120822-094542/ust/lt-hello-2056-20120822-094551/channel0_0 > >> >> ./mysession-20120822-094418/mysession-20120822-094542/ust/lt-hello-2056-20120822-094551/metadata > >> ./mysession-20120822-094418/kernel >> ./mysession-20120822-094418/kernel/channel0_0 >> ./mysession-20120822-094418/kernel/metadata There is a bug. I'll push a patch before RC2. A wrong pointer was used during the directory setting having the wrong behavior you are witnessing :). Good catch and thanks for the report! Cheers David > > The normal case and wht 2.1.x should to is provide you the same directory > structure as 2.0.x. Ex: > > ~/lttng-traces/test-20120822-091412/ust/lt-hello-4542-20120822-091415 > > Thanks! David > >>>> So as you see, under ust there is another sub directory with trace >>>> session name and date/time. I don't see a practical reason to have >>>> this sub directory, because the session name and date/time is already >>>> known from the top-level directory. Is it possible to revert this >>>> back? In Eclipse I would have to implement a special case for >>>> different LTTng-tools versions (in the import dialog). >>>> >>>> Thank you very much in advance. >>>> >>>> Best Regards Bernd >>>> >>>> >>>> On 08/17/2012 04:11 PM, David Goulet wrote: Greetings everyone >>>> (including LTTng elves), >>>> >>>> The lttng-tools project provides a session daemon (lttng-sessiond) >>>> that acts as a tracing registry, the "lttng" command line for tracing >>>> control, a lttng-ctl library for tracing control and a lttng-relayd >>>> for network streaming. >>>> >>>> This is the first release candidate for lttng-tools 2.1 which brings >>>> two exciting new features, network streaming and filtering support. >>>> >>>> It's the combination of a lot of work from the LTTng team so please, >>>> to help us improve our tools, report any bugs or misbehaving >>>> feature(s) that you may encounter through this mailing list or the >>>> bug tracker (https://bugs.lttng.org). >>>> >>>> - From now on, lttng-tools is in feature freeze mode unless an >>>> *IMMENSE* show stopper is found. >>>> >>>> 2012-08-17 lttng-tools 2.1.0-rc1 * Feature: Network Streaming * Add >>>> the lttng-relayd binary for network streaming * Support user space >>>> tracer filtering * Multiple fixes >>>> >>>> Project website: http://lttng.org/lttng2.0 Download link: >>>> http://lttng.org/files/lttng-tools/lttng-tools-2.1.0-rc1.tar.bz2 >>>> >>>> Cheers! David >>>>> _______________________________________________ lttng-dev mailing >>>>> list lttng-dev at lists.lttng.org >>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >>>> >>>> _______________________________________________ lttng-dev mailing >>>> list lttng-dev at lists.lttng.org >>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > > _______________________________________________ lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJQNQFJAAoJEELoaioR9I02XnEH/AlGnn2oryRAA5FWihIt+gcN 5Mk4Ry/6iCCeIvFBwqtZpG+ATG8hBBouzAopjfRMs0tzZSVxs4Nq+jLJchTD8Igh mPJyXhorBo4nCXFsH1FVaduQzTwos8hxsH2htpS6YU3DJhR4kLaVR8Qnk1Gt6t9q HoeoXFGwaP51NHrL9m4pMvdrlUTBOFkYYTeBLrUPV/ikmkhXwmSsnQZzGsFCMphE Qb9jEr1QebvyEk7CAocaKUMjh37j+qpaLQeOUYch6YdK+kuurotqgWqMWi/9a+UF F+/I5wvParmN6VyWcEfo4fUqlP/4x09x1tbOyyjYf/EQ8v5PKfO7gp62xV1QRzM= =E6zS -----END PGP SIGNATURE----- From david.goulet at polymtl.ca Wed Aug 22 12:45:44 2012 From: david.goulet at polymtl.ca (David Goulet) Date: Wed, 22 Aug 2012 12:45:44 -0400 Subject: [lttng-dev] Fwd: [RELEASE] LTTng-tools 2.1.0-rc2 In-Reply-To: <502EA58E.2060003@efficios.com> References: <502EA58E.2060003@efficios.com> Message-ID: <50350CB8.1040004@polymtl.ca> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Greetings everyone, The lttng-tools project provides a session daemon (lttng-sessiond) that acts as a tracing registry, the "lttng" command line for tracing control, a lttng-ctl library for tracing control and a lttng-relayd for network streaming. This is the release candidate version 2 for lttng-tools. 2012-08-22 lttng-tools 2.1.0-rc2 * Fix: put back 2.0 output text for lttng create cmd * Fix: remove set subdir call that uses bad ptr * Filter: Handle the unary bitwise not operator (~) with an unsupported * Fix: missing mutex unlock on register consumer err * Remove underscore from ifndef of lttng.h * Remove unused define in lttng.h * Standardize lttng command line usage text * Merge duplicate code in consumer for destroy relayd * Merge duplicate code in consumer for add relayd * Fix: Possible buffer overflows in strncat() usage * Move code out of main.c and fix some issues Project website: http://lttng.org/lttng2.0 Download link: http://lttng.org/files/lttng-tools/lttng-tools-2.1.0-rc2.tar.bz2 Cheers! David -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJQNQy4AAoJEELoaioR9I02RjQH+QGN3zpCpqjQ4s752ZtvFHd7 dthbrg/DhhSSew5YUBB4Wx7sQbGxG4RM3EATlzr482CmxqLmQpoWEl2PBtxJ762I 91PXm4+mwya0SIk8c70U+7/kNGT4ofBhmNwFOKxrNBAPnicxADZMP9n/w8IoJxFL dh9GawjqfRmftLltPQlYRAdbfoj5P+4KCwi+dWHMguSSpQK2shMN7f4DmSf8L/fY X1Z92NYV2I5Ez1/prAUKMh912yEQnKmH8d7bg7grMt92O2noBXRW5mldPw9p8MVV NWpGqV4vSpw8dy7zseWpCJ/YfJETEOv9Zdcu15Asfj/izoiQxOW/5k0WMetGl1s= =HZNa -----END PGP SIGNATURE----- From Bernd.Hufmann at ericsson.com Wed Aug 22 14:09:21 2012 From: Bernd.Hufmann at ericsson.com (Bernd Hufmann) Date: Wed, 22 Aug 2012 14:09:21 -0400 Subject: [lttng-dev] Fwd: [RELEASE] LTTng-tools 2.1.0-rc2 In-Reply-To: <50350CB8.1040004@polymtl.ca> References: <502EA58E.2060003@efficios.com> <50350CB8.1040004@polymtl.ca> Message-ID: <50352051.20303@ericsson.com> Hi David I tried RC2 and I was able to create a session and to import traces remotely with the current Eclipse Tracer Control. I only had to change the supported version in the Eclipse Tracer control. Thanks for the updates. I'll do some more tests. I'll let you know in case I find something else. Best Regards Bernd On 08/22/2012 12:45 PM, David Goulet wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Greetings everyone, > > The lttng-tools project provides a session daemon (lttng-sessiond) > that acts as a tracing registry, the "lttng" command line for tracing > control, a lttng-ctl library for tracing control and a lttng-relayd > for network streaming. > > This is the release candidate version 2 for lttng-tools. > > 2012-08-22 lttng-tools 2.1.0-rc2 > * Fix: put back 2.0 output text for lttng create cmd > * Fix: remove set subdir call that uses bad ptr > * Filter: Handle the unary bitwise not operator (~) with an unsupported > * Fix: missing mutex unlock on register consumer err > * Remove underscore from ifndef of lttng.h > * Remove unused define in lttng.h > * Standardize lttng command line usage text > * Merge duplicate code in consumer for destroy relayd > * Merge duplicate code in consumer for add relayd > * Fix: Possible buffer overflows in strncat() usage > * Move code out of main.c and fix some issues > > Project website: http://lttng.org/lttng2.0 > Download link: http://lttng.org/files/lttng-tools/lttng-tools-2.1.0-rc2.tar.bz2 > > Cheers! > David > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.12 (GNU/Linux) > > iQEcBAEBAgAGBQJQNQy4AAoJEELoaioR9I02RjQH+QGN3zpCpqjQ4s752ZtvFHd7 > dthbrg/DhhSSew5YUBB4Wx7sQbGxG4RM3EATlzr482CmxqLmQpoWEl2PBtxJ762I > 91PXm4+mwya0SIk8c70U+7/kNGT4ofBhmNwFOKxrNBAPnicxADZMP9n/w8IoJxFL > dh9GawjqfRmftLltPQlYRAdbfoj5P+4KCwi+dWHMguSSpQK2shMN7f4DmSf8L/fY > X1Z92NYV2I5Ez1/prAUKMh912yEQnKmH8d7bg7grMt92O2noBXRW5mldPw9p8MVV > NWpGqV4vSpw8dy7zseWpCJ/YfJETEOv9Zdcu15Asfj/izoiQxOW/5k0WMetGl1s= > =HZNa > -----END PGP SIGNATURE----- > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev From laijs at cn.fujitsu.com Wed Aug 22 20:57:28 2012 From: laijs at cn.fujitsu.com (Lai Jiangshan) Date: Thu, 23 Aug 2012 08:57:28 +0800 Subject: [lttng-dev] [RFC URCU PATCH] wfqueue: ABI v1, simplify implementation, 2.3x to 2.6x performance boost In-Reply-To: <50330371.6060401@cn.fujitsu.com> References: <20120815213107.GB17224@Krystal> <5032054D.7050508@cn.fujitsu.com> <20120820131642.GA551@Krystal> <50330371.6060401@cn.fujitsu.com> Message-ID: <50357FF8.9060704@cn.fujitsu.com> ping On 08/21/2012 11:41 AM, Lai Jiangshan wrote: > On 08/20/2012 09:16 PM, Mathieu Desnoyers wrote: >> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote: >>> On 08/16/2012 05:31 AM, Mathieu Desnoyers wrote: >>>> This work is derived from the patch from Lai Jiangshan submitted as >>>> "urcu: new wfqueue implementation" >>>> (http://lists.lttng.org/pipermail/lttng-dev/2012-August/018379.html) >>>> >>> >> >> Hi Lai, >> >>> Hi, Mathieu >>> >>> please add this part to your patch. >>> >>> Changes(item5 has alternative): >>> >>> 1) Reorder the head and the tail in struct cds_wfq_queue >>> Reason: wfq is enqueue-preference >> >> Is this a technical improvement over the original order ? Both cache >> lines are aligned, so it should not matter which one comes first. So I'm >> not sure why we should favor one order vs the other. >> >>> 2) Reorder the code of ___cds_wfq_next_blocking() >>> Reason: short code, better readability >> >> Yes, I agree with this change. Can you extract it into a separate patch ? >> >>> >>> 3) Add cds_wfq_dequeue_[un]lock >>> Reason: the fields of struct cds_wfq_queue are not exposed to users of lib, >>> but some APIs needs to have this lock held. Add this locking function >>> to allow the users use such APIs >> >> I agree with this change too. You can put it into the same patch as (2). >> >>> >>> 4) Add cds_wfq_node_sync_next(), cds_wfq_dequeue_all_blocking() >>> Reason: helper for for_each_* >> >> Points 4-5-6-7 will need more discussion. >> >> I don't like exposing "cds_wfq_node_sync_next()", which should remain an >> internal implementation detail not exposed to queue users. The >> "get_first"/"get_next" API, as well as the dequeue, are providing an >> abstraction that hide the node synchronization, which makes it harder >> for the user to make mistakes. Given that the fast-path of >> cds_wfq_node_sync_next is just a pointer load and test, I don't think >> that skipping this the second time we iterate on the same queue would >> bring any significant performance improvement, but exposing sync_next >> adds in API complexity, which I would like to avoid. > > If the name "cds_wfq_node_sync_next()" is bad and exposes the implementation, > we can rename it to "__cds_wfq_next_blocking_nocheck()" to hide the implementation. > > "get_next_nocheck": get next node of a queue, the caller ensure that the next node is existed. > "get_next": get next node of a queue, it also finds out that whether the next node is existed or not. > > Or just remove "cds_wfq_node_sync_next()", use "get_next" instead. > >> >> About cds_wfq_dequeue_all_blocking(): the "cds_wfq_splice()" I >> introduced performs the same action as "cds_wfq_dequeue_all_blocking()", >> but ensures that we can splice lists multiple times (rather than only >> once), which is a very neat side-effect. Let's imagine a use-case like >> Tree RCU, where each CPU produce a queue of call_rcu callbacks, and we >> need to merge pairs of queues together recursively until we reach the >> tree root: by doing splice() at each level of the tree, we can combine >> the queues into a single queue without ever needing to do an iteration >> on each node: no node synchronization is needed until we actually need >> to traverse the queue at the root node level. >> >> The cost of exposing dequeue_all is that we need to expose sync_next >> (more complexity for the user). As you exposed in your earlier email, >> the gain we get by exposing dequeue_all separately from splice is that >> dequeue_all is 1 xchg() operation, while splice is 2 xchg() operations. >> However, given that this operation is typically amortized over many >> nodes makes performance optimization a less compelling argument than >> simplicity of use. > > Reducing the temporary big struct is also my tempt. > > --- > Related to "internal", see the bottom. > >> >>> >>> 5) Add more for_each_*, which default semantic is dequeuing >>> Reason: __cds_wfq_for_each_blocking_undequeue is enough for undequeuing loops >>> don't need to add more. >>> looping and dequeuing are the most probable use-cases. >>> dequeuing for_each_* make the queue like a pipe. make it act as a channel.(GO language) >>> Alternative: rename these dequeuing __cds_wfq_for_each*() to __cds_wfq_dequeue_for_each* >> >> I notice, in the iterators you added, the presence of >> "cds_wfq_for_each_blocking_nobreak", and >> "cds_wfq_for_each_blocking_nobreak_internal", which document that this >> special flavor is needed for loops that do not have a "break" statement. >> That seems to be very much error-prone and counter-intuitive for users, >> and I would like to avoid that. > > __cds_wfq_for_each_blocking_safe(remove-all) is also "no-break"(*), is it error-prone and ...? > > So this is not a good reason to reject cds_wfq_for_each_blocking_nobreak(), if the user > don't know how to use it, he should use cds_wfq_for_each_blocking() (the version in my > patch which is dequeue()-based and we can break from the loop body) > > __cds_wfq_for_each_blocking_safe() and cds_wfq_for_each_blocking_nobreak() are exactly the > same semantic in "remove-all" and "no-break". but cds_wfq_for_each_blocking_nobreak() > does NOT CORRUPT the queue. > > So cds_wfq_for_each_blocking_nobreak() is yet another __cds_wfq_for_each_blocking_safe() > and it don't corrupt the queue, it is really safer than __cds_wfq_for_each_blocking_safe(). > > """"footnote: (*) > __cds_wfq_for_each_blocking_safe() should be used for "remove-none" or "remove-all" > We should use __cds_wfq_for_each_blocking_undequeue() instead of __cds_wfq_for_each_blocking_safe(remove-none) > """ > >> >> The "internal" flavor, along with "sync_next" are then used in the >> call_rcu code, since they are needed to get the best performance >> possible. I don't like to use special, internal APIs for call_rcu: the >> original motivation for refactoring the wfqueue code was testability, >> and I would really like to make sure that the APIs we use internally are >> also exposed to the outside world, mainly for testability, and also >> because if we need to directly access internal interfaces to have good >> performance, it means we did a poor job at exporting APIs that allow to >> do the same kind of highly-efficient use of the queue. Exposing an API >> with the "internal" keyword in it clearly discourages users from using >> it. > > Related to "internal" see the bottom. > >> >> >>> >>> 6) Remove __cds_wfq_for_each_blocking_safe >>> Reason: its semantic is not clear, hard to define a well-defined semantic to it. >>> when we use it, we have to delete none or delete all nodes, even when we delete all nodes, >>> struct cds_wfq_queue still becomes stale while looping or after loop. >> >> The typical use-case I envision is: >> >> - enqueue many nodes to queue A >> >> - then, a dequeuer thread splice the content of queue A into its temporary >> queue T (which head is local on its stack). >> >> - now, the thread is free to iterate on queue T, as many times as it >> likes, and it does not need to change the structure of the queue while >> iterating on it (read-only operation, without side-effect). The last >> time it iterates on queue T, it needs to use the _safe iteration and >> free the nodes. >> >> - after the nodes have been deleted, the queue T can be simply discarded >> (if on stack, function return will do so; if dynamically allocated, >> can be simply reinitialized to an empty queue, or freed). > > My for_each_*() can do exactly the same thing in this use-case since we still > have splice() and cds_wfq_for_each_blocking_nobreak()(safer **_safe()). Doesn't it? > > The memory represented a struct queue is big.(two cache line), something we need to > avoid to use it in the stack.(shorten the stack size or avoid to cost 2 or 3 additional cache line). > > -----------a use case > All these local field are hot in the stack(and for some reason, it is not in the registers) > int count; > struct cds_wfq_queue tmp; > struct cds_wfq_node *node, *next; > > It will use 4 cache line. Should we always recommend the users use splice()+tmp? > >> >> As you certainly noticed, we can iterate both on a queue being >> concurrently enqueued to, and on a queue that has been spliced into. >> Being able to use iterators and dequeue on any queue (initial enqueue, >> or queue spliced to) is a feature: we can therefore postpone the >> "sync_next" synchronization to the moment where loop iteration is really >> needed (lazy synchronization), which keeps optimal locality of reference >> of synchronization vs node data access. >> >>> >>> 7) Rename old __cds_wfq_for_each_blocking() to __cds_wfq_for_each_blocking_undequeue() >>> Reason: the default semantic of the other for_each_* is dequeuing. >> >> I'm not convinced that "__cds_wfq_for_each_blocking_nobreak", >> "__cds_wfq_for_each_blocking_nobreak_internal", and sync_next APIs are >> improvements over the API I propose. For call_rcu, this turns >> >> cds_wfq_enqueue() >> cds_wfq_splice_blocking() >> __cds_wfq_for_each_blocking_safe() for iteration >> >> into: >> >> cds_wfq_enqueue() >> __cds_wfq_dequeue_all_blocking() >> __cds_wfq_for_each_blocking_nobreak_internal() for iteration >> >> Maybe we could do some documentation improvement to >> "__cds_wfq_for_each_blocking(_safe)", __cds_wfq_first_blocking(), >> __cds_wfq_next_blocking() API members I proposed. Maybe I did not >> document them well enough ? >> > > It does be not enough for __cds_wfq_for_each_blocking_safe(). > > I *STRONGLY OPPOSE* __cds_wfq_for_each_blocking_safe()(although it is suggested by me). > > If you oppose the dequeue_all(), I agreed with this list: > Remove old __cds_wfq_for_each_blocking_safe() > Use manual splice()+get_first()+get_next() for call_rcu_thread(). > Remove my proposed cds_wfq_for_each_blocking_nobreak() > Remove my proposed sync_next(), dequeue_all(). > Keep dequeue-based for_each() (alternative: introduce it in future) > > If you also agreed this list, could you please partially merge my patch to your patch > and send it for review to reduce the review cycle. (don't forget the other minimal fixes in my patch) > > ========== > About internals: > > I want to expose the for_each_*, I don't want to expose these helpers: > cds_wfq_node_sync_next() > cds_wfq_dequeue_all_blocking() > __cds_wfq_dequeue_all_blocking() > __cds_wfq_for_each_blocking_nobreak_internal() > (and I want to disallow the users use these helpers directly). > > but the for_each_* use these helpers, so I have to declare it in the urcu/wfqueue.h, > and they become a part of ABI. > > So these helpers become internal things, how to handle internal things > in a exposed *.h? > > I found "__cds_list_del()" and "__tls_access_ ## name ()" are also > internals, and they are exposed. Are they exposed correctly? > > In my view, exposed internals is OK, what we can do is disallowing > the users use it directly and ensuring the compatibility. > > And we don't need to test exposed internals directly via testcases, because they are > internals, just like we don't need to test so much un-exposed internals. > When we test the exposed API, the internals will be also tested. > > The call_rcu_thread() use the internal directly and can't be tested as you mentioned. > It is not totally true. call_rcu_thread() need to inject a "synchronize_rcu()" > to __cds_wfq_for_each_blocking_nobreak(), so I have to use > __cds_wfq_for_each_blocking_nobreak_internal(). > Tests for __cds_wfq_for_each_blocking_nobreak() are enough for call_rcu_thread(). > > In one word, I don't agreed full of your disgust to the exposed internals, > but I agree to remove the ones introduced in my patch. > > Thanks, > Lai > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > From teawater at gmail.com Thu Aug 23 02:11:54 2012 From: teawater at gmail.com (Hui Zhu) Date: Thu, 23 Aug 2012 14:11:54 +0800 Subject: [lttng-dev] [PATCH] Fix trace-collection.h: No such file or directory that build code with libbabeltrace Message-ID: Hi, After I install the babeltrace and try to build a code, I got this: gcc -Wall -g 1.c -lbabeltrace -lbabeltrace-ctf In file included from 1.c:11:0: /usr/local/include/babeltrace/babeltrace.h:23:41: fatal error: babeltrace/trace-collection.h: No such file or directory compilation terminated. I check the code,looks most of function of trace-collection.h is covered by context.h. So I post a patch for it. Wish you like it. Thanks, Hui --- a/include/babeltrace/babeltrace.h +++ b/include/babeltrace/babeltrace.h @@ -20,7 +20,6 @@ #include #include #include -#include #include #endif /* _BABELTRACE_H */ From teawater at gmail.com Thu Aug 23 05:35:12 2012 From: teawater at gmail.com (Hui Zhu) Date: Thu, 23 Aug 2012 17:35:12 +0800 Subject: [lttng-dev] [babeltrace] About the type read and write Message-ID: Hi, I just tried to use libbabeltrace to read and write the CTF file. But I cannot find the api to read some type of CTF for example CTF_TYPE_ENUM, CTF_TYPE_STRUCT. And I cannot find the api file to write to the CTF file. (I check the code of babeltrace-log and found that it use the function that is not API) Could you help me with it? Thanks, Hui From mathieu.desnoyers at efficios.com Thu Aug 23 08:08:38 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 23 Aug 2012 08:08:38 -0400 Subject: [lttng-dev] [PATCH] Fix trace-collection.h: No such file or directory that build code with libbabeltrace In-Reply-To: References: Message-ID: <20120823120838.GA23356@Krystal> merged, thanks! Mathieu * Hui Zhu (teawater at gmail.com) wrote: > Hi, > > After I install the babeltrace and try to build a code, I got this: > gcc -Wall -g 1.c -lbabeltrace -lbabeltrace-ctf > In file included from 1.c:11:0: > /usr/local/include/babeltrace/babeltrace.h:23:41: fatal error: > babeltrace/trace-collection.h: No such file or directory > compilation terminated. > > I check the code,looks most of function of trace-collection.h is > covered by context.h. > > So I post a patch for it. Wish you like it. > > Thanks, > Hui > > --- a/include/babeltrace/babeltrace.h > +++ b/include/babeltrace/babeltrace.h > @@ -20,7 +20,6 @@ > #include > #include > #include > -#include > #include > > #endif /* _BABELTRACE_H */ > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 23 08:27:54 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 23 Aug 2012 08:27:54 -0400 Subject: [lttng-dev] [babeltrace] About the type read and write In-Reply-To: References: Message-ID: <20120823122754.GA23841@Krystal> * Hui Zhu (teawater at gmail.com) wrote: > Hi, > > I just tried to use libbabeltrace to read and write the CTF file. But > I cannot find the api to read some type of CTF for example > CTF_TYPE_ENUM, CTF_TYPE_STRUCT. AFAIU, the API bt_ctf_get_field_list allows to get the list of definitions from a struct, variant, array, or sequence (any compound type). Julien, I think the header documentation of this function should really be updated, can you look into that ? Indeed, I also cannot see any API to get the enum string from the field. Julien, can we add it ? > And I cannot find the api file to write to the CTF file. (I check the > code of babeltrace-log and found that it use the function that is not > API) Currently, babeltrace only does CTF-to-text and text-log-to-CTF conversions. As you noticed, it does not expose API to write into all CTF types at this stage. There is a prototype I am aware of that does just that (write custom CTF files): Javeltrace. See the post here http://lists.lttng.org/pipermail/lttng-dev/2012-August/018393.html for details. Thanks, Mathieu > > Could you help me with it? > > Thanks, > Hui > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From christian.babeux at efficios.com Thu Aug 23 18:21:14 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Thu, 23 Aug 2012 18:21:14 -0400 Subject: [lttng-dev] [PATCH lttng-tools] Filter: Notify the user when a filter is already enabled on event(s) Message-ID: <1345760474-29248-1-git-send-email-christian.babeux@efficios.com> Signed-off-by: Christian Babeux --- src/bin/lttng/commands/enable_events.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/src/bin/lttng/commands/enable_events.c b/src/bin/lttng/commands/enable_events.c index a2c8a68..7a65ee6 100644 --- a/src/bin/lttng/commands/enable_events.c +++ b/src/bin/lttng/commands/enable_events.c @@ -384,7 +384,17 @@ static int enable_events(char *session_name) ret = lttng_set_event_filter(handle, ev.name, channel_name, opt_filter); if (ret < 0) { - ERR("Error setting filter"); + switch (-ret) + { + case LTTCOMM_FILTER_EXIST: + ERR("Filter on events is already enabled (channel %s, session %s)", + channel_name, session_name); + break; + default: + ERR("Error setting filter"); + break; + } + ret = -1; goto error; } @@ -554,7 +564,17 @@ static int enable_events(char *session_name) ret = lttng_set_event_filter(handle, ev.name, channel_name, opt_filter); if (ret < 0) { - ERR("Error setting filter"); + switch (-ret) + { + case LTTCOMM_FILTER_EXIST: + ERR("Filter on event %s is already enabled (channel %s, session %s)", + event_name, channel_name, session_name); + break; + default: + ERR("Error setting filter"); + break; + } + ret = -1; goto error; } -- 1.7.11.4 From mathieu.desnoyers at efficios.com Thu Aug 23 18:45:41 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 23 Aug 2012 18:45:41 -0400 Subject: [lttng-dev] [PATCH lttng-tools] Filter: Notify the user when a filter is already enabled on event(s) In-Reply-To: <1345760474-29248-1-git-send-email-christian.babeux@efficios.com> References: <1345760474-29248-1-git-send-email-christian.babeux@efficios.com> Message-ID: <20120823224541.GA3495@Krystal> * Christian Babeux (christian.babeux at efficios.com) wrote: > > Signed-off-by: Christian Babeux > --- > src/bin/lttng/commands/enable_events.c | 24 ++++++++++++++++++++++-- > 1 file changed, 22 insertions(+), 2 deletions(-) > > diff --git a/src/bin/lttng/commands/enable_events.c b/src/bin/lttng/commands/enable_events.c > index a2c8a68..7a65ee6 100644 > --- a/src/bin/lttng/commands/enable_events.c > +++ b/src/bin/lttng/commands/enable_events.c > @@ -384,7 +384,17 @@ static int enable_events(char *session_name) > ret = lttng_set_event_filter(handle, ev.name, channel_name, > opt_filter); > if (ret < 0) { > - ERR("Error setting filter"); > + switch (-ret) bracket on same line as switch statement please. > + { > + case LTTCOMM_FILTER_EXIST: > + ERR("Filter on events is already enabled (channel %s, session %s)", > + channel_name, session_name); > + break; > + default: > + ERR("Error setting filter"); > + break; > + } > + > ret = -1; > goto error; > } > @@ -554,7 +564,17 @@ static int enable_events(char *session_name) > ret = lttng_set_event_filter(handle, ev.name, > channel_name, opt_filter); > if (ret < 0) { > - ERR("Error setting filter"); > + switch (-ret) same here. Can you describe the change in the changelog ? What was the behavior before, with some example cases (sequences of commands), and what this is changing. Thanks, Mathieu > + { > + case LTTCOMM_FILTER_EXIST: > + ERR("Filter on event %s is already enabled (channel %s, session %s)", > + event_name, channel_name, session_name); > + break; > + default: > + ERR("Error setting filter"); > + break; > + } > + > ret = -1; > goto error; > } > -- > 1.7.11.4 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From jdesfossez at efficios.com Thu Aug 23 20:53:09 2012 From: jdesfossez at efficios.com (Julien Desfossez) Date: Thu, 23 Aug 2012 20:53:09 -0400 Subject: [lttng-dev] =?utf-8?q?=5BBABELTRACE_PATCH=5D_API_documentation?= Message-ID: <1345769590-1350-1-git-send-email-jdesfossez@efficios.com> From: Danny Serres Signed-off-by: Danny Serres Signed-off-by: Julien Desfossez --- doc/API.txt | 247 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 247 insertions(+) create mode 100644 doc/API.txt diff --git a/doc/API.txt b/doc/API.txt new file mode 100644 index 0000000..f1780f0 --- /dev/null +++ b/doc/API.txt @@ -0,0 +1,247 @@ +Babeltrace API documentation + +Babeltrace provides trace read and write libraries, as well as a trace +converter. A plugin can be created for any trace format to allow its +conversion to/from another trace format. + +The main format expected to be converted to/from is the Common Trace +Format (CTF). The latest version of the CTF specification can be found at: + git tree: git://git.efficios.com/ctf.git + gitweb: http://git.efficios.com/?p=ctf.git + +This document describes the main concepts to use the libbabeltrace, +which exposes the Babeltrace trace reading capability. + + +TERMINOLOGY +??????????? + +* A "callback" is a reference to a piece of executable code (such as a + function) that is passed as an argument to another piece of code + (like another function). + +* A "context" is a structure that represents an object in which a trace + collection is opened. + +* An "iterator" is a structure that enables the user to traverse a trace. + +* A "trace handle" is a unique identifier representing a trace file. + It allows the user to manipulate a trace directly. + + + +USAGE +?????? + +Context: + +In order to use libbabeltrace to read a trace, the first step is to create a +context structure and to add a trace to it. This is done using the +bt_context_create() and bt_context_add_trace() functions. As long as this +context structure is allocated and the trace is valid, the trace can be +manipulated by the library. + +The context can be destroyed by calling one more bt_context_put() than +bt_context_get(), functions which respectively decrement and increment the +refcount of the context. These functions ensures that the context won't be +destroyed when it is in use. + +Once a trace is added to the context, it can be read and seeked using iterators +and callbacks. + + +Iterator: + +An iterator can be created using the bt_iter_create() function. As of now, only +ctf iterator are supported. These are used to traverse a ctf-formatted trace. +Such iterators can be created with bt_ctf_iter_create(). + +While creating an iterator, a begin and an end position may be specified. To do +so, one or two struct bt_iter_pos must be passed. Such struct have two +attributes: type and u. "type" is the seek type, can be either: + BT_SEEK_TIME + BT_SEEK_RESTORE + BT_SEEK_CUR + BT_SEEK_BEGIN + BT_SEEK_END +and "u" is a union of the seek time (if using BT_SEEK_TIME) and the restore +position (if using BT_SEEK_RESTORE). + +Once the iterator is created, various functions become available. We have +bt_ctf_iter_read_event() which returns the ctf event of the trace where the +iterator is set. There is also bt_ctf_iter_destroy() which frees the iterator. +Note that only one iterator can be created in a context at the same time. If +more than one iterator is being created for the same context, the second +creation will return NULL. The previous iterator must be destroyed before +creation of the new iterator. In the future, creation of multiples iterators +will be allowed. + +Finally, we have the bt_ctf_get_iter() function which returns a struct bt_iter +with which the iterator can be moved using one of these functions: + bt_iter_next(), moves the iterator to the next event + bt_iter_set_pos(), moves the iterator to the specified position + +To get the current position (struct bt_iter_pos) of the iterator, the function +bt_iter_get_pos() must be used. To create an arbitrary position based on a +specific time, bt_iter_create_time_pos() is the function to use. The +bt_iter_pos structure returned by these two functions must be freed with +bt_iter_free_pos() after use. + + +CTF Event: + +A CTF event is obtained from an iterator via the bt_ctf_iter_read_event() +function or via the call_data parameter of a callback. To read the data of a +CTF event : + * bt_ctf_event_name() returns the name of the event; + * bt_ctf_get_timestamp() returns the timestamp of the event + offsetted with the system clock + source (in ns); + * bt_ctf_get_cycles() returns the timestamp of the event as + written in the packet (in cycles). + +The payload of an event is divided in various scopes depending on the type of +information. There are six top-level scopes (defined in the bt_ctf_scope enum) +which can be accessed by the bt_ctf_get_top_level_scope() function : + BT_TRACE_PACKET_HEADER = 0, + BT_STREAM_PACKET_CONTEXT = 1, + BT_STREAM_EVENT_HEADER = 2, + BT_STREAM_EVENT_CONTEXT = 3, + BT_EVENT_CONTEXT = 4, + BT_EVENT_FIELDS = 5. + +In order to access a field or a field list, the user needs to pass a scope as +argument, this scope can be a top-level scope or a scope relative to an +arbitrary field in the case of compound types (array, sequence, structure or +variant) + +For more information on each scope, see the CTF specifications. + +The function to get a field list is the bt_ctf_get_field_list(). The function +to get the definition of a specific field is bt_ctf_get_field(). + +Once the field is obtained, we can obtain its name and type using the +bt_ctf_field_name() and bt_ctf_field_type() functions respectively. The +possible types are defined in the ctf_type_id enum: + CTF_TYPE_UNKNOWN = 0, + CTF_TYPE_INTEGER, + CTF_TYPE_FLOAT, + CTF_TYPE_ENUM, + CTF_TYPE_STRING, + CTF_TYPE_STRUCT, + CTF_TYPE_UNTAGGED_VARIANT, + CTF_TYPE_VARIANT, + CTF_TYPE_ARRAY, + CTF_TYPE_SEQUENCE, + NR_CTF_TYPES. + +Depending on the field type, we can get informations about the field with these +functions: + * bt_ctf_get_index() return the element at the index + position of an array of a sequence; + + * bt_ctf_get_array_len() return the length of an array; + + * bt_ctf_get_int_signedness() return the signedness of an integer; + + * bt_ctf_get_int_base() return the base of an integer; + + * bt_ctf_get_int_byte_order() return the byte order of an integer; + + * bt_ctf_get_int_len() return the size in bits of an integer; + + * bt_ctf_get_encoding() return the encoding of an int or a + string defined in the + ctf_string_encoding enum: + CTF_STRING_NONE = 0, + CTF_STRING_UTF8, + CTF_STRING_ASCII, + CTF_STRING_UNKNOWN. + +These functions give access to the value associated with a field : + * bt_ctf_get_uint64(); + * bt_ctf_get_int64(); + * bt_ctf_get_char_array(); + * bt_ctf_get_string(). + +If the field does not exist or is not of the type requested, the value returned +with these four functions is undefined. To check if an error occured, use the +bt_ctf_field_get_error() function after accessing a field. If no error +occured, the function will return 0. + +It is also possible to access the declaration fields, the same way as the +definition ones. bt_ctf_get_event_decl_list() sets a list to an array of +bt_ctf_event_decl pointers and bt_ctf_get_event_decl_fields() sets a list to an +array of bt_ctf_field_decl pointers. From the first type, the name of the +event can be obtained with bt_ctf_get_decl_event_name(). For the second type, +the field decl name is obtained with bt_ctf_get_decl_field_name(). + +The declaration functions allow the user to list the events, fields and +contexts fields enabled in the trace once it is opened, whereas the definition +functions apply on the current event being read. + + +Callback: + +The iterator allow the user to read the trace, in order to access the events +and fields, the user can either call the functions listed previously on each +event, or register callbacks functions that are called when specific (or all) +events are read. + +This is done with the bt_ctf_iter_add_callback() function. It requires a valid +ctf iterator as the first argument. Here are all arguments: + iter: trace collection iterator (input) + event: event to target. 0 for all events. + private_data: private data pointer to pass to the callback + flags: specific flags controlling the behavior of this callback + (or'd). + callback: function pointer to call + depends: struct bt_dependency detailing the required computation + results. Ends with 0. + weak_depends: struct bt_dependency detailing the optional computation + results that can be optionally consumed by this + callback. + provides: struct bt_dependency detailing the computation results + provided by this callback. + Ends with 0. + +"depends", "weak_depends" and "provides" memory is handled by the babeltrace +library after this call succeeds or fails. These objects can still be used by +the caller until the babeltrace iterator is destroyed, but they belong to the +babeltrace library. + +As of now the flags and dependencies are not used, the callbacks are +processed in FIFO order. + +Note: once implemented, the dependency graph will be calculated when +bt_ctf_iter_read_event() is executed after a bt_ctf_iter_add_callback(). It is +valid to create/add callbacks/read/add more callbacks/read some more. + +The callback function passed to bt_ctf_iter_add_callback() must return a +bt_cb_ret value: + BT_CB_OK = 0, + BT_CB_OK_STOP = 1, + BT_CB_ERROR_STOP = 2, + BT_CB_ERROR_CONTINUE = 3. + + +Trace handle: + +When a trace is added to a context, bt_context_add_trace() returns a trace +handle id. This id is associated with its corresponding trace handle. With +that id, it is possible to manipulate directly the trace. + + * bt_trace_handle_get_path() + -> returns the path of the trace handle (path to the trace). + + * bt_trace_handle_get_timestamp_begin() + * bt_trace_handle_get_timestamp_end() + -> return the creation/destruction timestamps (in ns or cycles + depending on the type specified) of the buffers of a + trace. + + * bt_ctf_event_get_handle_id() + -> returns the handle id associated with an event. + + +For more information on CTF, see the CTF documentation. -- 1.7.10.4 From christian.babeux at efficios.com Thu Aug 23 22:05:11 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Thu, 23 Aug 2012 22:05:11 -0400 Subject: [lttng-dev] [PATCH v2 lttng-tools] Filter: Notify the user when a filter is already enabled on event(s) In-Reply-To: <1345760474-29248-1-git-send-email-christian.babeux@efficios.com> References: <1345760474-29248-1-git-send-email-christian.babeux@efficios.com> Message-ID: <1345773912-30858-1-git-send-email-christian.babeux@efficios.com> Hi, Changelog from v1: - Style fixes. - Add missing changelog. Thanks, Christian Babeux (1): Filter: Notify the user when a filter is already enabled on event(s) src/bin/lttng/commands/enable_events.c | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) -- 1.7.11.4 From christian.babeux at efficios.com Thu Aug 23 22:05:12 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Thu, 23 Aug 2012 22:05:12 -0400 Subject: [lttng-dev] [PATCH v2 lttng-tools] Filter: Notify the user when a filter is already enabled on event(s) In-Reply-To: <1345773912-30858-1-git-send-email-christian.babeux@efficios.com> References: <1345760474-29248-1-git-send-email-christian.babeux@efficios.com> <1345773912-30858-1-git-send-email-christian.babeux@efficios.com> Message-ID: <1345773912-30858-2-git-send-email-christian.babeux@efficios.com> When using the enable-event command in conjunction with the --filter option, a user can specify a filter expression to refine the trace output. As stated in the lttng(1) man page, only the first activation of a filter on an event will work. Subsequent activation of any filter expression on the same event will fail. e.g: > lttng enable-event app:tp -s session -u --filter 'somefield > 42' # Case: invalid filter expression > lttng enable-event app:tp -s session -u --filter 'invalid expression' Error: Error setting filter Warning: Some command(s) went wrong > ... # Case: filter already enabled for event app:tp > lttng enable-event app:tp -s session -u --filter 'someotherfield < 42' Error: Error setting filter Warning: Some command(s) went wrong This commit differentiate the case where a filter was already set for the specified event from the generic 'Error setting filter' error message. e.g: > lttng enable-event app:tp -s session -u --filter 'somefield > 42' # Case: invalid filter expression > lttng enable-event app:tp -s session -u --filter 'invalid expression' Error: Error setting filter Warning: Some command(s) went wrong > ... # Case: filter already enabled for event app:tp > lttng enable-event app:tp -s session -u --filter 'someotherfield < 42' Error: Filter on event app:tp is already enabled (channel 0, session session) Warning: Some command(s) went wrong Signed-off-by: Christian Babeux --- src/bin/lttng/commands/enable_events.c | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/src/bin/lttng/commands/enable_events.c b/src/bin/lttng/commands/enable_events.c index a2c8a68..5a81fdb 100644 --- a/src/bin/lttng/commands/enable_events.c +++ b/src/bin/lttng/commands/enable_events.c @@ -384,7 +384,16 @@ static int enable_events(char *session_name) ret = lttng_set_event_filter(handle, ev.name, channel_name, opt_filter); if (ret < 0) { - ERR("Error setting filter"); + switch (-ret) { + case LTTCOMM_FILTER_EXIST: + ERR("Filter on events is already enabled (channel %s, session %s)", + channel_name, session_name); + break; + default: + ERR("Error setting filter"); + break; + } + ret = -1; goto error; } @@ -554,7 +563,16 @@ static int enable_events(char *session_name) ret = lttng_set_event_filter(handle, ev.name, channel_name, opt_filter); if (ret < 0) { - ERR("Error setting filter"); + switch (-ret) { + case LTTCOMM_FILTER_EXIST: + ERR("Filter on event %s is already enabled (channel %s, session %s)", + event_name, channel_name, session_name); + break; + default: + ERR("Error setting filter"); + break; + } + ret = -1; goto error; } -- 1.7.11.4 From mathieu.desnoyers at efficios.com Thu Aug 23 22:15:24 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 23 Aug 2012 22:15:24 -0400 Subject: [lttng-dev] [babeltrace] About the type read and write In-Reply-To: <20120823122754.GA23841@Krystal> References: <20120823122754.GA23841@Krystal> Message-ID: <20120824021524.GA13813@Krystal> * Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote: > * Hui Zhu (teawater at gmail.com) wrote: > > Hi, > > > > I just tried to use libbabeltrace to read and write the CTF file. But > > I cannot find the api to read some type of CTF for example > > CTF_TYPE_ENUM, CTF_TYPE_STRUCT. > > AFAIU, the API bt_ctf_get_field_list allows to get the list of > definitions from a struct, variant, array, or sequence (any compound > type). Julien, I think the header documentation of this function should > really be updated, can you look into that ? Updated in the master branch. > > Indeed, I also cannot see any API to get the enum string from the field. > Julien, can we add it ? Now supported in the master branch. Feedback welcome, Thanks! Mathieu > > > And I cannot find the api file to write to the CTF file. (I check the > > code of babeltrace-log and found that it use the function that is not > > API) > > Currently, babeltrace only does CTF-to-text and text-log-to-CTF > conversions. As you noticed, it does not expose API to write into all > CTF types at this stage. > > There is a prototype I am aware of that does just that (write custom CTF > files): Javeltrace. See the post here > http://lists.lttng.org/pipermail/lttng-dev/2012-August/018393.html for > details. > > Thanks, > > Mathieu > > > > > Could you help me with it? > > > > Thanks, > > Hui > > > > _______________________________________________ > > lttng-dev mailing list > > lttng-dev at lists.lttng.org > > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Thu Aug 23 22:17:21 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Thu, 23 Aug 2012 22:17:21 -0400 Subject: [lttng-dev] [PATCH v2 lttng-tools] Filter: Notify the user when a filter is already enabled on event(s) In-Reply-To: <1345773912-30858-2-git-send-email-christian.babeux@efficios.com> References: <1345760474-29248-1-git-send-email-christian.babeux@efficios.com> <1345773912-30858-1-git-send-email-christian.babeux@efficios.com> <1345773912-30858-2-git-send-email-christian.babeux@efficios.com> Message-ID: <20120824021721.GB13813@Krystal> * Christian Babeux (christian.babeux at efficios.com) wrote: > When using the enable-event command in conjunction with the --filter option, > a user can specify a filter expression to refine the trace output. > > As stated in the lttng(1) man page, only the first activation of a filter on > an event will work. Subsequent activation of any filter expression on the > same event will fail. > > e.g: > > > lttng enable-event app:tp -s session -u --filter 'somefield > 42' > > # Case: invalid filter expression > > lttng enable-event app:tp -s session -u --filter 'invalid expression' > Error: Error setting filter > Warning: Some command(s) went wrong > > > ... > > # Case: filter already enabled for event app:tp > > lttng enable-event app:tp -s session -u --filter 'someotherfield < 42' > Error: Error setting filter > Warning: Some command(s) went wrong > > This commit differentiate the case where a filter was already set for the > specified event from the generic 'Error setting filter' error message. > > e.g: > > > lttng enable-event app:tp -s session -u --filter 'somefield > 42' > > # Case: invalid filter expression > > lttng enable-event app:tp -s session -u --filter 'invalid expression' > Error: Error setting filter > Warning: Some command(s) went wrong > > > ... > > # Case: filter already enabled for event app:tp > > lttng enable-event app:tp -s session -u --filter 'someotherfield < 42' > Error: Filter on event app:tp is already enabled (channel 0, session session) > Warning: Some command(s) went wrong > > Signed-off-by: Christian Babeux Acked-by: Mathieu Desnoyers Thanks! Mathieu > --- > src/bin/lttng/commands/enable_events.c | 22 ++++++++++++++++++++-- > 1 file changed, 20 insertions(+), 2 deletions(-) > > diff --git a/src/bin/lttng/commands/enable_events.c b/src/bin/lttng/commands/enable_events.c > index a2c8a68..5a81fdb 100644 > --- a/src/bin/lttng/commands/enable_events.c > +++ b/src/bin/lttng/commands/enable_events.c > @@ -384,7 +384,16 @@ static int enable_events(char *session_name) > ret = lttng_set_event_filter(handle, ev.name, channel_name, > opt_filter); > if (ret < 0) { > - ERR("Error setting filter"); > + switch (-ret) { > + case LTTCOMM_FILTER_EXIST: > + ERR("Filter on events is already enabled (channel %s, session %s)", > + channel_name, session_name); > + break; > + default: > + ERR("Error setting filter"); > + break; > + } > + > ret = -1; > goto error; > } > @@ -554,7 +563,16 @@ static int enable_events(char *session_name) > ret = lttng_set_event_filter(handle, ev.name, > channel_name, opt_filter); > if (ret < 0) { > - ERR("Error setting filter"); > + switch (-ret) { > + case LTTCOMM_FILTER_EXIST: > + ERR("Filter on event %s is already enabled (channel %s, session %s)", > + event_name, channel_name, session_name); > + break; > + default: > + ERR("Error setting filter"); > + break; > + } > + > ret = -1; > goto error; > } > -- > 1.7.11.4 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From teawater at gmail.com Fri Aug 24 02:56:48 2012 From: teawater at gmail.com (Hui Zhu) Date: Fri, 24 Aug 2012 14:56:48 +0800 Subject: [lttng-dev] [babeltrace] About the type read and write In-Reply-To: <20120824021524.GA13813@Krystal> References: <20120823122754.GA23841@Krystal> <20120824021524.GA13813@Krystal> Message-ID: On Fri, Aug 24, 2012 at 10:15 AM, Mathieu Desnoyers wrote: > * Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote: >> * Hui Zhu (teawater at gmail.com) wrote: >> > Hi, >> > >> > I just tried to use libbabeltrace to read and write the CTF file. But >> > I cannot find the api to read some type of CTF for example >> > CTF_TYPE_ENUM, CTF_TYPE_STRUCT. >> >> AFAIU, the API bt_ctf_get_field_list allows to get the list of >> definitions from a struct, variant, array, or sequence (any compound >> type). Julien, I think the header documentation of this function should >> really be updated, can you look into that ? > > Updated in the master branch. > >> >> Indeed, I also cannot see any API to get the enum string from the field. >> Julien, can we add it ? > > Now supported in the master branch. > > Feedback welcome, > > Thanks! > > Mathieu > >> >> > And I cannot find the api file to write to the CTF file. (I check the >> > code of babeltrace-log and found that it use the function that is not >> > API) >> >> Currently, babeltrace only does CTF-to-text and text-log-to-CTF >> conversions. As you noticed, it does not expose API to write into all >> CTF types at this stage. >> >> There is a prototype I am aware of that does just that (write custom CTF >> files): Javeltrace. See the post here >> http://lists.lttng.org/pipermail/lttng-dev/2012-August/018393.html for >> details. >> >> Thanks, >> >> Mathieu >> >> > >> > Could you help me with it? >> > >> > Thanks, >> > Hui >> > >> > _______________________________________________ >> > lttng-dev mailing list >> > lttng-dev at lists.lttng.org >> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >> >> -- >> Mathieu Desnoyers >> Operating System Efficiency R&D Consultant >> EfficiOS Inc. >> http://www.efficios.com >> >> _______________________________________________ >> lttng-dev mailing list >> lttng-dev at lists.lttng.org >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com Great! Thanks for your help. Best, Hui From teawater at gmail.com Fri Aug 24 05:41:10 2012 From: teawater at gmail.com (Hui Zhu) Date: Fri, 24 Aug 2012 17:41:10 +0800 Subject: [lttng-dev] [babeltrace] About the type read and write In-Reply-To: <20120823122754.GA23841@Krystal> References: <20120823122754.GA23841@Krystal> Message-ID: On Thu, Aug 23, 2012 at 8:27 PM, Mathieu Desnoyers wrote: > * Hui Zhu (teawater at gmail.com) wrote: >> Hi, >> >> I just tried to use libbabeltrace to read and write the CTF file. But >> I cannot find the api to read some type of CTF for example >> CTF_TYPE_ENUM, CTF_TYPE_STRUCT. > > AFAIU, the API bt_ctf_get_field_list allows to get the list of > definitions from a struct, variant, array, or sequence (any compound > type). Julien, I think the header documentation of this function should > really be updated, can you look into that ? > > Indeed, I also cannot see any API to get the enum string from the field. > Julien, can we add it ? > >> And I cannot find the api file to write to the CTF file. (I check the >> code of babeltrace-log and found that it use the function that is not >> API) > > Currently, babeltrace only does CTF-to-text and text-log-to-CTF > conversions. As you noticed, it does not expose API to write into all > CTF types at this stage. Do you have plan to expose some API that can write CTF? Thanks, Hui > > There is a prototype I am aware of that does just that (write custom CTF > files): Javeltrace. See the post here > http://lists.lttng.org/pipermail/lttng-dev/2012-August/018393.html for > details. > > Thanks, > > Mathieu > >> >> Could you help me with it? >> >> Thanks, >> Hui >> >> _______________________________________________ >> lttng-dev mailing list >> lttng-dev at lists.lttng.org >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com From mathieu.desnoyers at efficios.com Fri Aug 24 07:38:52 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 24 Aug 2012 07:38:52 -0400 Subject: [lttng-dev] [babeltrace] About the type read and write In-Reply-To: References: <20120823122754.GA23841@Krystal> Message-ID: <20120824113852.GB12824@Krystal> * Hui Zhu (teawater at gmail.com) wrote: > On Thu, Aug 23, 2012 at 8:27 PM, Mathieu Desnoyers > wrote: > > * Hui Zhu (teawater at gmail.com) wrote: > >> Hi, > >> > >> I just tried to use libbabeltrace to read and write the CTF file. But > >> I cannot find the api to read some type of CTF for example > >> CTF_TYPE_ENUM, CTF_TYPE_STRUCT. > > > > AFAIU, the API bt_ctf_get_field_list allows to get the list of > > definitions from a struct, variant, array, or sequence (any compound > > type). Julien, I think the header documentation of this function should > > really be updated, can you look into that ? > > > > Indeed, I also cannot see any API to get the enum string from the field. > > Julien, can we add it ? > > > >> And I cannot find the api file to write to the CTF file. (I check the > >> code of babeltrace-log and found that it use the function that is not > >> API) > > > > Currently, babeltrace only does CTF-to-text and text-log-to-CTF > > conversions. As you noticed, it does not expose API to write into all > > CTF types at this stage. > > Do you have plan to expose some API that can write CTF? Not for 1.0, as it has nearly completed its development cycle. The intent is to someday add CTF write capability to babeltrace, but we are not there yet, and we have not received any requests/contracts to do. Thanks, Mathieu > > Thanks, > Hui > > > > > There is a prototype I am aware of that does just that (write custom CTF > > files): Javeltrace. See the post here > > http://lists.lttng.org/pipermail/lttng-dev/2012-August/018393.html for > > details. > > > > Thanks, > > > > Mathieu > > > >> > >> Could you help me with it? > >> > >> Thanks, > >> Hui > >> > >> _______________________________________________ > >> lttng-dev mailing list > >> lttng-dev at lists.lttng.org > >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > > > -- > > Mathieu Desnoyers > > Operating System Efficiency R&D Consultant > > EfficiOS Inc. > > http://www.efficios.com -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From henrik at fiktivkod.org Fri Aug 24 08:01:58 2012 From: henrik at fiktivkod.org (Henrik Hautakoski) Date: Fri, 24 Aug 2012 14:01:58 +0200 Subject: [lttng-dev] LTTNG bug when running on single user (root) system Message-ID: LTTng seems to work only when sessiond and the lttng process are owned by different users! Summary: sessiond enable-enevent WORKS? user user NO root user YES (No output in sessiond window) root root NO user root YES (No output in sessiond window) Details: sudo apt-get -F install uuid-dev libpopt-dev autoconf automake libtool mkdir ltt cd ltt/ git clone git://git.lttng.org/lttng-ust.git git clone git://git.lttng.org/lttng-tools.git git clone git://git.lttng.org/userspace-rcu.git cd userspace-rcu/ ./bootstrap ./configure make sudo make install cd .. cd lttng-ust/ ./bootstrap ./configure make sudo make install cd .. cd lttng-tools/ ./bootstrap ./configure make sudo make install =========================================================================== Testing as normal user(daemon normal, lttng normnal): ============================================================================ dfm01 at ubuntu:~$ lsof | grep ltt bash 2342 dfm01 cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools gedit 13840 dfm01 cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools dfm01 at ubuntu:~$ export LD_LIBRARY_PATH=/usr/local/lib dfm01 at ubuntu:~$ lttng-sessiond --no-kernel -vvv --verbose-consumer --consumerd32-path /usr/local/lib dfm01 at ubuntu:~/ltt/lttng-tools/tests$ lttng create Trace(s) output set to /home/dfm01/lttng-traces/auto-20120823-102454 Session created with default name auto-20120823-102454 Daemon log: DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] DEBUG1: Receiving data from client ... [in thread_manage_clients() at main.c:3031] DEBUG1: Nothing recv() from client... continuing [in thread_manage_clients() at main.c:3035] DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] DEBUG1: Receiving data from client ... [in thread_manage_clients() at main.c:3031] DEBUG1: Processing client command 8 [in process_client_msg() at main.c:2004] DEBUG1: Waiting for 1 URIs from client ... [in process_client_msg() at main.c:2505] DEBUG2: Trying to find session by name auto-20120823-102454 [in session_find_by_name() at session.c:122] DEBUG1: Tracing session auto-20120823-102454 created in (null) with ID 0 by UID 1000 GID 1000 [in session_create() at session.c:236] DEBUG2: Trying to find session by name auto-20120823-102454 [in session_find_by_name() at session.c:122] DEBUG3: Created hashtable size 4 at 0xb440a928 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG2: Setting trace directory path from URI to /home/dfm01/lttng-traces/auto-20120823-102454 [in add_uri_to_consumer() at cmd.c:395] DEBUG1: Sending response (size: 16, retcode: Success) [in thread_manage_clients() at main.c:3082] DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] dfm01 at ubuntu:~/ltt/lttng-tools/tests$ lttng enable-event -u all Error: Event all: 32-bit UST consumer start failed (channel channel0, session auto-20120823-102454) Warning: Some command(s) went wrong DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] DEBUG1: Receiving data from client ... [in thread_manage_clients() at main.c:3031] DEBUG1: Nothing recv() from client... continuing [in thread_manage_clients() at main.c:3035] DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] DEBUG1: Receiving data from client ... [in thread_manage_clients() at main.c:3031] DEBUG1: Processing client command 6 [in process_client_msg() at main.c:2004] DEBUG1: Getting session auto-20120823-102454 by name [in process_client_msg() at main.c:2074] DEBUG2: Trying to find session by name auto-20120823-102454 [in session_find_by_name() at session.c:122] DEBUG1: Creating UST session [in create_ust_session() at main.c:1891] DEBUG3: Created hashtable size 4 at 0xb440ba70 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG3: Created hashtable size 4 at 0xb440bb90 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG3: Created hashtable size 4 at 0xb440bcb0 of type 0 [in lttng_ht_new() at hashtable.c:96] DEBUG3: Created hashtable size 4 at 0xb4410e18 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG2: UST trace session create successful [in trace_ust_create_session() at trace-ust.c:143] DEBUG3: Copying tracing session consumer output in UST session [in copy_session_consumer() at main.c:1840] DEBUG3: Created hashtable size 4 at 0xb4415f80 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG3: Created hashtable size 4 at 0xb44160a0 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG2: Consumer subdir set to /auto-20120823-102454 [in consumer_set_subdir() at consumer.c:689] DEBUG3: Copy session consumer subdir /ust [in copy_session_consumer() at main.c:1861] DEBUG1: Spawning consumerd [in spawn_consumerd() at main.c:1547] DEBUG2: Consumer pid 10564 [in start_consumerd() at main.c:1718] DEBUG2: Spawning consumer control thread [in start_consumerd() at main.c:1721] DEBUG1: Using 32-bit UST consumer at: /usr/local/lib [in spawn_consumerd() at main.c:1665] Error: Spawning consumerd failed DEBUG1: Sending response (size: 16, retcode: 32-bit UST consumer start failed) [in thread_manage_clients() at main.c:3082] DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] DEBUG1: [thread] Manage consumer started [in thread_manage_consumer() at main.c:802] Fails dfm01 at ubuntu:~$ lsof | grep ltt bash 2342 dfm01 cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools lttng-ses 8435 dfm01 cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools lttng-ses 8435 dfm01 rtd DIR 8,1 4096 2 / lttng-ses 8435 dfm01 txt REG 8,1 660624 927277 /usr/local/bin/lttng-sessiond lttng-ses 8435 dfm01 mem REG 8,1 92016 405535 /lib/i386-linux-gnu/libnsl-2.15.so lttng-ses 8435 dfm01 mem REG 8,1 16833 917842 /usr/local/lib/liburcu-common.so.1.0.0 lttng-ses 8435 dfm01 mem REG 8,1 42652 405540 /lib/i386-linux-gnu/libnss_nis-2.15.so lttng-ses 8435 dfm01 mem REG 8,1 47040 405538 /lib/i386-linux-gnu/libnss_files-2.15.so lttng-ses 8435 dfm01 mem REG 8,1 62218 925784 /usr/local/lib/liburcu-bp.so.1.0.0 lttng-ses 8435 dfm01 mem REG 8,1 54778 925752 /usr/local/lib/liburcu.so.1.0.0 lttng-ses 8435 dfm01 mem REG 8,1 1713640 405529 /lib/i386-linux-gnu/libc-2.15.so lttng-ses 8435 dfm01 mem REG 8,1 393957 921931 /usr/local/lib/liblttng-ctl.so.0.0.0 lttng-ses 8435 dfm01 mem REG 8,1 572112 926835 /usr/local/lib/liblttng-ust-ctl.so.0.0.0 lttng-ses 8435 dfm01 mem REG 8,1 30684 405545 /lib/i386-linux-gnu/librt-2.15.so lttng-ses 8435 dfm01 mem REG 8,1 134344 393277 /lib/i386-linux-gnu/ld-2.15.so lttng-ses 8435 dfm01 mem REG 8,1 124663 405543 /lib/i386-linux-gnu/libpthread-2.15.so lttng-ses 8435 dfm01 mem REG 8,1 30520 405536 /lib/i386-linux-gnu/libnss_compat-2.15.so lttng-ses 8435 dfm01 mem REG 0,18 4096 71410 /run/shm/lttng-ust-apps-wait-1000 lttng-ses 8435 dfm01 0u CHR 136,2 0t0 5 /dev/pts/2 lttng-ses 8435 dfm01 1u CHR 136,2 0t0 5 /dev/pts/2 lttng-ses 8435 dfm01 2u CHR 136,2 0t0 5 /dev/pts/2 lttng-ses 8435 dfm01 3r FIFO 0,8 0t0 97133 pipe lttng-ses 8435 dfm01 4w FIFO 0,8 0t0 97133 pipe lttng-ses 8435 dfm01 5u unix 0x00000000 0t0 97134 /home/dfm01/.lttng/ustconsumerd64/error lttng-ses 8435 dfm01 6u unix 0x00000000 0t0 97135 /home/dfm01/.lttng/ustconsumerd32/error lttng-ses 8435 dfm01 7u unix 0x00000000 0t0 97136 /home/dfm01/.lttng/client-lttng-sessiond lttng-ses 8435 dfm01 8u unix 0x00000000 0t0 97137 /home/dfm01/.lttng/apps-lttng-sessiond lttng-ses 8435 dfm01 9r FIFO 0,8 0t0 97138 pipe lttng-ses 8435 dfm01 10w FIFO 0,8 0t0 97138 pipe lttng-ses 8435 dfm01 11r FIFO 0,8 0t0 97139 pipe lttng-ses 8435 dfm01 12w FIFO 0,8 0t0 97139 pipe lttng-ses 8435 dfm01 13u 0000 0,9 0 4765 anon_inode lttng-ses 8435 dfm01 14u 0000 0,9 0 4765 anon_inode lttng-ses 8435 dfm01 15u 0000 0,9 0 4765 anon_inode lttng-ses 8435 dfm01 16u unix 0x00000000 0t0 97140 /home/dfm01/.lttng/health.sock lttng-ses 8435 dfm01 17u 0000 0,9 0 4765 anon_inode lttng-ses 8435 dfm01 18u 0000 0,9 0 4765 anon_inode gedit 13840 dfm01 cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools ================================================================================ As root: for daemon ================================================================================ root at ubuntu:~$ export LD_LIBRARY_PATH=/usr/local/lib root at ubuntu:~$ sudo lttng-sessiond --no-kernel -vvv --verbose-consumer --consumerd32-path /usr/local/lib dfm01 at ubuntu:~/ltt/lttng-tools$ lsof | grep ltt bash 2342 dfm01 cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools lttng-ses 8494 root cwd unknown /proc/8494/cwd (readlink: Permission denied) lttng-ses 8494 root rtd unknown /proc/8494/root (readlink: Permission denied) lttng-ses 8494 root txt unknown /proc/8494/exe (readlink: Permission denied) lttng-ses 8494 root NOFD /proc/8494/fd (opendir: Permission denied) lsof 8504 dfm01 cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools grep 8505 dfm01 cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools lsof 8506 dfm01 cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools gedit 13840 dfm01 cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools sudo lsof | grep ltt dfm01 at ubuntu:~/ltt/lttng-tools$ sudo lsof | grep ltt [sudo] password for dfm01: lsof: WARNING: can't stat() fuse.gvfs-fuse-daemon file system /home/dfm01/.gvfs Output information may be incomplete. bash 2342 dfm01 cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools lttng-ses 8494 root cwd DIR 8,1 4096 270695 /home/dfm01 lttng-ses 8494 root rtd DIR 8,1 4096 2 / lttng-ses 8494 root txt REG 8,1 660624 927277 /usr/local/bin/lttng-sessiond lttng-ses 8494 root mem REG 8,1 92016 405535 /lib/i386-linux-gnu/libnsl-2.15.so lttng-ses 8494 root mem REG 8,1 47040 405538 /lib/i386-linux-gnu/libnss_files-2.15.so lttng-ses 8494 root mem REG 8,1 62218 925784 /usr/local/lib/liburcu-bp.so.1.0.0 lttng-ses 8494 root mem REG 8,1 134344 393277 /lib/i386-linux-gnu/ld-2.15.so lttng-ses 8494 root mem REG 8,1 42652 405540 /lib/i386-linux-gnu/libnss_nis-2.15.so lttng-ses 8494 root mem REG 8,1 1713640 405529 /lib/i386-linux-gnu/libc-2.15.so lttng-ses 8494 root mem REG 8,1 30684 405545 /lib/i386-linux-gnu/librt-2.15.so lttng-ses 8494 root mem REG 8,1 124663 405543 /lib/i386-linux-gnu/libpthread-2.15.so lttng-ses 8494 root mem REG 8,1 30520 405536 /lib/i386-linux-gnu/libnss_compat-2.15.so lttng-ses 8494 root mem REG 8,1 16833 917842 /usr/local/lib/liburcu-common.so.1.0.0 lttng-ses 8494 root mem REG 8,1 572112 926835 /usr/local/lib/liblttng-ust-ctl.so.0.0.0 lttng-ses 8494 root mem REG 8,1 54778 925752 /usr/local/lib/liburcu.so.1.0.0 lttng-ses 8494 root mem REG 8,1 393957 921931 /usr/local/lib/liblttng-ctl.so.0.0.0 lttng-ses 8494 root mem REG 0,18 4096 98026 /run/shm/lttng-ust-apps-wait lttng-ses 8494 root 0u CHR 136,3 0t0 6 /dev/pts/3 lttng-ses 8494 root 1u CHR 136,3 0t0 6 /dev/pts/3 lttng-ses 8494 root 2u CHR 136,3 0t0 6 /dev/pts/3 lttng-ses 8494 root 3r FIFO 0,8 0t0 98007 pipe lttng-ses 8494 root 4w FIFO 0,8 0t0 98007 pipe lttng-ses 8494 root 5u unix 0xf53d86c0 0t0 98010 /var/run/lttng/kconsumerd/error lttng-ses 8494 root 6u unix 0xf53d9440 0t0 98013 /var/run/lttng/ustconsumerd64/error lttng-ses 8494 root 7u unix 0xf53d9d40 0t0 98016 /var/run/lttng/ustconsumerd32/error lttng-ses 8494 root 8u unix 0xf53d9b00 0t0 98018 /var/run/lttng/client-lttng-sessiond lttng-ses 8494 root 9u unix 0xf53da1c0 0t0 98020 /var/run/lttng/apps-lttng-sessiond lttng-ses 8494 root 10r FIFO 0,8 0t0 98024 pipe lttng-ses 8494 root 11w FIFO 0,8 0t0 98024 pipe lttng-ses 8494 root 12r FIFO 0,8 0t0 98025 pipe lttng-ses 8494 root 13w FIFO 0,8 0t0 98025 pipe lttng-ses 8494 root 14u 0000 0,9 0 4765 anon_inode lttng-ses 8494 root 15u 0000 0,9 0 4765 anon_inode lttng-ses 8494 root 16u 0000 0,9 0 4765 anon_inode lttng-ses 8494 root 17u 0000 0,9 0 4765 anon_inode lttng-ses 8494 root 18u unix 0xf53d9680 0t0 98027 /var/run/lttng/health.sock lttng-ses 8494 root 19u 0000 0,9 0 4765 anon_inode sudo 8517 root cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools grep 8518 dfm01 cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools lsof 8519 root cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools lsof 8520 root cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools gedit 13840 dfm01 cwd DIR 8,1 4096 353314 /home/dfm01/ltt/lttng-tools =========================================================================== Testing as normal user(daemon is run as root, lttng as normnal): ============================================================================ dfm01 at ubuntu:~/ltt/lttng-tools/tests$ lttng create Trace(s) output set to /home/dfm01/lttng-traces/auto-20120823-102111 Session created with default name auto-20120823-102111 dfm01 at ubuntu:~/ltt/lttng-tools/tests$ lttng enable-event -u all UST event all created in channel channel0 Interesting, no DEBUG messages whatsoever in daemon window. =========== Test that fails: ========================== root at ubuntu:/home/dfm01/ltt/lttng-tools/tests# lttng create Trace(s) output set to /root/lttng-traces/auto-20120823-100802 Session created with default name auto-20120823-100802 Daemon log: DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] DEBUG1: Receiving data from client ... [in thread_manage_clients() at main.c:3031] DEBUG1: Nothing recv() from client... continuing [in thread_manage_clients() at main.c:3035] DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] DEBUG1: Receiving data from client ... [in thread_manage_clients() at main.c:3031] DEBUG1: Processing client command 8 [in process_client_msg() at main.c:2004] DEBUG1: Waiting for 1 URIs from client ... [in process_client_msg() at main.c:2505] DEBUG2: Trying to find session by name auto-20120823-101733 [in session_find_by_name() at session.c:122] DEBUG1: Tracing session auto-20120823-101733 created in (null) with ID 2 by UID 0 GID 0 [in session_create() at session.c:236] DEBUG2: Trying to find session by name auto-20120823-101733 [in session_find_by_name() at session.c:122] DEBUG3: Created hashtable size 4 at 0xb4416780 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG2: Setting trace directory path from URI to /root/lttng-traces/auto-20120823-101733 [in add_uri_to_consumer() at cmd.c:395] DEBUG1: Sending response (size: 16, retcode: Success) [in thread_manage_clients() at main.c:3082] DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] root at ubuntu:/home/dfm01/ltt/lttng-tools/tests# lttng enable-event -u all Error: Event all: 32-bit UST consumer start failed (channel channel0, session auto-20120823-100802) Warning: Some command(s) went wrong Daemon log: DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] DEBUG1: Receiving data from client ... [in thread_manage_clients() at main.c:3031] DEBUG1: Nothing recv() from client... continuing [in thread_manage_clients() at main.c:3035] DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] DEBUG1: Receiving data from client ... [in thread_manage_clients() at main.c:3031] DEBUG1: Processing client command 6 [in process_client_msg() at main.c:2004] DEBUG1: Getting session auto-20120823-101733 by name [in process_client_msg() at main.c:2074] DEBUG2: Trying to find session by name auto-20120823-101733 [in session_find_by_name() at session.c:122] DEBUG1: Creating UST session [in create_ust_session() at main.c:1891] DEBUG3: Created hashtable size 4 at 0xb44231b8 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG3: Created hashtable size 4 at 0xb44232d8 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG3: Created hashtable size 4 at 0xb44233f8 of type 0 [in lttng_ht_new() at hashtable.c:96] DEBUG3: Created hashtable size 4 at 0xb4428560 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG2: UST trace session create successful [in trace_ust_create_session() at trace-ust.c:143] DEBUG3: Copying tracing session consumer output in UST session [in copy_session_consumer() at main.c:1840] DEBUG3: Created hashtable size 4 at 0xb442d6c8 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG3: Created hashtable size 4 at 0xb442d7e8 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG2: Consumer subdir set to /auto-20120823-101733 [in consumer_set_subdir() at consumer.c:689] DEBUG3: Copy session consumer subdir /ust [in copy_session_consumer() at main.c:1861] DEBUG2: Trace UST channel channel0 not found by name [in trace_ust_find_channel_by_name() at trace-ust.c:52] DEBUG1: Enabling channel channel0 for session auto-20120823-101733 [in cmd_enable_channel() at cmd.c:743] DEBUG2: Trace UST channel channel0 not found by name [in trace_ust_find_channel_by_name() at trace-ust.c:52] DEBUG3: Created hashtable size 4 at 0xb442edd0 of type 0 [in lttng_ht_new() at hashtable.c:96] DEBUG3: Created hashtable size 4 at 0xb442eef0 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG2: Trace UST channel channel0 created [in trace_ust_create_channel() at trace-ust.c:205] DEBUG2: Channel channel0 being created in UST global domain [in channel_ust_create() at channel.c:267] DEBUG2: UST app adding channel channel0 to global domain for session id 2 [in ust_app_create_channel_glb() at ust-app.c:1984] DEBUG2: Channel channel0 created successfully [in channel_ust_create() at channel.c:292] DEBUG2: Trace UST channel channel0 found by name [in trace_ust_find_channel_by_name() at trace-ust.c:47] DEBUG2: Trace UST event NOT found by name all [in trace_ust_find_event_by_name() at trace-ust.c:79] DEBUG3: Created hashtable size 4 at 0xb442d908 of type 1 [in lttng_ht_new() at hashtable.c:96] DEBUG2: Trace UST event all, loglevel (0,-1) created [in trace_ust_create_event() at trace-ust.c:284] DEBUG1: UST app creating event all for all apps for session id 2 [in ust_app_create_event_glb() at ust-app.c:2108] DEBUG1: Event UST all created in channel channel0 [in event_ust_enable_tracepoint() at event.c:486] DEBUG1: Sending response (size: 16, retcode: Success) [in thread_manage_clients() at main.c:3082] DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] -- Henrik Hautakoski henrik at fiktivkod.org From teawater at gmail.com Fri Aug 24 08:48:21 2012 From: teawater at gmail.com (Hui Zhu) Date: Fri, 24 Aug 2012 20:48:21 +0800 Subject: [lttng-dev] [babeltrace] About the type read and write In-Reply-To: <20120824113852.GB12824@Krystal> References: <20120823122754.GA23841@Krystal> <20120824113852.GB12824@Krystal> Message-ID: On Fri, Aug 24, 2012 at 7:38 PM, Mathieu Desnoyers wrote: > * Hui Zhu (teawater at gmail.com) wrote: >> On Thu, Aug 23, 2012 at 8:27 PM, Mathieu Desnoyers >> wrote: >> > * Hui Zhu (teawater at gmail.com) wrote: >> >> Hi, >> >> >> >> I just tried to use libbabeltrace to read and write the CTF file. But >> >> I cannot find the api to read some type of CTF for example >> >> CTF_TYPE_ENUM, CTF_TYPE_STRUCT. >> > >> > AFAIU, the API bt_ctf_get_field_list allows to get the list of >> > definitions from a struct, variant, array, or sequence (any compound >> > type). Julien, I think the header documentation of this function should >> > really be updated, can you look into that ? >> > >> > Indeed, I also cannot see any API to get the enum string from the field. >> > Julien, can we add it ? >> > >> >> And I cannot find the api file to write to the CTF file. (I check the >> >> code of babeltrace-log and found that it use the function that is not >> >> API) >> > >> > Currently, babeltrace only does CTF-to-text and text-log-to-CTF >> > conversions. As you noticed, it does not expose API to write into all >> > CTF types at this stage. >> >> Do you have plan to expose some API that can write CTF? > > Not for 1.0, as it has nearly completed its development cycle. The > intent is to someday add CTF write capability to babeltrace, but we are > not there yet, and we have not received any requests/contracts to do. > OK. Thanks for your help, Mathieu. Best, Hui > Thanks, > > Mathieu > >> >> Thanks, >> Hui >> >> > >> > There is a prototype I am aware of that does just that (write custom CTF >> > files): Javeltrace. See the post here >> > http://lists.lttng.org/pipermail/lttng-dev/2012-August/018393.html for >> > details. >> > >> > Thanks, >> > >> > Mathieu >> > >> >> >> >> Could you help me with it? >> >> >> >> Thanks, >> >> Hui >> >> >> >> _______________________________________________ >> >> lttng-dev mailing list >> >> lttng-dev at lists.lttng.org >> >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >> > >> > -- >> > Mathieu Desnoyers >> > Operating System Efficiency R&D Consultant >> > EfficiOS Inc. >> > http://www.efficios.com > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com From mathieu.desnoyers at efficios.com Fri Aug 24 09:03:30 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Fri, 24 Aug 2012 09:03:30 -0400 Subject: [lttng-dev] LTTNG bug when running on single user (root) system In-Reply-To: References: Message-ID: <20120824130330.GB21676@Krystal> * Henrik Hautakoski (henrik at fiktivkod.org) wrote: > LTTng seems to work only when sessiond and the lttng process are owned > by different users! Please make sure that, when you test from a user account (not root), you don't have a per-user sessiond running. If a user is not part of the tracing group, it will spawn its own lttng-sessiond to trace its own application. This would explain why you don't see anything in the root sessiond debug log. Also, there seems to be an issue with starting the 32-bit consumerd, as far as I understand the logs. I'll let David look into this further. Best regards, Mathieu > > > Summary: > sessiond enable-enevent WORKS? > user user NO > root user YES (No output in sessiond window) > root root NO > user root YES (No output in sessiond window) > > > > Details: > > sudo apt-get -F install uuid-dev libpopt-dev autoconf automake libtool > mkdir ltt > cd ltt/ > git clone git://git.lttng.org/lttng-ust.git > git clone git://git.lttng.org/lttng-tools.git > git clone git://git.lttng.org/userspace-rcu.git > cd userspace-rcu/ > ./bootstrap > ./configure > make > sudo make install > cd .. > cd lttng-ust/ > ./bootstrap > ./configure > make > sudo make install > cd .. > cd lttng-tools/ > ./bootstrap > ./configure > make > sudo make install > =========================================================================== > Testing as normal user(daemon normal, lttng normnal): > ============================================================================ > dfm01 at ubuntu:~$ lsof | grep ltt > bash 2342 dfm01 cwd DIR 8,1 4096 > 353314 /home/dfm01/ltt/lttng-tools > gedit 13840 dfm01 cwd DIR 8,1 4096 353314 > /home/dfm01/ltt/lttng-tools > > dfm01 at ubuntu:~$ export LD_LIBRARY_PATH=/usr/local/lib > dfm01 at ubuntu:~$ lttng-sessiond --no-kernel -vvv --verbose-consumer > --consumerd32-path /usr/local/lib > > dfm01 at ubuntu:~/ltt/lttng-tools/tests$ lttng create > Trace(s) output set to /home/dfm01/lttng-traces/auto-20120823-102454 > Session created with default name auto-20120823-102454 > > Daemon log: > DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] > DEBUG1: Receiving data from client ... [in thread_manage_clients() at > main.c:3031] > DEBUG1: Nothing recv() from client... continuing [in > thread_manage_clients() at main.c:3035] > DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] > DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] > DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] > DEBUG1: Receiving data from client ... [in thread_manage_clients() at > main.c:3031] > DEBUG1: Processing client command 8 [in process_client_msg() at main.c:2004] > DEBUG1: Waiting for 1 URIs from client ... [in process_client_msg() at > main.c:2505] > DEBUG2: Trying to find session by name auto-20120823-102454 [in > session_find_by_name() at session.c:122] > DEBUG1: Tracing session auto-20120823-102454 created in (null) with ID > 0 by UID 1000 GID 1000 [in session_create() at session.c:236] > DEBUG2: Trying to find session by name auto-20120823-102454 [in > session_find_by_name() at session.c:122] > DEBUG3: Created hashtable size 4 at 0xb440a928 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG2: Setting trace directory path from URI to > /home/dfm01/lttng-traces/auto-20120823-102454 [in > add_uri_to_consumer() at cmd.c:395] > DEBUG1: Sending response (size: 16, retcode: Success) [in > thread_manage_clients() at main.c:3082] > DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] > DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] > > > > dfm01 at ubuntu:~/ltt/lttng-tools/tests$ lttng enable-event -u all > Error: Event all: 32-bit UST consumer start failed (channel channel0, > session auto-20120823-102454) > Warning: Some command(s) went wrong > > DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] > DEBUG1: Receiving data from client ... [in thread_manage_clients() at > main.c:3031] > DEBUG1: Nothing recv() from client... continuing [in > thread_manage_clients() at main.c:3035] > DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] > DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] > DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] > DEBUG1: Receiving data from client ... [in thread_manage_clients() at > main.c:3031] > DEBUG1: Processing client command 6 [in process_client_msg() at main.c:2004] > DEBUG1: Getting session auto-20120823-102454 by name [in > process_client_msg() at main.c:2074] > DEBUG2: Trying to find session by name auto-20120823-102454 [in > session_find_by_name() at session.c:122] > DEBUG1: Creating UST session [in create_ust_session() at main.c:1891] > DEBUG3: Created hashtable size 4 at 0xb440ba70 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG3: Created hashtable size 4 at 0xb440bb90 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG3: Created hashtable size 4 at 0xb440bcb0 of type 0 [in > lttng_ht_new() at hashtable.c:96] > DEBUG3: Created hashtable size 4 at 0xb4410e18 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG2: UST trace session create successful [in > trace_ust_create_session() at trace-ust.c:143] > DEBUG3: Copying tracing session consumer output in UST session [in > copy_session_consumer() at main.c:1840] > DEBUG3: Created hashtable size 4 at 0xb4415f80 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG3: Created hashtable size 4 at 0xb44160a0 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG2: Consumer subdir set to /auto-20120823-102454 [in > consumer_set_subdir() at consumer.c:689] > DEBUG3: Copy session consumer subdir /ust [in copy_session_consumer() > at main.c:1861] > DEBUG1: Spawning consumerd [in spawn_consumerd() at main.c:1547] > DEBUG2: Consumer pid 10564 [in start_consumerd() at main.c:1718] > DEBUG2: Spawning consumer control thread [in start_consumerd() at main.c:1721] > DEBUG1: Using 32-bit UST consumer at: /usr/local/lib [in > spawn_consumerd() at main.c:1665] > Error: Spawning consumerd failed > DEBUG1: Sending response (size: 16, retcode: 32-bit UST consumer start > failed) [in thread_manage_clients() at main.c:3082] > DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] > DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] > DEBUG1: [thread] Manage consumer started [in thread_manage_consumer() > at main.c:802] > > Fails > > > > dfm01 at ubuntu:~$ lsof | grep ltt > bash 2342 dfm01 cwd DIR 8,1 4096 353314 > /home/dfm01/ltt/lttng-tools > lttng-ses 8435 dfm01 cwd DIR 8,1 4096 353314 > /home/dfm01/ltt/lttng-tools > lttng-ses 8435 dfm01 rtd DIR 8,1 4096 2 / > lttng-ses 8435 dfm01 txt REG 8,1 660624 927277 > /usr/local/bin/lttng-sessiond > lttng-ses 8435 dfm01 mem REG 8,1 92016 405535 > /lib/i386-linux-gnu/libnsl-2.15.so > lttng-ses 8435 dfm01 mem REG 8,1 16833 917842 > /usr/local/lib/liburcu-common.so.1.0.0 > lttng-ses 8435 dfm01 mem REG 8,1 42652 405540 > /lib/i386-linux-gnu/libnss_nis-2.15.so > lttng-ses 8435 dfm01 mem REG 8,1 47040 405538 > /lib/i386-linux-gnu/libnss_files-2.15.so > lttng-ses 8435 dfm01 mem REG 8,1 62218 925784 > /usr/local/lib/liburcu-bp.so.1.0.0 > lttng-ses 8435 dfm01 mem REG 8,1 54778 925752 > /usr/local/lib/liburcu.so.1.0.0 > lttng-ses 8435 dfm01 mem REG 8,1 1713640 405529 > /lib/i386-linux-gnu/libc-2.15.so > lttng-ses 8435 dfm01 mem REG 8,1 393957 921931 > /usr/local/lib/liblttng-ctl.so.0.0.0 > lttng-ses 8435 dfm01 mem REG 8,1 572112 926835 > /usr/local/lib/liblttng-ust-ctl.so.0.0.0 > lttng-ses 8435 dfm01 mem REG 8,1 30684 405545 > /lib/i386-linux-gnu/librt-2.15.so > lttng-ses 8435 dfm01 mem REG 8,1 134344 393277 > /lib/i386-linux-gnu/ld-2.15.so > lttng-ses 8435 dfm01 mem REG 8,1 124663 405543 > /lib/i386-linux-gnu/libpthread-2.15.so > lttng-ses 8435 dfm01 mem REG 8,1 30520 405536 > /lib/i386-linux-gnu/libnss_compat-2.15.so > lttng-ses 8435 dfm01 mem REG 0,18 4096 71410 > /run/shm/lttng-ust-apps-wait-1000 > lttng-ses 8435 dfm01 0u CHR 136,2 0t0 5 /dev/pts/2 > lttng-ses 8435 dfm01 1u CHR 136,2 0t0 5 /dev/pts/2 > lttng-ses 8435 dfm01 2u CHR 136,2 0t0 5 /dev/pts/2 > lttng-ses 8435 dfm01 3r FIFO 0,8 0t0 97133 pipe > lttng-ses 8435 dfm01 4w FIFO 0,8 0t0 97133 pipe > lttng-ses 8435 dfm01 5u unix 0x00000000 0t0 97134 > /home/dfm01/.lttng/ustconsumerd64/error > lttng-ses 8435 dfm01 6u unix 0x00000000 0t0 97135 > /home/dfm01/.lttng/ustconsumerd32/error > lttng-ses 8435 dfm01 7u unix 0x00000000 0t0 97136 > /home/dfm01/.lttng/client-lttng-sessiond > lttng-ses 8435 dfm01 8u unix 0x00000000 0t0 97137 > /home/dfm01/.lttng/apps-lttng-sessiond > lttng-ses 8435 dfm01 9r FIFO 0,8 0t0 97138 pipe > lttng-ses 8435 dfm01 10w FIFO 0,8 0t0 97138 pipe > lttng-ses 8435 dfm01 11r FIFO 0,8 0t0 97139 pipe > lttng-ses 8435 dfm01 12w FIFO 0,8 0t0 97139 pipe > lttng-ses 8435 dfm01 13u 0000 0,9 0 4765 anon_inode > lttng-ses 8435 dfm01 14u 0000 0,9 0 4765 anon_inode > lttng-ses 8435 dfm01 15u 0000 0,9 0 4765 anon_inode > lttng-ses 8435 dfm01 16u unix 0x00000000 0t0 97140 > /home/dfm01/.lttng/health.sock > lttng-ses 8435 dfm01 17u 0000 0,9 0 4765 anon_inode > lttng-ses 8435 dfm01 18u 0000 0,9 0 4765 anon_inode > gedit 13840 dfm01 cwd DIR 8,1 4096 353314 > /home/dfm01/ltt/lttng-tools > > > ================================================================================ > As root: for daemon > ================================================================================ > > root at ubuntu:~$ export LD_LIBRARY_PATH=/usr/local/lib > root at ubuntu:~$ sudo lttng-sessiond --no-kernel -vvv > --verbose-consumer --consumerd32-path /usr/local/lib > > dfm01 at ubuntu:~/ltt/lttng-tools$ lsof | grep ltt > bash 2342 dfm01 cwd DIR 8,1 4096 353314 > /home/dfm01/ltt/lttng-tools > lttng-ses 8494 root cwd unknown > /proc/8494/cwd (readlink: Permission denied) > lttng-ses 8494 root rtd unknown > /proc/8494/root (readlink: Permission denied) > lttng-ses 8494 root txt unknown > /proc/8494/exe (readlink: Permission denied) > lttng-ses 8494 root NOFD > /proc/8494/fd (opendir: Permission denied) > lsof 8504 dfm01 cwd DIR 8,1 4096 353314 > /home/dfm01/ltt/lttng-tools > grep 8505 dfm01 cwd DIR 8,1 4096 353314 > /home/dfm01/ltt/lttng-tools > lsof 8506 dfm01 cwd DIR 8,1 4096 353314 > /home/dfm01/ltt/lttng-tools > gedit 13840 dfm01 cwd DIR 8,1 4096 353314 > /home/dfm01/ltt/lttng-tools > > sudo lsof | grep ltt > > dfm01 at ubuntu:~/ltt/lttng-tools$ sudo lsof | grep ltt > [sudo] password for dfm01: > lsof: WARNING: can't stat() fuse.gvfs-fuse-daemon file system /home/dfm01/.gvfs > Output information may be incomplete. > bash 2342 dfm01 cwd DIR 8,1 4096 > 353314 /home/dfm01/ltt/lttng-tools > lttng-ses 8494 root cwd DIR 8,1 4096 > 270695 /home/dfm01 > lttng-ses 8494 root rtd DIR 8,1 4096 2 / > lttng-ses 8494 root txt REG 8,1 660624 > 927277 /usr/local/bin/lttng-sessiond > lttng-ses 8494 root mem REG 8,1 92016 > 405535 /lib/i386-linux-gnu/libnsl-2.15.so > lttng-ses 8494 root mem REG 8,1 47040 > 405538 /lib/i386-linux-gnu/libnss_files-2.15.so > lttng-ses 8494 root mem REG 8,1 62218 > 925784 /usr/local/lib/liburcu-bp.so.1.0.0 > lttng-ses 8494 root mem REG 8,1 134344 > 393277 /lib/i386-linux-gnu/ld-2.15.so > lttng-ses 8494 root mem REG 8,1 42652 > 405540 /lib/i386-linux-gnu/libnss_nis-2.15.so > lttng-ses 8494 root mem REG 8,1 1713640 > 405529 /lib/i386-linux-gnu/libc-2.15.so > lttng-ses 8494 root mem REG 8,1 30684 > 405545 /lib/i386-linux-gnu/librt-2.15.so > lttng-ses 8494 root mem REG 8,1 124663 > 405543 /lib/i386-linux-gnu/libpthread-2.15.so > lttng-ses 8494 root mem REG 8,1 30520 > 405536 /lib/i386-linux-gnu/libnss_compat-2.15.so > lttng-ses 8494 root mem REG 8,1 16833 > 917842 /usr/local/lib/liburcu-common.so.1.0.0 > lttng-ses 8494 root mem REG 8,1 572112 > 926835 /usr/local/lib/liblttng-ust-ctl.so.0.0.0 > lttng-ses 8494 root mem REG 8,1 54778 > 925752 /usr/local/lib/liburcu.so.1.0.0 > lttng-ses 8494 root mem REG 8,1 393957 > 921931 /usr/local/lib/liblttng-ctl.so.0.0.0 > lttng-ses 8494 root mem REG 0,18 4096 > 98026 /run/shm/lttng-ust-apps-wait > lttng-ses 8494 root 0u CHR 136,3 0t0 > 6 /dev/pts/3 > lttng-ses 8494 root 1u CHR 136,3 0t0 > 6 /dev/pts/3 > lttng-ses 8494 root 2u CHR 136,3 0t0 > 6 /dev/pts/3 > lttng-ses 8494 root 3r FIFO 0,8 0t0 98007 pipe > lttng-ses 8494 root 4w FIFO 0,8 0t0 98007 pipe > lttng-ses 8494 root 5u unix 0xf53d86c0 0t0 > 98010 /var/run/lttng/kconsumerd/error > lttng-ses 8494 root 6u unix 0xf53d9440 0t0 > 98013 /var/run/lttng/ustconsumerd64/error > lttng-ses 8494 root 7u unix 0xf53d9d40 0t0 > 98016 /var/run/lttng/ustconsumerd32/error > lttng-ses 8494 root 8u unix 0xf53d9b00 0t0 > 98018 /var/run/lttng/client-lttng-sessiond > lttng-ses 8494 root 9u unix 0xf53da1c0 0t0 > 98020 /var/run/lttng/apps-lttng-sessiond > lttng-ses 8494 root 10r FIFO 0,8 0t0 98024 pipe > lttng-ses 8494 root 11w FIFO 0,8 0t0 98024 pipe > lttng-ses 8494 root 12r FIFO 0,8 0t0 98025 pipe > lttng-ses 8494 root 13w FIFO 0,8 0t0 98025 pipe > lttng-ses 8494 root 14u 0000 0,9 0 > 4765 anon_inode > lttng-ses 8494 root 15u 0000 0,9 0 > 4765 anon_inode > lttng-ses 8494 root 16u 0000 0,9 0 > 4765 anon_inode > lttng-ses 8494 root 17u 0000 0,9 0 > 4765 anon_inode > lttng-ses 8494 root 18u unix 0xf53d9680 0t0 > 98027 /var/run/lttng/health.sock > lttng-ses 8494 root 19u 0000 0,9 0 > 4765 anon_inode > sudo 8517 root cwd DIR 8,1 4096 > 353314 /home/dfm01/ltt/lttng-tools > grep 8518 dfm01 cwd DIR 8,1 4096 > 353314 /home/dfm01/ltt/lttng-tools > lsof 8519 root cwd DIR 8,1 4096 > 353314 /home/dfm01/ltt/lttng-tools > lsof 8520 root cwd DIR 8,1 4096 > 353314 /home/dfm01/ltt/lttng-tools > gedit 13840 dfm01 cwd DIR 8,1 4096 > 353314 /home/dfm01/ltt/lttng-tools > > =========================================================================== > Testing as normal user(daemon is run as root, lttng as normnal): > ============================================================================ > dfm01 at ubuntu:~/ltt/lttng-tools/tests$ lttng create > Trace(s) output set to /home/dfm01/lttng-traces/auto-20120823-102111 > Session created with default name auto-20120823-102111 > dfm01 at ubuntu:~/ltt/lttng-tools/tests$ lttng enable-event -u all > UST event all created in channel channel0 > > Interesting, no DEBUG messages whatsoever in daemon window. > > > > =========== Test that fails: ========================== > root at ubuntu:/home/dfm01/ltt/lttng-tools/tests# lttng create > Trace(s) output set to /root/lttng-traces/auto-20120823-100802 > Session created with default name auto-20120823-100802 > > Daemon log: > DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] > DEBUG1: Receiving data from client ... [in thread_manage_clients() at > main.c:3031] > DEBUG1: Nothing recv() from client... continuing [in > thread_manage_clients() at main.c:3035] > DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] > DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] > DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] > DEBUG1: Receiving data from client ... [in thread_manage_clients() at > main.c:3031] > DEBUG1: Processing client command 8 [in process_client_msg() at main.c:2004] > DEBUG1: Waiting for 1 URIs from client ... [in process_client_msg() at > main.c:2505] > DEBUG2: Trying to find session by name auto-20120823-101733 [in > session_find_by_name() at session.c:122] > DEBUG1: Tracing session auto-20120823-101733 created in (null) with ID > 2 by UID 0 GID 0 [in session_create() at session.c:236] > DEBUG2: Trying to find session by name auto-20120823-101733 [in > session_find_by_name() at session.c:122] > DEBUG3: Created hashtable size 4 at 0xb4416780 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG2: Setting trace directory path from URI to > /root/lttng-traces/auto-20120823-101733 [in add_uri_to_consumer() at > cmd.c:395] > DEBUG1: Sending response (size: 16, retcode: Success) [in > thread_manage_clients() at main.c:3082] > DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] > DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] > > > root at ubuntu:/home/dfm01/ltt/lttng-tools/tests# lttng enable-event -u all > Error: Event all: 32-bit UST consumer start failed (channel channel0, > session auto-20120823-100802) > Warning: Some command(s) went wrong > > > Daemon log: > DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] > DEBUG1: Receiving data from client ... [in thread_manage_clients() at > main.c:3031] > DEBUG1: Nothing recv() from client... continuing [in > thread_manage_clients() at main.c:3035] > DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] > DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] > DEBUG1: Wait for client response [in thread_manage_clients() at main.c:2992] > DEBUG1: Receiving data from client ... [in thread_manage_clients() at > main.c:3031] > DEBUG1: Processing client command 6 [in process_client_msg() at main.c:2004] > DEBUG1: Getting session auto-20120823-101733 by name [in > process_client_msg() at main.c:2074] > DEBUG2: Trying to find session by name auto-20120823-101733 [in > session_find_by_name() at session.c:122] > DEBUG1: Creating UST session [in create_ust_session() at main.c:1891] > DEBUG3: Created hashtable size 4 at 0xb44231b8 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG3: Created hashtable size 4 at 0xb44232d8 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG3: Created hashtable size 4 at 0xb44233f8 of type 0 [in > lttng_ht_new() at hashtable.c:96] > DEBUG3: Created hashtable size 4 at 0xb4428560 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG2: UST trace session create successful [in > trace_ust_create_session() at trace-ust.c:143] > DEBUG3: Copying tracing session consumer output in UST session [in > copy_session_consumer() at main.c:1840] > DEBUG3: Created hashtable size 4 at 0xb442d6c8 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG3: Created hashtable size 4 at 0xb442d7e8 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG2: Consumer subdir set to /auto-20120823-101733 [in > consumer_set_subdir() at consumer.c:689] > DEBUG3: Copy session consumer subdir /ust [in copy_session_consumer() > at main.c:1861] > DEBUG2: Trace UST channel channel0 not found by name [in > trace_ust_find_channel_by_name() at trace-ust.c:52] > DEBUG1: Enabling channel channel0 for session auto-20120823-101733 [in > cmd_enable_channel() at cmd.c:743] > DEBUG2: Trace UST channel channel0 not found by name [in > trace_ust_find_channel_by_name() at trace-ust.c:52] > DEBUG3: Created hashtable size 4 at 0xb442edd0 of type 0 [in > lttng_ht_new() at hashtable.c:96] > DEBUG3: Created hashtable size 4 at 0xb442eef0 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG2: Trace UST channel channel0 created [in > trace_ust_create_channel() at trace-ust.c:205] > DEBUG2: Channel channel0 being created in UST global domain [in > channel_ust_create() at channel.c:267] > DEBUG2: UST app adding channel channel0 to global domain for session > id 2 [in ust_app_create_channel_glb() at ust-app.c:1984] > DEBUG2: Channel channel0 created successfully [in channel_ust_create() > at channel.c:292] > DEBUG2: Trace UST channel channel0 found by name [in > trace_ust_find_channel_by_name() at trace-ust.c:47] > DEBUG2: Trace UST event NOT found by name all [in > trace_ust_find_event_by_name() at trace-ust.c:79] > DEBUG3: Created hashtable size 4 at 0xb442d908 of type 1 [in > lttng_ht_new() at hashtable.c:96] > DEBUG2: Trace UST event all, loglevel (0,-1) created [in > trace_ust_create_event() at trace-ust.c:284] > DEBUG1: UST app creating event all for all apps for session id 2 [in > ust_app_create_event_glb() at ust-app.c:2108] > DEBUG1: Event UST all created in channel channel0 [in > event_ust_enable_tracepoint() at event.c:486] > DEBUG1: Sending response (size: 16, retcode: Success) [in > thread_manage_clients() at main.c:3082] > DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:465] > DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:2950] > > -- > Henrik Hautakoski > henrik at fiktivkod.org > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From dgoulet at efficios.com Fri Aug 24 15:26:51 2012 From: dgoulet at efficios.com (David Goulet) Date: Fri, 24 Aug 2012 15:26:51 -0400 Subject: [lttng-dev] [PATCH v2 lttng-tools] Filter: Notify the user when a filter is already enabled on event(s) In-Reply-To: <1345773912-30858-2-git-send-email-christian.babeux@efficios.com> References: <1345760474-29248-1-git-send-email-christian.babeux@efficios.com> <1345773912-30858-1-git-send-email-christian.babeux@efficios.com> <1345773912-30858-2-git-send-email-christian.babeux@efficios.com> Message-ID: <5037D57B.7050203@efficios.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Merged! Christian Babeux: > When using the enable-event command in conjunction with the > --filter option, a user can specify a filter expression to refine > the trace output. > > As stated in the lttng(1) man page, only the first activation of a > filter on an event will work. Subsequent activation of any filter > expression on the same event will fail. > > e.g: > >> lttng enable-event app:tp -s session -u --filter 'somefield > >> 42' > > # Case: invalid filter expression >> lttng enable-event app:tp -s session -u --filter 'invalid >> expression' > Error: Error setting filter Warning: Some command(s) went wrong > >> ... > > # Case: filter already enabled for event app:tp >> lttng enable-event app:tp -s session -u --filter 'someotherfield >> < 42' > Error: Error setting filter Warning: Some command(s) went wrong > > This commit differentiate the case where a filter was already set > for the specified event from the generic 'Error setting filter' > error message. > > e.g: > >> lttng enable-event app:tp -s session -u --filter 'somefield > >> 42' > > # Case: invalid filter expression >> lttng enable-event app:tp -s session -u --filter 'invalid >> expression' > Error: Error setting filter Warning: Some command(s) went wrong > >> ... > > # Case: filter already enabled for event app:tp >> lttng enable-event app:tp -s session -u --filter 'someotherfield >> < 42' > Error: Filter on event app:tp is already enabled (channel 0, > session session) Warning: Some command(s) went wrong > > Signed-off-by: Christian Babeux > --- src/bin/lttng/commands/enable_events.c | 22 > ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 > deletions(-) > > diff --git a/src/bin/lttng/commands/enable_events.c > b/src/bin/lttng/commands/enable_events.c index a2c8a68..5a81fdb > 100644 --- a/src/bin/lttng/commands/enable_events.c +++ > b/src/bin/lttng/commands/enable_events.c @@ -384,7 +384,16 @@ > static int enable_events(char *session_name) ret = > lttng_set_event_filter(handle, ev.name, channel_name, opt_filter); > if (ret < 0) { - ERR("Error setting filter"); + switch (-ret) > { + case LTTCOMM_FILTER_EXIST: + ERR("Filter on events is > already enabled (channel %s, session %s)", + channel_name, > session_name); + break; + default: + ERR("Error setting > filter"); + break; + } + ret = -1; goto error; } @@ -554,7 > +563,16 @@ static int enable_events(char *session_name) ret = > lttng_set_event_filter(handle, ev.name, channel_name, opt_filter); > if (ret < 0) { - ERR("Error setting filter"); + switch (-ret) > { + case LTTCOMM_FILTER_EXIST: + ERR("Filter on event %s is > already enabled (channel %s, session %s)", + event_name, > channel_name, session_name); + break; + default: + > ERR("Error setting filter"); + break; + } + ret = -1; goto > error; } -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJQN9V4AAoJEELoaioR9I028ZgH/2MnMkgkidkFmv3ZZdBOCI58 NZzrdJzahoHKs7aKJ/xINbIiQqhMuCbucu1MUyStDp1A9ECG5b+Vy8bqG/Lwr+V5 pLPlZ75QexiaeqpwCOE5us8hJzgdoU32gADbFfCTIzv6Idzngadih7DiwYdXx4PD ha93R3tGMuk13UQhdhwsg0zq/yVudBbWa75b+ZmjZ9wMcfxtdokaADDJXOjGeQ0H WtsckN9W4vinLPNwXaCHb4CdQ6gFC0MazvOZ9d9kFZCnOJk8Raed8WFeRGc5vYB8 cn5Q3nJB4MPXhOkHJs5aC5eYgRbQkRwkO0i1MtFt1Wxw4xESd7JfYMj20GPWsOM= =l1J7 -----END PGP SIGNATURE----- From mathieu.desnoyers at efficios.com Mon Aug 27 08:21:11 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Mon, 27 Aug 2012 08:21:11 -0400 Subject: [lttng-dev] [RELEASE] Userspace RCU 0.7.4 Message-ID: <20120827122111.GA1935@Krystal> liburcu is a LGPLv2.1 userspace RCU (read-copy-update) library. This data synchronization library provides read-side access which scales linearly with the number of cores. It does so by allowing multiples copies of a given data structure to live at the same time, and by monitoring the data structure accesses to detect grace periods after which memory reclamation is possible. liburcu-cds provides efficient data structures based on RCU and lock-free algorithms. Those structures include hash tables, queues, stacks, and doubly-linked lists. The main change in this version is the added support for MIPS, contributed by Ralf Baechle. Changelog: 2012-08-27 Userspace RCU 0.7.4 * rculfhash API documentation: document destroy RCU read-lock constraint * Fix: rculfhash should be offline while waiting for resize to complete * Add missing entry to gitignore * urcu: move busy-wait code and name it ___cds_wfq_node_sync_next() * urcu: fix compat_futex_noasync() * urcu: add hint to DEFINE_URCU_TLS() for compound types * Fix: CAA_BUILD_BUG_ON should refer to CAA_BUILD_BUG_ON_ZERO * Add MIPS support * Compatibility: remove bash-ismsm from test scripts * Fix inappropriate lib behavior: don't call exit() * Fix: re-enable compatibility with autoconf < 2.64 * Fix c99 compatibility: use __asm__ and __volatile__ in public headers * Fix c99 compatibility: use __typeof__ instead of typeof in public headers * warning fix: tests urcutorture for NetBSD 5 -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Mon Aug 27 08:27:10 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Mon, 27 Aug 2012 08:27:10 -0400 Subject: [lttng-dev] [RELEASE] LTTng-UST 2.0.5 Message-ID: <20120827122710.GA2172@Krystal> LTTng-UST, the Linux Trace Toolkit Next Generation Userspace Tracer, is port of the low-overhead tracing capabilities of the LTTng kernel tracer to user-space. The library "liblttng-ust" enables tracing of applications and libraries. Changelog: 2012-08-27 lttng-ust 2.0.5 * Fix: threads should be created in DETACHED state * Fix UST SIGPIPE handling * Fix: Libtool fails to find dependent libraries when cross-compiling lttng-ust * Fix: remove unused texinfo dep from configure.ac * Fix C99 strict compatibility: don't use void * for function pointers * Fix c99 compatibility: tp_rcu_dereference_bp() should not use braced-groups within expressions * Fix c99 compatibility: use __typeof__ instead of typeof in public headers Project website: http://lttng.org Download link: http://lttng.org/download (please refer to the README file for installation instructions) -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Mon Aug 27 08:35:50 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Mon, 27 Aug 2012 08:35:50 -0400 Subject: [lttng-dev] [RELEASE] LTTng modules 2.0.5 Message-ID: <20120827123550.GA2311@Krystal> The LTTng modules provide Linux kernel tracing capability to the LTTng 2.0 tracer toolset. Changelog: 2012-08-27 LTTng modules 2.0.5 * Fix: statedump: disable vm maps enumeration * Fix: ensure userspace accesses are done with _inatomic * Fix: vppid context should test for current nsproxy Project website: http://lttng.org Download link: http://lttng.org/download (please refer to the README files for installation instructions and lttng-tools doc/quickstart.txt for usage information) -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Mon Aug 27 08:52:22 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Mon, 27 Aug 2012 08:52:22 -0400 Subject: [lttng-dev] [RELEASE] Babeltrace 1.0.0-rc5 Message-ID: <20120827125222.GA2619@Krystal> The Babeltrace project provides trace read and write libraries, as well as a trace converter. Plugins can be created for any trace format to allow its conversion to/from another trace format. The main format expected to be converted to/from is the Common Trace Format (CTF). The default input format of the "babeltrace" command is CTF, and its default output format is a human-readable text log. The "babeltrace-log" command converts from a text log to a CTF trace. Most of the changes in rc5 target libbabeltrace, for which the only known users outside the babeltrace tree at this stage are lttngtop and lttv (both under development). There is one noteworthy change to the babeltrace command line: it now allows specifying many input trace paths. Also, the default trace text printout will show the hostname, process name and vpid if those are available. (currently, hostname is only exported by master branches of lttng-modules and lttng-ust, not by stable-2.0). Along with the "--clock-force-correlate" command line option, we can now use babeltrace to show traces gathered on different machines at the same time with lttng's monotonic clock source, which approximate the offset from epoch. Changelog: 2012-08-27 Babeltrace 1.0.0-rc5 * Change default printout to add host, process names and vpid * Add support for trace:hostname field * Fix: allow specifying more than one input trace path * Fix: make warnings (partial errors) visible * Fix: --clock-force-correlate to handle trace collections gathered from v * Documentation: update API doc with enum functions * Fix: API: remove unsupported BT_SEEK_END from API * API documentation * Cleanup: shut up gcc uninitialized var warning * Fix: support large files on 32-bit systems * Fix: remove unused fts.h include * Fix: add missing enum support to API * Fix: handle clock offset with frequency different from 1GHz * Cleanup: update ifdef wrapper name * Fix: clarify bt_ctf_get_field_list * Fix trace-collection.h: No such file or directory that build code with l * Fix: check return value of bt_context_create * Fix: ensure mmap_base_offset is zeroed on initialization * Fix: Reswitch to FTW for add_traces_recursive * Fix: don't free unallocated index * Fix: don't close the metadata FD if a FP is passed * Add BT_SEEK_LAST type to bt_iter_pos * Fix: iterator.c BT_SEEK_RESTORE: check return value * Fix: complete error handling of babeltrace API * cleanup: protected -> hidden: cleanup symbol table * Fix: add mmap_base_offset to ctf_stream_pos * Fix: assign the current clock for mmap traces * Fix: libbabeltrace add missing static declaration * Fix: safety checks for opening mmap traces * Remove trace-collection.h from include_headers * Fix: protect visibility of ctf-parser functions * Fix: correct name of bt_ctf_field_get_error in comments and typo in man * Fix: wrong type in bt_ctf_get_uint64/int64 * API cleanup name get_timestamp and get_cycles * fix comment struct bt_saved_pos * Fix: Add missing clock-types.h * Get rid of clock-raw and use real clock * Cleanup (messages): Make the wording of the signedness warning clearer * Fix: error path if heap_init fails * Fix: Remove obsolete bt_iter_seek function * Make the signedness warning useful with the field name * Fix: Restore heap for SEEK_BEGIN * Fix: check if handle is valid * Fix: iterator set_pos * Fix: get rid of consumed flag * Fix: add missing heap_copy * Fix: babeltrace assert() triggered by directories within trace * Several fixes for bt_iter_pos related functions * Fix iterator: various fixes * Fix: remove duplicate yydebug var * Fix babeltrace iterator lib: seek at time 0 Project website: http://www.efficios.com/babeltrace Download link: http://www.efficios.com/files/babeltrace/ CTF specification: http://www.efficios.com/ctf -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From christian.babeux at efficios.com Mon Aug 27 14:48:18 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Mon, 27 Aug 2012 14:48:18 -0400 Subject: [lttng-dev] [PATCH lttng-tools 0/3] Filter allocation and large bytecode fixes Message-ID: <1346093301-25274-1-git-send-email-christian.babeux@efficios.com> Hi all, While testing the new filtering feature, some issues were discovered when generating large bytecode (more than 32768 bytes). The first patch fix an issue with the buffer size not being a power of 2, thus leading to odd-looking alloc buffer size. The second patch is a typo fix. The third patch fix an issue when generating a bytecode longer than 32768 bytes. The alloc_len of the underlying bytecode buffer was artificially limiting the bytecode size to 32768 because its length was overflowing the range of 16-bits values. Thanks, Christian Babeux (3): Fix: Filter bytecode alloc buffer size must be a power of 2 Fix: Typo in LTTNG_FILTER_MAX_LEN Fix: Generation of bytecode longer than 32768 bytes fails src/bin/lttng-sessiond/main.c | 2 +- src/common/sessiond-comm/sessiond-comm.h | 2 +- src/lib/lttng-ctl/filter/filter-bytecode.h | 2 +- .../filter/filter-visitor-generate-bytecode.c | 46 ++++++++++++++++++++-- 4 files changed, 46 insertions(+), 6 deletions(-) -- 1.7.11.4 From christian.babeux at efficios.com Mon Aug 27 14:48:19 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Mon, 27 Aug 2012 14:48:19 -0400 Subject: [lttng-dev] [PATCH lttng-tools 1/3] Fix: Filter bytecode alloc buffer size must be a power of 2 In-Reply-To: <1346093301-25274-1-git-send-email-christian.babeux@efficios.com> References: <1346093301-25274-1-git-send-email-christian.babeux@efficios.com> Message-ID: <1346093301-25274-2-git-send-email-christian.babeux@efficios.com> The current allocation policy for the filter bytecode buffer is to double the size each time the underlying buffer can no longer contain the entire bytecode plus padding. In some cases, the initial allocation length is not a multiple of 2, thus possibly leading to odd-looking allocation size each time the buffer size is doubled. Signed-off-by: Christian Babeux --- .../filter/filter-visitor-generate-bytecode.c | 41 +++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c b/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c index 36d35c5..71da21c 100644 --- a/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c +++ b/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c @@ -38,6 +38,45 @@ static int recursive_visit_gen_bytecode(struct filter_parser_ctx *ctx, struct ir_op *node); +static inline int fls(unsigned int x) +{ + int r = 32; + + if (!x) + return 0; + if (!(x & 0xFFFF0000U)) { + x <<= 16; + r -= 16; + } + if (!(x & 0xFF000000U)) { + x <<= 8; + r -= 8; + } + if (!(x & 0xF0000000U)) { + x <<= 4; + r -= 4; + } + if (!(x & 0xC0000000U)) { + x <<= 2; + r -= 2; + } + if (!(x & 0x80000000U)) { + x <<= 1; + r -= 1; + } + return r; +} + +static inline int get_count_order(unsigned int count) +{ + int order; + + order = fls(count) - 1; + if (count & (count - 1)) + order++; + return order; +} + static int bytecode_init(struct lttng_filter_bytecode_alloc **fb) { @@ -58,7 +97,7 @@ int32_t bytecode_reserve(struct lttng_filter_bytecode_alloc **fb, uint32_t align if ((*fb)->b.len + padding + len > (*fb)->alloc_len) { uint32_t new_len = - max_t(uint32_t, (*fb)->b.len + padding + len, + max_t(uint32_t, 1U << get_count_order((*fb)->b.len + padding + len), (*fb)->alloc_len << 1); uint32_t old_len = (*fb)->alloc_len; -- 1.7.11.4 From christian.babeux at efficios.com Mon Aug 27 14:48:20 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Mon, 27 Aug 2012 14:48:20 -0400 Subject: [lttng-dev] [PATCH lttng-tools 2/3] Fix: Typo in LTTNG_FILTER_MAX_LEN In-Reply-To: <1346093301-25274-1-git-send-email-christian.babeux@efficios.com> References: <1346093301-25274-1-git-send-email-christian.babeux@efficios.com> Message-ID: <1346093301-25274-3-git-send-email-christian.babeux@efficios.com> Signed-off-by: Christian Babeux --- src/bin/lttng-sessiond/main.c | 2 +- src/common/sessiond-comm/sessiond-comm.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/bin/lttng-sessiond/main.c b/src/bin/lttng-sessiond/main.c index 318da74..730ac65 100644 --- a/src/bin/lttng-sessiond/main.c +++ b/src/bin/lttng-sessiond/main.c @@ -2659,7 +2659,7 @@ skip_domain: { struct lttng_filter_bytecode *bytecode; - if (cmd_ctx->lsm->u.filter.bytecode_len > 65336) { + if (cmd_ctx->lsm->u.filter.bytecode_len > LTTNG_FILTER_MAX_LEN) { ret = LTTNG_ERR_FILTER_INVAL; goto error; } diff --git a/src/common/sessiond-comm/sessiond-comm.h b/src/common/sessiond-comm/sessiond-comm.h index 32ce384..ff22875 100644 --- a/src/common/sessiond-comm/sessiond-comm.h +++ b/src/common/sessiond-comm/sessiond-comm.h @@ -208,7 +208,7 @@ struct lttcomm_session_msg { } u; }; -#define LTTNG_FILTER_MAX_LEN 65336 +#define LTTNG_FILTER_MAX_LEN 65535 /* * Filter bytecode data. The reloc table is located at the end of the -- 1.7.11.4 From christian.babeux at efficios.com Mon Aug 27 14:48:21 2012 From: christian.babeux at efficios.com (Christian Babeux) Date: Mon, 27 Aug 2012 14:48:21 -0400 Subject: [lttng-dev] [PATCH lttng-tools 3/3] Fix: Generation of bytecode longer than 32768 bytes fails In-Reply-To: <1346093301-25274-1-git-send-email-christian.babeux@efficios.com> References: <1346093301-25274-1-git-send-email-christian.babeux@efficios.com> Message-ID: <1346093301-25274-4-git-send-email-christian.babeux@efficios.com> The bytecode buffer length field is currently limited to a uint16_t. A larger buffer, lttng_filter_bytecode_alloc, is the underlying storage for the bytecode. The current allocation policy dictate that the alloc buffer size must be doubled everytime the bytecode size plus padding exceeds its capacity. A problem arise when generating bytecode larger than 32768 bytes. e.g.: Legend * required_len: new bytecode len * old_len: current alloc_len * new_len: new alloc_len src/bin/lttng/lttng enable-event event:bla -s foo -u --filter "`perl -e 'print "intfield" . " && 1" x2730'`" UST event ust_tests_hello:tptest created in channel channel0 [debug liblttng-ctl] Generating IR... [debug liblttng-ctl] done [debug liblttng-ctl] Validating IR... [debug liblttng-ctl] done [debug liblttng-ctl] Generating bytecode... required_len = 11, old_len = 4, new_len = 16 required_len = 7, old_len = 4, new_len = 8 required_len = 16, old_len = 8, new_len = 16 required_len = 19, old_len = 16, new_len = 32 required_len = 40, old_len = 32, new_len = 64 required_len = 67, old_len = 64, new_len = 128 required_len = 136, old_len = 128, new_len = 256 required_len = 259, old_len = 256, new_len = 512 required_len = 520, old_len = 512, new_len = 1024 required_len = 1027, old_len = 1024, new_len = 2048 required_len = 2056, old_len = 2048, new_len = 4096 required_len = 4099, old_len = 4096, new_len = 8192 required_len = 8200, old_len = 8192, new_len = 16384 required_len = 16387, old_len = 16384, new_len = 32768 required_len = 32776, old_len = 32768, new_len = 65536 <-- Overflow 16-bits Generate bytecode error Error: Error setting filter The last new_len exceed the range of 16-bits values. In order to support the largest bytecode length (65535), the underlying alloc buffer len must be able to store more than 65535. Fix this by using a uint32_t for alloc_len. Also, add a check to ensure that a bytecode longer than LTTNG_FILTER_MAX_LEN (65535) bytes can't be generated. Signed-off-by: Christian Babeux --- src/lib/lttng-ctl/filter/filter-bytecode.h | 2 +- src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/src/lib/lttng-ctl/filter/filter-bytecode.h b/src/lib/lttng-ctl/filter/filter-bytecode.h index 5d2559d..d364ee2 100644 --- a/src/lib/lttng-ctl/filter/filter-bytecode.h +++ b/src/lib/lttng-ctl/filter/filter-bytecode.h @@ -176,7 +176,7 @@ struct return_op { } __attribute__((packed)); struct lttng_filter_bytecode_alloc { - uint16_t alloc_len; + uint32_t alloc_len; struct lttng_filter_bytecode b; }; diff --git a/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c b/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c index 71da21c..98f8375 100644 --- a/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c +++ b/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c @@ -95,14 +95,15 @@ int32_t bytecode_reserve(struct lttng_filter_bytecode_alloc **fb, uint32_t align int32_t ret; uint32_t padding = offset_align((*fb)->b.len, align); + if ((*fb)->b.len + padding + len > LTTNG_FILTER_MAX_LEN) + return -EINVAL; + if ((*fb)->b.len + padding + len > (*fb)->alloc_len) { uint32_t new_len = max_t(uint32_t, 1U << get_count_order((*fb)->b.len + padding + len), (*fb)->alloc_len << 1); uint32_t old_len = (*fb)->alloc_len; - if (new_len > 0xFFFF) - return -EINVAL; *fb = realloc(*fb, sizeof(struct lttng_filter_bytecode_alloc) + new_len); if (!*fb) return -ENOMEM; -- 1.7.11.4 From mathieu.desnoyers at efficios.com Mon Aug 27 15:15:43 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Mon, 27 Aug 2012 15:15:43 -0400 Subject: [lttng-dev] [PATCH lttng-tools 2/3] Fix: Typo in LTTNG_FILTER_MAX_LEN In-Reply-To: <1346093301-25274-3-git-send-email-christian.babeux@efficios.com> References: <1346093301-25274-1-git-send-email-christian.babeux@efficios.com> <1346093301-25274-3-git-send-email-christian.babeux@efficios.com> Message-ID: <20120827191542.GA7122@Krystal> * Christian Babeux (christian.babeux at efficios.com) wrote: > > Signed-off-by: Christian Babeux > --- > src/bin/lttng-sessiond/main.c | 2 +- > src/common/sessiond-comm/sessiond-comm.h | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/src/bin/lttng-sessiond/main.c b/src/bin/lttng-sessiond/main.c > index 318da74..730ac65 100644 > --- a/src/bin/lttng-sessiond/main.c > +++ b/src/bin/lttng-sessiond/main.c > @@ -2659,7 +2659,7 @@ skip_domain: > { > struct lttng_filter_bytecode *bytecode; > > - if (cmd_ctx->lsm->u.filter.bytecode_len > 65336) { > + if (cmd_ctx->lsm->u.filter.bytecode_len > LTTNG_FILTER_MAX_LEN) { > ret = LTTNG_ERR_FILTER_INVAL; > goto error; > } > diff --git a/src/common/sessiond-comm/sessiond-comm.h b/src/common/sessiond-comm/sessiond-comm.h > index 32ce384..ff22875 100644 > --- a/src/common/sessiond-comm/sessiond-comm.h > +++ b/src/common/sessiond-comm/sessiond-comm.h > @@ -208,7 +208,7 @@ struct lttcomm_session_msg { > } u; > }; > > -#define LTTNG_FILTER_MAX_LEN 65336 > +#define LTTNG_FILTER_MAX_LEN 65535 This should be 65536. Thanks, Mathieu > > /* > * Filter bytecode data. The reloc table is located at the end of the > -- > 1.7.11.4 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Mon Aug 27 15:16:58 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Mon, 27 Aug 2012 15:16:58 -0400 Subject: [lttng-dev] [PATCH lttng-tools 1/3] Fix: Filter bytecode alloc buffer size must be a power of 2 In-Reply-To: <1346093301-25274-2-git-send-email-christian.babeux@efficios.com> References: <1346093301-25274-1-git-send-email-christian.babeux@efficios.com> <1346093301-25274-2-git-send-email-christian.babeux@efficios.com> Message-ID: <20120827191658.GB7122@Krystal> * Christian Babeux (christian.babeux at efficios.com) wrote: > The current allocation policy for the filter bytecode buffer is to double > the size each time the underlying buffer can no longer contain the entire > bytecode plus padding. > > In some cases, the initial allocation length is not a multiple of 2, thus > possibly leading to odd-looking allocation size each time the buffer size > is doubled. > > Signed-off-by: Christian Babeux Acked-by: Mathieu Desnoyers > --- > .../filter/filter-visitor-generate-bytecode.c | 41 +++++++++++++++++++++- > 1 file changed, 40 insertions(+), 1 deletion(-) > > diff --git a/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c b/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c > index 36d35c5..71da21c 100644 > --- a/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c > +++ b/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c > @@ -38,6 +38,45 @@ static > int recursive_visit_gen_bytecode(struct filter_parser_ctx *ctx, > struct ir_op *node); > > +static inline int fls(unsigned int x) > +{ > + int r = 32; > + > + if (!x) > + return 0; > + if (!(x & 0xFFFF0000U)) { > + x <<= 16; > + r -= 16; > + } > + if (!(x & 0xFF000000U)) { > + x <<= 8; > + r -= 8; > + } > + if (!(x & 0xF0000000U)) { > + x <<= 4; > + r -= 4; > + } > + if (!(x & 0xC0000000U)) { > + x <<= 2; > + r -= 2; > + } > + if (!(x & 0x80000000U)) { > + x <<= 1; > + r -= 1; > + } > + return r; > +} > + > +static inline int get_count_order(unsigned int count) > +{ > + int order; > + > + order = fls(count) - 1; > + if (count & (count - 1)) > + order++; > + return order; > +} > + > static > int bytecode_init(struct lttng_filter_bytecode_alloc **fb) > { > @@ -58,7 +97,7 @@ int32_t bytecode_reserve(struct lttng_filter_bytecode_alloc **fb, uint32_t align > > if ((*fb)->b.len + padding + len > (*fb)->alloc_len) { > uint32_t new_len = > - max_t(uint32_t, (*fb)->b.len + padding + len, > + max_t(uint32_t, 1U << get_count_order((*fb)->b.len + padding + len), > (*fb)->alloc_len << 1); > uint32_t old_len = (*fb)->alloc_len; > > -- > 1.7.11.4 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From mathieu.desnoyers at efficios.com Mon Aug 27 15:17:35 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Mon, 27 Aug 2012 15:17:35 -0400 Subject: [lttng-dev] [PATCH lttng-tools 3/3] Fix: Generation of bytecode longer than 32768 bytes fails In-Reply-To: <1346093301-25274-4-git-send-email-christian.babeux@efficios.com> References: <1346093301-25274-1-git-send-email-christian.babeux@efficios.com> <1346093301-25274-4-git-send-email-christian.babeux@efficios.com> Message-ID: <20120827191735.GC7122@Krystal> * Christian Babeux (christian.babeux at efficios.com) wrote: > The bytecode buffer length field is currently limited to a uint16_t. > A larger buffer, lttng_filter_bytecode_alloc, is the underlying storage > for the bytecode. > > The current allocation policy dictate that the alloc buffer size must be > doubled everytime the bytecode size plus padding exceeds its capacity. > > A problem arise when generating bytecode larger than 32768 bytes. > > e.g.: > > Legend > * required_len: new bytecode len > * old_len: current alloc_len > * new_len: new alloc_len > > src/bin/lttng/lttng enable-event event:bla -s foo -u --filter "`perl -e 'print "intfield" . " && 1" x2730'`" > UST event ust_tests_hello:tptest created in channel channel0 > [debug liblttng-ctl] Generating IR... [debug liblttng-ctl] done > [debug liblttng-ctl] Validating IR... [debug liblttng-ctl] done > [debug liblttng-ctl] Generating bytecode... required_len = 11, old_len = 4, new_len = 16 > required_len = 7, old_len = 4, new_len = 8 > required_len = 16, old_len = 8, new_len = 16 > required_len = 19, old_len = 16, new_len = 32 > required_len = 40, old_len = 32, new_len = 64 > required_len = 67, old_len = 64, new_len = 128 > required_len = 136, old_len = 128, new_len = 256 > required_len = 259, old_len = 256, new_len = 512 > required_len = 520, old_len = 512, new_len = 1024 > required_len = 1027, old_len = 1024, new_len = 2048 > required_len = 2056, old_len = 2048, new_len = 4096 > required_len = 4099, old_len = 4096, new_len = 8192 > required_len = 8200, old_len = 8192, new_len = 16384 > required_len = 16387, old_len = 16384, new_len = 32768 > required_len = 32776, old_len = 32768, new_len = 65536 <-- Overflow 16-bits > Generate bytecode error > Error: Error setting filter > > The last new_len exceed the range of 16-bits values. In order to support > the largest bytecode length (65535), the underlying alloc buffer len must > be able to store more than 65535. Fix this by using a uint32_t for alloc_len. > > Also, add a check to ensure that a bytecode longer than LTTNG_FILTER_MAX_LEN > (65535) bytes can't be generated. > > Signed-off-by: Christian Babeux Acked-by: Mathieu Desnoyers > --- > src/lib/lttng-ctl/filter/filter-bytecode.h | 2 +- > src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c | 5 +++-- > 2 files changed, 4 insertions(+), 3 deletions(-) > > diff --git a/src/lib/lttng-ctl/filter/filter-bytecode.h b/src/lib/lttng-ctl/filter/filter-bytecode.h > index 5d2559d..d364ee2 100644 > --- a/src/lib/lttng-ctl/filter/filter-bytecode.h > +++ b/src/lib/lttng-ctl/filter/filter-bytecode.h > @@ -176,7 +176,7 @@ struct return_op { > } __attribute__((packed)); > > struct lttng_filter_bytecode_alloc { > - uint16_t alloc_len; > + uint32_t alloc_len; > struct lttng_filter_bytecode b; > }; > > diff --git a/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c b/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c > index 71da21c..98f8375 100644 > --- a/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c > +++ b/src/lib/lttng-ctl/filter/filter-visitor-generate-bytecode.c > @@ -95,14 +95,15 @@ int32_t bytecode_reserve(struct lttng_filter_bytecode_alloc **fb, uint32_t align > int32_t ret; > uint32_t padding = offset_align((*fb)->b.len, align); > > + if ((*fb)->b.len + padding + len > LTTNG_FILTER_MAX_LEN) > + return -EINVAL; > + > if ((*fb)->b.len + padding + len > (*fb)->alloc_len) { > uint32_t new_len = > max_t(uint32_t, 1U << get_count_order((*fb)->b.len + padding + len), > (*fb)->alloc_len << 1); > uint32_t old_len = (*fb)->alloc_len; > > - if (new_len > 0xFFFF) > - return -EINVAL; > *fb = realloc(*fb, sizeof(struct lttng_filter_bytecode_alloc) + new_len); > if (!*fb) > return -ENOMEM; > -- > 1.7.11.4 > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From salman.rafiq at esk.fraunhofer.de Tue Aug 28 06:44:55 2012 From: salman.rafiq at esk.fraunhofer.de (Salman Rafiq) Date: Tue, 28 Aug 2012 12:44:55 +0200 Subject: [lttng-dev] kernel Tracer Clock Message-ID: Hi Mathieu, By looking into traces generated using kernel tracer, I see the frequency in metadata file to be default. I would like to know if there is a way to change default clock frequency (1e9 Hz) for the tracer to current Core clock frequency? using lttng command line tool or any other way? Thanks. Best Regards, Salman -- Salman Rafiq Industrial Communication Fraunhofer-Einrichtung f?r Systeme der Kommunikationstechnik ESK Hansastra?e 32 | 80686 M?nchen Telefon, Fax: +49 89 547088-356 | +49 89 547088-66-356 E-Mail: salman.rafiq at esk.fraunhofer.de Internet: http://www.esk.fraunhofer.de http://www.facebook.com/FraunhoferESK http://www.twitter.com/FraunhoferESK -------------- next part -------------- An HTML attachment was scrubbed... URL: From mathieu.desnoyers at efficios.com Tue Aug 28 07:52:01 2012 From: mathieu.desnoyers at efficios.com (Mathieu Desnoyers) Date: Tue, 28 Aug 2012 07:52:01 -0400 Subject: [lttng-dev] kernel Tracer Clock In-Reply-To: References: Message-ID: <20120828115201.GB23818@Krystal> * Salman Rafiq (salman.rafiq at esk.fraunhofer.de) wrote: > Hi Mathieu, > > By looking into traces generated using kernel tracer, I see the > frequency in metadata file to be default. > I would like to know if there is a way to change default clock > frequency (1e9 Hz) for the tracer to current Core clock frequency? > using lttng command line tool or any other way? Hi Salman, You just have to modifiy lttng to implement your own trace clock, and use that instead of the monotonic clock. Then, you must, for your architecture, override the clock {} metadata description generated by lttng, ideally by providing an arch-specific implementation in a header added to lttng-modules (and moving a #define of the monotonic clock description to a generic header). Thanks, Mathieu > > Thanks. > > Best Regards, > Salman > > -- > Salman Rafiq > Industrial Communication > Fraunhofer-Einrichtung f?r Systeme der Kommunikationstechnik ESK > > Hansastra?e 32 | 80686 M?nchen > Telefon, Fax: +49 89 547088-356 | +49 89 547088-66-356 > E-Mail: salman.rafiq at esk.fraunhofer.de > > Internet: > http://www.esk.fraunhofer.de > http://www.facebook.com/FraunhoferESK > http://www.twitter.com/FraunhoferESK > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com From gerlando.falauto at keymile.com Tue Aug 28 09:00:15 2012 From: gerlando.falauto at keymile.com (Gerlando Falauto) Date: Tue, 28 Aug 2012 15:00:15 +0200 Subject: [lttng-dev] Timestamps in babeltrace In-Reply-To: <4EEB1A44.2020500@keymile.com> References: <4EEB1A44.2020500@keymile.com> Message-ID: <503CC0DF.7030607@keymile.com> Hi, picking up again from my own message... I'm try this again after these few months, and I still get some similar behavior. Again, it's a 32-bit PowerPC with 4ms system tick (CONFIG_HZ=250). The machine has neither NTP nor internal clock, therefore I set a given timestamp before starting the test. Traces were generated using the latest master branch of the three repositories (lttng-modules, lttng-tools, userspace-rcu). Running the following script on the target: date -s "2012-08-28 09:00:00" lttng create trace lttng enable-channel -k mychannel lttng enable-event sched_switch,sched_wakeup -k -c mychannel lttng start echo waiting 10 seconds sleep 10 lttng stop lttng destroy trace date (for which I get the following output): Tue Aug 28 09:00:00 UTC 2012 Session trace created. Traces will be written in /root/lttng-traces/trace-20120828-090000 Kernel channel mychannel enabled for session trace kernel event sched_switch created in channel mychannel kernel event sched_wakeup created in channel mychannel Tracing started for session trace waiting 10 seconds Tracing stopped for session trace Session trace destroyed Tue Aug 28 09:00:10 UTC 2012 I get the following trace: babeltrace --clock-date --clock-gmt /path/to/root/lttng-traces/trace-20120828-090000/ [2012-08-28 09:00:00.348477307] (+?.?????????) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "mng_clients", prev_tid = 452, prev_prio = 20, prev_state = 0, next_comm = "cons_recv_fds", next_tid = 461, next_prio = 20 } [2012-08-28 09:00:00.350913974] (+0.002436667) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = 20, prev_state = 0, next_comm = "mng_clients", next_tid = 452, next_prio = 20 } [2012-08-28 09:00:00.352444701] (+0.001530727) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "lttng", tid = 492, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.352584944] (+0.000140243) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "klogd", tid = 385, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.352691186] (+0.000106242) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.352853368] (+0.000162182) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "mng_clients", prev_tid = 452, prev_prio = 20, prev_state = 0, next_comm = "klogd", next_tid = 385, next_prio = 20 } [2012-08-28 09:00:00.353377247] (+0.000523879) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "syslogd", tid = 383, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.354078519] (+0.000701272) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "klogd", prev_tid = 385, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:00.354210095] (+0.000131576) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "syslogd", next_tid = 383, next_prio = 20 } [2012-08-28 09:00:00.355109428] (+0.000899333) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "syslogd", prev_tid = 383, prev_prio = 20, prev_state = 130, next_comm = "klogd", next_tid = 385, next_prio = 20 } [2012-08-28 09:00:00.355501731] (+0.000392303) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "syslogd", tid = 383, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.355620519] (+0.000118788) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "klogd", prev_tid = 385, prev_prio = 20, prev_state = 0, next_comm = "syslogd", next_tid = 383, next_prio = 20 } [2012-08-28 09:00:00.356259610] (+0.000639091) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "syslogd", prev_tid = 383, prev_prio = 20, prev_state = 130, next_comm = "klogd", next_tid = 385, next_prio = 20 } [2012-08-28 09:00:00.356807429] (+0.000547819) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "syslogd", tid = 383, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.357136883] (+0.000329454) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "klogd", prev_tid = 385, prev_prio = 20, prev_state = 1, next_comm = "syslogd", next_tid = 383, next_prio = 20 } [2012-08-28 09:00:00.357816762] (+0.000679879) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "syslogd", prev_tid = 383, prev_prio = 20, prev_state = 130, next_comm = "lttng", next_tid = 492, next_prio = 20 } [2012-08-28 09:00:00.358233853] (+0.000417091) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "syslogd", tid = 383, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.358371125] (+0.000137272) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 492, prev_prio = 20, prev_state = 0, next_comm = "syslogd", next_tid = 383, next_prio = 20 } [2012-08-28 09:00:00.360446034] (+0.002074909) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "syslogd", prev_tid = 383, prev_prio = 20, prev_state = 1, next_comm = "lttng", next_tid = 492, next_prio = 20 } [2012-08-28 09:00:00.361265065] (+0.000819031) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "ksoftirqd/0", tid = 3, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.361352459] (+0.000087394) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 492, prev_prio = 20, prev_state = 0, next_comm = "ksoftirqd/0", next_tid = 3, next_prio = 20 } [2012-08-28 09:00:00.361385974] (+0.000033515) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "lttng", tid = 492, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.361461186] (+0.000075212) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "ksoftirqd/0", prev_tid = 3, prev_prio = 20, prev_state = 1, next_comm = "lttng", next_tid = 492, next_prio = 20 } [2012-08-28 09:00:00.361540459] (+0.000079273) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "ksoftirqd/0", tid = 3, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.361600641] (+0.000060182) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 492, prev_prio = 20, prev_state = 0, next_comm = "ksoftirqd/0", next_tid = 3, next_prio = 20 } [2012-08-28 09:00:00.361685247] (+0.000084606) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "ksoftirqd/0", prev_tid = 3, prev_prio = 20, prev_state = 1, next_comm = "lttng", next_tid = 492, next_prio = 20 } [2012-08-28 09:00:00.362983004] (+0.001297757) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "sh", tid = 388, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.363067247] (+0.000084243) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 492, prev_prio = 20, prev_state = 64, next_comm = "sh", next_tid = 388, next_prio = 20 } [2012-08-28 09:00:00.363663428] (+0.000596181) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "ksoftirqd/0", tid = 3, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.363753004] (+0.000089576) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sh", prev_tid = 388, prev_prio = 20, prev_state = 0, next_comm = "ksoftirqd/0", next_tid = 3, next_prio = 20 } [2012-08-28 09:00:00.363788155] (+0.000035151) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "sh", tid = 388, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.363856580] (+0.000068425) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "ksoftirqd/0", prev_tid = 3, prev_prio = 20, prev_state = 1, next_comm = "sh", next_tid = 388, next_prio = 20 } [2012-08-28 09:00:00.363922519] (+0.000065939) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "ksoftirqd/0", tid = 3, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.364054580] (+0.000132061) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sh", prev_tid = 388, prev_prio = 20, prev_state = 0, next_comm = "ksoftirqd/0", next_tid = 3, next_prio = 20 } [2012-08-28 09:00:00.364119004] (+0.000064424) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "ksoftirqd/0", prev_tid = 3, prev_prio = 20, prev_state = 1, next_comm = "sh", next_tid = 388, next_prio = 20 } [2012-08-28 09:00:00.366897247] (+0.002778243) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sh", prev_tid = 388, prev_prio = 20, prev_state = 0, next_comm = "cons_recv_fds", next_tid = 461, next_prio = 20 } [2012-08-28 09:00:00.367404640] (+0.000507393) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = 20, prev_state = 1, next_comm = "sh", next_tid = 388, next_prio = 20 } [2012-08-28 09:00:00.367605549] (+0.000200909) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sh", prev_tid = 388, prev_prio = 20, prev_state = 1, next_comm = "mng_clients", next_tid = 452, next_prio = 20 } [2012-08-28 09:00:00.367966701] (+0.000361152) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "mng_clients", prev_tid = 452, prev_prio = 20, prev_state = 1, next_comm = "sh", next_tid = 495, next_prio = 20 } [2012-08-28 09:00:00.369751065] (+0.001784364) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sh", prev_tid = 495, prev_prio = 20, prev_state = 130, next_comm = "cons_recv_fds", next_tid = 494, next_prio = 20 } [2012-08-28 09:00:00.370164277] (+0.000413212) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "sh", tid = 495, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.371547186] (+0.001382909) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 494, prev_prio = 20, prev_state = 130, next_comm = "sh", next_tid = 495, next_prio = 20 } [2012-08-28 09:00:00.371978216] (+0.000431030) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "cons_recv_fds", tid = 494, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.372420277] (+0.000442061) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sh", prev_tid = 495, prev_prio = 20, prev_state = 130, next_comm = "cons_recv_fds", next_tid = 494, next_prio = 20 } [2012-08-28 09:00:00.372877853] (+0.000457576) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "sh", tid = 495, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.373435004] (+0.000557151) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 494, prev_prio = 20, prev_state = 130, next_comm = "sh", next_tid = 495, next_prio = 20 } [2012-08-28 09:00:00.373916641] (+0.000481637) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "cons_recv_fds", tid = 494, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.374868943] (+0.000952302) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sh", prev_tid = 495, prev_prio = 20, prev_state = 130, next_comm = "cons_recv_fds", next_tid = 494, next_prio = 20 } [2012-08-28 09:00:00.375289610] (+0.000420667) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "sh", tid = 495, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.375731186] (+0.000441576) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 494, prev_prio = 20, prev_state = 130, next_comm = "sh", next_tid = 495, next_prio = 20 } [2012-08-28 09:00:00.376117428] (+0.000386242) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "cons_recv_fds", tid = 494, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.378842034] (+0.002724606) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sleep", prev_tid = 495, prev_prio = 20, prev_state = 0, next_comm = "cons_recv_fds", next_tid = 494, next_prio = 20 } [2012-08-28 09:00:00.379333671] (+0.000491637) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 494, prev_prio = 20, prev_state = 130, next_comm = "sleep", next_tid = 495, next_prio = 20 } [2012-08-28 09:00:00.380850217] (+0.001516546) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sleep", prev_tid = 495, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:00.381269065] (+0.000418848) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "sleep", tid = 495, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.381361004] (+0.000091939) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "sleep", next_tid = 495, next_prio = 20 } [2012-08-28 09:00:00.386596822] (+0.005235818) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sleep", prev_tid = 495, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:00.394262762] (+0.007665940) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "cons_recv_fds", tid = 494, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.394357368] (+0.000094606) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "cons_recv_fds", next_tid = 494, next_prio = 20 } [2012-08-28 09:00:00.394719974] (+0.000362606) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "cons_recv_fds", tid = 461, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.395709549] (+0.000989575) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 494, prev_prio = 20, prev_state = 64, next_comm = "cons_recv_fds", next_tid = 461, next_prio = 20 } [2012-08-28 09:00:00.396408701] (+0.000699152) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "cons_poll_fds", tid = 462, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.398866640] (+0.002457939) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = 20, prev_state = 0, next_comm = "cons_poll_fds", next_tid = 462, next_prio = 20 } [2012-08-28 09:00:00.398973428] (+0.000106788) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_poll_fds", prev_tid = 462, prev_prio = 20, prev_state = 2, next_comm = "cons_recv_fds", next_tid = 461, next_prio = 20 } [2012-08-28 09:00:00.399076034] (+0.000102606) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "cons_poll_fds", tid = 462, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.399131186] (+0.000055152) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = 20, prev_state = 0, next_comm = "cons_poll_fds", next_tid = 462, next_prio = 20 } [2012-08-28 09:00:00.401086701] (+0.001955515) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_poll_fds", prev_tid = 462, prev_prio = 20, prev_state = 1, next_comm = "cons_recv_fds", next_tid = 461, next_prio = 20 } [2012-08-28 09:00:00.401670216] (+0.000583515) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = 20, prev_state = 1, next_comm = "cons_recv_fds", next_tid = 496, next_prio = 20 } [2012-08-28 09:00:00.402858156] (+0.001187940) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 496, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:00.403272398] (+0.000414242) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "cons_recv_fds", tid = 496, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.403363852] (+0.000091454) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "cons_recv_fds", next_tid = 496, next_prio = 20 } [2012-08-28 09:00:00.403830277] (+0.000466425) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 496, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:00.404193731] (+0.000363454) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "cons_recv_fds", tid = 496, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.404283852] (+0.000090121) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "cons_recv_fds", next_tid = 496, next_prio = 20 } [2012-08-28 09:00:00.404909247] (+0.000625395) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 496, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:00.405407065] (+0.000497818) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "cons_recv_fds", tid = 496, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.405496823] (+0.000089758) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "cons_recv_fds", next_tid = 496, next_prio = 20 } [2012-08-28 09:00:00.405854156] (+0.000357333) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "cons_recv_fds", tid = 461, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.405938822] (+0.000084666) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 496, prev_prio = 20, prev_state = 0, next_comm = "cons_recv_fds", next_tid = 461, next_prio = 20 } [2012-08-28 09:00:00.406084095] (+0.000145273) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = 20, prev_state = 1, next_comm = "cons_recv_fds", next_tid = 496, next_prio = 20 } [2012-08-28 09:00:00.407004519] (+0.000920424) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "cons_recv_fds", tid = 461, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.407075186] (+0.000070667) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 496, prev_prio = 20, prev_state = 64, next_comm = "cons_recv_fds", next_tid = 461, next_prio = 20 } [2012-08-28 09:00:00.407706034] (+0.000630848) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "cons_poll_fds", tid = 462, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:00.407897368] (+0.000191334) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = 20, prev_state = 1, next_comm = "cons_poll_fds", next_tid = 462, next_prio = 20 } [2012-08-28 09:00:00.408304580] (+0.000407212) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "cons_poll_fds", prev_tid = 462, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:04.547552602] (+4.139248022) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:04.547632966] (+0.000080364) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:04.547736542] (+0.000103576) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:04.691537693] (+0.143801151) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "flush-0:14", tid = 389, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:04.691615027] (+0.000077334) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "flush-0:14", next_tid = 389, next_prio = 20 } [2012-08-28 09:00:04.691705269] (+0.000090242) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "flush-0:14", prev_tid = 389, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:05.311535024] (+0.619829755) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:05.311613084] (+0.000078060) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:05.311882903] (+0.000269819) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:08.842504805] (+3.530621902) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:08.842583350] (+0.000078545) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:08.842664562] (+0.000081212) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:08.994502865] (+0.151838303) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:08.994581229] (+0.000078364) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:08.994670441] (+0.000089212) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:13.137477252] (+4.142806811) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:13.137557191] (+0.000079939) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:13.137656101] (+0.000098910) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:13.901469492] (+0.763813391) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:13.901548037] (+0.000078545) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:13.902056037] (+0.000508000) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:13.905507067] (+0.003451030) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "sync_supers", tid = 76, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:13.905589916] (+0.000082849) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "sync_supers", next_tid = 76, next_prio = 20 } [2012-08-28 09:00:13.905661613] (+0.000071697) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sync_supers", prev_tid = 76, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:17.432437394] (+3.526775781) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:17.432515636] (+0.000078242) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:17.432599636] (+0.000084000) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:21.727404869] (+4.294805233) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:21.727485232] (+0.000080363) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:21.727587050] (+0.000101818) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:22.491405717] (+0.763818667) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:22.491484384] (+0.000078667) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:22.491873414] (+0.000389030) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:26.022372104] (+3.530498690) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:26.022450467] (+0.000078363) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:26.022531134] (+0.000080667) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:26.166376770] (+0.143845636) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "flush-0:14", tid = 389, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:26.166453558] (+0.000076788) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "flush-0:14", next_tid = 389, next_prio = 20 } [2012-08-28 09:00:26.166543558] (+0.000090000) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "flush-0:14", prev_tid = 389, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:30.317347278] (+4.150803720) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:30.317427399] (+0.000080121) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:30.317528550] (+0.000101151) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:31.081338729] (+0.763810179) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:31.081417759] (+0.000079030) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:31.081881335] (+0.000463576) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:34.612306086] (+3.530424751) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:34.612384571] (+0.000078485) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:34.612465116] (+0.000080545) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:38.907282170] (+4.294817054) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:38.907362109] (+0.000079939) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:38.907461442] (+0.000099333) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:39.622817924] (+0.715356482) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "syslogd", tid = 383, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:39.622887803] (+0.000069879) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "syslogd", next_tid = 383, next_prio = 20 } [2012-08-28 09:00:39.623148106] (+0.000260303) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "syslogd", prev_tid = 383, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:39.671276530] (+0.048128424) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:39.671355864] (+0.000079334) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:39.671687682] (+0.000331818) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:39.675311742] (+0.003624060) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "sync_supers", tid = 76, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:39.675394409] (+0.000082667) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "sync_supers", next_tid = 76, next_prio = 20 } [2012-08-28 09:00:39.675466288] (+0.000071879) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sync_supers", prev_tid = 76, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.202240735] (+3.526774447) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.202318857] (+0.000078122) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } [2012-08-28 09:00:43.202402250] (+0.000083393) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.346233038] (+0.143830788) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "sleep", tid = 495, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.346330614] (+0.000097576) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "sleep", next_tid = 495, next_prio = 20 } [2012-08-28 09:00:43.347314432] (+0.000983818) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "sh", tid = 388, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.347383644] (+0.000069212) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sleep", prev_tid = 495, prev_prio = 20, prev_state = 64, next_comm = "sh", next_tid = 388, next_prio = 20 } [2012-08-28 09:00:43.349999825] (+0.002616181) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sh", prev_tid = 388, prev_prio = 20, prev_state = 1, next_comm = "sh", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.351726735] (+0.001726910) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sh", prev_tid = 497, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.352191159] (+0.000464424) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "sh", tid = 497, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.352283038] (+0.000091879) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "sh", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.353304492] (+0.001021454) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "sh", prev_tid = 497, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.353711038] (+0.000406546) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "sh", tid = 497, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.353802856] (+0.000091818) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "sh", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.357738795] (+0.003935939) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.358151644] (+0.000412849) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.358400311] (+0.000248667) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.358873644] (+0.000473333) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.359283644] (+0.000410000) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.359375523] (+0.000091879) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.360845098] (+0.001469575) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.361246492] (+0.000401394) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.361338977] (+0.000092485) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.361800674] (+0.000461697) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.362320250] (+0.000519576) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.362479463] (+0.000159213) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.363838068] (+0.001358605) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.364240553] (+0.000402485) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.364332553] (+0.000092000) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.364792189] (+0.000459636) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.365205038] (+0.000412849) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.365296916] (+0.000091878) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.366988129] (+0.001691213) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.367399705] (+0.000411576) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.367492371] (+0.000092666) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.368825704] (+0.001333333) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.369229038] (+0.000403334) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.369321280] (+0.000092242) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.370950675] (+0.001629395) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.371353220] (+0.000402545) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.371445462] (+0.000092242) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.380016917] (+0.008571455) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.380428129] (+0.000411212) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.380521159] (+0.000093030) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.381335462] (+0.000814303) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "mng_clients", tid = 452, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.381420795] (+0.000085333) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 0, next_comm = "mng_clients", next_tid = 452, next_prio = 20 } [2012-08-28 09:00:43.381826250] (+0.000405455) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "mng_clients", prev_tid = 452, prev_prio = 20, prev_state = 1, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.382007038] (+0.000180788) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "mng_clients", tid = 452, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.382081522] (+0.000074484) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 0, next_comm = "mng_clients", next_tid = 452, next_prio = 20 } [2012-08-28 09:00:43.382528190] (+0.000446668) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "mng_clients", prev_tid = 452, prev_prio = 20, prev_state = 1, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.383627341] (+0.001099151) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } [2012-08-28 09:00:43.384035280] (+0.000407939) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.384128189] (+0.000092909) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.385790613] (+0.001662424) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "mng_clients", tid = 452, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.385877704] (+0.000087091) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 0, next_comm = "mng_clients", next_tid = 452, next_prio = 20 } [2012-08-28 09:00:43.386442190] (+0.000564486) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "mng_clients", prev_tid = 452, prev_prio = 20, prev_state = 1, next_comm = "lttng", next_tid = 497, next_prio = 20 } [2012-08-28 09:00:43.386978008] (+0.000535818) mgcoge3ne sched_wakeup: { cpu_id = 0 }, { comm = "mng_clients", tid = 452, prio = 120, success = 1, target_cpu = 0 } [2012-08-28 09:00:43.387074129] (+0.000096121) mgcoge3ne sched_switch: { cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, prev_state = 0, next_comm = "mng_clients", next_tid = 452, next_prio = 20 } So again, I get these huge 3.3 (or so) seconds gaps, and a final timestamp which is 43 seconds instead of the 10 seconds which actually went by. BTW, I also tried with TMF under Eclipse, and here I get even weirder results. Here's a table of correspondence (as there must be some occult side to it!) for the timestamps as reported by babeltrace vs. Eclipse: [2012-08-28 09:00:00.348477307] -> 2033-05-07 16:49:22.035276997 ... [2012-08-28 09:00:00.408304580] -> 2033-05-07 16:49:22.112912324 .... [2012-08-28 09:00:04.691537693] -> 2039-11-23 09:06:50.748861638 Under Eclipse, the latest event of this 10-second trace lies sometime within year 2124. No clue, anyone? Thank you, Gerlando On 12/16/2011 11:15 AM, Gerlando Falauto wrote: > Hi, > > I am trying to use LLTng 2.0 (since I have no alternative for a 3.0 > kernel!) and I of course use babeltrace to read them. > Problem is, I can't make any sense out of the timestamps, as I get > something like (two CPU-hog bash processes): > > [1658841591493] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = > 647, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 646, > next_prio = 20 } > [1658845590645] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = > 646, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 647, > next_prio = 20 } > [1658849590039] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = > 647, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 646, > next_prio = 20 } > [1658853591069] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = > 646, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 647, > next_prio = 20 } > [1662152559395] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = > 647, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 646, > next_prio = 20 } > [1662156587941] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = > 646, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 647, > next_prio = 20 } > [1662160557698] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = > 647, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 646, > next_prio = 20 } > [1662164557517] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = > 646, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 647, > next_prio = 20 } > [....................................cut...........................] > [1663140553029] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = > 646, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 647, > next_prio = 20 } > [1663144555029] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = > 647, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 646, > next_prio = 20 } > [1663148552423] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = > 646, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 647, > next_prio = 20 } > [1666447519719] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = > 647, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 646, > next_prio = 20 } > [1666451561355] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = > 646, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid > = 94, next_prio = 20 } > [1666451863719] sched_switch: { 0 }, { prev_comm = "kworker/0:1", > prev_tid = 94, prev_prio = 20, prev_state = 1, next_comm = "bash", > next_tid = 647, next_prio = 20 } > > Which made me think timestamps were expressed in nanoseconds, as I am > running this on 4ms-tick PowerPC (CONFIG_HZ=250), and the difference > between timestamps is approximately 4,000,000. > However, I get these huge 3.3 second gaps between for instance between > 1658853591069 and 1662152559395. > Curiously enough, these leap 3.3 seconds occur exactly (well, within 4ms > precision...) every second. Kind of like the "seconds" part of the > timestamp is calculated on a wrong-base arithmetics or so.... > This also makes sense as the seconds' portion of the timestamp is > essentially 4.3 times bigger than the uptime (and any time stretch is > 4.3 times bigger than reality, for that matter). > > Any suggestions? > > Thanks! > Gerlando > > P.S. This also happens without my hack for flight recording mode... From gerlando.falauto at keymile.com Wed Aug 29 11:17:43 2012 From: gerlando.falauto at keymile.com (Gerlando Falauto) Date: Wed, 29 Aug 2012 17:17:43 +0200 Subject: [lttng-dev] Timestamps in babeltrace In-Reply-To: <503CC0DF.7030607@keymile.com> References: <4EEB1A44.2020500@keymile.com> <503CC0DF.7030607@keymile.com> Message-ID: <503E3297.8090703@keymile.com> Hi again, I tested this on an ARM board, and results look OK on both babeltrace and within TMF. So there must be something wrong with the way timestamps are calculated (and/or stored) by LLTng 2.x on the target. I looked at the timestamps again (having babeltrace dump the timestamp-cycles in HEX, sided by timestamp-seconds), and got the following: [000000000e3816c58e6e] [1346144403.557251024] (+?.?????????) [000000000e3816e58d02] [1346144403.559347812] (+0.002096788) [000000000e3816e887cb] [1346144403.559543085] (+0.000195273) [000000000e3816f08eb7] [1346144403.560069145] (+0.000526060) [000000000e3816f91981] [1346144403.560628963] (+0.000559818) [000000000e38170a3b65] [1346144403.561751751] (+0.001122788) [000000000e3817108c1b] [1346144403.562165629] (+0.000413878) [000000000e381712d427] [1346144403.562315145] (+0.000149516) [000000000e38172a770d] [1346144403.563864175] (+0.001549030) [000000000e381744a88b] [1346144403.565580781] (+0.001716606) [000000000e38176dd2e6] [1346144403.568278600] (+0.002697819) [000000000e38176f4e65] [1346144403.568375751] (+0.000097151) [000000000e381779ed61] [1346144403.569071811] (+0.000696060) [000000000e38177b3f36] [1346144403.569158296] (+0.000086485) [000000000e38177bc849] [1346144403.569193387] (+0.000035091) [000000000e38177ce3e7] [1346144403.569265993] (+0.000072606) [000000000e38177e25a3] [1346144403.569348357] (+0.000082364) [000000000e38177f214b] [1346144403.569412781] (+0.000064424) [000000000e3817804754] [1346144403.569488054] (+0.000075273) [000000000e381793d8cc] [1346144403.570770478] (+0.001282424) [000000000e3817951f07] [1346144403.570853993] (+0.000083515) [000000000e38179cc3b5] [1346144403.571354903] (+0.000500910) [000000000e38179ef04a] [1346144403.571497388] (+0.000142485) [000000000e3817a066d0] [1346144403.571593266] (+0.000095878) [000000000e3817a84ab3] [1346144403.572110357] (+0.000517091) [000000000e3817aaf798] [1346144403.572285690] (+0.000175333) [000000000e3817acc2a4] [1346144403.572403206] (+0.000117516) [000000000e3817bc26fb] [1346144403.573411933] (+0.001008727) [000000000e3817c223e3] [1346144403.573804357] (+0.000392424) [000000000e3817c3731d] [1346144403.573890175] (+0.000085818) [000000000e3817c3fb7a] [1346144403.573925084] (+0.000034909) [000000000e3817c50c71] [1346144403.573994963] (+0.000069879) [000000000e3817c64a26] [1346144403.574076296] (+0.000081333) [000000000e3817c74519] [1346144403.574140539] (+0.000064243) [000000000e3817c8684a] [1346144403.574215084] (+0.000074545) [000000000e3817d4a27f] [1346144403.575016417] (+0.000801333) [000000000e3817dbc719] [1346144403.575484539] (+0.000468122) [000000000e3817de86b2] [1346144403.575664660] (+0.000180121) [000000000e3817e98332] [1346144403.576384660] (+0.000720000) [000000000e3817eed6d5] [1346144403.576733751] (+0.000349091) [000000000e3817f0f079] [1346144403.576871387] (+0.000137636) [000000000e3817fa997c] [1346144403.577504478] (+0.000633091) [000000000e3817ffc467] [1346144403.577843145] (+0.000338667) [000000000e3818100692] [1346144403.578908660] (+0.001065515) [000000000e381817dcf7] [1346144403.579422297] (+0.000513637) [000000000e381821774c] [1346144403.580051630] (+0.000629333) [000000000e3818293e89] [1346144403.580561387] (+0.000509757) [000000000e38182d7c79] [1346144403.580839387] (+0.000278000) [000000000e38183661ef] [1346144403.581422417] (+0.000583030) [000000000e38183d1430] [1346144403.581861266] (+0.000438849) [000000000e3818433d9a] [1346144403.582265084] (+0.000403818) [000000000e38184a844a] [1346144403.582741932] (+0.000476848) [000000000e3818511160] [1346144403.583171266] (+0.000429334) [000000000e381857ed4b] [1346144403.583620781] (+0.000449515) [000000000e381861db33] [1346144403.584271509] (+0.000650728) [000000000e3818678ef3] [1346144403.584645205] (+0.000373696) [000000000e38186e85dc] [1346144403.585101630] (+0.000456425) [000000000e381875ec45] [1346144403.585586599] (+0.000484969) [000000000e38187af334] [1346144403.585916054] (+0.000329455) [000000000e3818a5f4ac] [1346144403.588734478] (+0.002818424) [000000000e3818bd10dd] [1346144403.590249023] (+0.001514545) [000000000e3818c35f73] [1346144403.590662357] (+0.000413334) [000000000e3818c4c52a] [1346144403.590753932] (+0.000091575) [000000000e3818cbb849] [1346144403.591209387] (+0.000455455) [000000000e3818cfcccb] [1346144403.591476781] (+0.000267394) [000000000e3818d54f12] [1346144403.591837812] (+0.000361031) [000000000e3818d81053] [1346144403.592018357] (+0.000180545) [000000000e3818d932cf] [1346144403.592092721] (+0.000074364) [000000000e3818de4f48] [1346144403.592427690] (+0.000334969) [000000000e3818df9af5] [1346144403.592512599] (+0.000084909) [000000000e38191378e5] [1346144403.595911751] (+0.003399152) [000000000e38191542c1] [1346144403.596028963] (+0.000117212) [000000000e3819189d6b] [1346144403.596248781] (+0.000219818) [000000000e38191a4715] [1346144403.596357751] (+0.000108970) [000000000e3819282a3e] [1346144403.597267872] (+0.000910121) [000000000e38194e6256] [1346144403.599772600] (+0.002504728) [000000000e3819516697] [1346144403.599970297] (+0.000197697) [000000000e381953f0eb] [1346144403.600136781] (+0.000166484) [000000000e38195791e5] [1346144403.600374599] (+0.000237818) [000000000e38196bff4c] [1346144403.601713326] (+0.001338727) [000000000e38196d4459] [1346144403.601796539] (+0.000083213) [000000000e38197290a5] [1346144403.602143751] (+0.000347212) [000000000e381975ccc5] [1346144403.602355751] (+0.000212000) [000000000e381976eac1] [1346144403.602428963] (+0.000073212) [000000000e38197bf486] [1346144403.602759144] (+0.000330181) [000000000e38197e224a] [1346144403.602901932] (+0.000142788) [000000000e38197f2364] [1346144403.602967750] (+0.000065818) [000000000e381986a674] [1346144403.603460054] (+0.000492304) [000000000e38198b8fcb] [1346144403.603781933] (+0.000321879) [000000000e38198cd550] [1346144403.603865266] (+0.000083333) [000000000e381992f086] [1346144403.604265448] (+0.000400182) [000000000e3819943d9f] [1346144403.604350721] (+0.000085273) [000000000e3819b686d8] [1346144403.606597690] (+0.002246969) [000000000e3819b8259e] [1346144403.606703872] (+0.000106182) [000000000e3819bb4684] [1346144403.606908902] (+0.000205030) [000000000e3819ec04bc] [1346144403.610103326] (+0.003194424) [000000000e3819eda61d] [1346144403.610210175] (+0.000106849) [000000000e3819f0c77c] [1346144403.610415326] (+0.000205151) [000000000e381a2178af] [1346144403.613606417] (+0.003191091) [000000000e381a2314da] [1346144403.613711932] (+0.000105515) [000000000e381a2633dc] [1346144403.613916478] (+0.000204546) [000000000e381a4f3892] [1346144403.616604660] (+0.002688182) [000000000e381a50ca8f] [1346144403.616707569] (+0.000102909) [000000000e381a53e5c7] [1346144403.616911145] (+0.000203576) [000000000e381a557437] [1346144403.617013145] (+0.000102000) [000000000e381a7d5dc8] [1346144403.619628842] (+0.002615697) [000000000e381a7fa51d] [1346144403.619778175] (+0.000149333) [000000000e381a864560] [1346144403.620212418] (+0.000434243) [000000000e381a9651cd] [1346144403.621264175] (+0.001051757) [000000000e381aa37b5b] [1346144403.622126781] (+0.000862606) [000000000e381ab84868] [1346144403.623489994] (+0.001363213) [000000000e381ac23926] [1346144403.624141448] (+0.000651454) [000000000e381adcda62] [1346144403.625886660] (+0.001745212) [000000000e381ae50f78] [1346144403.626424538] (+0.000537878) [000000000e381aebcd52] [1346144403.626866356] (+0.000441818) [000000000e381aeec673] [1346144403.627061205] (+0.000194849) [000000000e381b1e47ac] [1346144403.630174478] (+0.003113273) [000000000e381b33e1bd] [1346144403.631590175] (+0.001415697) [000000000e381b3a9348] [1346144403.632028842] (+0.000438667) [000000000e381b636d9d] [1346144403.634706175] (+0.002677333) [000000000e381b6b6304] [1346144403.635227750] (+0.000521575) [000000000e381b72078a] [1346144403.635663084] (+0.000435334) [000000000e381b7426dd] [1346144403.635802175] (+0.000139091) [000000000e381b7b720d] [1346144403.636280175] (+0.000478000) [000000000e381b839470] [1346144403.636813266] (+0.000533091) [000000000e381b85aca9] [1346144403.636950539] (+0.000137273) [000000000e381b8b5abb] [1346144403.637322781] (+0.000372242) [000000000e381b9a25e6] [1346144403.638292296] (+0.000969515) [000000000e381ba522a2] [1346144403.639012356] (+0.000720060) [000000000e381ba86e62] [1346144403.639228356] (+0.000216000) [000000000e381bb3f1d5] [1346144403.639982903] (+0.000754547) [000000000e381bb7c172] [1346144403.640232660] (+0.000249757) [000000000e3903588b84] [1346144407.526307046] (+3.886074386) [000000000e390359cae2] [1346144407.526388804] (+0.000081758) [000000000e39035b5e4b] [1346144407.526492077] (+0.000103273) [000000000e390d5bcfc7] [1346144407.694293289] (+0.167801212) [000000000e390d5cfa12] [1346144407.694369652] (+0.000076363) [000000000e390d5e5b4b] [1346144407.694460077] (+0.000090425) [000000000e390eca0c68] [1346144407.718294986] (+0.023834909) [000000000e390ecb3326] [1346144407.718370440] (+0.000075454) [000000000e390ecc897b] [1346144407.718458077] (+0.000087637) [000000000e390f9daeac] [1346144407.732164622] (+0.013706545) [000000000e390f9ea469] [1346144407.732227531] (+0.000062909) [000000000e390fa2983f] [1346144407.732486561] (+0.000259030) [000000000e3a003ef59a] [1346144411.769265916] (+4.036779355) [000000000e3a00402dde] [1346144411.769345856] (+0.000079940) [000000000e3a00458822] [1346144411.769696644] (+0.000350788) [000000000e3a03586436] [1346144411.821264280] (+0.051567636) [000000000e3a03598dcc] [1346144411.821340462] (+0.000076182) [000000000e3a035ac596] [1346144411.821420280] (+0.000079818) [000000000e3b0358576b] [1346144416.116228301] (+4.294808021) [000000000e3b03598dca] [1346144416.116307756] (+0.000079455) [000000000e3b035b1870] [1346144416.116408786] (+0.000101030) [000000000e3c003ee0c3] [1346144420.359195173] (+4.242786387) [000000000e3c00401228] [1346144420.359273354] (+0.000078181) [000000000e3c00480297] [1346144420.359793657] (+0.000520303) [000000000e3c035863f7] [1346144420.411198809] (+0.051405152) [000000000e3c03598e43] [1346144420.411275173] (+0.000076364) [000000000e3c035ac6ff] [1346144420.411355233] (+0.000080060) [000000000e3d03586b4c] [1346144424.706167982] (+4.294812749) [000000000e3d0359a7d2] [1346144424.706249012] (+0.000081030) [000000000e3d035b2f28] [1346144424.706349194] (+0.000100182) [000000000e3e003ee0fb] [1346144428.949129821] (+4.242780627) [000000000e3e00401317] [1346144428.949208185] (+0.000078364) [000000000e3e00495a90] [1346144428.949816306] (+0.000608121) [000000000e3e00f6261f] [1346144428.961140609] (+0.011324303) [000000000e3e00f79420] [1346144428.961234306] (+0.000093697) [000000000e3e00f8ba29] [1346144428.961309579] (+0.000075273) [000000000e3e03586522] [1346144429.001133700] (+0.039824121) [000000000e3e03598e7b] [1346144429.001209821] (+0.000076121) [000000000e3e035acd23] [1346144429.001291397] (+0.000081576) [000000000e3e0d5bc868] [1346144429.169127882] (+0.167836485) [000000000e3e0d5cf67d] [1346144429.169205215] (+0.000077333) [000000000e3e0d60bcdf] [1346144429.169452609] (+0.000247394) [000000000e3e0d645b02] [1346144429.169689700] (+0.000237091) [000000000e3e0d711730] [1346144429.170524306] (+0.000834606) [000000000e3e10139fd2] [1346144429.214730548] (+0.044206242) [000000000e3e10150f7b] [1346144429.214824669] (+0.000094121) [000000000e3e10190223] [1346144429.215083397] (+0.000258728) [000000000e3e111413cf] [1346144429.231537457] (+0.016454060) [000000000e3e11157581] [1346144429.231628003] (+0.000090546) [000000000e3e11193dc7] [1346144429.231875881] (+0.000247878) [000000000e3f03586de3] [1346144433.296103237] (+4.064227356) [000000000e3f03599e93] [1346144433.296181237] (+0.000078000) [000000000e3f035b2754] [1346144433.296281782] (+0.000100545) [000000000e40003ee171] [1346144437.539064531] (+4.242782749) [000000000e400040138c] [1346144437.539142894] (+0.000078363) [000000000e4000462587] [1346144437.539540713] (+0.000397819) [000000000e4003586469] [1346144437.591068107] (+0.051527394) [000000000e4003598f6a] [1346144437.591144652] (+0.000076545) [000000000e40035ac605] [1346144437.591224167] (+0.000079515) [000000000e4103585761] [1346144441.886032067] (+4.294807900) [000000000e4103598b62] [1346144441.886110916] (+0.000078849) [000000000e41035b16fa] [1346144441.886212188] (+0.000101272) [000000000e42003ee446] [1346144446.128999848] (+4.242787660) [000000000e4200401532] [1346144446.129077908] (+0.000078060) [000000000e420047efd9] [1346144446.129592635] (+0.000514727) [000000000e420358699b] [1346144446.181004029] (+0.051411394) [000000000e42035995cc] [1346144446.181080878] (+0.000076849) [000000000e42035ad0aa] [1346144446.181161484] (+0.000080606) [000000000e421c23e01f] [1346144446.596991361] (+0.415829877) [000000000e421c2558c7] [1346144446.597087785] (+0.000096424) [000000000e421c34e871] [1346144446.598107603] (+0.001019818) [000000000e421c35d936] [1346144446.598169240] (+0.000061637) [000000000e421c65189e] [1346144446.601265664] (+0.003096424) [000000000e421c661b24] [1346144446.601331846] (+0.000066182) [000000000e421c7fab69] [1346144446.603007179] (+0.001675333) [000000000e421c86a8f2] [1346144446.603465300] (+0.000458121) [000000000e421c880aa4] [1346144446.603555846] (+0.000090546) [000000000e421c8f701b] [1346144446.604040573] (+0.000484727) [000000000e421c95da63] [1346144446.604460997] (+0.000420424) [000000000e421c97371c] [1346144446.604550270] (+0.000089273) [000000000e421ca9fb16] [1346144446.605780088] (+0.001229818) [000000000e421cb03e8b] [1346144446.606190573] (+0.000410485) [000000000e421cb1a000] [1346144446.606281058] (+0.000090485) [000000000e421cb89ab2] [1346144446.606738452] (+0.000457394) [000000000e421cbefa16] [1346144446.607156088] (+0.000417636) [000000000e421cc057fe] [1346144446.607245664] (+0.000089576) [000000000e421cfc3af5] [1346144446.611170391] (+0.003924727) [000000000e421d02756b] [1346144446.611578573] (+0.000408182) [000000000e421d03d888] [1346144446.611669482] (+0.000090909) [000000000e421d0ad063] [1346144446.612126149] (+0.000456667) [000000000e421d114a0e] [1346144446.612550512] (+0.000424363) [000000000e421d12a68a] [1346144446.612639724] (+0.000089212) [000000000e421d2c1cc5] [1346144446.614308391] (+0.001668667) [000000000e421d3260f0] [1346144446.614719058] (+0.000410667) [000000000e421d33c44a] [1346144446.614810028] (+0.000090970) [000000000e421d3abcda] [1346144446.615266876] (+0.000456848) [000000000e421d410326] [1346144446.615678088] (+0.000411212) [000000000e421d425f29] [1346144446.615767179] (+0.000089091) [000000000e421d59acd5] [1346144446.617294391] (+0.001527212) [000000000e421d5ff7de] [1346144446.617706816] (+0.000412425) [000000000e421d615f7a] [1346144446.617798876] (+0.000092060) [000000000e421d6855ad] [1346144446.618255119] (+0.000456243) [000000000e421d6eb1fd] [1346144446.618671967] (+0.000416848) [000000000e421d700fe5] [1346144446.618761543] (+0.000089576) [000000000e421d86f308] [1346144446.620261482] (+0.001499939) [000000000e421d8d4124] [1346144446.620674694] (+0.000413212) [000000000e421d8ea405] [1346144446.620765543] (+0.000090849) [000000000e421da58d13] [1346144446.622266997] (+0.001501454) [000000000e421dabb0cf] [1346144446.622669361] (+0.000402364) [000000000e421dad1594] [1346144446.622760694] (+0.000091333) [000000000e421dc2edac] [1346144446.624192270] (+0.001431576) [000000000e421dc93edc] [1346144446.624606270] (+0.000414000) [000000000e421dcaa50d] [1346144446.624697967] (+0.000091697) [000000000e421e4f0056] [1346144446.633372088] (+0.008674121) [000000000e421e557e08] [1346144446.633797482] (+0.000425394) [000000000e421e56e162] [1346144446.633888452] (+0.000090970) [000000000e421e636796] [1346144446.634709240] (+0.000820788) [000000000e421e64a7e6] [1346144446.634791240] (+0.000082000) [000000000e421e6ae404] [1346144446.635199846] (+0.000408606) [000000000e421e6dafb0] [1346144446.635383058] (+0.000183212) [000000000e421e6ec24f] [1346144446.635453361] (+0.000070303) [000000000e421e73ea9f] [1346144446.635791361] (+0.000338000) [000000000e421e88f797] [1346144446.637170937] (+0.001379576) [000000000e421e8c65ab] [1346144446.637395725] (+0.000224788) [000000000e421e8dbcf2] [1346144446.637483604] (+0.000087879) [000000000e421ea85c85] [1346144446.639228391] (+0.001744787) [000000000e421ea9c3e5] [1346144446.639320391] (+0.000092000) [000000000e421eafbeac] [1346144446.639712270] (+0.000391879) [000000000e421eb7f386] [1346144446.640250088] (+0.000537818) [000000000e421eb93399] [1346144446.640332027] (+0.000081939) Since the most significant 32 bits of the timestamp span from 0xe38 to 0xe42, whose difference is exactly 0xa (10 in decimal), I would assume they are somehow representing the uptime in seconds, whereas the least significant 32 bits are (maybe?) really (clock?) cycles. It looks like the maximum value reached by the least significant 32 bits is somewehere near 0x1fff'ffff (so 29 bits as opposed to 32). It's also easy to spot that the huge (4.3 seconds) gaps occur whenever there's an increase in the uppermost 32-bit fraction (0xe38->0xe39, etc...). Anyway, there's definitely something wrong with the way LLTng actually picks up those timestamps. Could someone please spend a few words on how these timestamps are computed on PowerPC? Thanks a lot! Gerlando On 08/28/2012 03:00 PM, Gerlando Falauto wrote: > Hi, > > picking up again from my own message... > > I'm try this again after these few months, and I still get some similar > behavior. > Again, it's a 32-bit PowerPC with 4ms system tick (CONFIG_HZ=250). The > machine has neither NTP nor internal clock, therefore I set a given > timestamp before starting the test. > Traces were generated using the latest master branch of the three > repositories (lttng-modules, lttng-tools, userspace-rcu). > > Running the following script on the target: > > date -s "2012-08-28 09:00:00" > lttng create trace > lttng enable-channel -k mychannel > lttng enable-event sched_switch,sched_wakeup -k -c mychannel > lttng start > echo waiting 10 seconds > sleep 10 > lttng stop > lttng destroy trace > date > > (for which I get the following output): > > Tue Aug 28 09:00:00 UTC 2012 > Session trace created. > Traces will be written in /root/lttng-traces/trace-20120828-090000 > Kernel channel mychannel enabled for session trace > kernel event sched_switch created in channel mychannel > kernel event sched_wakeup created in channel mychannel > Tracing started for session trace > waiting 10 seconds > Tracing stopped for session trace > Session trace destroyed > Tue Aug 28 09:00:10 UTC 2012 > > I get the following trace: > > babeltrace --clock-date --clock-gmt > /path/to/root/lttng-traces/trace-20120828-090000/ > [2012-08-28 09:00:00.348477307] (+?.?????????) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "mng_clients", prev_tid = 452, prev_prio = > 20, prev_state = 0, next_comm = "cons_recv_fds", next_tid = 461, > next_prio = 20 } > [2012-08-28 09:00:00.350913974] (+0.002436667) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = > 20, prev_state = 0, next_comm = "mng_clients", next_tid = 452, next_prio > = 20 } > [2012-08-28 09:00:00.352444701] (+0.001530727) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "lttng", tid = 492, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.352584944] (+0.000140243) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "klogd", tid = 385, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.352691186] (+0.000106242) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.352853368] (+0.000162182) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "mng_clients", prev_tid = 452, prev_prio = > 20, prev_state = 0, next_comm = "klogd", next_tid = 385, next_prio = 20 } > [2012-08-28 09:00:00.353377247] (+0.000523879) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "syslogd", tid = 383, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.354078519] (+0.000701272) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "klogd", prev_tid = 385, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:00.354210095] (+0.000131576) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "syslogd", next_tid = 383, next_prio = 20 } > [2012-08-28 09:00:00.355109428] (+0.000899333) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "syslogd", prev_tid = 383, prev_prio = 20, > prev_state = 130, next_comm = "klogd", next_tid = 385, next_prio = 20 } > [2012-08-28 09:00:00.355501731] (+0.000392303) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "syslogd", tid = 383, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.355620519] (+0.000118788) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "klogd", prev_tid = 385, prev_prio = 20, > prev_state = 0, next_comm = "syslogd", next_tid = 383, next_prio = 20 } > [2012-08-28 09:00:00.356259610] (+0.000639091) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "syslogd", prev_tid = 383, prev_prio = 20, > prev_state = 130, next_comm = "klogd", next_tid = 385, next_prio = 20 } > [2012-08-28 09:00:00.356807429] (+0.000547819) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "syslogd", tid = 383, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.357136883] (+0.000329454) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "klogd", prev_tid = 385, prev_prio = 20, > prev_state = 1, next_comm = "syslogd", next_tid = 383, next_prio = 20 } > [2012-08-28 09:00:00.357816762] (+0.000679879) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "syslogd", prev_tid = 383, prev_prio = 20, > prev_state = 130, next_comm = "lttng", next_tid = 492, next_prio = 20 } > [2012-08-28 09:00:00.358233853] (+0.000417091) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "syslogd", tid = 383, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.358371125] (+0.000137272) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 492, prev_prio = 20, > prev_state = 0, next_comm = "syslogd", next_tid = 383, next_prio = 20 } > [2012-08-28 09:00:00.360446034] (+0.002074909) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "syslogd", prev_tid = 383, prev_prio = 20, > prev_state = 1, next_comm = "lttng", next_tid = 492, next_prio = 20 } > [2012-08-28 09:00:00.361265065] (+0.000819031) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "ksoftirqd/0", tid = 3, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.361352459] (+0.000087394) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 492, prev_prio = 20, > prev_state = 0, next_comm = "ksoftirqd/0", next_tid = 3, next_prio = 20 } > [2012-08-28 09:00:00.361385974] (+0.000033515) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "lttng", tid = 492, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.361461186] (+0.000075212) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "ksoftirqd/0", prev_tid = 3, prev_prio = 20, > prev_state = 1, next_comm = "lttng", next_tid = 492, next_prio = 20 } > [2012-08-28 09:00:00.361540459] (+0.000079273) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "ksoftirqd/0", tid = 3, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.361600641] (+0.000060182) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 492, prev_prio = 20, > prev_state = 0, next_comm = "ksoftirqd/0", next_tid = 3, next_prio = 20 } > [2012-08-28 09:00:00.361685247] (+0.000084606) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "ksoftirqd/0", prev_tid = 3, prev_prio = 20, > prev_state = 1, next_comm = "lttng", next_tid = 492, next_prio = 20 } > [2012-08-28 09:00:00.362983004] (+0.001297757) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "sh", tid = 388, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.363067247] (+0.000084243) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 492, prev_prio = 20, > prev_state = 64, next_comm = "sh", next_tid = 388, next_prio = 20 } > [2012-08-28 09:00:00.363663428] (+0.000596181) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "ksoftirqd/0", tid = 3, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.363753004] (+0.000089576) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sh", prev_tid = 388, prev_prio = 20, > prev_state = 0, next_comm = "ksoftirqd/0", next_tid = 3, next_prio = 20 } > [2012-08-28 09:00:00.363788155] (+0.000035151) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "sh", tid = 388, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.363856580] (+0.000068425) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "ksoftirqd/0", prev_tid = 3, prev_prio = 20, > prev_state = 1, next_comm = "sh", next_tid = 388, next_prio = 20 } > [2012-08-28 09:00:00.363922519] (+0.000065939) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "ksoftirqd/0", tid = 3, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.364054580] (+0.000132061) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sh", prev_tid = 388, prev_prio = 20, > prev_state = 0, next_comm = "ksoftirqd/0", next_tid = 3, next_prio = 20 } > [2012-08-28 09:00:00.364119004] (+0.000064424) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "ksoftirqd/0", prev_tid = 3, prev_prio = 20, > prev_state = 1, next_comm = "sh", next_tid = 388, next_prio = 20 } > [2012-08-28 09:00:00.366897247] (+0.002778243) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sh", prev_tid = 388, prev_prio = 20, > prev_state = 0, next_comm = "cons_recv_fds", next_tid = 461, next_prio = > 20 } > [2012-08-28 09:00:00.367404640] (+0.000507393) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = > 20, prev_state = 1, next_comm = "sh", next_tid = 388, next_prio = 20 } > [2012-08-28 09:00:00.367605549] (+0.000200909) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sh", prev_tid = 388, prev_prio = 20, > prev_state = 1, next_comm = "mng_clients", next_tid = 452, next_prio = 20 } > [2012-08-28 09:00:00.367966701] (+0.000361152) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "mng_clients", prev_tid = 452, prev_prio = > 20, prev_state = 1, next_comm = "sh", next_tid = 495, next_prio = 20 } > [2012-08-28 09:00:00.369751065] (+0.001784364) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sh", prev_tid = 495, prev_prio = 20, > prev_state = 130, next_comm = "cons_recv_fds", next_tid = 494, next_prio > = 20 } > [2012-08-28 09:00:00.370164277] (+0.000413212) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "sh", tid = 495, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.371547186] (+0.001382909) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 494, prev_prio = > 20, prev_state = 130, next_comm = "sh", next_tid = 495, next_prio = 20 } > [2012-08-28 09:00:00.371978216] (+0.000431030) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "cons_recv_fds", tid = 494, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:00.372420277] (+0.000442061) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sh", prev_tid = 495, prev_prio = 20, > prev_state = 130, next_comm = "cons_recv_fds", next_tid = 494, next_prio > = 20 } > [2012-08-28 09:00:00.372877853] (+0.000457576) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "sh", tid = 495, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.373435004] (+0.000557151) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 494, prev_prio = > 20, prev_state = 130, next_comm = "sh", next_tid = 495, next_prio = 20 } > [2012-08-28 09:00:00.373916641] (+0.000481637) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "cons_recv_fds", tid = 494, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:00.374868943] (+0.000952302) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sh", prev_tid = 495, prev_prio = 20, > prev_state = 130, next_comm = "cons_recv_fds", next_tid = 494, next_prio > = 20 } > [2012-08-28 09:00:00.375289610] (+0.000420667) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "sh", tid = 495, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.375731186] (+0.000441576) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 494, prev_prio = > 20, prev_state = 130, next_comm = "sh", next_tid = 495, next_prio = 20 } > [2012-08-28 09:00:00.376117428] (+0.000386242) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "cons_recv_fds", tid = 494, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:00.378842034] (+0.002724606) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sleep", prev_tid = 495, prev_prio = 20, > prev_state = 0, next_comm = "cons_recv_fds", next_tid = 494, next_prio = > 20 } > [2012-08-28 09:00:00.379333671] (+0.000491637) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 494, prev_prio = > 20, prev_state = 130, next_comm = "sleep", next_tid = 495, next_prio = 20 } > [2012-08-28 09:00:00.380850217] (+0.001516546) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sleep", prev_tid = 495, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:00.381269065] (+0.000418848) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "sleep", tid = 495, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:00.381361004] (+0.000091939) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "sleep", next_tid = 495, next_prio = 20 } > [2012-08-28 09:00:00.386596822] (+0.005235818) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sleep", prev_tid = 495, prev_prio = 20, > prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:00.394262762] (+0.007665940) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "cons_recv_fds", tid = 494, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:00.394357368] (+0.000094606) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "cons_recv_fds", next_tid = 494, next_prio = > 20 } > [2012-08-28 09:00:00.394719974] (+0.000362606) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "cons_recv_fds", tid = 461, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:00.395709549] (+0.000989575) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 494, prev_prio = > 20, prev_state = 64, next_comm = "cons_recv_fds", next_tid = 461, > next_prio = 20 } > [2012-08-28 09:00:00.396408701] (+0.000699152) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "cons_poll_fds", tid = 462, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:00.398866640] (+0.002457939) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = > 20, prev_state = 0, next_comm = "cons_poll_fds", next_tid = 462, > next_prio = 20 } > [2012-08-28 09:00:00.398973428] (+0.000106788) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_poll_fds", prev_tid = 462, prev_prio = > 20, prev_state = 2, next_comm = "cons_recv_fds", next_tid = 461, > next_prio = 20 } > [2012-08-28 09:00:00.399076034] (+0.000102606) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "cons_poll_fds", tid = 462, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:00.399131186] (+0.000055152) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = > 20, prev_state = 0, next_comm = "cons_poll_fds", next_tid = 462, > next_prio = 20 } > [2012-08-28 09:00:00.401086701] (+0.001955515) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_poll_fds", prev_tid = 462, prev_prio = > 20, prev_state = 1, next_comm = "cons_recv_fds", next_tid = 461, > next_prio = 20 } > [2012-08-28 09:00:00.401670216] (+0.000583515) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = > 20, prev_state = 1, next_comm = "cons_recv_fds", next_tid = 496, > next_prio = 20 } > [2012-08-28 09:00:00.402858156] (+0.001187940) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 496, prev_prio = > 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:00.403272398] (+0.000414242) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "cons_recv_fds", tid = 496, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:00.403363852] (+0.000091454) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "cons_recv_fds", next_tid = 496, next_prio = > 20 } > [2012-08-28 09:00:00.403830277] (+0.000466425) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 496, prev_prio = > 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:00.404193731] (+0.000363454) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "cons_recv_fds", tid = 496, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:00.404283852] (+0.000090121) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "cons_recv_fds", next_tid = 496, next_prio = > 20 } > [2012-08-28 09:00:00.404909247] (+0.000625395) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 496, prev_prio = > 20, prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:00.405407065] (+0.000497818) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "cons_recv_fds", tid = 496, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:00.405496823] (+0.000089758) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "cons_recv_fds", next_tid = 496, next_prio = > 20 } > [2012-08-28 09:00:00.405854156] (+0.000357333) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "cons_recv_fds", tid = 461, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:00.405938822] (+0.000084666) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 496, prev_prio = > 20, prev_state = 0, next_comm = "cons_recv_fds", next_tid = 461, > next_prio = 20 } > [2012-08-28 09:00:00.406084095] (+0.000145273) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = > 20, prev_state = 1, next_comm = "cons_recv_fds", next_tid = 496, > next_prio = 20 } > [2012-08-28 09:00:00.407004519] (+0.000920424) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "cons_recv_fds", tid = 461, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:00.407075186] (+0.000070667) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 496, prev_prio = > 20, prev_state = 64, next_comm = "cons_recv_fds", next_tid = 461, > next_prio = 20 } > [2012-08-28 09:00:00.407706034] (+0.000630848) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "cons_poll_fds", tid = 462, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:00.407897368] (+0.000191334) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_recv_fds", prev_tid = 461, prev_prio = > 20, prev_state = 1, next_comm = "cons_poll_fds", next_tid = 462, > next_prio = 20 } > [2012-08-28 09:00:00.408304580] (+0.000407212) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "cons_poll_fds", prev_tid = 462, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:04.547552602] (+4.139248022) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:04.547632966] (+0.000080364) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:04.547736542] (+0.000103576) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:04.691537693] (+0.143801151) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "flush-0:14", tid = 389, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:04.691615027] (+0.000077334) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "flush-0:14", next_tid = 389, next_prio = 20 } > [2012-08-28 09:00:04.691705269] (+0.000090242) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "flush-0:14", prev_tid = 389, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:05.311535024] (+0.619829755) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:05.311613084] (+0.000078060) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:05.311882903] (+0.000269819) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:08.842504805] (+3.530621902) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:08.842583350] (+0.000078545) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:08.842664562] (+0.000081212) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:08.994502865] (+0.151838303) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:08.994581229] (+0.000078364) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:08.994670441] (+0.000089212) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:13.137477252] (+4.142806811) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:13.137557191] (+0.000079939) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:13.137656101] (+0.000098910) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:13.901469492] (+0.763813391) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:13.901548037] (+0.000078545) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:13.902056037] (+0.000508000) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:13.905507067] (+0.003451030) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "sync_supers", tid = 76, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:13.905589916] (+0.000082849) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "sync_supers", next_tid = 76, next_prio = 20 } > [2012-08-28 09:00:13.905661613] (+0.000071697) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sync_supers", prev_tid = 76, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:17.432437394] (+3.526775781) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:17.432515636] (+0.000078242) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:17.432599636] (+0.000084000) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:21.727404869] (+4.294805233) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:21.727485232] (+0.000080363) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:21.727587050] (+0.000101818) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:22.491405717] (+0.763818667) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:22.491484384] (+0.000078667) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:22.491873414] (+0.000389030) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:26.022372104] (+3.530498690) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:26.022450467] (+0.000078363) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:26.022531134] (+0.000080667) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:26.166376770] (+0.143845636) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "flush-0:14", tid = 389, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:26.166453558] (+0.000076788) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "flush-0:14", next_tid = 389, next_prio = 20 } > [2012-08-28 09:00:26.166543558] (+0.000090000) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "flush-0:14", prev_tid = 389, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:30.317347278] (+4.150803720) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:30.317427399] (+0.000080121) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:30.317528550] (+0.000101151) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:31.081338729] (+0.763810179) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:31.081417759] (+0.000079030) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:31.081881335] (+0.000463576) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:34.612306086] (+3.530424751) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:34.612384571] (+0.000078485) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:34.612465116] (+0.000080545) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:38.907282170] (+4.294817054) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:38.907362109] (+0.000079939) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:38.907461442] (+0.000099333) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:39.622817924] (+0.715356482) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "syslogd", tid = 383, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:39.622887803] (+0.000069879) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "syslogd", next_tid = 383, next_prio = 20 } > [2012-08-28 09:00:39.623148106] (+0.000260303) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "syslogd", prev_tid = 383, prev_prio = 20, > prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:39.671276530] (+0.048128424) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:39.671355864] (+0.000079334) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:39.671687682] (+0.000331818) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:39.675311742] (+0.003624060) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "sync_supers", tid = 76, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:39.675394409] (+0.000082667) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "sync_supers", next_tid = 76, next_prio = 20 } > [2012-08-28 09:00:39.675466288] (+0.000071879) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sync_supers", prev_tid = 76, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.202240735] (+3.526774447) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "kworker/0:1", tid = 99, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.202318857] (+0.000078122) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "kworker/0:1", next_tid = 99, next_prio = 20 } > [2012-08-28 09:00:43.202402250] (+0.000083393) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "kworker/0:1", prev_tid = 99, prev_prio = > 20, prev_state = 1, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.346233038] (+0.143830788) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "sleep", tid = 495, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.346330614] (+0.000097576) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "sleep", next_tid = 495, next_prio = 20 } > [2012-08-28 09:00:43.347314432] (+0.000983818) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "sh", tid = 388, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.347383644] (+0.000069212) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sleep", prev_tid = 495, prev_prio = 20, > prev_state = 64, next_comm = "sh", next_tid = 388, next_prio = 20 } > [2012-08-28 09:00:43.349999825] (+0.002616181) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sh", prev_tid = 388, prev_prio = 20, > prev_state = 1, next_comm = "sh", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.351726735] (+0.001726910) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sh", prev_tid = 497, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.352191159] (+0.000464424) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "sh", tid = 497, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.352283038] (+0.000091879) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "sh", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.353304492] (+0.001021454) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "sh", prev_tid = 497, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.353711038] (+0.000406546) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "sh", tid = 497, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.353802856] (+0.000091818) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "sh", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.357738795] (+0.003935939) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.358151644] (+0.000412849) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.358400311] (+0.000248667) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.358873644] (+0.000473333) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.359283644] (+0.000410000) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.359375523] (+0.000091879) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.360845098] (+0.001469575) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.361246492] (+0.000401394) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.361338977] (+0.000092485) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.361800674] (+0.000461697) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.362320250] (+0.000519576) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.362479463] (+0.000159213) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.363838068] (+0.001358605) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.364240553] (+0.000402485) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.364332553] (+0.000092000) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.364792189] (+0.000459636) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.365205038] (+0.000412849) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.365296916] (+0.000091878) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.366988129] (+0.001691213) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.367399705] (+0.000411576) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.367492371] (+0.000092666) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.368825704] (+0.001333333) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.369229038] (+0.000403334) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.369321280] (+0.000092242) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.370950675] (+0.001629395) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.371353220] (+0.000402545) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.371445462] (+0.000092242) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.380016917] (+0.008571455) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.380428129] (+0.000411212) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.380521159] (+0.000093030) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.381335462] (+0.000814303) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "mng_clients", tid = 452, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:43.381420795] (+0.000085333) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 0, next_comm = "mng_clients", next_tid = 452, next_prio = 20 } > [2012-08-28 09:00:43.381826250] (+0.000405455) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "mng_clients", prev_tid = 452, prev_prio = > 20, prev_state = 1, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.382007038] (+0.000180788) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "mng_clients", tid = 452, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:43.382081522] (+0.000074484) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 0, next_comm = "mng_clients", next_tid = 452, next_prio = 20 } > [2012-08-28 09:00:43.382528190] (+0.000446668) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "mng_clients", prev_tid = 452, prev_prio = > 20, prev_state = 1, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.383627341] (+0.001099151) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 130, next_comm = "swapper", next_tid = 0, next_prio = 20 } > [2012-08-28 09:00:43.384035280] (+0.000407939) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "lttng", tid = 497, prio = 120, success = 1, > target_cpu = 0 } > [2012-08-28 09:00:43.384128189] (+0.000092909) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "swapper", prev_tid = 0, prev_prio = 20, > prev_state = 0, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.385790613] (+0.001662424) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "mng_clients", tid = 452, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:43.385877704] (+0.000087091) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 0, next_comm = "mng_clients", next_tid = 452, next_prio = 20 } > [2012-08-28 09:00:43.386442190] (+0.000564486) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "mng_clients", prev_tid = 452, prev_prio = > 20, prev_state = 1, next_comm = "lttng", next_tid = 497, next_prio = 20 } > [2012-08-28 09:00:43.386978008] (+0.000535818) mgcoge3ne sched_wakeup: { > cpu_id = 0 }, { comm = "mng_clients", tid = 452, prio = 120, success = > 1, target_cpu = 0 } > [2012-08-28 09:00:43.387074129] (+0.000096121) mgcoge3ne sched_switch: { > cpu_id = 0 }, { prev_comm = "lttng", prev_tid = 497, prev_prio = 20, > prev_state = 0, next_comm = "mng_clients", next_tid = 452, next_prio = 20 } > > So again, I get these huge 3.3 (or so) seconds gaps, and a final > timestamp which is 43 seconds instead of the 10 seconds which actually > went by. > > BTW, I also tried with TMF under Eclipse, and here I get even weirder > results. Here's a table of correspondence (as there must be some occult > side to it!) for the timestamps as reported by babeltrace vs. Eclipse: > > [2012-08-28 09:00:00.348477307] -> 2033-05-07 16:49:22.035276997 > ... > [2012-08-28 09:00:00.408304580] -> 2033-05-07 16:49:22.112912324 > .... > [2012-08-28 09:00:04.691537693] -> 2039-11-23 09:06:50.748861638 > > Under Eclipse, the latest event of this 10-second trace lies sometime > within year 2124. > > No clue, anyone? > > Thank you, > Gerlando > > On 12/16/2011 11:15 AM, Gerlando Falauto wrote: >> Hi, >> >> I am trying to use LLTng 2.0 (since I have no alternative for a 3.0 >> kernel!) and I of course use babeltrace to read them. >> Problem is, I can't make any sense out of the timestamps, as I get >> something like (two CPU-hog bash processes): >> >> [1658841591493] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = >> 647, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 646, >> next_prio = 20 } >> [1658845590645] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = >> 646, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 647, >> next_prio = 20 } >> [1658849590039] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = >> 647, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 646, >> next_prio = 20 } >> [1658853591069] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = >> 646, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 647, >> next_prio = 20 } >> [1662152559395] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = >> 647, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 646, >> next_prio = 20 } >> [1662156587941] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = >> 646, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 647, >> next_prio = 20 } >> [1662160557698] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = >> 647, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 646, >> next_prio = 20 } >> [1662164557517] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = >> 646, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 647, >> next_prio = 20 } >> [....................................cut...........................] >> [1663140553029] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = >> 646, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 647, >> next_prio = 20 } >> [1663144555029] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = >> 647, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 646, >> next_prio = 20 } >> [1663148552423] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = >> 646, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 647, >> next_prio = 20 } >> [1666447519719] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = >> 647, prev_prio = 20, prev_state = 0, next_comm = "bash", next_tid = 646, >> next_prio = 20 } >> [1666451561355] sched_switch: { 0 }, { prev_comm = "bash", prev_tid = >> 646, prev_prio = 20, prev_state = 0, next_comm = "kworker/0:1", next_tid >> = 94, next_prio = 20 } >> [1666451863719] sched_switch: { 0 }, { prev_comm = "kworker/0:1", >> prev_tid = 94, prev_prio = 20, prev_state = 1, next_comm = "bash", >> next_tid = 647, next_prio = 20 } >> >> Which made me think timestamps were expressed in nanoseconds, as I am >> running this on 4ms-tick PowerPC (CONFIG_HZ=250), and the difference >> between timestamps is approximately 4,000,000. >> However, I get these huge 3.3 second gaps between for instance between >> 1658853591069 and 1662152559395. >> Curiously enough, these leap 3.3 seconds occur exactly (well, within 4ms >> precision...) every second. Kind of like the "seconds" part of the >> timestamp is calculated on a wrong-base arithmetics or so.... >> This also makes sense as the seconds' portion of the timestamp is >> essentially 4.3 times bigger than the uptime (and any time stretch is >> 4.3 times bigger than reality, for that matter). >> >> Any suggestions? >> >> Thanks! >> Gerlando >> >> P.S. This also happens without my hack for flight recording mode... From henrik at fiktivkod.org Wed Aug 29 14:59:46 2012 From: henrik at fiktivkod.org (Henrik Hautakoski) Date: Wed, 29 Aug 2012 20:59:46 +0200 Subject: [lttng-dev] More LTTng problems In-Reply-To: <50324CD4.5050309@efficios.com> References: <20120817150806.GA15919@Krystal> <502E6022.60208@efficios.com> <50324CD4.5050309@efficios.com> Message-ID: This problem has been fixed. It was a linux-header version missmatch that caused the consumerd to not being able to connect to the error socket. On Mon, Aug 20, 2012 at 4:42 PM, David Goulet wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Hi, > > This is pretty odd... it seems the execl() of the consumer failed... > > We'll have to confirm this with the perror code I guess. Can you apply > this patch and re-run again. > > diff --git a/src/bin/lttng-sessiond/main.c b/src/bin/lttng-sessiond/main.c > index c952fc0..d3c378b 100644 > - --- a/src/bin/lttng-sessiond/main.c > +++ b/src/bin/lttng-sessiond/main.c > @@ -1782,6 +1782,7 @@ static pid_t spawn_consumerd(struct > consumer_data *consumer_data) > free(tmpnew); > } > if (ret) { > + PERROR("execl 32-bit consumer"); > goto error; > } > break; > > If you want, to speed things up, you can come on OFTC irc server on > #lttng and we'll be able to assist you in real time to find this problem. > > Cheers > David > > Henrik Hautakoski: >> Ok so I attached the output from lttng-sessiond (it's large). >> >> the commands was: >> >> # lttng-sessiond --vvv --no-kernel --verbose-consumer >> >> # lttng list -u >> >> # lttng create mysession >> >> # lttng enable-event -a -u >> >> Nothing shows up in dmesg. >> >> On Fri, Aug 17, 2012 at 5:15 PM, David Goulet >> wrote: Hi, >> >> Actually, this error means that a problem occurred *during* the 32 >> bit consumer spawning process... It seems the binary was found but >> failed to exec it. >> >> Running the session daemon (lttng-sessiond) with options "-vvv >> --verbose-consumer" will help us pin down the problem. >> >> Send me back the output, I'll be able to tell you what's going on >> (and also the series of command you guys did before the >> enable-event). >> >> Last thing, make sure the lttng-consumerd did not segfault (dmesg) >> or any other lttng-tools component. >> >> Thanks! David >> >> Mathieu Desnoyers: >>>>> It looks like you did not build a 32-bit consumer. I'm ccing >>>>> David Goulet, maintainer of lttng-tools. >>>>> >>>>> Thanks, >>>>> >>>>> Mathieu >>>>> >>>>> * Henrik Hautakoski (henrik at fiktivkod.org) wrote: >>>>>> Hi. got lttng working now. but when given the following >>>>>> command, we get an error. >>>>>> >>>>>> $ lttng enable-event -a -u Error: Events: 32-bit UST >>>>>> consumer start failed (channel channel0, session >>>>>> mysession) >>>>>> >>>>>> Could not find any information about this error massage. do >>>>>> you have any ideas? >>>>>> >>>>>> -- Henrik Hautakoski henrik at fiktivkod.org >>>>> >> >> >> > -----BEGIN PGP SIGNATURE----- > > iQEcBAEBCgAGBQJQMkzRAAoJEELoaioR9I02cTIIAIxjeagCL1uvHB4WT/SGAdOz > 5eIDXhRyjto+7af4VkjibAozT28XZpnJweWJKkeGJjtRLD/1KFMq06QWREUO6sSz > NqkU+i+5FHTSCAK4cRS80e0AcqmOCCAo0zfeLpurNCfeu654rHgzk+03XIj+Zn3a > QhY7Uw9S9SuJ0NPPP6f50bmlsdh6VGLHmJQMi2nNJQZPFiuwLHF9V7mUOO66ZWqF > ikqN5NGDMsmjOoY7XPwcalF7BK1vq9bMv+f9zyWImFV/ky8iHw0d1Cemy9O1cWw1 > cbM1wyEqzfDV5QIoFqa0ZKIsbuUzuwy4/bkk7pTrqOaTDLuwOOoqYs108yeJgUg= > =mDFB > -----END PGP SIGNATURE----- -- Henrik Hautakoski henrik at fiktivkod.org From sbohrer at rgmadvisors.com Wed Aug 29 16:39:59 2012 From: sbohrer at rgmadvisors.com (Shawn Bohrer) Date: Wed, 29 Aug 2012 15:39:59 -0500 Subject: [lttng-dev] [PATCH] lttng-gen-tp: Allow generation from templates in subdirectories Message-ID: <1346272799-23183-1-git-send-email-sbohrer@rgmadvisors.com> If a template is located in a subdirectory as is common in large software projects then the include guard will contain forward slash characters which are not allowed. This replaces those characters with an underscore. Additional the include guards previously started with an underscore and capital letter which is reserved thus this additionally removes the leading underscore. Fixes: #298 Signed-off-by: Shawn Bohrer --- tools/lttng-gen-tp | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/tools/lttng-gen-tp b/tools/lttng-gen-tp index d54559a..8cd3191 100755 --- a/tools/lttng-gen-tp +++ b/tools/lttng-gen-tp @@ -61,7 +61,7 @@ extern "C"{{ def write(self): outputFile = open(self.outputFilename,"w") - includeGuard = "_"+self.outputFilename.upper().replace(".","_") + includeGuard = self.outputFilename.upper().replace(".","_").replace("/","_") outputFile.write(HeaderFile.HEADER_TPL.format(providerName=self.template.domain, includeGuard = includeGuard, -- 1.7.7.6 -- --------------------------------------------------------------- This email, along with any attachments, is confidential. If you believe you received this message in error, please contact the sender immediately and delete all copies of the message. Thank you. From jsilver.pmc at gmail.com Fri Aug 24 13:57:15 2012 From: jsilver.pmc at gmail.com (jsilver.pmc at gmail.com) Date: Fri, 24 Aug 2012 11:57:15 -0600 Subject: [lttng-dev] Managing trace sizes Message-ID: <13959c7e325.1340654510569746171.-9152529687053830431@gmail.com> Are there any ways currently, or planned, or other suggestions, to address these within LTTng? - compression of trace data - rolling or rotating trace files - limiting the amount of disk space LTTng will use Thanks, Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.khouzam at ericsson.com Thu Aug 30 11:11:06 2012 From: matthew.khouzam at ericsson.com (Matthew Khouzam) Date: Thu, 30 Aug 2012 11:11:06 -0400 Subject: [lttng-dev] [babeltrace] About the type read and write In-Reply-To: References: Message-ID: <503F828A.1070601@ericsson.com> Hi, just a heads up, there will be a java version of a ctf writer coming out _soonish_ within in a couple months. On 12-08-23 05:35 AM, Hui Zhu wrote: > Hi, > > I just tried to use libbabeltrace to read and write the CTF file. But > I cannot find the api to read some type of CTF for example > CTF_TYPE_ENUM, CTF_TYPE_STRUCT. > And I cannot find the api file to write to the CTF file. (I check the > code of babeltrace-log and found that it use the function that is not > API) > > Could you help me with it? > > Thanks, > Hui > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev From gerlando.falauto at keymile.com Thu Aug 30 11:36:32 2012 From: gerlando.falauto at keymile.com (Gerlando Falauto) Date: Thu, 30 Aug 2012 17:36:32 +0200 Subject: [lttng-dev] [PATCH] fix timestamps on architectures without CONFIG_KTIME_SCALAR Message-ID: <1346340992-27307-1-git-send-email-gerlando.falauto@keymile.com> trace_clock_monotonic_wrapper() should return a u64 representing the number of nanoseconds since system startup. ktime_get() provides that value directly within its .tv64 field only on those architectures defining CONFIG_KTIME_SCALAR, whereas in all other cases (e.g. PowerPC) a ktime_to_ns() conversion (which translates back to .tv64 when CONFIG_KTIME_SCALAR is defined) becomes necessary. Signed-off-by: Gerlando Falauto --- wrapper/trace-clock.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/wrapper/trace-clock.h b/wrapper/trace-clock.h index 6ea9e81..bced61c 100644 --- a/wrapper/trace-clock.h +++ b/wrapper/trace-clock.h @@ -46,7 +46,7 @@ static inline u64 trace_clock_monotonic_wrapper(void) return (u64) -EIO; ktime = ktime_get(); - return (u64) ktime.tv64; + return ktime_to_ns(ktime); } static inline u32 trace_clock_read32(void) -- 1.7.10.1 From Dhiliphen.Arulraj at siemens.com Thu Aug 30 10:50:41 2012 From: Dhiliphen.Arulraj at siemens.com (Arulraj, Dhiliphen IN BLR STS) Date: Thu, 30 Aug 2012 20:20:41 +0530 Subject: [lttng-dev] LTTng run issues Message-ID: <18BECF85FEE8A1469B4AD1584AE2FF200AE61D8EC7@INBLRK77M2MSX.in002.siemens.net> Hello , I am new to LTTng usage. When trying to run lttng 2.0.4 for tracing my kernel I get the following issue - root at am335x-evm:/media/mmcblk0p1/lttng_bin/bin# ./arm-arago-linux-gnueabi-lttng start [ 1775.934997] LTTng: Failure to write metadata to buffers (timeout) Error: Starting kernel trace failed I checked the session daemon for startup issues and got the following - ./arm-arago-linux-gnueabi-lttng- sessiond -vvv --consumerd32-path /media/mmcblk0p1/lttng_bin/lib/lttng/libexec root at am335x-evm:/media/mmcblk0p1/lttng_bin/bin# ./arm-arago-linux-gnueabi-lttng- sessiond -vvv --consumerd32-path /media/mmcblk0p1/lttng_bin/lib/lttng/libexec DEBUG3: Creating LTTng run directory: /var/run/lttng [in create_lttng_rundir() a t main.c:4317] DEBUG2: Kernel consumer err path: /var/run/lttng/kconsumerd/error [in main() at main.c:4560] DEBUG2: Kernel consumer cmd path: /var/run/lttng/kconsumerd/command [in main() a t main.c:4562] DEBUG1: Client socket path /var/run/lttng/client-lttng-sessiond [in main() at ma in.c:4609] DEBUG1: Application socket path /var/run/lttng/apps-lttng-sessiond [in main() at main.c:4610] DEBUG1: LTTng run directory path: /var/run/lttng [in main() at main.c:4611] DEBUG2: UST consumer 32 bits err path: /var/run/lttng/ustconsumerd32/error [in m ain() at main.c:4620] DEBUG2: UST consumer 32 bits cmd path: /var/run/lttng/ustconsumerd32/command [in main() at main.c:4622] DEBUG2: UST consumer 64 bits err path: /var/run/lttng/ustconsumerd64/error [in m ain() at main.c:4631] DEBUG2: UST consumer 64 bits cmd path: /var/run/lttng/ustconsumerd64/command [in main() at main.c:4633] DEBUG2: Creating consumer directory: /var/run/lttng/kconsumerd [in set_consumer_ sockets() at main.c:4359] DEBUG1: Modprobe successfully lttng-tracer [in modprobe_lttng_control() at modpr obe.c:163] DEBUG2: Kernel tracer version validated (major version 2) [in kernel_validate_ve rsion() at kernel.c:675] DEBUG1: Modprobe successfully lttng-ftrace [in modprobe_lttng_data() at modprobe .c:199] DEBUG1: Modprobe successfully lttng-kprobes [in modprobe_lttng_data() at modprob e.c:199] DEBUG1: Modprobe successfully lttng-kretprobes [in modprobe_lttng_data() at modp robe.c:199] DEBUG1: Modprobe successfully lttng-lib-ring-buffer [in modprobe_lttng_data() at modprobe.c:199] DEBUG1: Modprobe successfully lttng-ring-buffer-client-discard [in modprobe_lttn g_data() at modprobe.c:199] DEBUG1: Modprobe successfully lttng-ring-buffer-client-overwrite [in modprobe_lt tng_data() at modprobe.c:199] DEBUG1: Modprobe successfully lttng-ring-buffer-metadata-client [in modprobe_ltt ng_data() at modprobe.c:199] DEBUG1: Modprobe successfully lttng-ring-buffer-client-mmap-discard [in modprobe _lttng_data() at modprobe.c:199] DEBUG1: Modprobe successfully lttng-ring-buffer-client-mmap-overwrite [in modpro be_lttng_data() at modprobe.c:199] DEBUG1: Modprobe successfully lttng-ring-buffer-metadata-mmap-client [in modprob e_lttng_data() at modprobe.c:199] DEBUG1: Modprobe successfully lttng-probe-lttng [in modprobe_lttng_data() at mod probe.c:199] DEBUG1: Modprobe successfully lttng-types [in modprobe_lttng_data() at modprobe. c:199] DEBUG1: Modprobe successfully lttng-probe-block [in modprobe_lttng_data() at mod probe.c:199] DEBUG1: Modprobe successfully lttng-probe-irq [in modprobe_lttng_data() at modpr obe.c:199] DEBUG1: Modprobe successfully lttng-probe-kvm [in modprobe_lttng_data() at modpr obe.c:199] DEBUG1: Modprobe successfully lttng-probe-sched [in modprobe_lttng_data() at mod probe.c:199] DEBUG1: Modprobe successfully lttng-probe-signal [in modprobe_lttng_data() at mo dprobe.c:199] DEBUG1: Modprobe successfully lttng-probe-statedump [in modprobe_lttng_data() at modprobe.c:199] DEBUG1: Modprobe successfully lttng-probe-timer [in modprobe_lttng_data() at mod probe.c:199] DEBUG1: Kernel tracer fd 6 [in init_kernel_tracer() at main.c:1887] DEBUG2: Creating consumer directory: /var/run/lttng/ustconsumerd64 [in set_consu mer_sockets() at main.c:4359] DEBUG2: Creating consumer directory: /var/run/lttng/ustconsumerd32 [in set_consu mer_sockets() at main.c:4359] DEBUG1: Signal handler set for SIGTERM, SIGPIPE and SIGINT [in set_signal_handle r() at main.c:4451] Warning: No tracing group detected DEBUG1: epoll set max size is 95800 [in compat_epoll_set_max_size() at compat-ep oll.c:224] DEBUG1: Thread manage kernel started [in thread_manage_kernel() at main.c:876] DEBUG1: Updating kernel poll set [in update_kernel_poll() at main.c:748] DEBUG1: Thread kernel polling on 2 fds [in thread_manage_kernel() at main.c:905] DEBUG1: [thread] Manage application started [in thread_manage_apps() at main.c:1 179] DEBUG1: Apps thread polling on 2 fds [in thread_manage_apps() at main.c:1200] DEBUG1: [thread] Manage application registration started [in thread_registration _apps() at main.c:1392] DEBUG1: Notifying applications of session daemon state: 1 [in notify_ust_apps() at main.c:687] DEBUG1: Got the wait shm fd 18 [in get_wait_shm() at shm.c:117] DEBUG1: Futex wait update active 1 [in futex_wait_update() at futex.c:62] DEBUG1: Accepting application registration [in thread_registration_apps() at mai n.c:1423] DEBUG1: [thread] Dispatch UST command started [in thread_dispatch_ust_registrati on() at main.c:1324] DEBUG1: Futex n to 1 prepare done [in futex_nto1_prepare() at futex.c:73] DEBUG1: Woken up but nothing in the UST command queue [in thread_dispatch_ust_re gistration() at main.c:1334] DEBUG1: [thread] Manage client started [in thread_manage_clients() at main.c:379 4] DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:3826] >From the above it seems I don't have any problems. Yes I get that failure when running lttng. I'm running this on a TI AM335x EVM with the lttng modules compiled against the arago 05.04 linux kernel(Version 3.2) and everything compiled using the arago toolchain. Thanks in advance Dhiliphen From matthew.khouzam at ericsson.com Fri Aug 31 10:34:48 2012 From: matthew.khouzam at ericsson.com (Matthew Khouzam) Date: Fri, 31 Aug 2012 10:34:48 -0400 Subject: [lttng-dev] Questions about CTF format In-Reply-To: References: <20120705142052.GA23162@Krystal> Message-ID: <5040CB88.3090203@ericsson.com> On 12-07-05 10:49 AM, Diego Dompe wrote: > Hi Mathieu, > > Thanks for the help. Here is my list of details: > > - Clocks: the spec doesn't explain properly that timestamps are an > offset from the base time of the clock they refer to. Since I was > using 64bit timestamps I somehow assumed that I was using absolute > timestamps from the epoch (although the spec doesn't says it either). This is acceptable AFAIK, just to be clear, you can use arbitrary time origins, that means epoch is a valid origin too. You can also use a scaling factor, just be careful applying it since a double has 53 bits and thus you may lose precision. > > - I saw that the lttng-generated traces for metadata are always a > multiple of 4k in size (at least the ones I generate for either kernel > or user space). I can't find where in the spec it mentions > requirements regarding metadata packet padding. I was generating > metadata packets that ended up right after my TSDL and eclipse wasn't > happy about it (although I didn't try babeltrace). Could you send the trace please, I can look into it. > Also I found that the lttng-generated traces have a "empty" metadata > packet after the metadata containing the TSDL, I didn't find either > any documentation regarding this. > > Regards, > > Diego > > On Thu, Jul 5, 2012 at 8:20 AM, Mathieu Desnoyers > > wrote: > > * Diego Dompe (ddompe at gmail.com ) wrote: > > Hi, > > > > I'm developing a custom tracer for an embedded product that will > generate > > CTF format. I was able to generate generic traces that can be > interpreted > > properly with babeltrace (but not with eclipse, I already file a > bug for > > that), but I found the CTF specification lacking in some aspects > (I had to > > peek into lttng-generated CTF traces to figure out some > details). I was > > wondering what is the proper mailing list to clear my questions > and provide > > feedback on the CTF specification for improvement in the areas > where the > > documentation is not detailed yet. I don't see any CTF-specific > mailing > > list, it's OK to discuss it here? Or maybe directly with a > developer(s)? > > Hi Diego, > > Yes, this mailing list would be the proper place, along maybe with > adding the MCA tiwg mailing list in CC, which I'm doing here. > > Thanks, > > Mathieu > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.khouzam at ericsson.com Fri Aug 31 11:21:58 2012 From: matthew.khouzam at ericsson.com (Matthew Khouzam) Date: Fri, 31 Aug 2012 11:21:58 -0400 Subject: [lttng-dev] Feature request: remove ust 0.x tests from lttng-ust Message-ID: <5040D696.9010301@ericsson.com> Hello tracing giants, I feel the tests that are no longer valid will just confuse users. Matthew.