<font size=2 face="sans-serif">Hi,</font>
<br>
<br><font size=2 face="sans-serif">Access beyond end of file seems to be
the issue IIUC.</font>
<br>
<br><font size=2 face="sans-serif">Let me try to give as much context as
I can and answer your questions one by one:</font>
<br>
<br><font size=2 face="sans-serif"><b><u>Versions</u></b></font>
<br>
<br><font size=2 face="sans-serif">lttng-ust 2.2.0, lttng-tools 2.2.0,
babeltrace - I used git hash cc26a15ac355d886bb507955fdaec9b218024713 with
a patch on babeltrace.i.in.</font>
<br>
<br><font size=2 face="sans-serif">[I cannot supply the patch as this requires
my employer's (IBM) approval - I am in the process of obtaining a permanent
contribution approval, but until then...]</font>
<br>
<br><font size=2 face="sans-serif">However, I can tell you exactly what
it does:</font>
<br><font size=2 face="sans-serif">1. Adds a typedef for 'ssize_t'</font>
<br><font size=2 face="sans-serif">2. Added a call to _bt_ctf_get_decl_from_def()
around the argument supplied to _bt_ctf_get_int_len()</font>
<br><font size=2 face="sans-serif">3. Changed all use of {} in string ".format()"
calls to add numbers (e.g. {0} instead of {}) - as this doesn't work for
Python 2.6.</font>
<br>
<br><font size=2 face="sans-serif">I am waiting for the contribution approval
and then I'll submit a patch.</font>
<br>
<br><font size=2 face="sans-serif">To provide a trace sample, I'll need
a special approval, which could take very long.</font>
<br>
<br><font size=2 face="sans-serif"><b><u>System Setup</u></b></font>
<br>
<br><font size=2 face="sans-serif">Processor: Xeon-family 6-cores (12 if
you count HT)</font>
<br><font size=2 face="sans-serif">RAM: 24GB of RAM, page size is 4KB.</font>
<br><font size=2 face="sans-serif">OS: Linux 2.6.32.12-211_11_4_0 based
on an MCP (IBM's SuSE derivative) build.</font>
<br>
<br><font size=2 face="sans-serif"><b><u>Tracing Setup</u></b></font>
<br>
<br><font size=2 face="sans-serif">Same system used to generate, collect
and read the traces.</font>
<br><font size=2 face="sans-serif">100% user-space.</font>
<br>
<br><font size=2 face="sans-serif"><b><u>Triggering</u></b></font>
<br>
<br><font size=2 face="sans-serif">This is not readily reproducible. Some
context:</font>
<br>
<br><font size=2 face="sans-serif">We have an in-house trace mechanism
including a code instrumentor and a curses-based viewer (actually urwid+python).</font>
<br><font size=2 face="sans-serif">My current project is to embed LTTng
logging and trace capabilites into several open-source projects, because
we couldn't use our in-house tracer without having GPL issues.</font>
<br><font size=2 face="sans-serif">I have written a back-end that enables
our in-house viewer to display Lttng traces, by linking with libbabeltrace
(allowed by its LGPL license).</font>
<br>
<br><font size=2 face="sans-serif">As far as I can tell SIGBUS happens
to me when I was doing a search. This literally reads the events in sequence
and tries to compare some fields against some criteria.</font>
<br>
<br><font size=2 face="sans-serif">And here's the main reason why I think
this is happening - I was adding a trace (using add_traces_recursive(trace_dir,'ctf'))
while the session is active (i.e. not stopped).</font>
<br>
<br><font size=2 face="sans-serif">I can see 2 problematic scenarios here:</font>
<br>
<br><font size=2 face="sans-serif"><u>Scenario 1</u></font>
<br>
<br><font size=2 face="sans-serif">When I add the trace to the context
babeltrace performs mmap() according to the size of the file(s) at that
moment.</font>
<br><font size=2 face="sans-serif">The files keep growing.</font>
<br><font size=2 face="sans-serif">When I try to access new packets they
may lie beyond the end of the mmap'ed region. I think this should lead
to SIGSEGV and not to SIGBUS.</font>
<br>
<br><font size=2 face="sans-serif"><u>Scenario 2</u></font>
<br>
<br><font size=2 face="sans-serif">Due to buffering issues, I could be
having a race with the session daemon, where my code reads the beginning
of a packet that was already written to the file, and try to access the
contents, which were not yet written. Again - could happen around the end
of the file, with the session still active.</font>
<br>
<br><font size=2 face="sans-serif">Amit</font>
<br>
<br><font size=2 color=#000080 face="sans-serif">Amit Margalit</font>
<br><font size=2 color=#808000 face="sans-serif">IBM XIV </font><font size=2 face="sans-serif">-
<i>Storage Reinvented</i></font>
<br><font size=2 face="sans-serif">XIV-NAS Development Team</font>
<br><font size=2 face="sans-serif">Tel. 03</font><font size=2 face="Arial">-689-7774</font>
<br><font size=2 face="Arial">Fax. 03-689-7230</font>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Mathieu Desnoyers <mathieu.desnoyers@efficios.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">Amit Margalit/Israel/IBM@IBMIL</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Cc:
</font><font size=1 face="sans-serif">lttng-dev@lists.lttng.org</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">07/24/2013 04:20 PM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [lttng-dev]
Getting SIGBUS in babeltrace - occasionally</font>
<br>
<hr noshade>
<br>
<br>
<br><tt><font size=2>* Mathieu Desnoyers (mathieu.desnoyers@efficios.com)
wrote:<br>
> * Amit Margalit (AMITM@il.ibm.com) wrote:<br>
> > Hi,<br>
> > <br>
> > I am getting occasional SIGBUS inside _aligned_integer_read()
in <br>
> > ./formats/ctf/types/integer.c<br>
> > <br>
> > Here is the backtrace, maybe someone could take a look and suggest
<br>
> > something:<br>
> <br>
> It looks like you're hitting the internal checks for overflow on the<br>
> buffer boundaries.<br>
<br>
Please forget about my previous email, I'm utterly wrong. That should<br>
teach me to reply before my first morning coffee ;)<br>
<br>
We indeed have internal checks for overflows in babeltrace, but those<br>
are not triggering SIGBUS ever, they return failure gracefully.<br>
<br>
What seems to happen in your case is documented here:<br>
<br>
mmap(2) manpage:<br>
<br>
SIGBUS Attempted access to a portion of the buffer
that does not corre$B!>(B<br>
spond to the
file (for example, beyond the end of the file,<br>
including the case where
another process has truncated the<br>
file).<br>
<br>
I'm curious to learn which versions of babeltrace/ust/modules you are<br>
using, if you modified them (and how), and if you can get us a sample<br>
trace that triggers the issue.<br>
<br>
Also, a bit of context about your setup: is this 32-bit or 64-bit<br>
user-space, which OS.<br>
<br>
Hrm.<br>
<br>
By any chance, is it possible that you record your trace on a platform<br>
with 4kB pages, and run babeltrace on it on a platform having 64kB<br>
pages? Everything points me to include/babeltrace/mmap-align.h<br>
mmap_align_addr(). We might be mmaping beyond the end of file in this<br>
case.<br>
<br>
Thanks!<br>
<br>
Mathieu<br>
<br>
> <br>
> On which babeltrace version can you reproduce it ? Did you do any<br>
> modification to babeltrace on your own on top ?<br>
> <br>
> Are you getting this with a lttng-ust or lttng-modules trace ? If
yes,<br>
> which version of the tool has generated the trace ?<br>
> <br>
> Can you provide a sample trace that triggers this issue ?<br>
> <br>
> Can you give us the detailed sequence of steps you use to reproduce
?<br>
> <br>
> Thanks,<br>
> <br>
> Mathieu<br>
> <br>
> > #0 _aligned_integer_read (definition=0x954320, ppos=0x106e028)
at <br>
> > integer.c:81<br>
> > #1 ctf_integer_read (ppos=0x106e028, definition=0x954320)
at <br>
> > integer.c:224<br>
> > #2 0x00007ffff070e36e in generic_rw (definition=<optimized
out>, <br>
> > pos=0x106e028) at ../include/babeltrace/types.h:133<br>
> > #3 bt_struct_rw (ppos=0x106e028, definition=0x9542e0) at
struct.c:56<br>
> > #4 0x00007ffff13f7fda in generic_rw (definition=<optimized
out>, <br>
> > pos=0x106e028) at ../../include/babeltrace/types.h:133<br>
> > #5 ctf_packet_seek (stream_pos=0x106e028, index=<optimized
out>, <br>
> > whence=<optimized out>) at ctf.c:860<br>
> > #6 0x00007ffff070a3f1 in seek_file_stream_by_timestamp
<br>
> > (cfs=cfs@entry=0x106cf90, timestamp=timestamp@entry=1374636821498972926)
<br>
> > at iterator.c:141<br>
> > #7 0x00007ffff070aa96 in seek_ctf_trace_by_timestamp <br>
> > (stream_heap=0x176d310, timestamp=1374636821498972926, tin=<optimized
<br>
> > out>) at iterator.c:188<br>
> > #8 bt_iter_set_pos (iter=iter@entry=0xfdafa0, iter_pos=0x2bf59f0)
at <br>
> > iterator.c:439<br>
> > #9 0x00007ffff1624f01 in _wrap__bt_iter_set_pos (self=<optimized
out>, <br>
> > args=<optimized out>) at babeltrace_wrap.c:3805<br>
> > #10 0x00007ffff7b28c0c in call_function (oparg=<optimized
out>, <br>
> > pp_stack=<optimized out>) at Python/ceval.c:3679<br>
> > #11 PyEval_EvalFrameEx (f=0x1c243c0, throwflag=<optimized
out>) at <br>
> > Python/ceval.c:2370<br>
> > #12 0x00007ffff7b2e0d2 in PyEval_EvalCodeEx (co=0xa8e5d0, <br>
> > globals=<optimized out>, locals=<optimized out>,
args=0x2088cc8, <br>
> > argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)<br>
> > at Python/ceval.c:2942<br>
> > .<br>
> > .<br>
> > .<br>
> > <br>
> > Any suggestions are welcome.<br>
> > <br>
> > Amit Margalit<br>
> > IBM XIV - Storage Reinvented<br>
> > XIV-NAS Development Team<br>
> > Tel. 03-689-7774<br>
> > Fax. 03-689-7230<br>
> > _______________________________________________<br>
> > lttng-dev mailing list<br>
> > lttng-dev@lists.lttng.org<br>
> > </font></tt><a href="http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev"><tt><font size=2>http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</font></tt></a><tt><font size=2><br>
> <br>
> <br>
> -- <br>
> Mathieu Desnoyers<br>
> EfficiOS Inc.<br>
> </font></tt><a href=http://www.efficios.com/><tt><font size=2>http://www.efficios.com</font></tt></a><tt><font size=2><br>
<br>
-- <br>
Mathieu Desnoyers<br>
EfficiOS Inc.<br>
</font></tt><a href=http://www.efficios.com/><tt><font size=2>http://www.efficios.com</font></tt></a><tt><font size=2><br>
<br>
</font></tt>
<br>