[lttng-dev] lost trace events

[Mathieu Desnoyers]

Hi Alok, 

With a snapshot trace, you can end up with some per-cpu buffers that contain information 
going further back in time compared to other per-cpu buffers. This depends on the level of 
system activity and tracing throughput for each CPU. 

The situation you experience can very well be triggered by having the begin event on one CPU, 
being migrated to another CPU, and having the end event on another CPU. If either the 
source or destination CPU has been more active than the other, you may find that you 
have only the begin or the end event in your trace. 

This is why we implemented the "stream intersection" mode in babeltrace. It ensures that 
babeltrace cuts away trace data belonging to time spans that only appear in some of 
the streams. It is not however activated by default. 

To try it out, you can do this: 

babeltrace --stream-intersection path/to/trace 

Does it help ? 

Thanks, 

Mathieu 

----- On Nov 7, 2018, at 7:12 PM, Alok Priyadarshi <alokpr at gmail.com> wrote: 

> A custom trace event class emits "begin" and "end" events in its constructor and
> destructor respectively. So I do not think this is due to conditional path.

> Sequence of commands to capture a snapshot are:
> lttng create --snapshot
> lttng enable-event --userspace --all
> lttng add-context --userspace -t vpid -t vtid
> lttng start
> lttng stop
> lttng snapshot record

> We create a new session for each snapshot to make sure that the buffer does not
> have any left-over events from the previous session. But I am curious if lttng
> shares buffers between sessions or re-uses buffers from an old session without
> flushing?

> Note that I am using the default LTTNG_BUFFER_PER_UID mode. All processes for
> our application run under one UID, and only one tracing session is active at a
> time.

> On Wed, Nov 7, 2018 at 12:09 PM Jonathan Rajotte-Julien < [
> mailto:jonathan.rajotte-julien at efficios.com |
> jonathan.rajotte-julien at efficios.com ] > wrote:

>> Hi Alok,

>> On Wed, Nov 07, 2018 at 11:53:25AM -0800, Alok Priyadarshi wrote:
>> > Hi Jonathan,

>> > Thanks for your response.

>> > We are tracing function scopes. Each scope emits two events - begin and
>> > end. We noticed that some begin events did not have corresponding end
>> > events in the trace.

>> Seems like a solid ground as long as the corresponding ending event is not
>> inside any conditional paths. I'm sure you already validated that but we never
>> know.


>> > I have the trace with this problem available. Would that provide any clue?

>> It could help but I presume it might contains confidential or sensible
>> information. Let's define the scenario first and we will go back to actual
>> trace data if needed.

>> > If not, I will try to extract a small reproducer.

>> That would be best. In the meantime, could you provide the exact lttng commands
>> used to setup everything and describe subsequent lttng commands used for
>> snapshot etc.

>> You can omit event name or replace them with dummy name if necessary.

>> Cheers

>> --
>> Jonathan Rajotte-Julien
>> EfficiOS

> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20181108/cdeaf2a1/attachment.html>