[lttng-dev] lost trace events

[Alok Priyadarshi]

Jonathan: The system is x86_64, 16 cores, no cpu affinity set. Not too sure
about load yet.

Mathieu: The explanation makes sense. Running babeltrace with
--stream-intersection flag did filter out a lot of events. However it is
hard to verify the filtered result because we do not yet know how to read
the text format. Does the babeltrace python API support stream intersection
by any chance?


On Thu, Nov 8, 2018 at 9:52 AM Mathieu Desnoyers <
mathieu.desnoyers at efficios.com> wrote:

> Hi Alok,
>
> With a snapshot trace, you can end up with some per-cpu buffers that
> contain information
> going further back in time compared to other per-cpu buffers. This depends
> on the level of
> system activity and tracing throughput for each CPU.
>
> The situation you experience can very well be triggered by having the
> begin event on one CPU,
> being migrated to another CPU, and having the end event on another CPU. If
> either the
> source or destination CPU has been more active than the other, you may
> find that you
> have only the begin or the end event in your trace.
>
> This is why we implemented the "stream intersection" mode in babeltrace.
> It ensures that
> babeltrace cuts away trace data belonging to time spans that only appear
> in some of
> the streams. It is not however activated by default.
>
> To try it out, you can do this:
>
> babeltrace --stream-intersection path/to/trace
>
> Does it help ?
>
> Thanks,
>
> Mathieu
>
> ----- On Nov 7, 2018, at 7:12 PM, Alok Priyadarshi <alokpr at gmail.com>
> wrote:
>
> A custom trace event class emits "begin" and "end" events in its
> constructor and destructor respectively. So I do not think this is due to
> conditional path.
>
> Sequence of commands to capture a snapshot are:
> lttng create --snapshot
> lttng enable-event --userspace --all
> lttng add-context --userspace -t vpid -t vtid
> lttng start
> lttng stop
> lttng snapshot record
>
> We create a new session for each snapshot to make sure that the buffer
> does not have any left-over events from the previous session. But I am
> curious if lttng shares buffers between sessions or re-uses buffers from an
> old session without flushing?
>
> Note that I am using the default LTTNG_BUFFER_PER_UID mode. All processes
> for our application run under one UID, and only one tracing session is
> active at a time.
>
> On Wed, Nov 7, 2018 at 12:09 PM Jonathan Rajotte-Julien <
> jonathan.rajotte-julien at efficios.com> wrote:
>
>> Hi Alok,
>>
>> On Wed, Nov 07, 2018 at 11:53:25AM -0800, Alok Priyadarshi wrote:
>> > Hi Jonathan,
>> >
>> > Thanks for your response.
>> >
>> > We are tracing function scopes. Each scope emits two events - begin and
>> > end. We noticed that some begin events did not have corresponding end
>> > events in the trace.
>>
>> Seems like a solid ground as long as the corresponding ending event is not
>> inside any conditional paths. I'm sure you already validated that but we
>> never
>> know.
>>
>> >
>> > I have the trace with this problem available. Would that provide any
>> clue?
>>
>> It could help but I presume it might contains confidential or sensible
>> information. Let's define the scenario first and we will go back to actual
>> trace data if needed.
>>
>> > If not, I will try to extract a small reproducer.
>>
>> That would be best. In the meantime, could you provide the exact lttng
>> commands
>> used to setup everything and describe subsequent lttng commands used for
>> snapshot etc.
>>
>> You can omit event name or replace them with dummy name if necessary.
>>
>> Cheers
>>
>> --
>> Jonathan Rajotte-Julien
>> EfficiOS
>>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20181108/d5be3bb0/attachment-0001.html>