[lttng-dev] Processing large trace files

[Shehab Elsayed]

I haven't tried the C API yet.

A smaller trace of about 2.7GB took +5 hrs before I had to kill the
process.

As for trace cutting, that sounds promising. How broken is it in babeltrace
2? I mean, can I use it to some extent or is it not functioning at all?

Thanks!

Shehab Y. Elsayed, MSc.
PhD Student
The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
University of Toronto
E-mail: shehabyomn at gmail.com
<https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>

On Tue, Jul 10, 2018 at 9:37 AM, Jonathan Rajotte-Julien <
jonathan.rajotte-julien at efficios.com> wrote:

> Hi Shehab,
>
> On Mon, Jul 09, 2018 at 06:48:05PM -0400, Shehab Elsayed wrote:
> > Hello All,
> >
> > I was wondering if anyone had any suggestions on how to speedup trace
> > processing. Right now I am using babeltrace in python to read the traces
> > and process them. Some of the traces I am dealing with reach up to 10GB.
>
> Well Babeltrace also provide a C API. You might get some performance gains
> there.
>
> Depending of the data you extract you might be able to split the 10 GB
> trace
> into smaller chunk and parallelize your analysis. Babeltrace 2 will support
> trace cutting, it seems that it is broken for now. Babeltrace 1 does not
> support
> trace cutting.
>
> You might want to look at ways to reduce the size of your traces using
> tracing
> mechanism such as the snapshot feature and enabling/recording only
> pertinent
> event if it is not already the case.
>
> Could you share with us the time it take for analyzing your bigger trace?
>
> Cheers
>
> >
> > I read the trace, extract the info I need and write them to a csv gzipped
> > file (all in python). I fill up a list first with all extracted info for
> > each tracked event and then write once after the script has gone through
> > all events in the trace.
> >
> > Thanks,
> > Shehab
>
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev at lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
> --
> Jonathan Rajotte-Julien
> EfficiOS
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20180710/2b00af36/attachment.html>