[lttng-dev] Merge two LTTng CTF traces into one

Fri Jan 9 12:08:31 EST 2015

On 01/08/2015 12:28 PM, Divya Vyas wrote:
>
>
> On Jan 8, 2015 10:12 PM, "Geneviève Bastien" <gbastien at versatic.net 
> <mailto:gbastien at versatic.net>> wrote:
> >
> > Hi Divya,
> >
> > I'm putting you back on the lttng-dev mailing list, please reply to 
> that list in the future so other people can answer, because I think 
> that trace synchronization is not exactly what you are looking for here...
>
> I am sorry I clicked reply instead of reply to all.
>
> I have a host and on top of that I have multiple guests. Suppose like 
> from 9 to 9:30 I want to analyze the host information as well as guest 
> information in single view and in a synchronized way. As the guest 
> time is different from host . I need the guest traces referenced at 
> host time . and I tracecompass I can see the host traces as well as 
> guest traces in a synchronized way.
>
Then your traces are taken at the same time.  The ideal in your case is 
to have your machines communicate, at least a few TCP packets during the 
duration of the trace so they can be synchronized. If you trace a 
virtual machine guest and host for 30 minutes, chances are the time 
drift between the clocks is important so simple time offsetting at a 
point that you know (either the beginnings of the trace or a precise 
point in the trace) will correlate justly the traces at this point but 
the further you go from there, the less accurate the time will be. Trace 
synchronization accounts for the time drift so if you have accurate 
synchronization, then the same accuracy applies to the whole trace.

Also, I may point you to a recently contributed Virtual Machine 
Analysis. Part of it is in master, some still on gerrit (the view for 
instance). The documentation is still unexistent. It works with the 
Qemu/KVM virtualizer, and needs some custom compilation of lttng-modules 
on both guest and host. It is still experimental. It allows to associate 
a host process with a guest CPU so that you can see on the guest whether 
the CPU was actually running or preempted on the host. If you're using 
Qemu/KVM and are interested in this analysis, I can work on the 
documentation sooner than later.

> I hope now I gave the clear picture of what I am doing. Please let me 
> know if u need some explanation.
> >
> >> I tried the sessions at same time and communication events in 
> between and it worked :)
> >> I am getting quality as accurate and I can see both traces merged .
> >
> > Excellent!
> >
> >>
> >> Now I come at my second  point i.e taking traces from lttng 
> sessions at different machines and at different time and may be 
> communication events or might be not.
> >
> > If the sessions are taken are different moments, they can _never_ 
> synchronize.
>
> Yes I think you told that they can be analyzed by applying offset but 
> not synchronized.
>
> Trace synchronization first matches events such that one single event 
> from a trace (for example "sending a packet") can be correlated to 
> exactly one event in the second trace (the reception of that packet). 
> If that is the case, the first event can have a timestamp of now and 
> the second be timed to January 1st 1970, trace synchronization will 
> work and the 2 traces will have a common time reference after 
> synchronization.
>
> What is the criteria of timestamps to be synchronized? How much two 
> timestamps matched for synchronization
>
There is no criteria for the timestamp. Matching events is the important 
part of the process. If you are interested in trace synchronization, I 
invite you to read this article by Masoume Jabbarifar [1]. It is at the 
base of what is implemented in Trace Compass.

[1] http://link.springer.com/article/10.1007/s10922-014-9331-7#page-1

Geneviève

> >
> >
> >>
> >> So In my opinion I need to apply time offset on one of the trace to 
> match the beginning so that they can be synchronized .
> >>
> >> So like first trace timestamp starts at 07:13:59 .499 222 870 and 
> other starts at 11:21:35.537 682 885 . Then I can apply timeoffset ( 
> How much offset ? In apply time offset advanced option If reference 
> time is 7:13.... then target time will be 11:21:35... it will 
> calculate the offset ) on one of the trace to match traces .
> >>
> >> Is it possible to synchronize the traces then ? Actually I tried 
> these steps but not getting the synchronized traces but then both 
> traces beginning is exactly same ( obviously it will as applied by the 
> offset)
> >>
> >> Please let me know how can I synchronize the traces from different 
> machines and at different time
> >
> > I'm not sure what your end goal is exactly. Do you want to compare 
> trace executions? I am cc'ing François Doray who is currently working 
> on a trace comparison utility. Here's a presentation he did recently: 
> http://ahls.dorsal.polymtl.ca/system/files/Comparison%20of%20Traces%20to%20Diagnose%20Performance%20Variations.pdf
> >
> > His work is still experimental and available on github. If that's 
> what you're interested in, he might give you some information on how 
> to use it.
> >
> > Otherwise, please explain what you are trying to achieve.
> >
> > Geneviève
> >
> >
> >
> >>
> >>
> >>
> >> On Thu, Jan 8, 2015 at 12:39 AM, Geneviève Bastien 
> <gbastien at versatic.net <mailto:gbastien at versatic.net>> wrote:
> >>>
> >>> On 15-01-07 11:32 AM, Divya Vyas wrote:
> >>>>
> >>>> Hi ,
> >>>>
> >>>> I have some comments below and thanks a lot for your help
> >>>>
> >>>> On Jan 7, 2015 7:55 PM, "Geneviève Bastien" 
> <gbastien at versatic.net <mailto:gbastien at versatic.net>> wrote:
> >>>> >
> >>>> > Hi Divya,
> >>>> >
> >>>> >
> >>>> > On 15-01-07 01:24 AM, Divya Vyas wrote:
> >>>> >>
> >>>> >>
> >>>> >> Hi Genevieve,
> >>>> >>
> >>>> >> My target to is merge the traces of two different machines 
> taken on different time or same time .
> >>>> >
> >>>> > Trace synchronization works by first matching events 
> corresponding to the sending and the reception of a TCP packet from 
> both traces, and then taking those matched events and computing a 
> formula that will transform timestamps from one trace to the same 
> clock as another trace. So only traces taken at the same time (meaning 
> together, not the same clock time of course) will work.
> >>>>
> >>>> I am not getting this last point , where the traces are taken 
> together. You mean that we start lttng session on two machines at 
> around same time and then during  session use some communication 
> events . Then take the traces into tracecompass for synchronization in 
> experiment.  I am not getting in which cases time offset is needed?
> >>>
> >>>
> >>> Yes, two sessions on two machines at the same time and they 
> communicate during those sessions. Then you can synchronize traces. 
> The following traces are used in unit tests to test trace 
> synchronization if you wish to see what it looks like: 
> http://archive.eclipse.org/tracecompass/test-traces/ctf/synctraces.tar.gz
> >>>
> >>>> In what traces I need to apply time offset so that their 
> beginning matches. Is it applicable on the above case which I am 
> thinking above.
> >>>
> >>>
> >>> You never _need_ to apply time offset. It is if you have traces 
> that do not contain communication events, and you want one of them to 
> be offsetted by a certain time. For example, you have a trace taken 
> GST-5 and one at GST, you cannot synchronize them because they don't 
> have communication events but you want to analyze them with the same 
> time reference, you would offset the one at GST-5 by +5 hours.
> >>>
> >>> Or if you have a custom event that you know happened at the same 
> time on both traces, but synchronization does not support that event, 
> then you can offset one trace so that both events from both traces 
> have the same timestamp.
> >>>
> >>> Time offsetting is a manual process by the user, while trace 
> synchronization is an automatic analysis using pre-defined events to 
> compute a formula to transform timestamps from one trace to the same 
> clock as the other.
> >>>
> >>>> >
> >>>> >
> >>>> >>
> >>>> >> Actually synchronization was not working on different machine 
> so I tried on same machine.  But now I got it correctly I think so :)
> >>>> >
> >>>> > Indeed, you have the right events now, with the net_dev_queue 
> and netif_receive_skb. So you should be OK. But I *think* this method 
> works only when on a LAN. If your computers communicate through 
> internet, you're better off with the inet_sock_local_[in|out] method
> >>>>
> >>>> They are connected through LAN only.
> >>>>
> >>>> >
> >>>> >>
> >>>> >> For different time we need to apply offset so that their 
> beginning matches. I think I need to do this .
> >>>> >>
> >>>> >> I tried applying offset on eclipse luna ( I added linux tools 
> in install new software)   but I am not getting this option " Apply 
> time offset ".
> >>>> >
> >>>> > It is in master only so far, so you should use tracecompass to 
> have this option.
> >>>> >
> >>>> >>
> >>>> >> I tried on tracecompass also, where I am getting time offset 
> option but the synchronization view is empty . The window is not 
> showing anything. The tracecompass version is 0.1.0
> >>>> >
> >>>> > The time offset does not involve synchronization at all, it is 
> just an offset, so you would not see anything in the synchronization 
> view. But in the "Properties" view, when you select the trace, the 
> time offset is displayed.
> >>>>
> >>>> I meant to say that I tried synchronizing the traces in 
> tracecompass without time offset . The synchronizing view is empty.
> >>>> It should show at least quality absent etc.
> >>>
> >>>
> >>> Quality absent would show only if the traces are from different 
> machines. If you are trying to synchronize traces from the same 
> machine, that view will stay empty.
> >>>
> >>> Geneviève
> >>>
> >>>
> >>
> >
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20150109/a304ee8b/attachment.html>