[ltt-dev] Using LTTng to trace a host and its virtual machines

Thu Apr 16 12:32:23 EDT 2009

Hello Zengshan,

There are two things here.

1. Tracing a host and a virtual machine simultaneously
2. Following a packet through those machines

Tracing a host and a virtual machine is not too difficult. It's normally
just a matter of recording a trace on the host and recording traces in
the virtual machines, all at the same time. Then you open them both in
LTTV and see them merged. But at least two things can cause problems.
The first is that you need a common time source between all these
machines. On a single core x86, you can use the TSC, which is supposed
to be available in the virtual machines too. But you must make sure the
virtualization system does not change the TSC before it presents it to
the virtual machine. For example, it could save it when the virtual
machine is scheduled out and restore it to that value when it's
scheduled back in, in other words "virtualizing" the TSC. I think kvm
does this or some other form of mangling by default. You need to do
whatever needs to be done on your specific architecture to get a common
time source. The other thing that can cause problems is that your
virtual cpus might think they are not running exactly at the same speed
as host cpus. Frequencies are determined when the system boots. The
frequency is written in the trace and is used to convert TSCs in
seconds. If the host and VMs have the same time source but think they
are running at slightly different frequencies, the merge will not work
properly in LTTV. I commited a patch in lttv a while ago so that the
frequency of the first trace is used for all the other traces that are
loaded. So this shouldn't be a problem anymore.

The other part of your question concerns following of packets through
the virtualization border. For that, you will likely need to add
supplementary instrumentation. Perhaps you will need to instrument kvm
itself. One way to do it would be assigning identifiers to packets and
to refer to those identifiers in events.

I hope this helps a bit.

pmf

zengshan227 wrote:
>  
> hi Pierre-Marc :
>    Mathieu Desnoyers (CC-ed in this mail) who is so kind to tell me that
> you have  done some work to make sure LTTV can show traces
> from both the host kernel and a virtual machine together, since I am now
> pursing tracing network I/O path under the environment of KVM,  what I
> want to know exactly is the ingressing path of a packet from the host
> machine to the virtual machine , Can your work allow the kind of analysis?
>  
> Thank you very much ~
> Best Regards!
>  
> 2009-04-10
> ------------------------------------------------------------------------
> zengshan227
> ------------------------------------------------------------------------
> *发件人：* zengshan227
> *发送时间：* 2009-04-10 10:34:08
> *收件人：* pierre-marc.fournier at polymtl.ca
> *抄送：* Mathieu Desnoyers
> *主题：* Re: Re: LTTng 0.115 for Linux 2.6.29
>  
> hi Pierre-Marc :
>    I know from Mathieu Desnoyers (CC-ed in this mail) that you
> have  done some work to make sure LTTV can show traces
> from both the host kernel and a virtual machine together, since I am now
> pursing tracing network I/O path under the environment of KVM,  what I
> want to know exactly is the ingressing path of a packet from the host
> machine to the virtual machine , Can your work allow the kind of analysis?
>  
> Thank you very much ~
> Best Regards!
>  
>  
> 2009-04-10
> ------------------------------------------------------------------------
> zengshan227
> ------------------------------------------------------------------------
> *发件人：* Mathieu Desnoyers
> *发送时间：* 2009-04-10  02:56:09
> *收件人：* 曾珊; pierre-marc.fournier at polymtl.ca
> *抄送：* ltt-dev at lists.casi.polymtl.ca; linux-kernel at vger.kernel.org
> *主题：* Re: LTTng 0.115 for Linux 2.6.29
> * 曾珊 (zengshan227 at gmail.com) wrote:
>> can ltt been used in network IO tracing under the environment of KVM?
>> 
>> hi:
>>   what I meant path tracing is that we want to get the ingressing path of a
>> packet under the environment of KVM (Kernel based Virtual Machine ), what we
>> want to know exactly is the ingressing path of a packet from the host
>> machine to the virtual machine ,
>> ps: KVM used Qemu to process its network I/O , which is actually a process
>> in the host OS.
>>   Any ideas?
>>   Thank you very much~
>  
> Yes, Pierre-Marc has done some work to make sure LTTV can show traces
> from both the host kernel and a virtual machine together and therefore
> allow the kind of analysis you are looking for. I am putting him in CC
> so he can explain how to use his work.
>  
> Mathieu
>  
>> 
>> 2009/3/24 Mathieu Desnoyers  <compudj at krystal.dyndns.org >
>> 
>>  > Hi,
>>  >
>>  > I just released LTTng 0.115 for Linux 2.6.29. Happy tracing. :)
>>  >
>>  > It is, as always, available both as a patchset and as a git tree. See
>>  > http://www.lttng.org for details <http://www.lttng.org for details>.
>>  >
>>  > Mathieu
>>  >
>>  >
>>  > --
>>  > Mathieu Desnoyers
>> 
>> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
>>  > --
>> 
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>  > the body of a message to majordomo at vger.kernel.org
>>  > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>  > Please read the FAQ at  http://www.tux.org/lkml/
>>  >
>  
> -- 
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68