[lttng-dev] RFC: lttng-module: how to track the link between host and guest traces

[Genevieve Bastien]

Hi all,


Here included is a conversation I had on the kvm mailing list regarding
how to link traces from host and guests when tracing virtual
environments. Following Stefan's last suggestions, here is a suggestion
on what can be done in lttng-modules, it's not intrusive and does not
require modifications to the kernel. Let me know what you think:


* If we want to trace guests _and_ host and linek them together, we
require to set the VM's -uuid parameter. I think virsh does that by
default anyway.

    - We can then obtain this uuid in lttng via
dmi_get_system_info(DMI_PRODUCT_UUID). We can add it to a statedump
event, or in the 'env' section of the metadata (which might be easier to
manager and does not require activating an additional event).

* Guest statedump: on host, we can access the guests by parsing the
struct dentry* kvm_debugfs_dir for entries with the <pid>-xx format. We
can then parse the /proc/<pid>/cmdline to obtain the uuid and associate
it to the guest.

* Guest lifetime during the trace: We can add a lttng-module that adds a
udev rule to intercept VM creation/destruction. We can trace those
events, adding the pid/uuid in the fields.


What do you think of this approach?


Thanks,

Geneviève



-------- Forwarded Message --------
Subject: 	Re: Shared value between host and guests
Date: 	Fri, 26 Jan 2018 14:03:48 +0000
From: 	Stefan Hajnoczi <stefanha at gmail.com>
To: 	Geneviève Bastien <gbastien at versatic.net>
CC: 	kvm at vger.kernel.org



On Thu, Jan 25, 2018 at 10:44:12AM -0500, Geneviève Bastien wrote:
> On 2018-01-25 04:29 AM, Stefan Hajnoczi wrote:
> > On Wed, Jan 24, 2018 at 10:56:26AM -0500, Geneviève Bastien wrote:
> >> Thanks for the hint, I didn't know about those. But the uevents are only
> >> for the host right? There is no specific value in debugfs for guest.
> > I'm not sure I understand the question.
> >
> > Uevent are emitted on the host.  Given the PID field, it should be
> > possible to correlate them to a specific guest (e.g. by looking at
> > information from the user or VM management tools, or simply by looking
> > at /proc/$PID/cmdline for the QEMU -name argument that can be used to
> > identify guests on the host side).
> >
> > Stefan
> With the information from the host and those uevents, we can indeed
> easily figure out in our traces which processes are associated with a
> guest and get its command line and a lot more information.
> 
> The missing piece is in the guest trace. For example, we have lttng
> traces taken on the guest and the host. We may have multiple guests and
> multiple hosts to trace. It's easy to know which traces are from hosts
> from the events, but which are from guests? And which guest trace goes
> with which of the kvm processes on which host? The guest trace contains
> no information that can be linked to the host.
> 
> I was thinking that the guest could do a hypercall to the host after
> bootup to share some unique information, for instance its bootid, that
> the host could store somewhere. That information would be available in
> the traces so that we can easily associate the guest with its host
> process and the states of its virtual CPUs to that of the corresponding
> threads, and much more.
> 
> I hope this describes our need a little better.

The QEMU -uuid <uuid> option makes a UUID available to the guest via
SMBIOS and fw_cfg on x86.  Inside the guest you can print it like this:

  # dmidecode -s system-uuid
  01020304-0506-0708-090A-0B0C0D0E0F10

Maybe you can base the guest trace filename off the UUID:

  guest-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat

On the host you can either find the UUIDs in the libvirt domain XML:

  # virsh dump my-domain
  <domain ...>
    <uuid>0102030405060708090a0b0c0d0e0f10</uuid>
    ...

Or you can use the kvm.ko uevent to find the QEMU PID and then check
/proc/$PID/cmdline for the uuid.  The UUID can also be fetched via the
query-uuid QMP command if you don't want to search /proc/$PID/cmdline
for -uuid <uuid>.

Maybe you can base the host trace filename off the UUID too:

  host-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat

Then the ask of correlating traces becomes pretty easy for
post-processing scripts since they can look at the filenames.

Stefan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20180129/c8368647/attachment.html>