[lttng-dev] RFC: lttng-module: how to track the link between host and guest traces
gbastien+lttng at versatic.net
Mon Jan 29 16:32:17 UTC 2018
Here included is a conversation I had on the kvm mailing list regarding
how to link traces from host and guests when tracing virtual
environments. Following Stefan's last suggestions, here is a suggestion
on what can be done in lttng-modules, it's not intrusive and does not
require modifications to the kernel. Let me know what you think:
* If we want to trace guests _and_ host and linek them together, we
require to set the VM's -uuid parameter. I think virsh does that by
- We can then obtain this uuid in lttng via
dmi_get_system_info(DMI_PRODUCT_UUID). We can add it to a statedump
event, or in the 'env' section of the metadata (which might be easier to
manager and does not require activating an additional event).
* Guest statedump: on host, we can access the guests by parsing the
struct dentry* kvm_debugfs_dir for entries with the <pid>-xx format. We
can then parse the /proc/<pid>/cmdline to obtain the uuid and associate
it to the guest.
* Guest lifetime during the trace: We can add a lttng-module that adds a
udev rule to intercept VM creation/destruction. We can trace those
events, adding the pid/uuid in the fields.
What do you think of this approach?
-------- Forwarded Message --------
Subject: Re: Shared value between host and guests
Date: Fri, 26 Jan 2018 14:03:48 +0000
From: Stefan Hajnoczi <stefanha at gmail.com>
To: Geneviève Bastien <gbastien at versatic.net>
CC: kvm at vger.kernel.org
On Thu, Jan 25, 2018 at 10:44:12AM -0500, Geneviève Bastien wrote:
> On 2018-01-25 04:29 AM, Stefan Hajnoczi wrote:
> > On Wed, Jan 24, 2018 at 10:56:26AM -0500, Geneviève Bastien wrote:
> >> Thanks for the hint, I didn't know about those. But the uevents are only
> >> for the host right? There is no specific value in debugfs for guest.
> > I'm not sure I understand the question.
> > Uevent are emitted on the host. Given the PID field, it should be
> > possible to correlate them to a specific guest (e.g. by looking at
> > information from the user or VM management tools, or simply by looking
> > at /proc/$PID/cmdline for the QEMU -name argument that can be used to
> > identify guests on the host side).
> > Stefan
> With the information from the host and those uevents, we can indeed
> easily figure out in our traces which processes are associated with a
> guest and get its command line and a lot more information.
> The missing piece is in the guest trace. For example, we have lttng
> traces taken on the guest and the host. We may have multiple guests and
> multiple hosts to trace. It's easy to know which traces are from hosts
> from the events, but which are from guests? And which guest trace goes
> with which of the kvm processes on which host? The guest trace contains
> no information that can be linked to the host.
> I was thinking that the guest could do a hypercall to the host after
> bootup to share some unique information, for instance its bootid, that
> the host could store somewhere. That information would be available in
> the traces so that we can easily associate the guest with its host
> process and the states of its virtual CPUs to that of the corresponding
> threads, and much more.
> I hope this describes our need a little better.
The QEMU -uuid <uuid> option makes a UUID available to the guest via
SMBIOS and fw_cfg on x86. Inside the guest you can print it like this:
# dmidecode -s system-uuid
Maybe you can base the guest trace filename off the UUID:
On the host you can either find the UUIDs in the libvirt domain XML:
# virsh dump my-domain
Or you can use the kvm.ko uevent to find the QEMU PID and then check
/proc/$PID/cmdline for the uuid. The UUID can also be fetched via the
query-uuid QMP command if you don't want to search /proc/$PID/cmdline
for -uuid <uuid>.
Maybe you can base the host trace filename off the UUID too:
Then the ask of correlating traces becomes pretty easy for
post-processing scripts since they can look at the filenames.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the lttng-dev