<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi all,</p>
<p><br>
</p>
<p>Here included is a conversation I had on the kvm mailing list
regarding how to link traces from host and guests when tracing
virtual environments. Following Stefan's last suggestions, here is
a suggestion on what can be done in lttng-modules, it's not
intrusive and does not require modifications to the kernel. Let me
know what you think:</p>
<p><br>
</p>
<p>* If we want to trace guests _and_ host and linek them together,
we require to set the VM's -uuid parameter. I think virsh does
that by default anyway.</p>
<p> - We can then obtain this uuid in lttng via
dmi_get_system_info(DMI_PRODUCT_UUID). We can add it to a
statedump event, or in the 'env' section of the metadata (which
might be easier to manager and does not require activating an
additional event).</p>
<p>* Guest statedump: on host, we can access the guests by parsing
the struct dentry* kvm_debugfs_dir for entries with the
<pid>-xx format. We can then parse the
/proc/<pid>/cmdline to obtain the uuid and associate it to
the guest.</p>
<p>* Guest lifetime during the trace: We can add a lttng-module that
adds a udev rule to intercept VM creation/destruction. We can
trace those events, adding the pid/uuid in the fields.</p>
<p><br>
</p>
<p>What do you think of this approach?</p>
<p><br>
</p>
<p>Thanks,</p>
<p>Geneviève<br>
</p>
<br>
<div class="moz-forward-container"><br>
-------- Forwarded Message --------
<table class="moz-email-headers-table" border="0" cellspacing="0"
cellpadding="0">
<tbody>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">Subject:
</th>
<td>Re: Shared value between host and guests</td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">Date: </th>
<td>Fri, 26 Jan 2018 14:03:48 +0000</td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">From: </th>
<td>Stefan Hajnoczi <a class="moz-txt-link-rfc2396E" href="mailto:stefanha@gmail.com"><stefanha@gmail.com></a></td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">To: </th>
<td>Geneviève Bastien <a class="moz-txt-link-rfc2396E" href="mailto:gbastien@versatic.net"><gbastien@versatic.net></a></td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">CC: </th>
<td><a class="moz-txt-link-abbreviated" href="mailto:kvm@vger.kernel.org">kvm@vger.kernel.org</a></td>
</tr>
</tbody>
</table>
<br>
<br>
<pre>On Thu, Jan 25, 2018 at 10:44:12AM -0500, Geneviève Bastien wrote:
> On 2018-01-25 04:29 AM, Stefan Hajnoczi wrote:
> > On Wed, Jan 24, 2018 at 10:56:26AM -0500, Geneviève Bastien wrote:
> >> Thanks for the hint, I didn't know about those. But the uevents are only
> >> for the host right? There is no specific value in debugfs for guest.
> > I'm not sure I understand the question.
> >
> > Uevent are emitted on the host. Given the PID field, it should be
> > possible to correlate them to a specific guest (e.g. by looking at
> > information from the user or VM management tools, or simply by looking
> > at /proc/$PID/cmdline for the QEMU -name argument that can be used to
> > identify guests on the host side).
> >
> > Stefan
> With the information from the host and those uevents, we can indeed
> easily figure out in our traces which processes are associated with a
> guest and get its command line and a lot more information.
>
> The missing piece is in the guest trace. For example, we have lttng
> traces taken on the guest and the host. We may have multiple guests and
> multiple hosts to trace. It's easy to know which traces are from hosts
> from the events, but which are from guests? And which guest trace goes
> with which of the kvm processes on which host? The guest trace contains
> no information that can be linked to the host.
>
> I was thinking that the guest could do a hypercall to the host after
> bootup to share some unique information, for instance its bootid, that
> the host could store somewhere. That information would be available in
> the traces so that we can easily associate the guest with its host
> process and the states of its virtual CPUs to that of the corresponding
> threads, and much more.
>
> I hope this describes our need a little better.
The QEMU -uuid <uuid> option makes a UUID available to the guest via
SMBIOS and fw_cfg on x86. Inside the guest you can print it like this:
# dmidecode -s system-uuid
01020304-0506-0708-090A-0B0C0D0E0F10
Maybe you can base the guest trace filename off the UUID:
guest-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
On the host you can either find the UUIDs in the libvirt domain XML:
# virsh dump my-domain
<domain ...>
<uuid>0102030405060708090a0b0c0d0e0f10</uuid>
...
Or you can use the kvm.ko uevent to find the QEMU PID and then check
/proc/$PID/cmdline for the uuid. The UUID can also be fetched via the
query-uuid QMP command if you don't want to search /proc/$PID/cmdline
for -uuid <uuid>.
Maybe you can base the host trace filename off the UUID too:
host-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
Then the ask of correlating traces becomes pretty easy for
post-processing scripts since they can look at the filenames.
Stefan
</pre>
</div>
</body>
</html>