[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Masami Hiramatsu writes:

> On Mon, 05 Sep 2016 16:37:01 +0200
> Lluís Vilanova <vilanova at ac.upc.edu> wrote:

>> Stefan Hajnoczi writes:
>> 
>> > On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
>> >> >> Also, I'm still not sure how to interact with QEMU's monitor interface from
>> >> >> within the probe code (probes execute in kernel mode, including "guru mode"
>> >> >> code).
>> >> 
>> >> > When SystemTap is used the QEMU monitor interface does nothing.
>> >> 
>> >> That's not what I've experienced. I was able to use a stap script to change the
>> >> tracing state of events:
>> >> 
>> >> #!/usr/bin/env stap
>> >> 
>> >> %{
>> >> #include </home/lluis/Projects/qemu-dbi-test/test.h>
>> >> %}
>> >> 
>> >> function event:long(cpu:long, addr:long, info:long)
>> >> %{
>> >> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>> >> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>> >> STAP_RETURN(0);
>> >> %}
>> >> 
>> >> probe begin {
>> >> printf("hello\n")
>> >> }
>> >> probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
>> >> {
>> >> printf("%x %d %d\n", $arg1, $arg2, $arg3)
>> >> event($arg1, $arg2, $arg3)
>> >> exit()
>> >> }
>> >> 
>> >> The only caveat is that you must pass the "-g" argument to stap.
>> >> 
>> >> Also, for some reason the printf in the probe always prints zeros, no matter
>> >> what the actual event receives (I've debugged QEMU down to the call to the
>> >> auto-generated stap functions). Could this be an error in systemtap?
>> 
>> > It's strange that arguments do not have valid values.  Debugging the
>> > stap functions is the next step if you want to figure out what happened.
>> > I've never had this issue before so maybe something with Debian
>> > SystemTap userspace probes is broken.
>> 
>> I already debugged it, to the point where QEMU executes the trap injected by
>> systemtap, and the register values that were supposed to hold the arguments are
>> correct.
>> 
>> I suppose that if you execute the stap script I pasted it will show the proper
>> values. Then it's definitely a problem with Debian's userspace probes.

> Would you have tried to update your kernel to mainline and tested it ?

I've compiled the tarball for 4.8-rc5 (.config from "make localmodconfig") and a
printf of the probe arguments still shows zeroes. Also, I've had to add a small
patch to [1] to properly lock/unlock inodes in this kernel version (using
inode_lock/unlock instead of mutex_lock/unlock on inode->i_mutex).

[1] /usr/share/systemtap/runtime/transport/transport.c


> If it occurs, you also should try to use a raw uprobe via ftrace(uprobe_events)
> and perftools.
> If you have the latest perf (maybe you'll need checkout the latest tip tree),
> you can use SDT as below (currently it doesn't support args, so you'll need
> debuginfo.)

> # perf buildid-cache --add ./install/vanilla/bin/qemu-system-i386
> # perf probe -x ./install/vanilla/bin/qemu-system-i386 -a 'guest_mem_before_exec $vars'

> And you'll see new event is registered which can be traced by ftrace or perf.

It does show something (I'm interested in stap probe "guest_hypertrace", raised
on function "trace_guest_hypertrace"), but is incorrect:

    50.00%  (55e36d5ee32b) __cpu=0x55e370446fd0 arg1=0xcafe
    50.00%  (55e36d5ee32b) __cpu=0x7f2ee0a10b20 arg1=0x198

My test app calls "trace_guest_hypertrace" twice, always with the same "__cpu"
and "arg1" argument values. Just in case, this is runnign a QEMU compiled with
"-O0".

Running "record" multiple times shows different "random" values on the
arguments, and keeps changing which of the two trace elements shows incorrect
values.


Thanks,
  Lluis