[lttng-dev] Tracepoints overhead in x86_64

[Mathieu Desnoyers]

----- Original Message -----

> From: "Gianluca Borello" <g.borello at gmail.com>
> To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
> Cc: lttng-dev at lists.lttng.org
> Sent: Saturday, August 31, 2013 6:16:35 PM
> Subject: Re: [lttng-dev] Tracepoints overhead in x86_64

> On Aug 31, 2013 1:21 PM, "Mathieu Desnoyers" < mathieu.desnoyers at efficios.com
> > wrote:
> >
> > My recommendation would be to implement a dummy trace_sys_enter and
> > trace_sys_exit, and then benchmark the overhead of enabling system call
> > tracing (this will set TIF_SYSCALL_TRACEPOINT in every thread). So at
> > least it would help you pinpoint where most of the overhead comes from.
> >

> Yes, I already did that before writing the patch I attached in the previous
> mail, and I can confirm that simply setting TIF_SYSCALL_TRACEPOINT in all
> threads (without registering any tracepoint) shows the same overhead.
Good to know! 

> > Indeed, if you come up with ways to shrink the overhead of this "slow
> > path", I'm sure the LTTng community would gladly welcome this. Of
> > course, these changes would have to be proposed to the Linux kernel
> > community. You will need to keep in mind that they will frown upon
> > pretty much _any_ slowdown of the fast path (common case, no tracing)
> > for improving the speed of the slow (uncommon) case.
> >

> That might be interesting.
> I guess I have a couple of questions:

> 1) Where do you think I could find an answer to the question "can
> trace_sys_enter be safely called from the fast path?". In my tests it worked
> just fine, but I'm concerned that calling the tracepoint without using the
> IRET slow path could lead to the potential DoS the commit was fixing.
I'm concerned that it could not be called safely from fast path. You might want to ask the x86 Linux kernel maintainers about this, CCing linux-kernel at vger.kernel.org 

> 2) How do you think the kernel community would react to a patch that adds the
> tracepoint call to the fast path? Basically, if it works, I'd check the
> TIF_SYSCALL_TRACEPOINT directly in the fast path and, if set, I would
> proceed in saving the registers and calling the probes, so in the common
> case there would be just one more check against the current thread's flags.
As soon as they can notice any overhead on some crazy workloads, they will probably refuse the change. You should try, and ask them which workloads they care about, and try to prove them that it does not have any perceivable impact on performances. However, success is far from guaranteed. 

Another approach you could take is to wrap this call in a macro, that expands either to the tracing call if CONFIG_TRACE_SYSCALL_FASTPATH is set, or to nothing otherwise (default to nothing). Therefore, if yourself, or a distribution, care about tracing speed more than usual, they can tune their kernel at compile time. 

Thanks, 

Mathieu 

> Thank you again!
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20131003/a0098dda/attachment-0001.html>