<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Gianluca Borello" <g.borello@gmail.com><br><b>To: </b>"Mathieu Desnoyers" <mathieu.desnoyers@efficios.com><br><b>Cc: </b>lttng-dev@lists.lttng.org<br><b>Sent: </b>Saturday, August 31, 2013 6:16:35 PM<br><b>Subject: </b>Re: [lttng-dev] Tracepoints overhead in x86_64<br><div><br></div><p dir="ltr"><br>
On Aug 31, 2013 1:21 PM, "Mathieu Desnoyers" <<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>> wrote:<br>
><br>
> My recommendation would be to implement a dummy trace_sys_enter and<br>
> trace_sys_exit, and then benchmark the overhead of enabling system call<br>
> tracing (this will set TIF_SYSCALL_TRACEPOINT in every thread). So at<br>
> least it would help you pinpoint where most of the overhead comes from.<br>
></p><p dir="ltr">Yes, I already did that before writing the patch I attached in the previous mail, and I can confirm that simply setting TIF_SYSCALL_TRACEPOINT in all threads (without registering any tracepoint) shows the same overhead.</p></blockquote><div><br></div><div>Good to know!<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><p dir="ltr">> Indeed, if you come up with ways to shrink the overhead of this "slow<br>
> path", I'm sure the LTTng community would gladly welcome this. Of<br>
> course, these changes would have to be proposed to the Linux kernel<br>
> community. You will need to keep in mind that they will frown upon<br>
> pretty much _any_ slowdown of the fast path (common case, no tracing)<br>
> for improving the speed of the slow (uncommon) case.<br>
></p><p dir="ltr">That might be interesting.<br>
I guess I have a couple of questions:</p><p dir="ltr">1) Where do you think I could find an answer to the question "can trace_sys_enter be safely called from the fast path?". In my tests it worked just fine, but I'm concerned that calling the tracepoint without using the IRET slow path could lead to the potential DoS the commit was fixing.</p></blockquote><div><br></div><div>I'm concerned that it could not be called safely from fast path. You might want to ask the x86 Linux kernel maintainers about this, CCing linux-kernel@vger.kernel.org<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><p dir="ltr">2) How do you think the kernel community would react to a patch that adds the tracepoint call to the fast path? Basically, if it works, I'd check the TIF_SYSCALL_TRACEPOINT directly in the fast path and, if set, I would proceed in saving the registers and calling the probes, so in the common case there would be just one more check against the current thread's flags.</p></blockquote><div><br></div><div>As soon as they can notice any overhead on some crazy workloads, they will probably refuse the change. You should try, and ask them which workloads they care about, and try to prove them that it does not have any perceivable impact on performances. However, success is far from guaranteed.<br></div><div><br></div><div>Another approach you could take is to wrap this call in a macro, that expands either to the tracing call if CONFIG_TRACE_SYSCALL_FASTPATH is set, or to nothing otherwise (default to nothing). Therefore, if yourself, or a distribution, care about tracing speed more than usual, they can tune their kernel at compile time.<br></div><div><br></div><div>Thanks,<br></div><div><br></div><div>Mathieu<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><p dir="ltr">Thank you again!</p><br>_______________________________________________<br>lttng-dev mailing list<br>lttng-dev@lists.lttng.org<br>http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev<br></blockquote><div><br><br></div><div><br></div><div>-- <br></div><div><span name="x"></span>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com<span name="x"></span><br></div></div></body></html>