[ltt-dev] LTTng-UST vs SystemTap userspace tracing benchmarks

[Mathieu Desnoyers]

* Stefan Hajnoczi (stefanha at gmail.com) wrote:
> On Tue, Feb 15, 2011 at 4:26 PM, Frank Ch. Eigler <fche at redhat.com> wrote:
> >
> > Julien Desfossez <julien.desfossez at polymtl.ca> writes:
> >
> >> LTTng-UST vs SystemTap userspace tracing benchmarks
> >
> > Thank you.
> >
> >> [...]  For flight recorder tracing, UST is 289 times faster than
> >> SystemTap on an 8-core system with a LTTng kernel and 279 times with
> >> a vanilla+utrace kernel.
> >
> > This is not that surprising, considering how the two tools work.  UST
> > does its work in userspace, and is therefore focused on an individual
> > process's activities.  Systemtap does its work in kernelspace, and can
> > therefore focus on many different processes and the kernel at the same
> > time.  This entails some ring transitions.
> >
> > (One may imagine a future version of systemtap where scripts that
> > happen to independently probe single processes are executed with a
> > pure userspace backend, but this is not in our immediate roadmap.)
> 

Hi Stefan,

> What is the fundamental mechanism that UST and SystemTap use for tracing?
> 
> e.g. Here's a guess:
> UST: a conditional function call within the same process

Yes, UST can manage to stay within the same process because tracing is buffered:
it only has to write the trace data into shared-memory buffers. Therefore, a
simple function call is sufficient.

> SystemTap: a software interrupt on x86

Yep, AFAIK, SystemTap needs to receive everything at the kernel-level to perform
its system-wide data processing at kernel-level, without buffering between the
data extraction from the instrumented applications and the in-kernel execution
of SystemTap. This leads to a strong dependency on using a software interrupt
for every event.

Thanks,

Mathieu

> I don't know the implementations details but would be interested in
> understanding this.
> 
> Stefan

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com