[lttng-dev] Reducing profiling overhead

[Deneau, Tom]

I am trying to use lttng to simultaneously trace some kernel and user events.
The kernel events are a few syscalls (sendto, futex).
The user events are the pthread ones in lttng-ust-libc-wrapper.

Looking for recommended lttng configurations to:
   * avoid dropping any events
   * avoid too much overhead to slow down the profiled application.

As background info, the profiled application might be running as many as 64 software threads.
Each thread might be doing as many as 4000 "operations"/ sec so approximately 250,000 total 
operations/sec.  A single operation involves about 60-70 trace events (the number is variable
depending on the amount of lock contention).

All the threads are very similar so there is no need to profile all the threads.

Right now I don't get any messages from lttng-stop nor from babeltrace saying
that any events are being dropped so that's good.

I am however seeing a pretty significant performance drop in the app with profiling.
The 250,000 total ops/sec can go down to 170,000 ops/sec.  I have measured this
by profiling for several seconds and observing the effect on the workload periodic performance reports.
And also by measuring the number of ops/sec per thread in the traces themselves.  (An op always has
one sendto syscall).

The subbuf setup I am currently using probably is not  optimal:
        lttng enable-channel k -k --num-subbuf 64 --subbuf-size 512k
        lttng enable-channel u -u --num-subbuf 64 --subbuf-size 512k
        lttng add-context -k -c k -t tid -t pid
        lttng add-context -u -c u -t vtid -t vpid

When I enable-events, I generally filter on either a pid/vpid (for both kernel and user)
or on a small number of tids/vtids (to keep the trace smaller).

Any pointers for subbuf configuration or anything to reduce the profiling overhead?
Or is the 33% perf hit for this kind of profiling load pretty expected?

-- Tom Deneau