[lttng-dev] Cost of user-space tracepoint()

[ChitlurSrinivasa, Venkatesh]

I sent this mail few days back and didn't receive response. So I am sending it again.

We are planning to use LTTng UST as it supports lot of interesting features, but have some performance concerns (as compared with our in-house tracing tool). Please point me to the latest benchmark tests and performance results.

I noticed that the tracepoint is making clock_gettime() and sched_getcpu() system calls. With Linux kernel v3.6.11 and libc v2.13-38+deb7u3, I see that these system calls are not going through VDSO and hence costing more. I tried to add wrapper functions for these system calls to call __vdso_clock_gettime and __vdso_getcpu as upgrading libc was not an option. With this change the cost of tracepoint() recording one integer dropped to 795 cycles (= 284 nsec on Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz). Still this number seems to be higher than earlier published numbers. 

VBabu

________________________________________
From: ChitlurSrinivasa, Venkatesh
Sent: Wednesday, August 20, 2014 12:30 PM
To: lttng-dev at lists.lttng.org
Subject: Cost of user-space tracepoint()

I am sure you guys will do regular performance runs on different versions of LTTng, but I couldn't find any links to the latest results. I was trying to find the cost of a tracepoint() recording one integer. On CPU Intel Xeon E5-2680 v2 @ 2.80GHz​, running Linux 3.6.11​ and lttng 2.4.1,  I am getting about 927 cycles (9270692144 cycles for 10000000​ iterations). This seems to be lot higher than the documented results. In the paper https://lttng.org/files/papers/desnoyers.pdf  the average cost of tracepoint() with older ltt-usertrace-fast tracepoint is 297 cycles. Another link http://lttng.org/files/thesis/desnoyers-thesis-defense-2009-12-e1.pdf says cache hot tracepoint() cost is 238 cycles.

In spite of better cpu, newer version of lttng and kernel I am seeing higher cost. Am I missing something here? Do I need to enable certain features of kernel/cpu ?

VBabu