[lttng-dev] LTTng UST Benchmarks

[Kienan Stewart]

Hi Aditya,

It has been suggested to me that the following publication[1] would also 
be of interest. It gives a good comparison of micro-benchmarking tracers.

[1]: https://dl.acm.org/doi/10.1145/3158644

thanks,
kienan

On 4/25/24 1:53 PM, Kienan Stewart via lttng-dev wrote:
> Hi Aditya,
> 
> On 4/24/24 11:25 AM, Aditya Kurdunkar via lttng-dev wrote:
>> Hello everyone, I was working on a use case where I am working on 
>> enabling LTTng on an embedded ARM device running the OpenBMC linux 
>> distribution. I have enabled the lttng yocto recipe and I am able to 
>> trace my code. The one thing I am concerned about is the performance 
>> overhead. Although the documentation mentions that LTTng has the 
>> lowest overhead amongst all the available solutions, I am concerned 
>> about the overhead of the LTTng UST in comparison to 
>> other available tracers/profilers. I have used the benchmarking setup 
>> from lttng-ust/tests/benchmark at master · lttng/lttng-ust 
>> (github.com) 
>> <https://github.com/lttng/lttng-ust/tree/master/tests/benchmark> to 
>> benchmark the overhead of the tracepoints (on the device). The 
>> benchmark, please correct me if I am wrong, gives the overhead of a 
>> single tracepoint in your code.
> 
> This seems to be what it does.
> 
> Although this might be fine for now, I
>> was just wondering if there are any published benchmarks comparing 
>> LTTng with the available tracing/profiling solutions. 
> 
> I don't know of any published ones that do an exhaustive comparison.
> 
> There is this one[1] which references a comparison with some parts of 
> eBPF. The source for the benchmarking is also available[2].
> 
> If not, how can I go
>> about benchmarking the overhead of the applications?
>>
> 
> I'm not really sure how to answer you here.
> 
> I guess the most pertinent to your use case is to test your application 
> with and without tracing to see the complete effect?
> 
> It would be good to have a dedicated system, disable CPU frequency 
> scaling, and to perform the tests repeatedly and measure the mean, 
> median, and standard deviation.
> 
> You could pull methodological inspiration from prior publications[3], 
> which while outdated in terms of software version and hardware 
> demonstrate the process of creating and comparing benchmarks.
> 
> It would also be useful to identify how your application and tracing 
> setup works, and to understand which parts of the system you are 
> interested in measuring.
> 
> For example, the startup time of tracing rapidly spawning processes will 
> depend on the type of buffering scheme in use, if the tracing 
> infrastructure is loaded before or after forking, etc.
> 
> Your case might be a long running application and you aren't interested 
> in startup time performance but more concretely the impact of the static 
> instrumentation on one of your hot paths.
> 
> If you're not sure what kind of tracing setups work best in your case, 
> or would like us to characterize at certain aspect of the tool-set's 
> performance, EfficiOS[4] offers consultation and support for 
> instrumentation and performance in applications.
> 
>> I have come across the lttng/lttng-ust-benchmarks (github.com) 
>> <https://github.com/lttng/lttng-ust-benchmarks> repository which has 
>> no documentation on how to run it, apart from one commit message on 
>> how to run the benchmark script.
>>
> 
> To run those benchmarks when you have babeltrace2, lttng-tools, urcu, 
> lttng-ust, and optional lttng-modules installed:
> 
> ```
> $ make
> $ python3 ./benchmark.py
> ```
> 
> This should produce a file, `benchmarks.json`
> 
> You can also inspect how the CI job runs it: 
> https://ci.lttng.org/view/LTTng-ust/job/lttng-ust-benchmarks_master_linuxbuild/
> 
>> Any help is really appreciated. Thank you.
>>
>> Regards,
>> Aditya
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev at lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 
> [1]: 
> https://tracingsummit.org/ts/2022/files/Tracing_Summit_2022-LTTng_Beyond_Ring-Buffer_Based_Tracing_Jeremie_Galarneau_.pdf
> [2]: https://github.com/jgalar/LinuxCon2022-Benchmarks
> [3]: https://www.dorsal.polymtl.ca/files/publications/desnoyers.pdf
> [4]: https://www.efficios.com/contact/
> 
> thanks,
> kienan
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev