[lttng-dev] Profiling LTTng tracepoint latency on different arm platforms

[Mathieu Desnoyers]

On 6/21/23 01:39, Yitschak, Yehuda wrote:
>> On 6/20/23 10:20, Mathieu Desnoyers via lttng-dev wrote:
>>> On 6/20/23 06:27, Mousa, Anas via lttng-dev wrote:
>>>> Hello,
>>>
>>>>
>>>>
>> Arethereanysuggestionstorootcausethehighlatencyandpotentiallyimproveito
>> n*platform****1*?
>>>>
>>>> Thanks and best regards,
>>>>
>>>> Anas.
>>>>
>>>
>>> I recommend using "perf" when tracing with the sample program in a
>>> loop to figure out the hot spots. With that information on the "fast"
>>> and "slow" system, we might be able to figure out what differs.
>>>
>>> Also, comparing the kernel configurations of the two systems can help.
>>> Also comparing the glibc versions of the two systems would be relevant.
>>>
>>
>> Also make sure you benchmark the lttng "snapshot" mode [1] to make sure
>> you don't run into a situation where the disk/network I/O throughput cannot
>> cope with the generated event throughput, thus causing the ring buffer to
>> discard events. This would therefore "speed up" tracing from the application
>> perspective because discarding an event is faster than writing it to a ring
>> buffer.
> 
> You mean we should avoid the "discard" loss mode and use "overwrite" loss mode since discard mode can fake fast performance ?

Yes. In addition to use "overwrite-when-buffer-full" mode, the 
"snapshot" session also ensures that no consumer daemon extracts the 
trace data (unless an explicit snapshot record is performed), which 
allows comparing the ring buffer producer performance with minimal noise.

If you really want to benchmark the discard-when-buffer-full mode and 
the the consumer daemon I/O behavior, then you need to take into account 
event discarded counts and the actual trace data size that was written 
to disk.

Thanks,

Mathieu

> 
>>
>> Thanks,
>>
>> Mathieu
>>
>> [1] https://lttng.org/docs/v2.13/#doc-taking-a-snapshot
>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> https://www.efficios.com
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev at lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com