<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p></p>
<p style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
Hey Mathieu,</p>
<p style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
We see that upon recording a tracepoint, there are multiple stages of reserve-commit-write,</p>
<p style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
where atomics and shared memory accesses take up a big part of the recording time,</p>
<p style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
we're wondering, is there a "light-mode" of recording a tracepoint involving less logic or</p>
<p style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
a mode which can potentially have lower latency?</p>
<p style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
<br>
</p>
<p style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
Also, are there any recent docs to share regarding tracepoint latency?</p>
<p style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
<br>
</p>
<p style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
Regards,</p>
<p style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
Anas.</p>
<br>
<p></p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Yitschak, Yehuda<br>
<b>Sent:</b> Wednesday, June 21, 2023 5:21:35 PM<br>
<b>To:</b> Mathieu Desnoyers; Mousa, Anas; lttng-dev@lists.lttng.org<br>
<b>Subject:</b> RE: [EXTERNAL][lttng-dev] Profiling LTTng tracepoint latency on different arm platforms</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText"><br>
> On 6/21/23 01:39, Yitschak, Yehuda wrote:<br>
> >> On 6/20/23 10:20, Mathieu Desnoyers via lttng-dev wrote:<br>
> >>> On 6/20/23 06:27, Mousa, Anas via lttng-dev wrote:<br>
> >>>> Hello,<br>
> >>><br>
> >>>><br>
> >>>><br>
> >><br>
> Arethereanysuggestionstorootcausethehighlatencyandpotentiallyimprovei<br>
> >> to<br>
> >> n*platform****1*?<br>
> >>>><br>
> >>>> Thanks and best regards,<br>
> >>>><br>
> >>>> Anas.<br>
> >>>><br>
> >>><br>
> >>> I recommend using "perf" when tracing with the sample program in a<br>
> >>> loop to figure out the hot spots. With that information on the "fast"<br>
> >>> and "slow" system, we might be able to figure out what differs.<br>
> >>><br>
> >>> Also, comparing the kernel configurations of the two systems can help.<br>
> >>> Also comparing the glibc versions of the two systems would be relevant.<br>
> >>><br>
> >><br>
> >> Also make sure you benchmark the lttng "snapshot" mode [1] to make<br>
> >> sure you don't run into a situation where the disk/network I/O<br>
> >> throughput cannot cope with the generated event throughput, thus<br>
> >> causing the ring buffer to discard events. This would therefore<br>
> >> "speed up" tracing from the application perspective because<br>
> >> discarding an event is faster than writing it to a ring buffer.<br>
> ><br>
> > You mean we should avoid the "discard" loss mode and use "overwrite"<br>
> loss mode since discard mode can fake fast performance ?<br>
> <br>
> Yes. In addition to use "overwrite-when-buffer-full" mode, the "snapshot"<br>
> session also ensures that no consumer daemon extracts the trace data<br>
> (unless an explicit snapshot record is performed), which allows comparing<br>
> the ring buffer producer performance with minimal noise.<br>
> <br>
> If you really want to benchmark the discard-when-buffer-full mode and the<br>
> the consumer daemon I/O behavior, then you need to take into account<br>
> event discarded counts and the actual trace data size that was written to<br>
> disk.<br>
<br>
Since you mentioned this, is there any "stat" command which lists events such as discards and disk writes, etc ?
<br>
I looked this up in the past but couldn't find anything<br>
<br>
> <br>
> Thanks,<br>
> <br>
> Mathieu<br>
> <br>
> ><br>
> >><br>
> >> Thanks,<br>
> >><br>
> >> Mathieu<br>
> >><br>
> >> [1] <a href="https://lttng.org/docs/v2.13/#doc-taking-a-snapshot">https://lttng.org/docs/v2.13/#doc-taking-a-snapshot</a><br>
> >><br>
> >>> Thanks,<br>
> >>><br>
> >>> Mathieu<br>
> >>><br>
> >>><br>
> >><br>
> >> --<br>
> >> Mathieu Desnoyers<br>
> >> EfficiOS Inc.<br>
> >> <a href="https://www.efficios.com">https://www.efficios.com</a><br>
> >><br>
> >> _______________________________________________<br>
> >> lttng-dev mailing list<br>
> >> lttng-dev@lists.lttng.org<br>
> >> <a href="https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev">https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><br>
> <br>
> --<br>
> Mathieu Desnoyers<br>
> EfficiOS Inc.<br>
> <a href="https://www.efficios.com">https://www.efficios.com</a><br>
<br>
</div>
</span></font>
</body>
</html>