<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Emoji";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-CA" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi Doug<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Great to see your interest this area.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal">>> BTW, my focus on libbabeltrace is to allow for a full range of tooling for our users using the language of their choice. The python binding is particularly interesting. As we move forward into the new world of IDEs, I can see a node.js
binding being interesting as well. And it may even make sense to use it in Java tooling like TraceCompass using JNA. That's the vision at least, but first, baby steps
<span style="font-family:"Segoe UI Emoji",serif">☺</span>.<o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">For the integration with Trace Compass in Eclipse, we chose to implement a CTF parser in Java to avoid using JNI/JNA to libbabeltrace. This was for performance reasons over JNI/JNA interface. We
had implemented that at the beginning when Trace Compass supported LTTng version 0.x (prior CTF) and the performance was not sufficient enough.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">For the integration in next generation IDEs, we are working on defining a Trace Server Protocol (TSP) as an interface between a client and server for exchanging of trace date for visualization purposes.
It’s a similar idea as the LSP (Language Server Protocol) or DAP (Debug Server protocol). The TSP interface is on a higher level than what a trace reading interface like the CTF trace reader in Java or over your suggested JNA interface over libbabeltrace would
provide. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">We have some promising results for a Typescript front-end based on the next generation IDE Theia that uses the TSP to a trace server using Trace Compass core features as backend. We can already visualize
various Trace Compass views. It’s still preliminary, but we could demo it if you’re interested.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">BR<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Bernd<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> tracecompass-dev-bounces@eclipse.org <tracecompass-dev-bounces@eclipse.org>
<b>On Behalf Of </b>Doug Schaefer<br>
<b>Sent:</b> March-12-19 11:23 AM<br>
<b>To:</b> tracecompass-dev@eclipse.org; jonathan.rajotte-julien@efficios.com<br>
<b>Cc:</b> lttng-dev@lists.lttng.org<br>
<b>Subject:</b> Re: [tracecompass-dev] Scalability<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">It should be easy to generate a random one with the class I mentioned below and fill it with 16 million events. BTW, two of the int fields are 5 and 10 bits respectively, but I'm not sure that matters (or at least it shouldn't).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'll also take a look at the babeltrace code and see what I can see.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">BTW, my focus on libbabeltrace is to allow for a full range of tooling for our users using the language of their choice. The python binding is particularly interesting. As we move forward into the new world of IDEs, I can see a node.js
binding being interesting as well. And it may even make sense to use it in Java tooling like TraceCompass using JNA. That's the vision at least, but first, baby steps
<span style="font-family:"Segoe UI Emoji",serif">☺</span>.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Doug.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">On Tue, 2019-03-12 at 11:03 -0400, Jonathan Rajotte-Julien wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #729FCF 1.5pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<pre>Hi Doug,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>On Tue, Mar 12, 2019 at 02:56:26PM +0000, Doug Schaefer wrote:<o:p></o:p></pre>
<pre>Thanks, Matthew, I ended up flushing every 100K events and that seemed OK.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>My biggest worry is on the read side. Babeltrace blew up on that file. It's a pretty simple trace (for now) with a single event class with 4 ints and a sequence of 32-bit unsigned ints which is usually only 2 elements long.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>This is not I expect. Still eithet jgalar or eep might have a more insight on<o:p></o:p></pre>
<pre>this. CCing the lttng-dev mailing list.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Is there any way to share us a similar trace? Either with a generator or we can<o:p></o:p></pre>
<pre>provide a link for you to upload it. The current limit on bugs.lttng.org is a<o:p></o:p></pre>
<pre>bit too small for such trace.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Aside from that, am very pleased with the how easy CTF is to work with. Looking forward to doing more.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Doug.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>On Tue, 2019-03-12 at 14:41 +0000, Matthew Khouzam wrote:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Hi Doug,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Great to hear you're coming to a standard! I don't know if trace compass will scale properly as I don't know what the trace configuration is.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>I have one of our trace which has 16 million events, the size of each event is around 32 bytes giving me a 540MB file. My first attempt at simply writing out the CTF events without a flush ran out of virtual memory. I then flushed after every event which made each event take 32K. So I found a middle ground and the resulting stream file is close to the same size.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>My suggestion is to have 1 MB packets. This makes seeking very efficient. If each event is 32 bytes, basically flush every 25k events or so.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Please keep us posted!<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Matthew.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>________________________________<o:p></o:p></pre>
<pre>From: <a href="mailto:tracecompass-dev-bounces@eclipse.org">tracecompass-dev-bounces@eclipse.org</a> <<a href="mailto:tracecompass-dev-bounces@eclipse.org">tracecompass-dev-bounces@eclipse.org</a>> on behalf of Doug Schaefer <<a href="mailto:dschaefer@blackberry.com">dschaefer@blackberry.com</a>><o:p></o:p></pre>
<pre>Sent: Tuesday, March 12, 2019 10:30:15 AM<o:p></o:p></pre>
<pre>To: <a href="mailto:tracecompass-dev@eclipse.org">tracecompass-dev@eclipse.org</a><o:p></o:p></pre>
<pre>Subject: [tracecompass-dev] Scalability<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Hey gang,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>We're finally starting to look at converting our custom traces into CTF so we can leverage tools like TraceCompass and, of course, contribute to it. One thing I quickly ran into is a scalability issue I'm seeing with libbabeltrace.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>I have one of our trace which has 16 million events, the size of each event is around 32 bytes giving me a 540MB file. My first attempt at simply writing out the CTF events without a flush ran out of virtual memory. I then flushed after every event which made each event take 32K. So I found a middle ground and the resulting stream file is close to the same size.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>But when I use babeltrace to print it out, I ran out of virtual memory. I then hand coded a reader and simply adding the trace to the context caused the memory issue. It really looks like libbabeltrace (version 1.5 from the Ubuntu 18.04 distro), tries to inflate the events into it's internal representation for the entire trace. I need to do more investigation to confirm that.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>So my question for this list, would TraceCompass do better? Does it have it's own parsing libraries?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Thanks,<o:p></o:p></pre>
<pre>Doug<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>tracecompass-dev mailing list<o:p></o:p></pre>
<pre><a href="mailto:tracecompass-dev@eclipse.org">tracecompass-dev@eclipse.org</a><o:p></o:p></pre>
<pre>To change your delivery options, retrieve your password, or unsubscribe from this list, visit<o:p></o:p></pre>
<pre><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.eclipse.org_mailman_listinfo_tracecompass-2Ddev&d=DwIBAg&c=yzoHOc_ZK-sxl-kfGNSEvlJYanssXN3q-lhj0sp26wE&r=NrrbvTHWa2Nbp_kAN0Hl1o3lM1WAwSes64uBjxjNhMc&m=ZoCx61VE_sHGa6j3DXehbqjX5P1NGEuDtEFaGORCh9k&s=qi9q1zvDaC9cJCci0y123O-j66M643YwxRJccJCzg_c&e=">https://urldefense.proofpoint.com/v2/url?u=https-3A__www.eclipse.org_mailman_listinfo_tracecompass-2Ddev&d=DwIBAg&c=yzoHOc_ZK-sxl-kfGNSEvlJYanssXN3q-lhj0sp26wE&r=NrrbvTHWa2Nbp_kAN0Hl1o3lM1WAwSes64uBjxjNhMc&m=ZoCx61VE_sHGa6j3DXehbqjX5P1NGEuDtEFaGORCh9k&s=qi9q1zvDaC9cJCci0y123O-j66M643YwxRJccJCzg_c&e=</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
</blockquote>
</div>
</body>
</html>