<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Jun 2, 2017 at 10:24 PM, craig harmer <span dir="ltr"><<a href="mailto:charmster@gmail.com" target="_blank">charmster@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:"trebuchet ms",sans-serif">hi all,<br><br></div><div style="font-family:"trebuchet ms",sans-serif">the suggestion to write a pre-processor for LTTng tracepoints is interesting and may be a good idea. however, i'm not sure that it will provide a general purpose solution.<br><br></div><div style="font-family:"trebuchet ms",sans-serif">the problem i for see is that tracepoint invocations, at least, should be written in the syntax of the language they're being used in (a programmer would would expect the tracepoint() function call to look like any other function call in the language).<br><br>so the tracepoint parser needs to understand the function invocation syntax of each language. it also needs to understand at least some of the rest of the syntax of the language -- in the case of Go it at least needs to be able recognize "import" statements and import those files and it needs to understand when something that looks like a tracepoint() is actually embedded in a comment or string. it also needs to emit code that matches the syntax of the language.<br><br>so a general purpose lttng tracepoint parsing tool would need at least a limited understanding of the syntax and grammar of each language for reading the source file in and then writing the source file out. there could be code in the middle that's common across languages, but probably not a lot. perhaps an lttng tracepoint parsing tool could be better structured as a common framework for language specific tracepoint parsers to be plugged in to.<br><br></div><div style="font-family:"trebuchet ms",sans-serif">but writing a parser that actually understands the grammar of a language is not easy. if i was really going to do this, i'd be tempted to see if i could just take this from gcc (which has front-ends that supports C, C++, Go, and other languages (i think)).<br></div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">and then there's the tracepoint definitions. they could be written using the barectf() tool that Philippe mentions, but i'm not sure if that would be satisfactory for most programmers, who would prefer to write tracepoint definitions in the same language that they are programming in. they would also need to support the data types of the language (i'm thinking Python here as an example, where tracepoints would be expected to support most of the Python basic data types).<br></div></div></blockquote><div><br></div><div>There's a little misunderstanding here.</div><div><br></div><div>The idea is to have a universal tracepoint definition language (and what I suggested is that a subset of CTF 2, of which the syntax is JSON (or YAML which translates to JSON directly), could be used for this) which translates to functions, classes, macros, etc. in selected programming languages. Depending on the destination language, we can generate serialization functions directly or wrappers which use LTTng-UST behind the scenes. So in your case, this tool would generate Go source files which would contain public functions which correspond to tracepoints. We never considered parsing an existing programming language to infer tracepoints.</div><div><br></div><div>Of course this requires the developer to define tracepoints in this universal language. That's an additional developing step needed to allow the developer to use tracepoints in his/her favorite language. But there are other advantages, like generating tracepoints in two different languages from the same tracepoint definitions. Like Mathieu wrote, it's the same spirit as Flex and Bison. With Bison, you don't use the C preprocessor to write your grammar rules: the tool reads a Bison input file and generates a parser source file. Just like barectf reads a YAML configuration file which defines stream and event classes and generates tracer source files.</div><div><br></div><div>Phil</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:"trebuchet ms",sans-serif"></div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">because Go seems to be similar to C i'm strongly tempted to just co-opt the C pre-processor and existing LTTng tracepoint machinery and modify the LTTng header files to generate valid Go code for a tracepoint definition. this does add a LTTng tracepoint tool to the steps required to compile Go, except that the tracepoint tool is called "cpp" (plus header files).<br><br></div><div style="font-family:"trebuchet ms",sans-serif">i don't really know Go yet and i hae not been assigned to get LTTng to work in Go, so this is all quite preliminary. but i think the project i'm about to start on really needs LTTng or LTTng-like tracepoint functionality so i'm motivated to see if this can work.<span style="font-family:arial,sans-serif"> </span></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">i'm going to investigate some more.<span class="gmail-m_-6643529984790646395HOEnZb"><font color="#888888"><br></font></span></div><span class="gmail-m_-6643529984790646395HOEnZb"><font color="#888888"><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">--craig<br></div><div style="font-family:"trebuchet ms",sans-serif"><br></div></font></span></div><div class="gmail-m_-6643529984790646395HOEnZb"><div class="gmail-m_-6643529984790646395h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 2, 2017 at 10:05 AM, Philippe Proulx <span dir="ltr"><<a href="mailto:eeppeliteloop@gmail.com" target="_blank">eeppeliteloop@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><span><div><div class="gmail-m_-6643529984790646395m_5091517559717538205m_790898205284245872gmail_signature">On Fri, Jun 2, 2017 at 11:58 AM, Mathieu Desnoyers <span dir="ltr"><<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.co<wbr>m</a>></span> wrote:<br></div></div></span><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><span class="gmail-m_-6643529984790646395m_5091517559717538205m_790898205284245872gmail-"><div><br></div><span id="gmail-m_-6643529984790646395m_5091517559717538205m_790898205284245872gmail-m_-5040175781309588822zwchr">----- On Jun 2, 2017, at 2:27 AM, craig harmer <<a href="mailto:charmster@gmail.com" target="_blank">charmster@gmail.com</a>> wrote:<br></span></span><div><span class="gmail-m_-6643529984790646395m_5091517559717538205m_790898205284245872gmail-"><blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;padding-left:5px;color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div style="font-family:"trebuchet ms",sans-serif">hi all,<br><br></div><div style="font-family:"trebuchet ms",sans-serif">has anyone looked at porting LTTng to <a href="https://en.wikipedia.org/wiki/Go_%28programming_language%29" target="_blank">Go</a> (AKA <a href="https://golang.org/" target="_blank">Golang</a>)?</div></div></blockquote><div><br></div></span><div>Not yet ! I'm glad someone is looking into it. :)<br></div><span class="gmail-m_-6643529984790646395m_5091517559717538205m_790898205284245872gmail-"><div><br></div><blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;padding-left:5px;color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div style="font-family:"trebuchet ms",sans-serif"><br><br></div><div style="font-family:"trebuchet ms",sans-serif">Go is a language very similar to C, but it has intrinsic support for concurrency and better memory protection (at the expense of including garbage collection). its from Google and is pretty hot these days for big data and "web scale" distributed systems. the similarity to C is not surprising since two of the three authors were also involved in the development of UNIX in at AT&T in the 1970's: Rob Pike and Ken Thompson (Robert Griesemer is the third author).<br><br>Go is able to link with C object files, so implementing support for tracef(3) should not be very difficult. <br></div><div style="font-family:"trebuchet ms",sans-serif"><br>but i want to have full support for user-defined tracepoint() events. that looks to be much trickier since the LTTng tracepoint() functionality relies heavily on (some might say abuses</div></div></blockquote></span></div></div></div></blockquote></span><div>"abuses" is appropriate ;-) </div><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div><span class="gmail-m_-6643529984790646395m_5091517559717538205m_790898205284245872gmail-"><blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;padding-left:5px;color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div style="font-family:"trebuchet ms",sans-serif">) the C pre-processor, and Go does not have an equivalent of the C pre-processor. however, i don't see an obvious reason why Go source files that contain tracepoints couldn't be run through the C pre-processor -- except that Go specific LTTng header files would be required.<br><br><a href="https://tour.golang.org/welcome/1" target="_blank">here is an example of "hello, world" in Go</a>. (note the use of "import" rather then "#include").<br><br></div><div style="font-family:"trebuchet ms",sans-serif">i've spent more than a few hours looking at the C code generated by the C pre-processor trying to track down exactly why a tracepoint() definition is causing a compiler error, so i think i have an idea of just how hard it would be to development equivalent header files for Go + C pre-processor. it may be the case that the Go "header files", once developed, would be compiler specific, i.e only work with <a href="https://golang.org/doc/install/gccgo" target="_blank">gccgo</a> (the gcc front-end for Go).<br></div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">anyway, i'm wondering if anybody has attempted this and/or what your thoughts would be.</div></div></blockquote></span><div>In the case of C/C++ code, using the C preprocessor got the job done without requiring any additional<br></div><div>dependency. Given the context you describe, perhaps it would be wise to consider introducing a dedicated</div><div>"lttng probe description" parser. The idea here would be to parse the tracepoint probe definitions and</div><div>translate those into native code for various languages (e.g. Golang).<br></div><div><br></div><div>This would create a new lttng utility that would be required to to produce the language-specific files from</div><div>those descriptions. I would allow us to port the "tracepoint" concept to many more languages easily, and</div><div>not be so much tied to the C preprocessor anymore.<span style="font-family:arial,sans-serif;font-size:small;color:rgb(34,34,34)"> </span></div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div><div></div><div><br></div><div>So perhaps adding this new tool as a dependency that needs to be invoked prior to compilation might</div><div>not be too much of an issue ? I see it as being slightly similar to the role accomplished by Flex and<br></div><div>Bison: they are required to translate from source files to an intermediary language, and then the<br></div><div>resulting files can be included into the distribution source packages, so only those who aim at<br></div><div>changing the source descriptions need to have the translation tool installed.<br></div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div><div><br></div><div>Thoughts ?<br></div></div></div></div></blockquote><div><br></div></span><div>Good point and we thought about this before. In fact this is just what barectf <<a href="http://barectf.org/" target="_blank">http://barectf.org/</a>> does, for example. What I see in the future is that a subset of (eventual) CTF 2's metadata language (JSON) could be used here (or an equivalent, human-friendly YAML, like barectf) to describe LTTng event and stream classes and translate them to what's needed to record such events for a given programming language. CTF 2's user attributes can be used to insert additional, language-specific properties, like parameter types and names, and other options.</div><div><br></div><div>Phil </div><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div><div></div><div><br></div><div>Thanks,<br></div><div><br></div><div>Mathieu<br></div><div><br></div><div><br></div><div><br></div><blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;padding-left:5px;color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">--craig<br></div></div><br>______________________________<wbr>_________________<br>lttng-dev mailing list<br><a href="mailto:lttng-dev@lists.lttng.org" target="_blank">lttng-dev@lists.lttng.org</a><br><a href="https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" target="_blank">https://lists.lttng.org/cgi-bi<wbr>n/mailman/listinfo/lttng-dev</a><span class="gmail-m_-6643529984790646395m_5091517559717538205m_790898205284245872gmail-HOEnZb"><font color="#888888"><br></font></span></blockquote></div><span class="gmail-m_-6643529984790646395m_5091517559717538205m_790898205284245872gmail-HOEnZb"><font color="#888888"><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a></div></font></span></div></div><br>______________________________<wbr>_________________<br>
lttng-dev mailing list<br>
<a href="mailto:lttng-dev@lists.lttng.org" target="_blank">lttng-dev@lists.lttng.org</a><br>
<a href="https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" rel="noreferrer" target="_blank">https://lists.lttng.org/cgi-bi<wbr>n/mailman/listinfo/lttng-dev</a><br>
<br></blockquote></span></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>