[lttng-dev] porting LTTng to Go(lang)?

Sat Jun 3 19:45:39 UTC 2017

On Fri, Jun 2, 2017 at 10:24 PM, craig harmer <charmster at gmail.com> wrote:

> hi all,
>
> the suggestion to write a pre-processor for LTTng tracepoints is
> interesting and may be a good idea.  however, i'm not sure that it will
> provide a general purpose solution.
>
> the problem i for see is that tracepoint invocations, at least, should be
> written in the syntax of the language they're being used in (a programmer
> would would expect the tracepoint() function call to look like any other
> function call in the language).
>
> so the tracepoint parser needs to understand the function invocation
> syntax of each language.  it also needs to understand at least some of the
> rest of the syntax of the language -- in the case of Go it at least needs
> to be able recognize "import" statements and import those files and it
> needs to understand when something that looks like a tracepoint() is
> actually embedded in a comment or string.  it also needs to emit code that
> matches the syntax of the language.
>
> so a general purpose lttng tracepoint parsing tool would need at least a
> limited understanding of the syntax and grammar of each language for
> reading the source file in and then writing the source file out.  there
> could be code in the middle that's common across languages, but probably
> not a lot.  perhaps an lttng tracepoint parsing tool could be better
> structured as a common framework for language specific tracepoint parsers
> to be plugged in to.
>
> but writing a parser that actually understands the grammar of a language
> is not easy.  if i was really going to do this, i'd be tempted to see if i
> could just take this from gcc (which has front-ends that supports C, C++,
> Go, and other languages (i think)).
>
> and then there's the tracepoint definitions.  they could be written using
> the barectf() tool that Philippe mentions, but i'm not sure if that would
> be satisfactory for most programmers, who would prefer to write tracepoint
> definitions in the same language that they are programming in.  they would
> also need to support the data types of the language (i'm thinking Python
> here as an example, where tracepoints would be expected to support most of
> the Python basic data types).
>

There's a little misunderstanding here.

The idea is to have a universal tracepoint definition language (and what I
suggested is that a subset of CTF 2, of which the syntax is JSON (or YAML
which translates to JSON directly), could be used for this) which
translates to functions, classes, macros, etc. in selected programming
languages. Depending on the destination language, we can generate
serialization functions directly or wrappers which use LTTng-UST behind the
scenes. So in your case, this tool would generate Go source files which
would contain public functions which correspond to tracepoints. We never
considered parsing an existing programming language to infer tracepoints.

Of course this requires the developer to define tracepoints in this
universal language. That's an additional developing step needed to allow
the developer to use tracepoints in his/her favorite language. But there
are other advantages, like generating tracepoints in two different
languages from the same tracepoint definitions. Like Mathieu wrote, it's
the same spirit as Flex and Bison. With Bison, you don't use the C
preprocessor to write your grammar rules: the tool reads a Bison input file
and generates a parser source file. Just like barectf reads a YAML
configuration file which defines stream and event classes and generates
tracer source files.

Phil

>
>
> because Go seems to be similar to C i'm strongly tempted to just co-opt
> the C pre-processor and existing LTTng tracepoint machinery and modify the
> LTTng header files to generate valid Go code for a tracepoint definition.
> this does add a LTTng tracepoint tool to the steps required to compile Go,
> except that the tracepoint tool is called "cpp" (plus header files).
>
> i don't really know Go yet and i hae not been assigned to get LTTng to
> work in Go, so this is all quite preliminary.  but i think the project i'm
> about to start on really needs LTTng or LTTng-like tracepoint functionality
> so i'm motivated to see if this can work.
>

> i'm going to investigate some more.
>
> --craig
>
>
> On Fri, Jun 2, 2017 at 10:05 AM, Philippe Proulx <eeppeliteloop at gmail.com>
> wrote:
>
>> On Fri, Jun 2, 2017 at 11:58 AM, Mathieu Desnoyers <
>> mathieu.desnoyers at efficios.com> wrote:
>>
>>>
>>> ----- On Jun 2, 2017, at 2:27 AM, craig harmer <charmster at gmail.com>
>>> wrote:
>>>
>>> hi all,
>>>
>>> has anyone looked at porting LTTng to Go
>>> <https://en.wikipedia.org/wiki/Go_%28programming_language%29> (AKA
>>> Golang <https://golang.org/>)?
>>>
>>>
>>> Not yet ! I'm glad someone is looking into it. :)
>>>
>>>
>>>
>>> Go is a language very similar to C, but it has intrinsic support for
>>> concurrency and better memory protection (at the expense of including
>>> garbage collection).  its from Google and is pretty hot these days for big
>>> data and "web scale" distributed systems.  the similarity to C is not
>>> surprising since two of the three authors were also involved in the
>>> development of UNIX in at AT&T in the 1970's: Rob Pike and Ken Thompson
>>> (Robert Griesemer is the third author).
>>>
>>> Go is able to link with C object files, so implementing support for
>>> tracef(3) should not be very difficult.
>>>
>>> but i want to have full support for user-defined tracepoint() events.
>>> that looks to be much trickier since the LTTng tracepoint() functionality
>>> relies heavily on (some might say abuses
>>>
>>> "abuses" is appropriate ;-)
>>
>>> ) the C pre-processor, and Go does not have an equivalent of the C
>>> pre-processor.  however, i don't see an obvious reason why Go source files
>>> that contain tracepoints couldn't be run through the C pre-processor --
>>> except that Go specific LTTng header files would be required.
>>>
>>> here is an example of "hello, world" in Go
>>> <https://tour.golang.org/welcome/1>.  (note the use of "import" rather
>>> then "#include").
>>>
>>> i've spent more than a few hours looking at the C code generated by the
>>> C pre-processor trying to track down exactly why a tracepoint() definition
>>> is causing a compiler error, so i think i have an idea of just how hard it
>>> would be to development equivalent header files for Go + C pre-processor.
>>> it may be the case that the Go "header files", once developed, would be
>>> compiler specific, i.e only work with gccgo
>>> <https://golang.org/doc/install/gccgo> (the gcc front-end for Go).
>>>
>>> anyway, i'm wondering if anybody has attempted this and/or what your
>>> thoughts would be.
>>>
>>> In the case of C/C++ code, using the C preprocessor got the job done
>>> without requiring any additional
>>> dependency. Given the context you describe, perhaps it would be wise to
>>> consider introducing a dedicated
>>> "lttng probe description" parser. The idea here would be to parse the
>>> tracepoint probe definitions and
>>> translate those into native code for various languages (e.g. Golang).
>>>
>>> This would create a new lttng utility that would be required to to
>>> produce the language-specific files from
>>> those descriptions. I would allow us to port the "tracepoint" concept to
>>> many more languages easily, and
>>> not be so much tied to the C preprocessor anymore.
>>>
>>
>>> So perhaps adding this new tool as a dependency that needs to be invoked
>>> prior to compilation might
>>> not be too much of an issue ? I see it as being slightly similar to the
>>> role accomplished by Flex and
>>> Bison: they are required to translate from source files to an
>>> intermediary language, and then the
>>> resulting files can be included into the distribution source packages,
>>> so only those who aim at
>>> changing the source descriptions need to have the translation tool
>>> installed.
>>>
>>
>>> Thoughts ?
>>>
>>
>> Good point and we thought about this before. In fact this is just what
>> barectf <http://barectf.org/> does, for example. What I see in the
>> future is that a subset of (eventual) CTF 2's metadata language (JSON)
>> could be used here (or an equivalent, human-friendly YAML, like barectf) to
>> describe LTTng event and stream classes and translate them to what's needed
>> to record such events for a given programming language. CTF 2's user
>> attributes can be used to insert additional, language-specific properties,
>> like parameter types and names, and other options.
>>
>> Phil
>>
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>
>>>
>>>
>>>
>>> --craig
>>>
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev at lists.lttng.org
>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>
>>>
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
>>>
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev at lists.lttng.org
>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20170603/6aa97b1e/attachment.html>