[lttng-dev] porting LTTng to Go(lang)?

Sun Jun 4 16:05:19 UTC 2017

----- On Jun 3, 2017, at 3:45 PM, Philippe Proulx <eeppeliteloop at gmail.com> wrote: 

> On Fri, Jun 2, 2017 at 10:24 PM, craig harmer < [ mailto:charmster at gmail.com |
> charmster at gmail.com ] > wrote:

>> hi all,

>> the suggestion to write a pre-processor for LTTng tracepoints is interesting and
>> may be a good idea. however, i'm not sure that it will provide a general
>> purpose solution.

>> the problem i for see is that tracepoint invocations, at least, should be
>> written in the syntax of the language they're being used in (a programmer would
>> would expect the tracepoint() function call to look like any other function
>> call in the language).

>> so the tracepoint parser needs to understand the function invocation syntax of
>> each language. it also needs to understand at least some of the rest of the
>> syntax of the language -- in the case of Go it at least needs to be able
>> recognize "import" statements and import those files and it needs to understand
>> when something that looks like a tracepoint() is actually embedded in a comment
>> or string. it also needs to emit code that matches the syntax of the language.

>> so a general purpose lttng tracepoint parsing tool would need at least a limited
>> understanding of the syntax and grammar of each language for reading the source
>> file in and then writing the source file out. there could be code in the middle
>> that's common across languages, but probably not a lot. perhaps an lttng
>> tracepoint parsing tool could be better structured as a common framework for
>> language specific tracepoint parsers to be plugged in to.

>> but writing a parser that actually understands the grammar of a language is not
>> easy. if i was really going to do this, i'd be tempted to see if i could just
>> take this from gcc (which has front-ends that supports C, C++, Go, and other
>> languages (i think)).

>> and then there's the tracepoint definitions. they could be written using the
>> barectf() tool that Philippe mentions, but i'm not sure if that would be
>> satisfactory for most programmers, who would prefer to write tracepoint
>> definitions in the same language that they are programming in. they would also
>> need to support the data types of the language (i'm thinking Python here as an
>> example, where tracepoints would be expected to support most of the Python
>> basic data types).

> There's a little misunderstanding here.

> The idea is to have a universal tracepoint definition language (and what I
> suggested is that a subset of CTF 2, of which the syntax is JSON (or YAML which
> translates to JSON directly), could be used for this) which translates to
> functions, classes, macros, etc. in selected programming languages. Depending
> on the destination language, we can generate serialization functions directly
> or wrappers which use LTTng-UST behind the scenes. So in your case, this tool
> would generate Go source files which would contain public functions which
> correspond to tracepoints. We never considered parsing an existing programming
> language to infer tracepoints.

> Of course this requires the developer to define tracepoints in this universal
> language. That's an additional developing step needed to allow the developer to
> use tracepoints in his/her favorite language. But there are other advantages,
> like generating tracepoints in two different languages from the same tracepoint
> definitions. Like Mathieu wrote, it's the same spirit as Flex and Bison. With
> Bison, you don't use the C preprocessor to write your grammar rules: the tool
> reads a Bison input file and generates a parser source file. Just like barectf
> reads a YAML configuration file which defines stream and event classes and
> generates tracer source files.

The problem is that this is often not enough, and we sometimes need to have 
pre/post code run before/after the serialization to prepare the arguments. 
This happens for instance in lttng-modules with the LTTNG_TRACEPOINT_EVENT_CODE() 
macro. 

Ideally we should find a way to glue together this high-level description 
and language-native helper code. 

Thoughts ? 

Thanks, 

Mathieu 

> Phil

>> because Go seems to be similar to C i'm strongly tempted to just co-opt the C
>> pre-processor and existing LTTng tracepoint machinery and modify the LTTng
>> header files to generate valid Go code for a tracepoint definition. this does
>> add a LTTng tracepoint tool to the steps required to compile Go, except that
>> the tracepoint tool is called "cpp" (plus header files).

>> i don't really know Go yet and i hae not been assigned to get LTTng to work in
>> Go, so this is all quite preliminary. but i think the project i'm about to
>> start on really needs LTTng or LTTng-like tracepoint functionality so i'm
>> motivated to see if this can work.

>> i'm going to investigate some more.

>> --craig

>> On Fri, Jun 2, 2017 at 10:05 AM, Philippe Proulx < [
>> mailto:eeppeliteloop at gmail.com | eeppeliteloop at gmail.com ] > wrote:

>>> On Fri, Jun 2, 2017 at 11:58 AM, Mathieu Desnoyers < [
>>> mailto:mathieu.desnoyers at efficios.com | mathieu.desnoyers at efficios.com ] >
>>> wrote:

>>>> ----- On Jun 2, 2017, at 2:27 AM, craig harmer < [ mailto:charmster at gmail.com |
>>>> charmster at gmail.com ] > wrote:

>>>>> hi all,

>>>>> has anyone looked at porting LTTng to [
>>>>> https://en.wikipedia.org/wiki/Go_%28programming_language%29 | Go ] (AKA [
>>>>> https://golang.org/ | Golang ] )?

>>>> Not yet ! I'm glad someone is looking into it. :)

>>>>> Go is a language very similar to C, but it has intrinsic support for concurrency
>>>>> and better memory protection (at the expense of including garbage collection).
>>>>> its from Google and is pretty hot these days for big data and "web scale"
>>>>> distributed systems. the similarity to C is not surprising since two of the
>>>>> three authors were also involved in the development of UNIX in at AT&T in the
>>>>> 1970's: Rob Pike and Ken Thompson (Robert Griesemer is the third author).

>>>>> Go is able to link with C object files, so implementing support for tracef(3)
>>>>> should not be very difficult.

>>>>> but i want to have full support for user-defined tracepoint() events. that looks
>>>>> to be much trickier since the LTTng tracepoint() functionality relies heavily
>>>>> on (some might say abuses

>>> "abuses" is appropriate ;-)

>>>>> ) the C pre-processor, and Go does not have an equivalent of the C
>>>>> pre-processor. however, i don't see an obvious reason why Go source files that
>>>>> contain tracepoints couldn't be run through the C pre-processor -- except that
>>>>> Go specific LTTng header files would be required.

>>>>> [ https://tour.golang.org/welcome/1 | here is an example of "hello, world" in Go
>>>>> ] . (note the use of "import" rather then "#include").

>>>>> i've spent more than a few hours looking at the C code generated by the C
>>>>> pre-processor trying to track down exactly why a tracepoint() definition is
>>>>> causing a compiler error, so i think i have an idea of just how hard it would
>>>>> be to development equivalent header files for Go + C pre-processor. it may be
>>>>> the case that the Go "header files", once developed, would be compiler
>>>>> specific, i.e only work with [ https://golang.org/doc/install/gccgo | gccgo ]
>>>>> (the gcc front-end for Go).

>>>>> anyway, i'm wondering if anybody has attempted this and/or what your thoughts
>>>>> would be.

>>>> In the case of C/C++ code, using the C preprocessor got the job done without
>>>> requiring any additional
>>>> dependency. Given the context you describe, perhaps it would be wise to consider
>>>> introducing a dedicated
>>>> "lttng probe description" parser. The idea here would be to parse the tracepoint
>>>> probe definitions and
>>>> translate those into native code for various languages (e.g. Golang).

>>>> This would create a new lttng utility that would be required to to produce the
>>>> language-specific files from
>>>> those descriptions. I would allow us to port the "tracepoint" concept to many
>>>> more languages easily, and
>>>> not be so much tied to the C preprocessor anymore.

>>>> So perhaps adding this new tool as a dependency that needs to be invoked prior
>>>> to compilation might
>>>> not be too much of an issue ? I see it as being slightly similar to the role
>>>> accomplished by Flex and
>>>> Bison: they are required to translate from source files to an intermediary
>>>> language, and then the
>>>> resulting files can be included into the distribution source packages, so only
>>>> those who aim at
>>>> changing the source descriptions need to have the translation tool installed.

>>>> Thoughts ?

>>> Good point and we thought about this before. In fact this is just what barectf <
>>> [ http://barectf.org/ | http://barectf.org/ ] > does, for example. What I see
>>> in the future is that a subset of (eventual) CTF 2's metadata language (JSON)
>>> could be used here (or an equivalent, human-friendly YAML, like barectf) to
>>> describe LTTng event and stream classes and translate them to what's needed to
>>> record such events for a given programming language. CTF 2's user attributes
>>> can be used to insert additional, language-specific properties, like parameter
>>> types and names, and other options.

>>> Phil

>>>> Thanks,

>>>> Mathieu

>>>>> --craig

>>>>> _______________________________________________
>>>>> lttng-dev mailing list
>>>>> [ mailto:lttng-dev at lists.lttng.org | lttng-dev at lists.lttng.org ]
>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>> _______________________________________________
>>>> lttng-dev mailing list
>>>> [ mailto:lttng-dev at lists.lttng.org | lttng-dev at lists.lttng.org ]
>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20170604/b84c3045/attachment-0001.html>