[lttng-dev] Golang agent for LTTng-ust

[Loïc Gelle]

Hi Jeremie,

Thanks for your answer. I roughly estimated the overhead of calling an  
empty C function (passing two integer arguments) from Go to 50ns per  
call. Maybe not a big deal for a lot of use cases, but more  
problematic if you want to trace performance-critical parts of Go like  
its runtime itself. The overhead could even be bigger when it involves  
passing strings or arrays that have different memory layouts in Golang  
and C. What was the overhead that you observed for Python and Java?

 From what I understand, it will always be a problem to have agents  
for languages different than C, especially if you want to keep relying  
on existing C code. Even if the sessiond part is independant from the  
agent itself, there are tons of implementation issues that are  
specific to the language itself. The problem with Go is that calling C  
functions is really a hack that does not integrate well with the build  
system that was designed for Go.

Did I provide more context?

Cheers,
Loïc.

Jérémie Galarneau <jeremie.galarneau at efficios.com> a écrit :

> On 4 May 2018 at 06:03, Loïc Gelle <loic.gelle at polymtl.ca> wrote:
>
>> Hi,
>>
>> There has been a previous discussion on the mailing list about porting
>> LTTng to Golang, about a year ago: https://lists.lttng.org/
>> pipermail/lttng-dev/2017-June/027203.html . This new topic is to discuss
>> more precisely about implementation possibilities.
>>
>> Currently, one has to use the the C UST agent from LTTng in order to
>> instrument Golang programs, and to compile the whole thing using custom
>> Makefiles and cgo. Here is a recent example that I wrote:
>> https://github.com/loicgelle/jaeger-go-lttng-instr
>>
>> As you can guess, there are a low of drawbacks in that approach. It is
>> actually a hack and cannot be integrated into more complex Golang program
>> that use a more complex build process (e.g. the Golang runtime itself),
>> because of the compiler instructions that you have to include at the top of
>> the Golang files. There is also a big concern about the performance of this
>> solution, as calling a C function from Go requires to do a full context
>> switch on the stack, because the calling conventions in C and Golang are
>> different.
>>
>
>> I think a more integrated and performant solution is needed. We can’t
>> really ignore a language such as Golang that is now widely adopted for
>> cloud applications. LTTng is really the best solution out there in terms of
>> overhead per tracepoint, and could benefit from being made available to
>> such a large community. My question to the experts on this mailing list:
>> how much would it take to write a Golang agent for LTTng?
>>
>
>
> Hi Loïc,
>
> Without having performed any measurements myself, it does seem like calling
> C from Go is very expensive. In that context, I can see that LTTng would
> probably lose its performance advantage over any native Go solution.
> However, it wouldn't hurt to measure the impact and see if it really is a
> deal breaker.
>
> We faced the same dilemma when implementing the Java and Python support in
> lttng-ust. In those cases, we ended up calling C code, with the performance
> penalties it implies. The correlation with other applications' and the
> kernel's events, along with the rest of LTTng's features, provided enough
> value to make that solution worthwhile.
>
> There aren't a ton of solutions if we can't call existing C code. We
> basically have to reimplement a ring-buffer and the setup/communication
> infrastructure to interact with the lttng-sessiond. The communication with
> the session daemon is not a big concern as the protocol is fairly
> straightforward.
>
> The "hairy" part is that lttng-ust and lttng-consumerd use a shared memory
> map to produce and consume the tracing buffers. This means that all changes
> to that memory layout would need to be replicated in the Go tracer, making
> future evolution more difficult. Also, I don't know how easy it would be to
> synchronize C and Go applications interacting in a shared memory map given
> those languages have different memory models. My knowledge of Go doesn't go
> that far.
>
> A more viable solution could be to introduce a Go-native consumer daemon
> implementing its own synchronization with Go applications. This way, that
> implementation could evolve on its own and could also start with a simpler
> ring buffer than lttng-ust's.
>
> Still, it is not a small undertaking and it basically means maintaining a
> third tracer implementation.
>
>
> What do you think?
>
> Thanks!
> Jérémie
>
>
>> Cheers,
>> Loïc.
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev at lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>>
>
>
> --
> Jérémie Galarneau
> EfficiOS Inc.
> http://www.efficios.com