[lttng-dev] Shared memory circular buffer and K8s pods

[Lemay, Paul]

Merci Jérémie!
Pour te mettre en contexte, nous somme L'organisation télécom 5G de Dell. Je regarde LTTng présentement et je suis intéressé par la fonctone que vous décrivez comme étant "Flight Recorder". 

Je vais lire ces document et merci encore pour la réponse rapide!


Internal Use - Confidential

-----Original Message-----
From: Jérémie Galarneau <jeremie.galarneau at efficios.com> 
Sent: Tuesday, October 18, 2022 2:34 PM
To: Lemay, Paul
Cc: lttng-dev at lists.lttng.org
Subject: Re: [lttng-dev] Shared memory circular buffer and K8s pods


[EXTERNAL EMAIL] 

Hi Paul, 

We don’t have much documentation about deploying in Kubernetes; we’re working on it.

It’s hard to provide very specific guidance without knowing your setup and constraints. Since you mention applications, I’m assuming you are mostly interested in user space tracing. There are two main approaches for that.

First, you can deploy the full LTTng stack in every container: the daemons of LTTng-tools, and LTTng-UST + your app. That’s probably the simplest way to get going, but it goes against K8s’ model of having one process per container, which can become annoying.

The other approach, which I would recommend you look into, is deploying an LTTng sidecar per pod that runs the session daemon, and installing LTTng-UST (the tracer libraries) along your instrumented applications in the other containers of the pod.

Michael Jeanson has a demo showing how he deployed LTTng in K8s using that approach. Hopefully, that can help you get started:

https://urldefense.com/v3/__https://github.com/mjeanson/gcloud-lttng__;!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye8lWbYy9w$  [github[.]com]

There are a couple things to note in this configuration:
  - A “memory” volume, devshm, is shared between containers. It has to be
    large enough to accommodate your buffer configuration, and
  - the containers share the same pid namespace across the pod
    (https://urldefense.com/v3/__https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/__;!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye-CE3jM7Q$  [kubernetes[.]io]).

Then that leaves the question of where you want to store traces. There’s a ton of options there. By default, Michael's demo writes them to the lttng-home volume. You may want to look into network streaming to send traces over the network to a relay daemon (possibly running in another pod):

https://urldefense.com/v3/__https://lttng.org/docs/v2.13/*doc-lttng-relayd__;Iw!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye8DDrqZHg$  [lttng[.]org]

Otherwise, you may also be interested in session rotations to manage traces pretty much how you would handle logs:

https://urldefense.com/v3/__https://lttng.org/blog/2019/10/15/lttng-session-rotation/__;!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye8jbW2hug$  [lttng[.]org]

https://urldefense.com/v3/__https://lttng.org/docs/v2.13/*doc-session-rotation__;Iw!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye_2vJ9Z8g$  [lttng[.]org]

Let me know if you have more questions,
Jérémie

On Tue, Oct 18, 2022 at 01:00:47PM +0000, Lemay, Paul via lttng-dev wrote:
> Hello There,
> 
> We are looking at LTTng for a development here at Dell. As I understand the LTTng, there are shared memories of circular buffer traces created between  an application and the consumer daemon. After looking at some presentation that you did, I see that you have started working on containerization and pods.
> 
> So our applications run in pods, and am not sure how to organize the pods knowing about those shared memories? Can you help us better understand how our code should be organized in the context of Kubernetes? Any documentation that we can access?
> 
> Best Regards
> 
> 
> Internal Use - Confidential

> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://urldefense.com/v3/__https://lists.lttng.org/cgi-bin/mailman/li
> stinfo/lttng-dev__;!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9
> KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye8k-nnC1A$  [lists[.]lttng[.]org]