[lttng-dev] Shared memory circular buffer and K8s pods

[Jérémie Galarneau]

Hi Paul, 

We don’t have much documentation about deploying in Kubernetes; we’re working on
it.

It’s hard to provide very specific guidance without knowing your setup and
constraints. Since you mention applications, I’m assuming you are mostly
interested in user space tracing. There are two main approaches for that.

First, you can deploy the full LTTng stack in every container: the daemons of
LTTng-tools, and LTTng-UST + your app. That’s probably the simplest way to get
going, but it goes against K8s’ model of having one process per container, which
can become annoying.

The other approach, which I would recommend you look into, is deploying an LTTng
sidecar per pod that runs the session daemon, and installing LTTng-UST (the
tracer libraries) along your instrumented applications in the other containers
of the pod.

Michael Jeanson has a demo showing how he deployed LTTng in K8s using that
approach. Hopefully, that can help you get started:

https://github.com/mjeanson/gcloud-lttng

There are a couple things to note in this configuration:
  - A “memory” volume, devshm, is shared between containers. It has to be
    large enough to accommodate your buffer configuration, and
  - the containers share the same pid namespace across the pod
    (https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/).

Then that leaves the question of where you want to store traces. There’s a ton
of options there. By default, Michael's demo writes them to the lttng-home
volume. You may want to look into network streaming to send traces over the
network to a relay daemon (possibly running in another pod):

https://lttng.org/docs/v2.13/#doc-lttng-relayd

Otherwise, you may also be interested in session rotations to manage traces
pretty much how you would handle logs:

https://lttng.org/blog/2019/10/15/lttng-session-rotation/

https://lttng.org/docs/v2.13/#doc-session-rotation

Let me know if you have more questions,
Jérémie

On Tue, Oct 18, 2022 at 01:00:47PM +0000, Lemay, Paul via lttng-dev wrote:
> Hello There,
> 
> We are looking at LTTng for a development here at Dell. As I understand the LTTng, there are shared memories of circular buffer traces created between  an application and the consumer daemon. After looking at some presentation that you did, I see that you have started working on containerization and pods.
> 
> So our applications run in pods, and am not sure how to organize the pods knowing about those shared memories? Can you help us better understand how our code should be organized in the context of Kubernetes? Any documentation that we can access?
> 
> Best Regards
> 
> 
> Internal Use - Confidential

> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev