[lttng-dev] Seeking Assistance on LTTng for Kubernetes Performance Research

[Michael Jeanson]

On 2023-04-06 15:56, Benjamin Saint-Cyr via lttng-dev wrote:
> Dear LTTng Developers,
> 
> I am currently conducting performance research on Kubernetes for my master's 
> thesis and am using LTTng for data collection and Trace Compass for analysis. 
> I have a couple of questions and concerns, and I would appreciate your 
> guidance to ensure I am using your tools effectively.
> 
>      1. In the Control Flow view, I can see the CPU throttling due to the 
> quota defined in the Cgroup as the pod runs and then waits for the CPU 
> periodically. However, it is unclear whether the pod is waiting due to a full 
> CPU or because of timeshares. I can easily find this information in 
> /sys/fs/cgroup/cpu,cpuacct/kubepods/<burstable or 
> besteffort>/pod<UID>/cpu.stat if the pod is being throttled. Is there a way to 
> extract this information using LTTng or Trace Compass?
> 
>      2. I am attempting to determine which PID is running a specific 
> Container/Pod UID. Currently, I can deduce this by observing the process named 
> runc[*:INIT], as it creates the cgroup and clones the process for the 
> container. However, this approach is fragile and only works due to an 
> implementation detail. It also fails if tracing starts after deployment. This 
> information is easily available in 
> /sys/fs/cgroup/cpu,cpuacct/kubepods/<burstable or 
> besteffort>/pod<UID>/<container ID>/cgroup.procs. Is there an existing method 
> within LTTng to obtain this information that I may not be aware of?

The PID namespace context is probably the closest thing to what you are 
looking for, see 
"https://lttng.org/man/3/lttng-ust/v2.13/#doc-_context_information". The 
hierachy of the namespaces can be recorded in a statedump.

> 
> I have been informed that it might be possible to read the information in 
> cpu.stat or cgroup.procs using a state dump at the beginning or end, but the 
> process remains unclear. Alternatively, I could instrument the kernel or the 
> process handling the accounting, but this would be quite challenging. Another 
> option would be to use something similar to cAdvisor, but its efficiency is 
> questionable since it dedicates an entire process to reading a few files.

I don't think we have anything cgroup related in the statedump at the moment, 
not that it's impossible but it was never implemented.

> 
> I would greatly appreciate any guidance or suggestions you may have on 
> improving my approach and effectively utilizing LTTng for my research.
> 
> Thank you for your time and assistance.
> 
> Best regards,
> 
> Benjamin Saint-Cyr