[lttng-dev] Deleted files held open by LTTng

[Mathieu Desnoyers]

----- On Oct 12, 2020, at 5:04 AM, lttng-dev lttng-dev at lists.lttng.org wrote:

> Hi all,
> 
> I have recently started using LTTng and stumbled upon the fact, that
> LTTng is keeping a lot of open file pointers to a single deleted file.
> 
> Specifically the process that holds these open is the lttng-consumerd and its
> threads. The file that is held open, but is deleted is
> /dev/shm/ust-shm-consumer-${PID} with the PID of the lttng-comsumerd. The
> number of open file descriptors is dependent on the number of CPUs the
> system has and results to 16 + (#CPUs * 16). Is this behavior expected?
> 
> # lsof | grep lttng | grep DEL
> lttng-con  2280             root  DEL       REG               0,18
> 11128
>        /dev/shm/ust-shm-consumer-2280
> lttng-con  2280             root  DEL       REG               0,18
> 11127
>        /dev/shm/ust-shm-consumer-2280
> lttng-con  2280             root  DEL       REG               0,18
> 11126
>        /dev/shm/ust-shm-consumer-2280
> [...]
> 
> I have tested this on a Xen-Dom0 system running SLES 12 SP5 and on a
> SLES12 SP5 without Xen.

Yes, it works exactly as designed. The inodes to each of those files stay
allocated as along as there is at least one file descriptor referencing them.
We use this scheme to facilitate cleaning up when the consumer daemon is
killed (including with SIGKILL) so it does not leave stale files hanging
around.

Note that on Linux, and starting from Linux kernel 3.17, we could instead
use the memfd_create system call to create those shared mappings, which AFAIU
would remove the need for unlinking the shm files after their creation.
However, considering that LTTng-tools and LTTng-UST also works on older Linux
kernels, and on BSDs, using the POSIX shm still appears to be appropriate.

On the reason for having one shm file per cpu (per channel/per uid/per session):
having one file per cpu is mainly done to facilitate NUMA-local memory allocation.
That being said, I wonder if as a future improvement we could combine all per-cpu
buffers into a single shm file, and issue numa_set_preferred piecewise when
mmapping regions of the files. This would lessen the number of file descriptors
needed by a significant amount. I suspect it would work, but we'd need to do some
prototyping to validate this approach first.

Thanks,

Mathieu

> 
> Best regards,
> Nils Döring
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com