[lttng-dev] Deleted files held open by LTTng

[nils.doering at fujitsu.com]

>> Hi all,
>> 
>> I have recently started using LTTng and stumbled upon the fact, that
>> LTTng is keeping a lot of open file pointers to a single deleted file.
>> 
>> Specifically the process that holds these open is the lttng-consumerd and its
>> threads. The file that is held open, but is deleted is
>> /dev/shm/ust-shm-consumer-${PID} with the PID of the lttng-comsumerd. The
>> number of open file descriptors is dependent on the number of CPUs the
>> system has and results to 16 + (#CPUs * 16). Is this behavior expected?
>> 
>> # lsof | grep lttng | grep DEL
>> lttng-con  2280             root  DEL       REG               0,18
>> 11128
>>        /dev/shm/ust-shm-consumer-2280
>> lttng-con  2280             root  DEL       REG               0,18
>> 11127
>>        /dev/shm/ust-shm-consumer-2280
>> lttng-con  2280             root  DEL       REG               0,18
>> 11126
>>        /dev/shm/ust-shm-consumer-2280
>> [...]
>> 
>> I have tested this on a Xen-Dom0 system running SLES 12 SP5 and on a
>> SLES12 SP5 without Xen.
> 
> Yes, it works exactly as designed. The inodes to each of those files stay
> allocated as along as there is at least one file descriptor referencing them.
> We use this scheme to facilitate cleaning up when the consumer daemon is
> killed (including with SIGKILL) so it does not leave stale files hanging
> around.
> 
> 
> Note that on Linux, and starting from Linux kernel 3.17, we could instead
> use the memfd_create system call to create those shared mappings, which AFAIU
> would remove the need for unlinking the shm files after their creation.
> However, considering that LTTng-tools and LTTng-UST also works on older Linux
> kernels, and on BSDs, using the POSIX shm still appears to be appropriate. >
>
> On the reason for having one shm file per cpu (per channel/per uid/per session):
> having one file per cpu is mainly done to facilitate NUMA-local memory allocation.
> That being said, I wonder if as a future improvement we could combine all per-cpu
> buffers into a single shm file, and issue numa_set_preferred piecewise when
> mmapping regions of the files. This would lessen the number of file descriptors
> needed by a significant amount. I suspect it would work, but we'd need to do some
> prototyping to validate this approach first.
>
> Thanks,
>
> Mathieu
>

Thank you for your explanation. This helps a lot in understanding.

Best wishes,
Nils Döring

>> 
>> Best regards,
>> Nils Döring
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev at lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com