[lttng-dev] Tracing directly into shared memory
mathieu.desnoyers at efficios.com
Tue Mar 5 11:24:12 EST 2019
----- On Mar 5, 2019, at 4:43 AM, Thomas Schickentanz <thomas.schickentanz at harman.com> wrote:
> Hi Mathieu,
> Thanks for clarifying the restrictions of ‘shm-path’ option!
> Our purpose is to use exclusively reserved memory for tracing. Our customers
> request a certain amount of free memory on the system for future development.
> In development stage chunks of this “free” memory are grabbed by the
> applications and tracing lacks memory. We want to allocate this “free” memory
> in an early boot stage to ensure that we get the amount of memory we need for
> tracing. E.g. if we have reserved 512 Mbyte of memory, we know that we can
> start lttng at a later point of time with 16 buffers of 4 Mbyte on a 8 core
> system. But this works only, if we can provide the previously reserved memory
> to lttng for buffer allocation.
> Is there some general concern about using POSIX shared memory for buffers in
> kernel tracing mode (apart from performance reasons)?
The kernel tracer does not use POSIX shared memory at all at the moment. Only lttng-ust does. lttng-modules allocates memory for its buffers
directly from the kernel page allocator all by itself.
Switching to a design where an application process (kernel consumerd) would map memory associated to files on a filesystem, and then
hand over that memory to the kernel so it can trace there would be a rather large design change.
For the needs you have, I think a simpler option would be to implement a new setup argument to lttng-modules so it can reserve some memory
from the page allocator when the lttng modules are loaded (e.g. at system start), and then use that pre-reserved pool to allocate the buffers
when a tracing session is created. All this would need to be NUMA-aware, with e.g. one pool per NUMA node.
This would be a new feature request against lttng-modules.
In order to reserve memory for lttng-ust, you could ensure your /dev/shm mountpoint is a tmpfs filesystem which contains enough memory
to hold the lttng-ust buffers. An alternative there would be to use pmem + dax to hold the buffers with the --shm-path option.
> I used lttng-crash only for testing, because I wanted to verify, if tracing data
> is available from provided shared memory. If this is ensured, I agree with you,
> that ‘lttng snapshot’ can do the work.
> From: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
> Sent: Montag, 4. März 2019 17:01
> To: Schickentanz, Thomas <thomas.schickentanz at harman.com>
> Cc: lttng-dev <lttng-dev at lists.lttng.org>
> Subject: [EXTERNAL] Re: [lttng-dev] Tracing directly into shared memory
> ----- On Mar 4, 2019, at 4:28 AM, Thomas Schickentanz < [
> mailto:thomas.schickentanz at harman.com | thomas.schickentanz at harman.com ] >
>> Hi all,
>> My idea is to pre-allocate memory on system startup which can later be used by
>> lttng for tracing. It should work like flight recorder mode without extracting
>> data from ringbuffers to trace file while recording data. Data extraction of
>> shm ringbuffers should be done when recording has been finished.
>> As per documention with option `--shm-path‘ ringbuffer for tracing can be
>> created in shared memory. With lttng-crash data can be extracted from
>> Can this option be used with shared memory in volatile memory (default ram),
>> too? And is it possible to record kernel space traces with this option?
>> I tried
>> lttng create testshm --shm-path=/dev/shm --snapshot
>> lttng enable-channel c0 -s testshm -k
>> lttng enable-event -s testshm -c c0 -k -a
>> lttng start
>> lttng stop
>> lttng-crash -extract=/home/test /dev/shm
>> With lttng and lttng-crash everything seems to work fine (no error messages).
>> However the extract folder keeps empty.
>> Has someone researched into this option and can help to clarify the points?
> Hi Thomas,
> About "--shm-path" and lttng-crash, those are discussed here: [
> https://lttng.org/docs/v2.10/#doc-persistent-memory-file-systems ]
> I'm not sure I completely understand your use-case and whether your intent is to
> extract trace data after a
> system crash or if the "snapshot" mode is enough for your needs.
> Note that the "shm-path" option only applies to UST (user-space) traces at the
> moment. It has not been implemented
> for kernel tracing yet.
> The way we typically use it for extraction of trace data from user-space when a
> system crashes is to use the pmem driver to
> create a block device out of a range of the system memory, and then use ext2 or
> ext4 with the DAX mount option to ensure
> the filesystem bypasses the page cache.
> Then we ensure that kexec is used to boot a new kernel on crash, or make sure
> the BIOS does not erase the
> memory content on soft reboot, which gives access to the shm-path buffers after
> We don't usually use /dev/shm as an argument to --shm-path. This path is
> normally used to hold POSIX shared memory
> files. I would expect a different mount point path as argument to --shm-path
> which points to a DAX-enabled filesystem
> backed by a PMEM block device.
> Mathieu Desnoyers
> EfficiOS Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the lttng-dev