[lttng-dev] Tracing directly into shared memory

[Schickentanz, Thomas]

Hi Mathieu,

Thanks for clarifying the restrictions of ‘shm-path’ option!
Our purpose is to use exclusively reserved memory for tracing. Our customers request a certain amount of free memory on the system for future development. In development stage chunks of this “free” memory are grabbed by the applications and tracing lacks memory. We want to allocate this “free” memory in an early boot stage to ensure that we get the amount of memory we need for tracing. E.g. if we have reserved 512 Mbyte of memory, we know that we can start lttng at a later point of time with 16 buffers of 4 Mbyte on a 8 core system. But this works only, if we can provide the previously reserved memory to lttng for buffer allocation.
Is there some general concern about using POSIX shared memory for buffers in kernel tracing mode (apart from performance reasons)?

I used lttng-crash only for testing, because I wanted to verify, if tracing data is available from provided shared memory. If this is ensured, I agree with you, that ‘lttng snapshot’ can do the work.

Regards,
Thomas

From: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
Sent: Montag, 4. März 2019 17:01
To: Schickentanz, Thomas <thomas.schickentanz at harman.com>
Cc: lttng-dev <lttng-dev at lists.lttng.org>
Subject: [EXTERNAL] Re: [lttng-dev] Tracing directly into shared memory


----- On Mar 4, 2019, at 4:28 AM, Thomas Schickentanz <thomas.schickentanz at harman.com<mailto:thomas.schickentanz at harman.com>> wrote:

Hi all,
My idea is to pre-allocate memory on system startup which can later be used by lttng for tracing. It should work like flight recorder mode without extracting data from ringbuffers to trace file while recording data. Data extraction of shm ringbuffers should be done when recording has been finished.
As per documention with option `--shm-path‘ ringbuffer for tracing can be created in shared memory. With lttng-crash data can be extracted from ringbuffer.
Can this option be used with shared memory in volatile memory (default ram), too? And is it possible to record kernel space traces with this option?
I tried
lttng create testshm --shm-path=/dev/shm --snapshot
lttng enable-channel c0 -s testshm -k
lttng enable-event -s testshm -c c0 -k -a
lttng start
lttng stop
lttng-crash -extract=/home/test /dev/shm
With lttng and lttng-crash everything seems to work fine (no error messages).
However the extract folder keeps empty.
Has someone researched into this option and can help to clarify the points?
Hi Thomas,

About "--shm-path" and lttng-crash, those are discussed here: https://lttng.org/docs/v2.10/#doc-persistent-memory-file-systems<https://clicktime.symantec.com/3YZ15pbJVSD47yeXxhXiwqE7Vc?u=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.10%2F%23doc-persistent-memory-file-systems>

I'm not sure I completely understand your use-case and whether your intent is to extract trace data after a
system crash or if the "snapshot" mode is enough for your needs.

Note that the "shm-path" option only applies to UST (user-space) traces at the moment. It has not been implemented
for kernel tracing yet.

The way we typically use it for extraction of trace data from user-space when a system crashes is to use the pmem driver to
create a block device out of a range of the system memory, and then use ext2 or ext4 with the DAX mount option to ensure
the filesystem bypasses the page cache.

Then we ensure that kexec is used to boot a new kernel on crash, or make sure the BIOS does not erase the
memory content on soft reboot, which gives access to the shm-path buffers after crash/reboot.

We don't usually use /dev/shm as an argument to --shm-path. This path is normally used to hold POSIX shared memory
files. I would expect a different mount point path as argument to --shm-path which points to a DAX-enabled filesystem
backed by a PMEM block device.

Thanks,

Mathieu


--
Mathieu Desnoyers
EfficiOS Inc.
https://clicktime.symantec.com/32vnSPUsVCfHCkkwoyPymVx7Vc?u=http%3A%2F%2Fwww.efficios.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20190305/58960f72/attachment-0001.html>