[lttng-dev] Tracing directly into shared memory

[Mathieu Desnoyers]

----- On Mar 5, 2019, at 4:43 AM, Thomas Schickentanz <thomas.schickentanz at harman.com> wrote: 

> Hi Mathieu,

> Thanks for clarifying the restrictions of ‘shm-path’ option!

> Our purpose is to use exclusively reserved memory for tracing. Our customers
> request a certain amount of free memory on the system for future development.
> In development stage chunks of this “free” memory are grabbed by the
> applications and tracing lacks memory. We want to allocate this “free” memory
> in an early boot stage to ensure that we get the amount of memory we need for
> tracing. E.g. if we have reserved 512 Mbyte of memory, we know that we can
> start lttng at a later point of time with 16 buffers of 4 Mbyte on a 8 core
> system. But this works only, if we can provide the previously reserved memory
> to lttng for buffer allocation.

> Is there some general concern about using POSIX shared memory for buffers in
> kernel tracing mode (apart from performance reasons)?

The kernel tracer does not use POSIX shared memory at all at the moment. Only lttng-ust does. lttng-modules allocates memory for its buffers 
directly from the kernel page allocator all by itself. 

Switching to a design where an application process (kernel consumerd) would map memory associated to files on a filesystem, and then 
hand over that memory to the kernel so it can trace there would be a rather large design change. 

For the needs you have, I think a simpler option would be to implement a new setup argument to lttng-modules so it can reserve some memory 
from the page allocator when the lttng modules are loaded (e.g. at system start), and then use that pre-reserved pool to allocate the buffers 
when a tracing session is created. All this would need to be NUMA-aware, with e.g. one pool per NUMA node. 

This would be a new feature request against lttng-modules. 

In order to reserve memory for lttng-ust, you could ensure your /dev/shm mountpoint is a tmpfs filesystem which contains enough memory 
to hold the lttng-ust buffers. An alternative there would be to use pmem + dax to hold the buffers with the --shm-path option. 

Thanks, 

Mathieu 

> I used lttng-crash only for testing, because I wanted to verify, if tracing data
> is available from provided shared memory. If this is ensured, I agree with you,
> that ‘lttng snapshot’ can do the work.

> Regards,

> Thomas

> From: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
> Sent: Montag, 4. März 2019 17:01
> To: Schickentanz, Thomas <thomas.schickentanz at harman.com>
> Cc: lttng-dev <lttng-dev at lists.lttng.org>
> Subject: [EXTERNAL] Re: [lttng-dev] Tracing directly into shared memory

> ----- On Mar 4, 2019, at 4:28 AM, Thomas Schickentanz < [
> mailto:thomas.schickentanz at harman.com | thomas.schickentanz at harman.com ] >
> wrote:

>> Hi all,

>> My idea is to pre-allocate memory on system startup which can later be used by
>> lttng for tracing. It should work like flight recorder mode without extracting
>> data from ringbuffers to trace file while recording data. Data extraction of
>> shm ringbuffers should be done when recording has been finished.

>> As per documention with option `--shm-path‘ ringbuffer for tracing can be
>> created in shared memory. With lttng-crash data can be extracted from
>> ringbuffer.

>> Can this option be used with shared memory in volatile memory (default ram),
>> too? And is it possible to record kernel space traces with this option?

>> I tried

>> lttng create testshm --shm-path=/dev/shm --snapshot

>> lttng enable-channel c0 -s testshm -k

>> lttng enable-event -s testshm -c c0 -k -a

>> lttng start

>> lttng stop

>> lttng-crash -extract=/home/test /dev/shm

>> With lttng and lttng-crash everything seems to work fine (no error messages).

>> However the extract folder keeps empty.

>> Has someone researched into this option and can help to clarify the points?
> Hi Thomas,

> About "--shm-path" and lttng-crash, those are discussed here: [
> https://clicktime.symantec.com/3YZ15pbJVSD47yeXxhXiwqE7Vc?u=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.10%2F%23doc-persistent-memory-file-systems
> |
> https://lttng.org/docs/v2.10/#doc-persistent-memory-file-systems ]

> I'm not sure I completely understand your use-case and whether your intent is to
> extract trace data after a

> system crash or if the "snapshot" mode is enough for your needs.

> Note that the "shm-path" option only applies to UST (user-space) traces at the
> moment. It has not been implemented

> for kernel tracing yet.

> The way we typically use it for extraction of trace data from user-space when a
> system crashes is to use the pmem driver to

> create a block device out of a range of the system memory, and then use ext2 or
> ext4 with the DAX mount option to ensure

> the filesystem bypasses the page cache.

> Then we ensure that kexec is used to boot a new kernel on crash, or make sure
> the BIOS does not erase the

> memory content on soft reboot, which gives access to the shm-path buffers after
> crash/reboot.

> We don't usually use /dev/shm as an argument to --shm-path. This path is
> normally used to hold POSIX shared memory

> files. I would expect a different mount point path as argument to --shm-path
> which points to a DAX-enabled filesystem

> backed by a PMEM block device.

> Thanks,

> Mathieu

> --

> Mathieu Desnoyers
> EfficiOS Inc.
> [
> https://clicktime.symantec.com/32vnSPUsVCfHCkkwoyPymVx7Vc?u=http%3A%2F%2Fwww.efficios.com
> |
> https://clicktime.symantec.com/32vnSPUsVCfHCkkwoyPymVx7Vc?u=http%3A%2F%2Fwww.efficios.com
> ]

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20190305/ef0208d6/attachment.html>