<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div><style><!--
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style></div><div><br></div><div><br></div><div><span id="zwchr" data-marker="__DIVIDER__">----- On Mar 5, 2019, at 4:43 AM, Thomas Schickentanz <thomas.schickentanz@harman.com> wrote:<br></span></div><div data-marker="__QUOTED_TEXT__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1F497D">Hi Mathieu,</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1F497D" lang="EN-US">Thanks for clarifying the restrictions of ‘shm-path’ option!</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1F497D" lang="EN-US">Our purpose is to use exclusively reserved memory for tracing. Our customers request a certain amount of free memory on the system for future development. In development stage chunks
of this “free” memory are grabbed by the applications and tracing lacks memory. We want to allocate this “free” memory in an early boot stage to ensure that we get the amount of memory we need for tracing. E.g. if we have reserved 512 Mbyte of memory, we know
that we can start lttng at a later point of time with 16 buffers of 4 Mbyte on a 8 core system. But this works only, if we can provide the previously reserved memory to lttng for buffer allocation.</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1F497D" lang="EN-US">Is there some general concern about using POSIX shared memory for buffers in kernel tracing mode (apart from performance reasons)?
</span></p></div></blockquote><div>The kernel tracer does not use POSIX shared memory at all at the moment. Only lttng-ust does. lttng-modules allocates memory for its buffers</div><div>directly from the kernel page allocator all by itself.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Switching to a design where an application process (kernel consumerd) would map memory associated to files on a filesystem, and then<br data-mce-bogus="1"></div><div>hand over that memory to the kernel so it can trace there would be a rather large design change.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>For the needs you have, I think a simpler option would be to implement a new setup argument to lttng-modules so it can reserve some memory<br data-mce-bogus="1"></div><div>from the page allocator when the lttng modules are loaded (e.g. at system start), and then use that pre-reserved pool to allocate the buffers<br data-mce-bogus="1"></div><div>when a tracing session is created. All this would need to be NUMA-aware, with e.g. one pool per NUMA node.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>This would be a new feature request against lttng-modules.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>In order to reserve memory for lttng-ust, you could ensure your /dev/shm mountpoint is a tmpfs filesystem which contains enough memory<br data-mce-bogus="1"></div><div>to hold the lttng-ust buffers. An alternative there would be to use pmem + dax to hold the buffers with the --shm-path option.</div><div><br data-mce-bogus="1"></div><div>Thanks,<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Mathieu</div><div><br data-mce-bogus="1"></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div class="WordSection1">
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1F497D" lang="EN-US">I used lttng-crash only for testing, because I wanted to verify, if tracing data is available from provided shared memory. If this is ensured, I agree with you, that ‘lttng snapshot’
can do the work.</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1F497D" lang="EN-US">Regards,</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1F497D" lang="EN-US">Thomas
</span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="mso-fareast-language:DE" lang="EN-US">From:</span></b><span style="mso-fareast-language:DE" lang="EN-US"> Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
<br><b>Sent:</b> Montag, 4. März 2019 17:01<br><b>To:</b> Schickentanz, Thomas <thomas.schickentanz@harman.com><br><b>Cc:</b> lttng-dev <lttng-dev@lists.lttng.org><br><b>Subject:</b> [EXTERNAL] Re: [lttng-dev] Tracing directly into shared memory</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<div>
<div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">----- On Mar 4, 2019, at 4:28 AM, Thomas Schickentanz <<a href="mailto:thomas.schickentanz@harman.com" target="_blank">thomas.schickentanz@harman.com</a>>
wrote:<br><br></span></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #1010FF 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Hi all,</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black" lang="EN-US">My idea is to pre-allocate memory on system startup which can later be used by lttng for tracing. It should work like flight recorder mode without extracting data from ringbuffers
to trace file while recording data. Data extraction of shm ringbuffers should be done when recording has been finished.</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black" lang="EN-US">As per documention with option
</span><span style="font-size:12.0pt;font-family:"Courier New";color:black" lang="EN-US">`--shm-path‘</span><span style="font-size:12.0pt;color:black" lang="EN-US"> ringbuffer for tracing can be created in shared memory. With
</span><span style="font-size:12.0pt;font-family:"Courier New";color:black" lang="EN-US">lttng-crash</span><span style="font-size:12.0pt;color:black" lang="EN-US"> data can be extracted from ringbuffer.</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black" lang="EN-US">Can this option be used with shared memory in volatile memory (default ram), too? And is it possible to record kernel space traces with this option?</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black" lang="EN-US">I tried
</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Courier New";color:black" lang="EN-US">lttng create testshm --shm-path=/dev/shm --snapshot</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Courier New";color:black" lang="EN-US">lttng enable-channel c0 -s testshm -k</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Courier New";color:black" lang="EN-US">lttng enable-event -s testshm -c c0 -k -a</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Courier New";color:black" lang="EN-US">lttng start</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Courier New";color:black" lang="EN-US">lttng stop</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Courier New";color:black" lang="EN-US">lttng-crash -extract=/home/test /dev/shm</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black" lang="EN-US">With lttng and lttng-crash everything seems to work fine (no error messages).</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black" lang="EN-US">However the extract folder keeps empty.</span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black" lang="EN-US">Has someone researched into this option and can help to clarify the points?</span></p>
</blockquote>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">Hi Thomas,</span></p>
</div>
<div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">About "--shm-path" and lttng-crash, those are discussed here:
<a href="https://clicktime.symantec.com/3YZ15pbJVSD47yeXxhXiwqE7Vc?u=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.10%2F%23doc-persistent-memory-file-systems" target="_blank">
https://lttng.org/docs/v2.10/#doc-persistent-memory-file-systems</a></span><br data-mce-bogus="1"></p>
</div>
<div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">I'm not sure I completely understand your use-case and whether your intent is to extract trace data after a</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">system crash or if the "snapshot" mode is enough for your needs.</span></p>
</div>
<div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">Note that the "shm-path" option only applies to UST (user-space) traces at the moment. It has not been implemented</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">for kernel tracing yet.</span></p>
</div>
<div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">The way we typically use it for extraction of trace data from user-space when a system crashes is to use the pmem driver to</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">create a block device out of a range of the system memory, and then use ext2 or ext4 with the DAX mount option to ensure</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">the filesystem bypasses the page cache.</span></p>
</div>
<div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">Then we ensure that kexec is used to boot a new kernel on crash, or make sure the BIOS does not erase the</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">memory content on soft reboot, which gives access to the shm-path buffers after crash/reboot.</span></p>
</div>
<div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">We don't usually use /dev/shm as an argument to --shm-path. This path is normally used to hold POSIX shared memory</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">files. I would expect a different mount point path as argument to --shm-path which points to a DAX-enabled filesystem</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">backed by a PMEM block device.</span></p>
</div>
<div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">Thanks,</span></p>
</div>
<div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">Mathieu</span></p>
</div>
<div>
</div>
</div>
<div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">--
</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE">Mathieu Desnoyers<br>
EfficiOS Inc.<br><a href="https://clicktime.symantec.com/32vnSPUsVCfHCkkwoyPymVx7Vc?u=http%3A%2F%2Fwww.efficios.com" target="_blank">https://clicktime.symantec.com/32vnSPUsVCfHCkkwoyPymVx7Vc?u=http%3A%2F%2Fwww.efficios.com</a></span><br data-mce-bogus="1"></p>
</div>
</div>
</div><br></blockquote></div><div><br></div><div data-marker="__SIG_POST__">-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com</div></div></body></html>