<div dir="ltr">Thanks Kienan for these quick suggestions, <br>we'll investigate the pmem route (I was not aware of the lttng-cash utility, it's pretty nice) even if I'm not sure how fast it would burn through our SSD, it might still be worth trying.<br>As for kexec-tool, it's not officially supported on our embedded modules unfortunately, so we might be SOL there. We may have to try to add our own trace-point in kernel to use as trigger.<br>Cheers<div>Damien</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 16, 2024 at 8:12 AM Kienan Stewart <<a href="mailto:kstewart@efficios.com">kstewart@efficios.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Damien,<br>
<br>
I want to expand on one of the options that could work for your case.<br>
<br>
On 5/16/24 9:37 AM, Kienan Stewart via lttng-dev wrote:<br>
> Hi Damien,<br>
> <br>
> <br>
> On 5/15/24 6:24 PM, Damien Berget via lttng-dev wrote:<br>
>> Good day,<br>
>> we have been using LTTng successfully to capture snapshots on user <br>
>> defined tracepoints and it did provide invaluable to debug our issues. <br>
>> Thanks to all the contributors of this project!<br>
>><br>
>> We'd like to know if it would be possible to trigger on a kernel <br>
>> panic? I might be dubiously possible as you would still need to have <br>
>> the file-system working to write the results but I should ask.<br>
>><br>
> <br>
> For userspace tracing, I think the recommendation is usually to use a <br>
> dax/pmem device and have the buffers for the session mapped there. After <br>
> a panic, the contents of the buffers can be restored using lttng-crash[1].<br>
> <br>
> Note that dax/pem isn't supported by the kernel space tracer at this time.<br>
> <br>
> If I recall, there are other ways to things in the panic sequence (that <br>
> aren't lttng specific), but I'm personally not as familiar with the <br>
> details of that stage of linux.<br>
> <br>
<br>
It's possible to kexec-tools to load a new kernel post-panic[1]. If your <br>
system uses kexec, the contents of RAM aren't necessarily flushed, and <br>
if both the initial kernel and post-panic kernel started by kexec have <br>
the same configuration for an emulated PMEM device using the memmap <br>
paramenter [2,3] that region of memory can have a daxfs created in it <br>
post-clean boot.<br>
<br>
Note: some systems may not flush the memory during a warm reboot, but <br>
this is dependent on the BIOS.<br>
<br>
When your system boots you could do something like the following:<br>
<br>
* If it's a clean boot, create the daxfs<br>
* If it's an "unclean" boot (e.g. the daxfs already exists, or a <br>
kernel parameter informs you that it started post-panic) then you can <br>
copy/move/use lttng-crash to persistent storage for analysis<br>
* Start tracing using a snapshot session and the userspace buffers on <br>
the daxfs.<br>
<br>
In this type of situation the "snapshot" command is never invoked <br>
directly, but the recovery of the buffers to create a snapshot is possible.<br>
<br>
[1]: <a href="https://www.kernel.org/doc/html/latest/admin-guide/kdump/kdump.html" rel="noreferrer" target="_blank">https://www.kernel.org/doc/html/latest/admin-guide/kdump/kdump.html</a><br>
[2]: <br>
<a href="https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html" rel="noreferrer" target="_blank">https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html</a><br>
[3]: <br>
<a href="https://docs.pmem.io/persistent-memory/getting-started-guide/creating-development-environments/linux-environments/linux-memmap" rel="noreferrer" target="_blank">https://docs.pmem.io/persistent-memory/getting-started-guide/creating-development-environments/linux-environments/linux-memmap</a><br>
<br>
thanks,<br>
kienan<br>
<br>
>> Looking at available kernel syscall, the "reboot" one seems like a <br>
>> good candidate, however I was not able to capture a snapshot on it. I <br>
>> have tested the setup below with "--name=chdir" syscall and it <br>
>> works, "cd" to a directory will create a trace. But no dice with reboot.<br>
>><br>
> <br>
> The details of how this work will depend on your system. For example, my <br>
> installations tend to use systemd as PID 1. The broad strokes seem to <br>
> be: `/usr/sbin/reboot` is actually a link to `systemctl`, which I <br>
> believe then kicks off the reboot.service, the PID 1 is swapped to <br>
> /usr/lib/systemd/systemd-shutdown, sigterm then sigkill are sent to all <br>
> processes, unmounts, syncs, calls the reboot system call [2,3].<br>
> <br>
> As both the sigterm and the unmounts are done before the syscall, <br>
> lttng-sessiond and the consumers will have already shutdown by the time <br>
> it enters.<br>
> <br>
> While this doesn't necessarily help your original question of panics, if <br>
> you want to snapshot before shutdown or reboot and are using systemd, <br>
> it's possible to leave a script or binary in a known directory so that <br>
> it's invoked prior to the rest of the shutdown sequence[4].<br>
> <br>
> [1]: <a href="https://lttng.org/docs/v2.13/#doc-persistent-memory-file-systems" rel="noreferrer" target="_blank">https://lttng.org/docs/v2.13/#doc-persistent-memory-file-systems</a><br>
> [2]: <br>
> <a href="https://github.com/systemd/systemd/blob/6533c14997700f74e9ea42121303fc1f5c63e62b/src/shutdown/shutdown.c" rel="noreferrer" target="_blank">https://github.com/systemd/systemd/blob/6533c14997700f74e9ea42121303fc1f5c63e62b/src/shutdown/shutdown.c</a><br>
> [3]: <br>
> <a href="https://github.com/systemd/systemd/blob/main/src/shared/reboot-util.c#L77" rel="noreferrer" target="_blank">https://github.com/systemd/systemd/blob/main/src/shared/reboot-util.c#L77</a><br>
> [4]: <a href="https://www.systutorials.com/docs/linux/man/8-systemd-reboot/" rel="noreferrer" target="_blank">https://www.systutorials.com/docs/linux/man/8-systemd-reboot/</a><br>
> <br>
> hope this helps,<br>
> kienan<br>
> <br>
>> Would you have any suggestions?<br>
>> Thanks for your help,<br>
>> Cheers<br>
>> Damien<br>
>><br>
>> ============================<br>
>><br>
>> # Prep output dir<br>
>> mkdir /application/trace/<br>
>> rm -rf /application/trace/*<br>
>><br>
>> # Create session<br>
>> sudo lttng destroy snapshot-trace-session<br>
>> sudo lttng create snapshot-trace-session --snapshot <br>
>> --output="/application/trace/"<br>
>> sudo lttng enable-channel --kernel --num-subbuf=8 channelk<br>
>> sudo lttng enable-channel --userspace --num-subbuf=8 channelu<br>
>><br>
>> # Configure session<br>
>> sudo lttng enable-event --kernel --syscall --all --channel channelk<br>
>> sudo lttng enable-event --kernel --tracepoint "sched*" --channel channelk<br>
>> sudo lttng enable-event --userspace --all --channel channelu<br>
>> sudo lttng add-context -u -t vtid -t procname<br>
>> sudo lttng remove-trigger trig_reboot<br>
>> sudo lttng add-trigger --name=trig_reboot \<br>
>> --condition=event-rule-matches --type=kernel:syscall:entry \<br>
>> --name=reboot\<br>
>> --action=snapshot-session snapshot-trace-session \<br>
>> --rate-policy=once-after:1<br>
>><br>
>> # start & list info<br>
>> sudo lttng start<br>
>> sudo lttng list snapshot-trace-session<br>
>> sudo lttng list-triggers<br>
>><br>
>> #======== test it...<br>
>> sudo reboot<br>
>><br>
>> #======= reconnect and Nothing :(<br>
>> $ ls -alu /application/trace/<br>
>> drwxr-xr-x 2 u u 4096 May 15 2024 .<br>
>> drwxr-xr-x 10 u u 4096 May 15 2024 ..<br>
>><br>
>><br>
>> _______________________________________________<br>
>> lttng-dev mailing list<br>
>> <a href="mailto:lttng-dev@lists.lttng.org" target="_blank">lttng-dev@lists.lttng.org</a><br>
>> <a href="https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" rel="noreferrer" target="_blank">https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><br>
> _______________________________________________<br>
> lttng-dev mailing list<br>
> <a href="mailto:lttng-dev@lists.lttng.org" target="_blank">lttng-dev@lists.lttng.org</a><br>
> <a href="https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" rel="noreferrer" target="_blank">https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><br>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><b>Damien Berget</b></div></div>