[lttng-dev] Problems setting up lttng on PREEMPT_RT Raspberry Pi

[Mathieu Desnoyers]

----- On Apr 15, 2019, at 7:07 AM, Nicholas Sadjoli <nick.sadjoli at gmail.com> wrote: 

> Dear Lttng dev Team,
> I am currently interested in using lttng to measure the interrupt or hardware
> real-time latencies experienced within my user-space applications.
> Unfortunately I have been having issues installing lttng on my Raspberry Pi
> that is using the PREEMPT_RT patched kernel, version 4.14.91-rt49-v7+. The
> problems so far can be seen documented in the RPi Linux github forum page here:
> [ https://github.com/raspberrypi/linux/issues/2930 |
> https://github.com/raspberrypi/linux/issues/2930 ]

> Initially I was about to just install the Debian package of the lttng onto the
> RPi, and just proceeded with the commands ' sudo apt-get install
> lttng-modules-dkms liblttng-ust-dev lttng-tools' However, I was unable to use
> lttng package at all and found out that for the RPi, a build from source is
> necessary for lttng, as described here: [
> https://stackoverflow.com/questions/52386448/linux-trace-tool-lttng-on-raspbian-kernel-tracer-not-available
> |
> https://stackoverflow.com/questions/52386448/linux-trace-tool-lttng-on-raspbian-kernel-tracer-not-available
> ]

> Following their suggestion, I then tried to build the lttng packages from
> source, and even re-building the RPi kernel locally to ensure the kernel
> headers are in the RPi and linked up properly to the source packages.

> Unfortunately when I tried to do this for the lttng-modules package, it instead
> gave me the following warning in my latest attempt:

> I n file included from
> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/lttng-kprobes.c:31:0:
> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/../blacklist/kprobes.h:19:4:
> error: #error "Your kernel is known to have buggy optimized kprobes
> implementation. Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c \"ARM:
> 8834/1: Fix: kprobes: optimized kprobes illegal instruction\" in Linux. Disable
> CONFIG_OPTPROBES or upgrade your kernel."
> # error "Your kernel is known to have buggy optimized kprobes implementation.
> Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c \"ARM: 8834/1: Fix:
> kprobes: optimized kprobes illegal instruction\" in Linux. Disable
> CONFIG_OPTPROBES or upgrade your kernel."

> I then tried to turn off the CONFIG_OPTPROBES kernel config, but it seemed to be
> automatically overwritten during kernel building and is considered an important
> part of the real-time RPi kernels. Has such problems been reported with this
> version of the PREEMPT_RT kernel in the past (again, the current kernel version
> that I am using is 4.14.91-rt49-v7+)? If there are I would like to know how
> this issue was resolved so that I can properly build and use the lttng package
> for my project.

Hi Nicholas, 

It looks like our lttng-modules kernel version blacklist is acting exactly as it should. Looking at this 
git tree: 

git clone https://github.com/raspberrypi/linux.git -b rpi-4.14.y-rt 
I notice that in the git log, we find this bogus commit: 

commit f9e0bc710347fadad55910846ee36f0681d0dca8 
Author: Kees Cook <keescook at chromium.org> 
Date: Tue Oct 30 22:12:56 2018 +0100 

ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE 

commit e46daee53bb50bde38805f1823a182979724c229 upstream. 

But not the fix for it (upstream): 

commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c 
Author: Mathieu Desnoyers <mathieu.desnoyers at efficios.com> 
Date: Tue Feb 5 16:37:40 2019 +0100 

ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction 

So lttng-modules refuses to build against your kernel configuration with CONFIG_OPTPROBES=y 
to save you the trouble of having kprobes trigger illegal instruction OOPS at runtime. 

I'm unusure who maintains that raspberrypi rt kernel tree. However, the upstream rt tree 
for 4.14-rt maintained by Tom Zanussi seems to contain the fix: 

[ https://kernel.googlesource.com/pub/scm/linux/kernel/git/rt/linux-stable-rt/+/refs/heads/v4.14-rt | https://kernel.googlesource.com/pub/scm/linux/kernel/git/rt/linux-stable-rt/+/refs/heads/v4.14-rt ] 

(currently at 4.14.109-rt57). 

So you should contact your raspberrypi linux rt tree clone maintainer and ask him 
to upgrade, or, as stated in the #error generated by lttng-modules, simply ensure 
CONFIG_OPTPROBES is unset when you build your kernel. You should ask the 
raspberrypi community advice on how to ensure this configuration option is not 
overridden by the raspberrypi build system. 

Thanks, 

Mathieu 

> Thank you so much for your attention and looking forward to your reply soon.

> Best Regards,

> Nicholas Sadjoli

> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20190415/8769df25/attachment-0001.html>