[lttng-dev] LTTng system call tracing on ARM

[Jan Glauber]

On Wed, Apr 10, 2013 at 09:38:33AM -0400, Pierre-Luc St-Charles wrote:
> Here are the links I stashed a while back about previous patching attempts
> to add syscall tracepoints to the ARM kernel :
> 
> Most promising:
> http://www.spinics.net/lists/arm-kernel/msg166419.html
> 
> Earlier draft(s):
> http://lists.infradead.org/pipermail/linux-arm-kernel/2012-February/086974.html
> https://lkml.org/lkml/2011/12/1/131
> http://comments.gmane.org/gmane.linux.ports.arm.kernel/141933
> 
> I believe the first link shows the most refined patch there is out there,
> but it might take some minor tinkering to apply it to a different kernel
> version. I briefly tried to apply it to the 3.0.31 kernel, but it's a bit
> out of my 'tinkering' range, and I never finished it.
> 
> 
> Good luck!

Thanks, fortunately I'm targeting 3.4. It wasn't too hard, got the kernel running
and ftrace already works.

Now I'm facing something hilarious, on loading the lttng-tracer module I get:

Error: could not insert module lttng-tracer.ko: Cannot allocate memory

[15912.259399] alloc_vmap_area: 2 callbacks suppressed
[15912.264953] vmap allocation for size 15998976 failed: use vmalloc=<size> to increase size.
[15912.273895] vmalloc: allocation failure: 15991653 bytes
[15912.279602] insmod: page allocation failure: order:0, mode:0xd0

Never saw vmalloc fail on loading a kernel module before. Hmmm.
Looking at the module reveals:
-rw-rw-r-- 1 jang jang 16036281 Apr 10 17:48 lttng-tracer.ko

Wow. 16 megs for a kernel module ?!? readelf says its all in .rodata.

Doing an objdump on that
Disassembly of section .rodata:

00000000 <sc_table>:
...
    1714:       00f0a788        rscseq  sl, r0, r8, lsl #15
    1718:       00f0a7a0        rscseq  sl, r0, r0, lsr #15
    171c:       00000004        andeq   r0, r0, r4
        ...
  f00054:       00f0a860        rscseq  sl, r0, r0, ror #16
  f00058:       00f0a878        rscseq  sl, r0, r8, ror r8
  f0005c:       00000002        andeq   r0, r0, r2

Now whats that? A big hole in the system call table?

Digging through the various defines and headers of lttng-modules I've found:

instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h:
...
#define OVERRIDE_TABLE_32_sys_set_tls
TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)

Weird ARM apparently expanded the system call area above 0xf0000. I don't
understand why this override logic is needed but removing that define results
in a kernel module with a much more sane size.

LTTng system call tracing works with that change for me.

Any idea on how the override problem can be solved with the magic system calls?

--Jan

> -PL
> 
> 
> 
> On Wed, Apr 10, 2013 at 9:02 AM, Jan Glauber <jan.glauber at gmail.com> wrote:
> 
> > On Wed, Apr 10, 2013 at 08:37:18AM -0400, PLSTC wrote:
> > > Hey Jan,
> > >
> > > I cannot speak on behalf of everyone here, but during our attempt to port
> > > LTTng to Android, we also noticed that the kernels we were using (3.0.x)
> > > were nowhere near the requirements for syscall tracepoints support. I
> > > believe such support was added on x86/64 way earlier (early 3.x) than on
> > > ARM, which is why it was included in LTTng's modules a while ago. Simply
> > > put, the ARM kernel is late.
> >
> > OK, so for ARM kernel version 3.6 is the minimum unless the syscall
> > tracepoint
> > support is backported.
> >
> > > There are a few actuals ways to 'enable' syscall tracepoints support on
> > > early ARM kernels, but they all including a bit of kernel
> > hacking/patching.
> > > I could send you some links if you're interested in that.
> >
> > Yes, sure!
> >
> > --Jan
> >
> > > -PL
> > > On Apr 10, 2013 4:40 AM, "Jan Glauber" <jan.glauber at gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I want to use LTTng for system call tracing on ARM. Now lttng-modules
> > seems
> > > > to support system call tracing on ARM already since
> > > > "8f4f80e LTTng Modules ARM syscall instrumentation".
> > > >
> > > > But I wonder how that worked since lttng-syscalls.c is only build under
> > > > CONFIG_HAVE_SYSCALL_TRACEPOINTS and that was added to ARM only with
> > kernel
> > > > 3.6
> > > > (much after than the lttng-modules commit).
> > > >
> > > > Am I missing something? Is system call tracing working on ARM with the
> > > > upstream
> > > > LTTng version?
> > > >
> > > > thanks,
> > > > Jan
> > > >
> > > > _______________________________________________
> > > > lttng-dev mailing list
> > > > lttng-dev at lists.lttng.org
> > > > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> > > >
> >