[lttng-dev] LTTng system call tracing on ARM

Mathieu Desnoyers mathieu.desnoyers at efficios.com
Wed Apr 10 12:20:12 EDT 2013


* Jan Glauber (jan.glauber at gmail.com) wrote:
> On Wed, Apr 10, 2013 at 09:38:33AM -0400, Pierre-Luc St-Charles wrote:
> > Here are the links I stashed a while back about previous patching attempts
> > to add syscall tracepoints to the ARM kernel :
> > 
> > Most promising:
> > http://www.spinics.net/lists/arm-kernel/msg166419.html
> > 
> > Earlier draft(s):
> > http://lists.infradead.org/pipermail/linux-arm-kernel/2012-February/086974.html
> > https://lkml.org/lkml/2011/12/1/131
> > http://comments.gmane.org/gmane.linux.ports.arm.kernel/141933
> > 
> > I believe the first link shows the most refined patch there is out there,
> > but it might take some minor tinkering to apply it to a different kernel
> > version. I briefly tried to apply it to the 3.0.31 kernel, but it's a bit
> > out of my 'tinkering' range, and I never finished it.
> > 
> > 
> > Good luck!
> 
> Thanks, fortunately I'm targeting 3.4. It wasn't too hard, got the kernel running
> and ftrace already works.
> 
> Now I'm facing something hilarious, on loading the lttng-tracer module I get:
> 
> Error: could not insert module lttng-tracer.ko: Cannot allocate memory
> 
> [15912.259399] alloc_vmap_area: 2 callbacks suppressed
> [15912.264953] vmap allocation for size 15998976 failed: use vmalloc=<size> to increase size.
> [15912.273895] vmalloc: allocation failure: 15991653 bytes
> [15912.279602] insmod: page allocation failure: order:0, mode:0xd0
> 
> Never saw vmalloc fail on loading a kernel module before. Hmmm.
> Looking at the module reveals:
> -rw-rw-r-- 1 jang jang 16036281 Apr 10 17:48 lttng-tracer.ko
> 
> Wow. 16 megs for a kernel module ?!? readelf says its all in .rodata.
> 
> Doing an objdump on that
> Disassembly of section .rodata:
> 
> 00000000 <sc_table>:
> ...
>     1714:       00f0a788        rscseq  sl, r0, r8, lsl #15
>     1718:       00f0a7a0        rscseq  sl, r0, r0, lsr #15
>     171c:       00000004        andeq   r0, r0, r4
>         ...
>   f00054:       00f0a860        rscseq  sl, r0, r0, ror #16
>   f00058:       00f0a878        rscseq  sl, r0, r8, ror r8
>   f0005c:       00000002        andeq   r0, r0, r2
> 
> Now whats that? A big hole in the system call table?
> 
> Digging through the various defines and headers of lttng-modules I've found:
> 
> instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h:
> ...
> #define OVERRIDE_TABLE_32_sys_set_tls
> TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)
> 
> Weird ARM apparently expanded the system call area above 0xf0000. I don't
> understand why this override logic is needed but removing that define results
> in a kernel module with a much more sane size.
> 
> LTTng system call tracing works with that change for me.
> 
> Any idea on how the override problem can be solved with the magic system calls?

Thanks for the detailed report !!

Please try with this commit:

commit 11fa478f18241fedee86f7dc7820a91c629c9e7e
Author: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
Date:   Wed Apr 10 12:14:38 2013 -0400

    Fix: remove ARM set_tls system call override
    
    We'll need to find a better way to instrument ARM-specific system calls
    located at a far offset from the standard systems calls. A 16MB
    lttng-modules kernel module is really not acceptable.
    
    Removing this instrumentation for now. sys_set_tls will appear as
    sys_unknown.
    
    Fixes #472
    
    CC: Ryan Kyser <Ryan.Kyser at jci.com>
    Ref: http://lists.lttng.org/pipermail/lttng-dev/2013-April/019990.html
    Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>

diff --git a/instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h b/instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h
index 93b8674..895370f 100644
--- a/instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h
+++ b/instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h
@@ -2,7 +2,6 @@
 
 #define OVERRIDE_TABLE_32_sys_arm_fadvise64_64
 #define OVERRIDE_TABLE_32_sys_sync_file_range2
-#define OVERRIDE_TABLE_32_sys_set_tls
 
 #ifndef CREATE_SYSCALL_TABLE
 
@@ -38,18 +37,6 @@ SC_TRACE_EVENT(sys_sync_file_range2,
 	TP_printk()
 )
 
-SC_TRACE_EVENT(sys_set_tls,
-	TP_PROTO(unsigned int tid, unsigned long tls),
-	TP_ARGS(tid, tls),
-	TP_STRUCT__entry(
-		__field(unsigned int, tid)
-		__field_hex(unsigned int, tls)),
-	TP_fast_assign(
-		tp_assign(tid, tid)
-		tp_assign(tls, tls)),
-	TP_printk()
-)
-
 #else	/* CREATE_SYSCALL_TABLE */
 
 #define OVVERRIDE_TABLE_32_sys_mmap
@@ -59,9 +46,6 @@ TRACE_SYSCALL_TABLE(sys_mmap, sys_mmap, 90, 6)
 TRACE_SYSCALL_TABLE(sys_arm_fadvise64_64, sys_arm_fadvise64_64, 270, 4)
 #define OVERRIDE_TABLE_32_sys_sync_file_range2
 TRACE_SYSCALL_TABLE(sys_sync_file_range2, sys_sync_file_range2, 341, 4)
-#define OVERRIDE_TABLE_32_sys_set_tls
-TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)
-
 
 #endif /* CREATE_SYSCALL_TABLE */
 


> 
> --Jan
> 
> > -PL
> > 
> > 
> > 
> > On Wed, Apr 10, 2013 at 9:02 AM, Jan Glauber <jan.glauber at gmail.com> wrote:
> > 
> > > On Wed, Apr 10, 2013 at 08:37:18AM -0400, PLSTC wrote:
> > > > Hey Jan,
> > > >
> > > > I cannot speak on behalf of everyone here, but during our attempt to port
> > > > LTTng to Android, we also noticed that the kernels we were using (3.0.x)
> > > > were nowhere near the requirements for syscall tracepoints support. I
> > > > believe such support was added on x86/64 way earlier (early 3.x) than on
> > > > ARM, which is why it was included in LTTng's modules a while ago. Simply
> > > > put, the ARM kernel is late.
> > >
> > > OK, so for ARM kernel version 3.6 is the minimum unless the syscall
> > > tracepoint
> > > support is backported.
> > >
> > > > There are a few actuals ways to 'enable' syscall tracepoints support on
> > > > early ARM kernels, but they all including a bit of kernel
> > > hacking/patching.
> > > > I could send you some links if you're interested in that.
> > >
> > > Yes, sure!
> > >
> > > --Jan
> > >
> > > > -PL
> > > > On Apr 10, 2013 4:40 AM, "Jan Glauber" <jan.glauber at gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I want to use LTTng for system call tracing on ARM. Now lttng-modules
> > > seems
> > > > > to support system call tracing on ARM already since
> > > > > "8f4f80e LTTng Modules ARM syscall instrumentation".
> > > > >
> > > > > But I wonder how that worked since lttng-syscalls.c is only build under
> > > > > CONFIG_HAVE_SYSCALL_TRACEPOINTS and that was added to ARM only with
> > > kernel
> > > > > 3.6
> > > > > (much after than the lttng-modules commit).
> > > > >
> > > > > Am I missing something? Is system call tracing working on ARM with the
> > > > > upstream
> > > > > LTTng version?
> > > > >
> > > > > thanks,
> > > > > Jan
> > > > >
> > > > > _______________________________________________
> > > > > lttng-dev mailing list
> > > > > lttng-dev at lists.lttng.org
> > > > > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> > > > >
> > >
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com



More information about the lttng-dev mailing list