[ltt-dev] Problems with LTT Probe Install (Kernel Panic)

[Mathieu Desnoyers]

Hi Thomas,

I just re-installed a new ML server, sorry for the delay.

2.6.22-rc4-mm2 is a rather "test-only" release.

Can you try with the latest versions ? 2.6.25.4 and follow the
compatibility list on ltt.polymtl.ca

Thanks,

Mathieu

> 
> Hi -
> 
> 
> 
> I am currently in the process of trying to get the
> 
> LTTng going, but am having some trouble with Probe
> 
> installs that seems rather basic/fundamental to
> 
> the operation.  My versions of everything are:
> 
> 
> 
> Linux 2.6.22.23 SMP x86_64 
> 
> 
> 
> patch-2.6.22-rc4-mm2-lttng-0.9.10
> 
> 
> 
> ltt-control-0.42-16072007
> 
> 
> 
> LinuxTraceToolkitViewer-0.8.84-12082007
> 
> 
> 
> My Kernel configuration is:
> 
> 
> 
> #
> 
> # Instrumentation Support
> 
> #
> 
> # CONFIG_PROFILING is not set
> 
> CONFIG_KPROBES=y
> 
> # CONFIG_IMMEDIATE is not set
> 
> CONFIG_MARKERS=y
> 
> CONFIG_LTT=y
> 
> CONFIG_LTT_TRACER=y
> 
> CONFIG_LTT_RELAY=y
> 
> CONFIG_LTT_SERIALIZE=y
> 
> # CONFIG_LTT_ALIGNMENT is not set
> 
> # CONFIG_LTT_HEARTBEAT is not set
> 
> # CONFIG_LTT_TEST_TSC is not set
> 
> # CONFIG_LTT_USERSPACE_GENERIC is not set
> 
> CONFIG_LTT_NETLINK_CONTROL=y
> 
> CONFIG_LTT_STATEDUMP=y
> 
> 
> 
> #
> 
> # Probes
> 
> #
> 
> CONFIG_LTT_PROBE_CORE=m
> 
> CONFIG_LTT_PROBE_FS=m
> 
> CONFIG_LTT_PROBE_KERNEL=m
> 
> CONFIG_LTT_PROBE_ARCH=m
> 
> CONFIG_LTT_PROBE_MM=m
> 
> CONFIG_LTT_PROBE_NET=m
> 
> CONFIG_LTT_PROBE_LIST=m
> 
> CONFIG_LTT_PROBE_LOCKING=m
> 
> # CONFIG_LTT_PROBE_STACK is not set
> 
> 
> 
> During boot, I see the following messages:
> 
> 
> 
> LTT : ltt-facilities init
> 
> LTT : ltt-relay init
> 
> ltt-control init
> 
> LTT : ltt-facility-statedump init
> 
> 
> 
> Then, I tried to install <ltt-probe-core.ko>;
> 
> this gave me a Kernel Panic.  I did some looking around,
> 
> commented out some areas of the code so that I learn a
> 
> few things, and found the following:
> 
> 
> 
> In <marker.c>, this is where the fault is occuring:
> 
> 
> 
> static struct marker_entry *_get_marker(const char *name)
> 
> {
> 
>     struct hlist_node *node;
> 
>     struct marker_entry *e; 
> 
> 
> 
>     hlist_for_each_entry(e, node, &marker_list, hlist)
> 
>         printk( "_get_marker(): name %x, e-name %s\n",
> 
>                 name, e->name);
> 
>  
> 
> /*      if (!strcmp(name, e->name))
> 
>             return e;
> 
> tbesemer
> 
> */
> 
>     
> 
>     return NULL;
> 
> }   
> 
> 
> 
> Through the printk() I put in, instead of the strcmp(), I found
> 
> that "name" is passed in as NULL.  More so, if you look at the
> 
> following dump from dmesg for <ltt-probe-kernel.ko>, you can
> 
> see not only is "name" NULL at times, but it looks like the same
> 
> stuff keeps getting operated on:
> 
> 
> 
> _get_marker(): name 8058a9c0, e-name kernel_process_signal
> 
> _get_marker(): name 8058a9c0, e-name kernel_process_wait
> 
> _get_marker(): name 8058a9c0, e-name kernel_process_free
> 
> _get_marker(): name 8058a9c0, e-name kernel_process_exit
> 
> _get_marker(): name 8058a9c0, e-name kernel_process_fork
> 
> _get_marker(): name 0, e-name kernel_irq_exit
> 
> _get_marker(): name 0, e-name kernel_irq_entry
> 
> _get_marker(): name 0, e-name kernel_module_free
> 
> _get_marker(): name 0, e-name kernel_module_load
> 
> _get_marker(): name 0, e-name kernel_kthread_stop_ret
> 
> _get_marker(): name 0, e-name kernel_kthread_stop
> 
> _get_marker(): name 0, e-name kernel_tasklet_high_exit
> 
> _get_marker(): name 0, e-name kernel_tasklet_high_entry
> 
> _get_marker(): name 0, e-name kernel_tasklet_low_exit
> 
> _get_marker(): name 0, e-name kernel_tasklet_low_entry
> 
> _get_marker(): name 0, e-name kernel_softirq_exit
> 
> _get_marker(): name 0, e-name kernel_softirq_entry
> 
> _get_marker(): name 0, e-name kernel_timer_timeout
> 
> _get_marker(): name 0, e-name kernel_timer_update_time
> 
> _get_marker(): name 0, e-name kernel_timer_set
> 
> _get_marker(): name 0, e-name kernel_timer_itimer_set
> 
> _get_marker(): name 0, e-name kernel_timer_itimer_expired
> 
> _get_marker(): name 0, e-name kernel_vprintk
> 
> _get_marker(): name 0, e-name kernel_printk
> 
> _get_marker(): name 0, e-name kernel_sched_schedule
> 
> _get_marker(): name 0, e-name kernel_sched_migrate_task
> 
> _get_marker(): name 0, e-name kernel_sched_wakeup_new_task
> 
> _get_marker(): name 0, e-name kernel_sched_try_wakeup
> 
> _get_marker(): name 0, e-name kernel_sched_wait_task
> 
> _get_marker(): name 0, e-name kernel_process_signal
> 
> _get_marker(): name 0, e-name kernel_process_wait
> 
> _get_marker(): name 0, e-name kernel_process_free
> 
> _get_marker(): name 0, e-name kernel_process_exit
> 
> _get_marker(): name 0, e-name kernel_process_fork
> 
> _get_marker(): name 802510f8, e-name kernel_irq_exit
> 
> _get_marker(): name 802510f8, e-name kernel_irq_entry
> 
> _get_marker(): name 802510f8, e-name kernel_module_free
> 
> _get_marker(): name 802510f8, e-name kernel_module_load
> 
> _get_marker(): name 802510f8, e-name kernel_kthread_stop_ret
> 
> 
> 
> [snip lot's of output until the end]
> 
> 
> 
> _get_marker(): name 0, e-name kernel_process_signal
> 
> _get_marker(): name 0, e-name kernel_process_wait
> 
> _get_marker(): name 0, e-name kernel_process_free
> 
> _get_marker(): name 0, e-name kernel_process_exit
> 
> _get_marker(): name 0, e-name kernel_process_fork
> 
> _get_marker(): name 802510f8, e-name kernel_irq_exit
> 
> _get_marker(): name 802510f8, e-name kernel_irq_entry
> 
> _get_marker(): name 802510f8, e-name kernel_module_free
> 
> _get_marker(): name 802510f8, e-name kernel_module_load
> 
> _get_marker(): name 802510f8, e-name kernel_kthread_stop_ret
> 
> _get_marker(): name 802510f8, e-name kernel_kthread_stop
> 
> _get_marker(): name 802510f8, e-name kernel_tasklet_high_exit
> 
> _get_marker(): name 802510f8, e-name kernel_tasklet_high_entry
> 
> _get_marker(): name 802510f8, e-name kernel_tasklet_low_exit
> 
> _get_marker(): name 802510f8, e-name kernel_tasklet_low_entry
> 
> _get_marker(): name 802510f8, e-name kernel_softirq_exit
> 
> _get_marker(): name 802510f8, e-name kernel_softirq_entry
> 
> _get_marker(): name 802510f8, e-name kernel_timer_timeout
> 
> _get_marker(): name 802510f8, e-name kernel_timer_update_time
> 
> _get_marker(): name 802510f8, e-name kernel_timer_set
> 
> _get_marker(): name 802510f8, e-name kernel_timer_itimer_set
> 
> _get_marker(): name 802510f8, e-name kernel_timer_itimer_expired
> 
> _get_marker(): name 802510f8, e-name kernel_vprintk
> 
> _get_marker(): name 802510f8, e-name kernel_printk
> 
> _get_marker(): name 802510f8, e-name kernel_sched_schedule
> 
> _get_marker(): name 802510f8, e-name kernel_sched_migrate_task
> 
> _get_marker(): name 802510f8, e-name kernel_sched_wakeup_new_task
> 
> _get_marker(): name 802510f8, e-name kernel_sched_try_wakeup
> 
> _get_marker(): name 802510f8, e-name kernel_sched_wait_task
> 
> _get_marker(): name 802510f8, e-name kernel_process_signal
> 
> _get_marker(): name 802510f8, e-name kernel_process_wait
> 
> _get_marker(): name 802510f8, e-name kernel_process_free
> 
> _get_marker(): name 802510f8, e-name kernel_process_exit
> 
> _get_marker(): name 802510f8, e-name kernel_process_fork
> 
> (END) 
> 
> 
> 
> --
> 
> So, the summary is:
> 
> 
> 
>  1. I have applied the patch set.  Some areas were fuzzy
> 
>     patches, I have not looked at all those.
> 
>  2. Where the patch set FAILED, I resovled by hand.
> 
>  3. I configured/built Kernel, looks like LTT at Kernel
> 
>     level is somewhat okay - does not crash Kernel.
> 
>  4. I start to add probes, and I get Kernel Panics because
> 
>     of "name" being passed to _get_marker(const char *name)
> 
>     is NULL.
> 
> 
> 
> All help appreciated - always willing to give back any changes I
> 
> come up with, and keep you in the loop of how it is working for
> 
> me.
> 
> 
> 
> Many thanks,
> 
> 
> 
> Tom
> 
-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68