[ltt-dev] Problems with LTT Probe Install (Kernel Panic)

[Thomas Besemer]

Hi -



I am currently in the process of trying to get the

LTTng going, but am having some trouble with Probe

installs that seems rather basic/fundamental to

the operation.  My versions of everything are:



Linux 2.6.22.23 SMP x86_64 



patch-2.6.22-rc4-mm2-lttng-0.9.10



ltt-control-0.42-16072007



LinuxTraceToolkitViewer-0.8.84-12082007



My Kernel configuration is:



#

# Instrumentation Support

#

# CONFIG_PROFILING is not set

CONFIG_KPROBES=y

# CONFIG_IMMEDIATE is not set

CONFIG_MARKERS=y

CONFIG_LTT=y

CONFIG_LTT_TRACER=y

CONFIG_LTT_RELAY=y

CONFIG_LTT_SERIALIZE=y

# CONFIG_LTT_ALIGNMENT is not set

# CONFIG_LTT_HEARTBEAT is not set

# CONFIG_LTT_TEST_TSC is not set

# CONFIG_LTT_USERSPACE_GENERIC is not set

CONFIG_LTT_NETLINK_CONTROL=y

CONFIG_LTT_STATEDUMP=y



#

# Probes

#

CONFIG_LTT_PROBE_CORE=m

CONFIG_LTT_PROBE_FS=m

CONFIG_LTT_PROBE_KERNEL=m

CONFIG_LTT_PROBE_ARCH=m

CONFIG_LTT_PROBE_MM=m

CONFIG_LTT_PROBE_NET=m

CONFIG_LTT_PROBE_LIST=m

CONFIG_LTT_PROBE_LOCKING=m

# CONFIG_LTT_PROBE_STACK is not set



During boot, I see the following messages:



LTT : ltt-facilities init

LTT : ltt-relay init

ltt-control init

LTT : ltt-facility-statedump init



Then, I tried to install <ltt-probe-core.ko>;

this gave me a Kernel Panic.  I did some looking around,

commented out some areas of the code so that I learn a

few things, and found the following:



In <marker.c>, this is where the fault is occuring:



static struct marker_entry *_get_marker(const char *name)

{

    struct hlist_node *node;

    struct marker_entry *e; 



    hlist_for_each_entry(e, node, &marker_list, hlist)

        printk( "_get_marker(): name %x, e-name %s\n",

                name, e->name);

 

/*      if (!strcmp(name, e->name))

            return e;

tbesemer

*/

    

    return NULL;

}   



Through the printk() I put in, instead of the strcmp(), I found

that "name" is passed in as NULL.  More so, if you look at the

following dump from dmesg for <ltt-probe-kernel.ko>, you can

see not only is "name" NULL at times, but it looks like the same

stuff keeps getting operated on:



_get_marker(): name 8058a9c0, e-name kernel_process_signal

_get_marker(): name 8058a9c0, e-name kernel_process_wait

_get_marker(): name 8058a9c0, e-name kernel_process_free

_get_marker(): name 8058a9c0, e-name kernel_process_exit

_get_marker(): name 8058a9c0, e-name kernel_process_fork

_get_marker(): name 0, e-name kernel_irq_exit

_get_marker(): name 0, e-name kernel_irq_entry

_get_marker(): name 0, e-name kernel_module_free

_get_marker(): name 0, e-name kernel_module_load

_get_marker(): name 0, e-name kernel_kthread_stop_ret

_get_marker(): name 0, e-name kernel_kthread_stop

_get_marker(): name 0, e-name kernel_tasklet_high_exit

_get_marker(): name 0, e-name kernel_tasklet_high_entry

_get_marker(): name 0, e-name kernel_tasklet_low_exit

_get_marker(): name 0, e-name kernel_tasklet_low_entry

_get_marker(): name 0, e-name kernel_softirq_exit

_get_marker(): name 0, e-name kernel_softirq_entry

_get_marker(): name 0, e-name kernel_timer_timeout

_get_marker(): name 0, e-name kernel_timer_update_time

_get_marker(): name 0, e-name kernel_timer_set

_get_marker(): name 0, e-name kernel_timer_itimer_set

_get_marker(): name 0, e-name kernel_timer_itimer_expired

_get_marker(): name 0, e-name kernel_vprintk

_get_marker(): name 0, e-name kernel_printk

_get_marker(): name 0, e-name kernel_sched_schedule

_get_marker(): name 0, e-name kernel_sched_migrate_task

_get_marker(): name 0, e-name kernel_sched_wakeup_new_task

_get_marker(): name 0, e-name kernel_sched_try_wakeup

_get_marker(): name 0, e-name kernel_sched_wait_task

_get_marker(): name 0, e-name kernel_process_signal

_get_marker(): name 0, e-name kernel_process_wait

_get_marker(): name 0, e-name kernel_process_free

_get_marker(): name 0, e-name kernel_process_exit

_get_marker(): name 0, e-name kernel_process_fork

_get_marker(): name 802510f8, e-name kernel_irq_exit

_get_marker(): name 802510f8, e-name kernel_irq_entry

_get_marker(): name 802510f8, e-name kernel_module_free

_get_marker(): name 802510f8, e-name kernel_module_load

_get_marker(): name 802510f8, e-name kernel_kthread_stop_ret



[snip lot's of output until the end]



_get_marker(): name 0, e-name kernel_process_signal

_get_marker(): name 0, e-name kernel_process_wait

_get_marker(): name 0, e-name kernel_process_free

_get_marker(): name 0, e-name kernel_process_exit

_get_marker(): name 0, e-name kernel_process_fork

_get_marker(): name 802510f8, e-name kernel_irq_exit

_get_marker(): name 802510f8, e-name kernel_irq_entry

_get_marker(): name 802510f8, e-name kernel_module_free

_get_marker(): name 802510f8, e-name kernel_module_load

_get_marker(): name 802510f8, e-name kernel_kthread_stop_ret

_get_marker(): name 802510f8, e-name kernel_kthread_stop

_get_marker(): name 802510f8, e-name kernel_tasklet_high_exit

_get_marker(): name 802510f8, e-name kernel_tasklet_high_entry

_get_marker(): name 802510f8, e-name kernel_tasklet_low_exit

_get_marker(): name 802510f8, e-name kernel_tasklet_low_entry

_get_marker(): name 802510f8, e-name kernel_softirq_exit

_get_marker(): name 802510f8, e-name kernel_softirq_entry

_get_marker(): name 802510f8, e-name kernel_timer_timeout

_get_marker(): name 802510f8, e-name kernel_timer_update_time

_get_marker(): name 802510f8, e-name kernel_timer_set

_get_marker(): name 802510f8, e-name kernel_timer_itimer_set

_get_marker(): name 802510f8, e-name kernel_timer_itimer_expired

_get_marker(): name 802510f8, e-name kernel_vprintk

_get_marker(): name 802510f8, e-name kernel_printk

_get_marker(): name 802510f8, e-name kernel_sched_schedule

_get_marker(): name 802510f8, e-name kernel_sched_migrate_task

_get_marker(): name 802510f8, e-name kernel_sched_wakeup_new_task

_get_marker(): name 802510f8, e-name kernel_sched_try_wakeup

_get_marker(): name 802510f8, e-name kernel_sched_wait_task

_get_marker(): name 802510f8, e-name kernel_process_signal

_get_marker(): name 802510f8, e-name kernel_process_wait

_get_marker(): name 802510f8, e-name kernel_process_free

_get_marker(): name 802510f8, e-name kernel_process_exit

_get_marker(): name 802510f8, e-name kernel_process_fork

(END) 



--

So, the summary is:



 1. I have applied the patch set.  Some areas were fuzzy

    patches, I have not looked at all those.

 2. Where the patch set FAILED, I resovled by hand.

 3. I configured/built Kernel, looks like LTT at Kernel

    level is somewhat okay - does not crash Kernel.

 4. I start to add probes, and I get Kernel Panics because

    of "name" being passed to _get_marker(const char *name)

    is NULL.



All help appreciated - always willing to give back any changes I

come up with, and keep you in the loop of how it is working for

me.



Many thanks,



Tom