[lttng-dev] lttng Crash seen after stopping the traces.

Jonnavithula Sharma sarmajprs at gmail.com
Mon Jul 29 09:45:27 EDT 2013 

Hi

We are currently running lttng on our CPE with 64 MB RAM.We are observing a
crash after stopping the traces and even if we leave our CPE for few min
after starting the trace we are observing this crash.Please find the
attachment for the crash log.

Let us know if any one has faced this issue and your help is highly
appreciated in resolving this issue.


Regards,
Sarma J
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20130729/60eea380/attachment.html>
-------------- next part --------------
root at ltqcpe:~# lttng-sessiond -d -vvv
root at ltqcpe:~# lttng create demo
Session demo created.
Traces will be written in /root/lttng-traces/demo-20130722-191207
root at ltqcpe:~# lttng list demo
Tracing session demo: [inactive]
    Trace path: /root/lttng-traces/demo-20130722-191207

root at ltqcpe:~# lttng enable-event sched_switch -k --tracepoint
kernel event sched_switch created in channel channel0
root at ltqcpe:~# lttng list demo
Tracing session demo: [inactive]
    Trace path: /root/lttng-traces/demo-20130722-191207

=== Domain: Kernel ===

Channels:
-------------
- channel0: [enabled]

    Attributes:
      overwrite mode: 0
      subbufers size: 262144
      number of subbufers: 4
      switch timer interval: 0
      read timer interval: 200000
      output: splice()

    Events:
      sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint) [enabled]

root at ltqcpe:~# lttng disable-event sched_switch -k --tracepoint
usage: lttng disable-event NAME[,NAME2,...] [-k|-u] [OPTIONS]

Options:
  -h, --help               Show this help
      --list-options       Simple listing of options
  -s, --session NAME       Apply to session name
  -c, --channel NAME       Apply to this channel
  -a, --all-events         Disable all tracepoints
  -k, --kernel             Apply for the kernel tracer
  -u, --userspace          Apply to the user-space tracer

Error: Undefined command
root at ltqcpe:~# lttng disable-event sched_switch  -k
kernel event sched_switch disabled in channel channel0 for session demo
root at ltqcpe:~# lttng list demo
Tracing session demo: [inactive]
    Trace path: /root/lttng-traces/demo-20130722-191207

=== Domain: Kernel ===

Channels:
-------------
- channel0: [enabled]

    Attributes:
      overwrite mode: 0
      subbufers size: 262144
      number of subbufers: 4
      switch timer interval: 0
      read timer interval: 200000
      output: splice()

    Events:
      sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint) [disabled]

root at ltqcpe:~# lttng enable-event sched_switch -k --function lttng_calibrate_kre
tprobe
kernel event sched_switch created in channel channel0
root at ltqcpe:~# lttng list demo
Tracing session demo: [inactive]
    Trace path: /root/lttng-traces/demo-20130722-191207

=== Domain: Kernel ===

Channels:
-------------
- channel0: [enabled]

    Attributes:
      overwrite mode: 0
      subbufers size: 262144
      number of subbufers: 4
      switch timer interval: 0
      read timer interval: 200000
      output: splice()

    Events:
      sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint) [enabled]

root at ltqcpe:~# lttng start
Tracing started for session demo
root at ltqcpe:~# ping 192.168.1.3
PING 192.168.1.3 (192.168.1.3): 56 data bytes
64 bytes from 192.168.1.3: seq=0 ttl=128 time=1.083 ms
64 bytes from 192.168.1.3: seq=1 ttl=128 time=0.541 ms
64 bytes from 192.168.1.3: seq=2 ttl=128 time=0.498 ms
64 bytes from 192.168.1.3: seq=3 ttl=128 time=0.527 ms
64 bytes from 192.168.1.3: seq=4 ttl=128 time=0.557 ms
64 bytes from 192.168.1.3: seq=5 ttl=128 time=0.504 ms
64 bytes from 192.168.1.3: seq=6 ttl=128 time=0.515 ms

--- 192.168.1.3 ping statistics ---
7 packets transmitted, 7 packets received, 0% packet loss
round-trip min/avg/max = 0.498/0.603/1.083 ms
root at ltqcpe:~# lttng stop
[ 1045.520000] CPU 0 Unable to handle kernel paging request at virtual address ffff979c, epc == 831a979c, ra == 831a7fec
[ 1045.520000] Oops[#1]:
[ 1045.520000] Cpu 0
[ 1045.520000] $ 0   : 00000000 7751fef0 00000001 00000000
[ 1045.520000] $ 4   : 8307da00 83209e60 80455e48 00000000
[ 1045.520000] $ 8   : 80455e4c 00002486 00000000 00000004
[ 1045.520000] $12   : 7751fde8 00001000 00000000 00407948
[ 1045.520000] $16   : 8333b000 00000000 00000000 00001000
[ 1045.520000] $20   : 00001000 00000001 80455e48 83b7b694
[ 1045.520000] $24   : 00000000 831a979c
[ 1045.520000] $28   : 83208000 83209d50 80500000 831a7fec
[ 1045.520000] Hi    : 0000024d
[ 1045.520000] Lo    : 0001e678
[ 1045.520000] epc   : 831a979c cleanup_module+0xfbc/0x18 [lttng_lib_ring_buffer]
[ 1045.520000]     Tainted: G           O
[ 1045.520000] ra    : 831a7fec lib_ring_buffer_ioctl+0x530/0x560 [lttng_lib_ring_buffer]
[ 1045.520000] Status: 1100fc03    KERNEL EXL IE
[ 1045.520000] Cause : 00800008
[ 1045.520000] BadVA : ffff979c
[ 1045.520000] PrId  : 00019556 (MIPS 34Kc)
[ 1045.520000] Modules linked in: lttng_probe_workqueue(O) lttng_probe_udp(O) lttng_probe_timer(O) lttng_probe_statedump(O) lttng_probe_sock(O) lttng_probe_skb(O) lttng_probe_signal(O) lttng_probe_scsi(O) lttng_probe_sched(O) lttng_probe_rcu(O) lttng_probe_random(O) lttng_probe_printk(O) lttng_probe_power(O) lttng_probe_net(O) lttng_probe_napi(O) lttng_probe_module(O) lttng_probe_kmem(O) lttng_probe_jbd2(O) lttng_probe_irq(O) lttng_probe_gpio(O) lttng_probe_ext4(O) lttng_probe_compaction(O) lttng_probe_block(O) lttng_types(O) lttng_ring_buffer_metadata_mmap_client(O) lttng_ring_buffer_client_mmap_overwrite(O) lttng_ring_buffer_client_mmap_discard(O) lttng_ring_buffer_metadata_client(O) lttng_ring_buffer_client_overwrite(O) lttng_ring_buffer_client_discard(O) lttng_tracer(O) lttng_lib_ring_buffer(O) lttng_statedump(O)
[ 1045.520000] Process lttng-consumerd (pid: 1963, threadinfo=83208000, task=832195e8, tls=77527970)
[ 1045.520000] Stack : 00000000 00000020 830d53b8 8320c774 00000000 00001000 83b84860 00000029
        830d5380 832195e8 00000001 800b2704 8312b700 80002420 8307da80 8317b800
        8320c774 00000029 775205b4 00000029 83b84860 775205b4 00000029 8001ea88
        0000001b 00000000 00acc000 83b75dec 83209fe0 0000fc00 8004ab00 00000004
        7f84aed0 775ca030 7f84af40 00460000 83b75dec 83209e98 00000000 00acbff8
        ...
[ 1045.520000] Call Trace:
[ 1045.520000] [<800b2704>] handle_mm_fault+0xb0/0xd0
[ 1045.520000] [<80002420>] ret_from_exception+0x0/0x10
[ 1045.520000] [<8001ea88>] do_page_fault+0x2f4/0x4dc
[ 1045.520000] [<8004ab00>] sys_ni_syscall+0x0/0x8
[ 1045.520000] [<801016ac>] ep_send_events_proc+0x94/0x16c
[ 1045.520000] [<80177094>] security_file_permission+0xa4/0xcc
[ 1045.520000] [<831a7de0>] lib_ring_buffer_ioctl+0x324/0x560 [lttng_lib_ring_buffer]
[ 1045.520000] [<831a8094>] lib_ring_buffer_splice_read+0x78/0x104 [lttng_lib_ring_buffer]
[ 1045.520000] [<800edaf8>] do_splice_to+0x4c/0xcc
[ 1045.520000] [<80103070>] sys_epoll_wait+0x30c/0x3ac
[ 1045.520000] [<80102d10>] sys_epoll_ctl+0x82c/0x880
[ 1045.520000] [<800ef7fc>] sys_splice+0x554/0x5a4
[ 1045.520000] [<80056d0c>] default_wake_function+0x0/0x8
[ 1045.520000] [<800d49f4>] sys_ioctl+0x74/0xa4
[ 1045.520000] [<800041b0>] stack_done+0x20/0x44
[ 1045.520000] [<8004ab00>] sys_ni_syscall+0x0/0x8
[ 1045.520000]
[ 1045.520000]
Code: 00000000  83144a80  831af480 <831a979c> 831a979c  839f1198  00000000  81082880  00000000
[ 1045.820000] ---[ end trace 1489f875219fcc4e ]---
[ 1045.840000] CPU 0 Unable to handle kernel paging request at virtual address ffff979c, epc == 831a979c, ra == 831a7fec
[ 1045.840000] Oops[#2]:
[ 1045.840000] Cpu 0
[ 1045.840000] $ 0   : 00000000 7731fe48 00000010 00000000
[ 1045.840000] $ 4   : 8307d680 8321de60 80455e48 00000000
[ 1045.840000] $ 8   : 80455e4c 00002479 00000001 ffffffff
[ 1045.840000] $12   : 00200200 00100100 00000003 00460000
[ 1045.840000] $16   : 83326000 00006000 00000000 00010000
[ 1045.840000] $20   : 00001000 00000010 80455e48 81084e6c
[ 1045.840000] $24   : 00000000 831a979c
[ 1045.840000] $28   : 8321c000 8321dd50 80500000 831a7fec
[ 1045.840000] Hi    : 000000f3
[ 1045.840000] Lo    : 80ebf400
[ 1045.840000] epc   : 831a979c cleanup_module+0xfbc/0x18 [lttng_lib_ring_buffer]
[ 1045.840000]     Tainted: G      D    O
[ 1045.840000] ra    : 831a7fec lib_ring_buffer_ioctl+0x530/0x560 [lttng_lib_ring_buffer]
[ 1045.840000] Status: 1100fc03    KERNEL EXL IE
[ 1045.840000] Cause : 00800008
[ 1045.840000] BadVA : ffff979c
[ 1045.840000] PrId  : 00019556 (MIPS 34Kc)
[ 1045.840000] Modules linked in: lttng_probe_workqueue(O) lttng_probe_udp(O) lttng_probe_timer(O) lttng_probe_statedump(O) lttng_probe_sock(O) lttng_probe_skb(O) lttng_probe_signal(O) lttng_probe_scsi(O) lttng_probe_sched(O) lttng_probe_rcu(O) lttng_probe_random(O) lttng_probe_printk(O) lttng_probe_power(O) lttng_probe_net(O) lttng_probe_napi(O) lttng_probe_module(O) lttng_probe_kmem(O) lttng_probe_jbd2(O) lttng_probe_irq(O) lttng_probe_gpio(O) lttng_probe_ext4(O) lttng_probe_compaction(O) lttng_probe_block(O) lttng_types(O) lttng_ring_buffer_metadata_mmap_client(O) lttng_ring_buffer_client_mmap_overwrite(O) lttng_ring_buffer_client_mmap_discard(O) lttng_ring_buffer_metadata_client(O) lttng_ring_buffer_client_overwrite(O) lttng_ring_buffer_client_discard(O) lttng_tracer(O) lttng_lib_ring_buffer(O) lttng_statedump(O)
[ 1045.840000] Process lttng-consumerd (pid: 1964, threadinfo=8321c000, task=832191b8, tls=77327970)
[ 1045.840000] Stack : 00000000 00000000 00000000 8320c774 00000000 00001000 83115498 00000000
        00001000 832191b8 00000000 00001000 8380e590 00000000 00001000 8317b000
        00000000 00001000 777b705c 00000000 00001000 777b705c 00000000 00001000
        51ed84c0 00000000 00001000 00000000 00000000 00001000 00030001 00000000
        00001000 00000000 00000000 00001000 00000001 00000000 00001000 8307d7b0
        ...
[ 1045.840000] Call Trace:
[ 1045.840000] [<80052acc>] __wake_up_common+0x58/0xc0
[ 1045.840000] [<831a7de0>] lib_ring_buffer_ioctl+0x324/0x560 [lttng_lib_ring_buffer]
[ 1045.840000] [<831a8094>] lib_ring_buffer_splice_read+0x78/0x104 [lttng_lib_ring_buffer]
[ 1045.840000] [<800edaf8>] do_splice_to+0x4c/0xcc
[ 1045.840000] [<800ef7fc>] sys_splice+0x554/0x5a4
[ 1045.840000] [<800d49f4>] sys_ioctl+0x74/0xa4
[ 1045.840000] [<800d675c>] sys_poll+0x94/0xf8
[ 1045.840000] [<800041b0>] stack_done+0x20/0x44
[ 1045.840000] [<8003f748>] sys_getcpu+0x0/0x70
[ 1045.840000]
[ 1045.840000]
Code: 00000000  83144a80  831af480 <831a979c> 831a979c  839f1198  00000000  81082880  00000000
[ 1046.110000] ---[ end trace 1489f875219fcc4f ]---



root at ltqcpe:~# ls lttng-traces/
demo-20130722-191207  tr-20130722-191911
root at ltqcpe:~# ls lttng-traces/demo-20130722-191207/
kernel
root at ltqcpe:~# ls lttng-traces/demo-20130722-191207/kernel/
channel0_0  metadata
root at ltqcpe:~# ls lttng-traces/demo-20130722-191207/kernel/channel0_0
lttng-traces/demo-20130722-191207/kernel/channel0_0
root at ltqcpe:~# ls lttng-traces/demo-20130722-191207/kernel/channel0_0
root at ltqcpe:~#
root at ltqcpe:~#

More information about the lttng-dev mailing list