[ltt-dev] LTTng 0.193 fixes RT kernel support

[naresh kamboju]

On Tue, Feb 23, 2010 at 9:22 PM, Mathieu Desnoyers
<mathieu.desnoyers at efficios.com> wrote:
> * naresh kamboju (naresh.kernel at gmail.com) wrote:
>> On Tue, Feb 23, 2010 at 6:00 PM, Mathieu Desnoyers
>> >>
>> >> with out this patch on SMP reported the previous bug as BUG: sleeping
>> >> function called from invalid context at kernel/rtmutex.c:685
>> >>
>> >> However, i'll investigate.
>> >
>> > Hrm, we should turn the arch/{arm/mach-omap2,x86/kernel}/trace-clock.c:
>> > trace_clock_lock into a mutex, and kernel/trace/trace-clock-32-to-64.c:
>> > synthetic_tsc_lock into a mutex too.
>>
>> I have modified kernel/trace/trace-clock-32-to-64.c spin_lock to
>> mutex_lock to all the calls
>>
>> -static DEFINE_SPINLOCK(synthetic_tsc_lock);
>> +static DEFINE_MUTEX(synthetic_tsc_lock);
>>
>> -       spin_lock(&synthetic_tsc_lock);
>> +       mutex_lock(&synthetic_tsc_lock);
>>
>> for arch/{arm/mach-omap2/kernel}/trace-clock.c is already modified as
>> above from the patch
>> omap-trace-clock-fix-mutex.patch
>> from LTTng patches 02-Feb-2009.
>> this patch was prepared by you to fix
>> BUG: sleeping function called from invalid context at kernel/mutex.c:207.
>>
>> Still reporting same bug at my end :-(
>>
>> let me try in all possible ways.
>
Hi Mathieu,

> Please note that at the end of 2009, I moved the ARM LTTng omap trace clock to a
> neater per-cpu design.
yes.
i'll review per-cpu related patch and investigating.
> So it's possible that the old implementation you use has
> problems with the mutex. This could be fixed with the newer implementation.

currently i am using lttng-0.158 version patches of course these are OCT-2009.
now i'll check the with latest LTTng version patches and changes made
in latest repo.

>
> If you can tell us more about the BUG you get with the mutex-based synthetic tsc
> (what the callers are) I could tell you if this problem really goes away in
> newer version.

As I said previously, i was able to fix this BUG on ARM-UP kernels
after your fix patches :-)
currently checking the what could the issue making BUG on ARM-SMP,
doing workarounds on what are the function calls are differing in the
context of smp call which are making delay in spin locks and how the
lttng will lookafter timing issues w.r.t to RT kernels when i execute
lttng test cases.

If you have any inputs regarding lttng vs timers n locking, please welcome :-)

FYI
of course,  it would be difficult to have same env to reproduce this
BUG at your end just want to share my analysis report.

generic_smp_call_function_single_interrupt ---------- kernel/smp.c
|
\/
disable_synthetic_tsc_ip ----------------------------------
kernel/trace/trace-clock-32-to-64.c
|
\/
del_timer ------------------------------------------------- kernel/timer.c
|
\/
lock_timer_base ---------------------------------------  kernel/timer.c
|
\/
rt_spin_lock --------------------------------kernel/rtmutex.c
 |
\/
__might_sleep ------------------------ kernel/sched.c


BUG: sleeping function called from invalid context at kernel/rtmutex.c:685
in_atomic(): 1, irqs_disabled(): 128, pid: 720, name: lttd
Backtrace:
[<c002d434>] (dump_backtrace+0x0/0x10c) from [<c03a75d8>] (dump_stack+0x18/0x1c)
 r7:000002ad r6:c045da78 r5:00001116 r4:c04ba400
[<c03a75c0>] (dump_stack+0x0/0x1c) from [<c0041028>] (__might_sleep+0x120/0x14c)
[<c0040f08>] (__might_sleep+0x0/0x14c) from [<c03a9b18>]
(rt_spin_lock+0x38/0x68)
 r7:ce319d04 r6:c0763660 r5:c05107a0 r4:c05107a0
[<c03a9ae0>] (rt_spin_lock+0x0/0x68) from [<c00570b0>]
(lock_timer_base+0x30/0x54)
 r4:c05107a0
[<c0057080>] (lock_timer_base+0x0/0x54) from [<c00571b4>] (del_timer+0x2c/0x6c)
 r8:c0023570 r7:ce319d38 r6:00740000 r5:ceb19ca4 r4:c0763660
[<c0057188>] (del_timer+0x0/0x6c) from [<c008e5ec>]
(disable_synthetic_tsc_ipi+0x24/0x30)
 r5:ceb19ca4 r4:00000001
[<c008e5c8>] (disable_synthetic_tsc_ipi+0x0/0x30) from [<c0072e00>]
(generic_smp_call_function_single_interrupt+0x98/0xf4)
[<c0072d68>] (generic_smp_call_function_single_interrupt+0x0/0xf4)
from [<c0028368>] (do_IPI+0xc8/0x15c)
[<c00282a0>] (do_IPI+0x0/0x15c) from [<c00280c4>] (_text+0xc4/0x128)

Thank you.

Best regards
Naresh Kamboju
>
> Thanks,
>
> Mathieu
>\