[lttng-dev] Odd behaviour of lttng 2.0 kernel event listing

[Mathieu Desnoyers]

Hi Daniel,

What is your lttng-modules git commit ID ?

And we will need your full kernel .config file to see what CONFIG_*
options are enabled.

Thanks,

Mathieu

* David Goulet (david.goulet at polymtl.ca) wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi Daniel,
> 
> On 12-01-18 02:12 PM, Thibault, Daniel wrote:
> > 
> >    Well, the good news is that the new head (lttng-tools-2.0-pre16+-76150f6 2012-01-18 17:21) compiles correctly (no warnings, no errors in bootstrap, configure, make and install logs).  The bad news is that it still misbehaves.  Here's my session log, edited for brevity:
> > 
> 
> Hmmmm... this is really unusual. I'll try to test this issue on a 2.6.38 kernel
> because apart from that, I have no idea what can cause this. We can't reproduce
> it here... Some of my dev. is on Lucid 10.04 and Debian testing so I doubt that
> the Ubuntu version makes difference here. I'll let you know for sure.
> 
> > $ sudo lttng list -k
> > Spawning a session daemon
> > Kernel events:
> > -------------
> >       sched_kthread_stop (type: tracepoint)
> >       sched_kthread_stop_ret (type: tracepoint)
> >       [...]
> >       kvm_ioapic_set_irq (type: tracepoint)
> >       kvm_msi_set_irq (type: tracepoint)
> >        (type: tracepoint) [disabled]
> >        (type: tracepoint) [disabled]
> >       [... 27 more instances ...]
> >        (type: tracepoint) [disabled]
> >        (type: tracepoint) [disabled]
> > 
> > $ sudo lttng list -k
> > Kernel events:
> > -------------
> >       sched_kthread_stop (type: tracepoint)
> >       sched_kthread_stop_ret (type: tracepoint)
> >       [...]
> >       kvm_ioapic_set_irq (type: tracepoint)
> >       kvm_msi_set_irq (type: tracepoint)
> >       kvm_ack_irq (type: tracepoint)
> >       kvm_mmio (type: tracepoint)
> >       [...]
> >       block_bio_remap (type: tracepoint)
> >       block_rq_remap (type: tracepoint)
> > 
> >    When I kill the lttng-sessiond daemon, all 17 lttng-modules are unloaded.
> > 
> >    I tried doing 'sudo modprobe lttng_probe_block' before calling 'sudo lttng list -k' to see if that wouldn't un-anonymise the block_* tracepoints (which I think are serviced by the lttng_probe_block module).  The module loaded along with a handful of other modules it depends on (ltt_relay, lttng_kretprobes, lttng_kprobes, lttng_ftrace, lib_ring_buffer) but 'sudo lttng list -k' yielded precisely the same output.
> > 
> > -----
> > 
> >    I'd like to take this opportunity to point out a problem when running lttng from an unelevated account:
> > 
> > $ lttng list -k
> > Spawning a session daemon
> > FATAL: Error inserting lttng_ftrace (/lib/modules/2.6.38-020638-generic/extra/probes/lttng-ftrace.ko): Operation not permitted
> > FATAL: Error inserting lttng_kprobes (/lib/modules/2.6.38-020638-generic/extra/probes/lttng-kprobes.ko): Operation not permitted
> > [... 14 more FATAL errors interleaved with 30 WARNINGs ...]
> > WARNING: Error inserting ltt_relay (/lib/modules/2.6.38-020638-generic/extra/ltt-relay.ko): Operation not permitted
> > FATAL: Error inserting lttng_probe_sched (/lib/modules/2.6.38-020638-generic/extra/probes/lttng-probe-sched.ko): Operation not permitted
> > Error: Unable to list kernel events
> > Error: Kernel tracer not available
> > 
> >    If the error is FATAL, why does lttng keep going?  The final error message could be improved to read something like "Error: Kernel tracer not available or not accessible at your privilege level", in order to at least hint at what the source of the problem is.
> > 
> 
> I did fix that with latest commit :
> 
> Author: David Goulet <dgoulet at efficios.com>
> Date:   Thu Jan 19 13:43:57 2012 -0500
> 
>     Don't init kernel tracer if not root (UID=0)
> 
>     No point of trying to init kernel tracer if the session daemon is not
>     root. At this time, only UID=0 can load kernel modules and interact with
>     the lttng kernel tracer.
> 
>     Reported-by: Daniel Thibault <daniel.thibault at drdc-rddc.gc.ca>
>     Signed-off-by: David Goulet <dgoulet at efficios.com>
> 
> Thanks of the report!
> 
> Note that after this commit, I'll release lttng-tools pre17 and completely
> feature freeze it until February 16th 2012. We are going into stable mode aiming
> at the stable packages for Ubuntu next LTS.
> 
> Cheers!
> David
> 
> > Daniel U. Thibault
> > R & D pour la défense Canada - Valcartier (RDDC Valcartier) / Defence R&D Canada - Valcartier (DRDC Valcartier)
> > Système de systèmes (SdS) / System of Systems (SoS)
> > Solutions informatiques et expérimentations (SIE) / Computing Solutions and Experimentations (CSE)
> > 2459 Boul. Pie XI Nord
> > Québec, QC  G3J 1X5
> > CANADA
> > Vox : (418) 844-4000 x4245
> > Fax : (418) 844-4538
> > NAC: 918V QSDJ
> > Gouvernement du Canada / Government of Canada
> > <http://www.valcartier.drdc-rddc.gc.ca/>
> > 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> 
> iQEcBAEBAgAGBQJPGGcaAAoJEELoaioR9I028EgH/217mx6gMoPm07ylxWwUBbg1
> J5FfpdIGCFiuspT38zUXhlM3+J+W4oMY8giONGTlVuBkdK34PWNflXf0QyBfg3XW
> i1ZmBBLGg/3vVbjGA74MGrYDVIUUr7Cpm+cycM2XD9k7xhe3RWbPdzP7JY8RWmzH
> IqKDUxSvPHp9GnbNaRHdZ1G6nF89w6xXIHO5gaMJP4087AhGvDo8m52nCHBXIEJA
> +uaB49NhvoOES5LPTxOJbaYPEpLgzEdGGjx1uriDX3BGGnfPRMmAS27Nf+4iiusI
> H7B6Oye104hTZFqR4a/15T0v+FKR3dlGZvJga2MBHqiNoDRX5CuIohHBG0YDsvk=
> =lD76
> -----END PGP SIGNATURE-----
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com