[ltt-dev] kmemtrace

[Mathieu Desnoyers]

* Pekka Enberg (penberg at cs.helsinki.fi) wrote:
> Hi Ingo,
> 
> (Adding some cc's.)
> 
> On Tue, 2008-12-16 at 20:44 +0100, Ingo Molnar wrote:
> > Have you thought about converting kmemtrace into an ftrace plugin?
> > 
> > It would avoid infrastructure duplication and it would be enriched by all 
> > the ftrace gadgets we have. For example you could add ad-hoc 
> > ftrace_printk() calls to your code and it would show up in the kmemtrace 
> > output. Or the function-graph-tracer:
> > 
> >      0) ! 108.528 us  |            }
> >      0)               |            irq_exit() {
> >      0)               |              do_softirq() {
> >      0)               |                __do_softirq() {
> >      0)   0.895 us    |                  __local_bh_disable();
> >      0)               |                  run_timer_softirq() {
> >      0)   0.827 us    |                    hrtimer_run_pending();
> >      0)   1.226 us    |                    _spin_lock_irq();
> >      0)               |                    _spin_unlock_irq() {
> >      0)   6.550 us    |                  }
> >      0)   0.924 us    |                  _local_bh_enable();
> >      0) + 12.129 us   |                }
> >      0) + 13.911 us   |              }
> >      0)   0.707 us    |              idle_cpu();
> >      0) + 17.009 us   |            }
> >      0) ! 137.419 us  |          }
> > 
> > Could be mixed with kmemtrace perhaps too.
> 
> I started the initial work on kmemtrace (which Eduard used as some sort
> of base for his work) before ftrace appeared so, no, I haven't thought
> about this. However, as far as I understood, ftrace is really not for
> pushing out tons of data? For kmemtrace, we really want to push out all
> the data to userspace for further inspection. The reason being that when
> analyzing memory usage, there are multiple (slightly conflicting) use
> cases:
> 
>   - What call-sites have the biggest internal fragmentations and _why_?
>     This is, of course, mostly because of bad fitting in the SLAB
>     allocators but we need to know the exact numbers to work it out.
> 
>   - What call-site is doing the most allocations?
> 
>   - What call-site is hanging on to the biggest chunks of memory?
> 
>   - What call-sites are doing the most kmalloc()/kfree() CPU ping-pong?
>     What about ping-pong between NUMA nodes?
> 
>   - What call-sites are doing the most GFP_ATOMIC (or other expensive)
>     allocations?
> 
> There's also some debugging things we are interested in like is someone
> kfree()'ing kmem_cache_alloc()'d memory and so on.
> 
> So as far as I can tell, while some of these things can certainly be
> converted to use the ftrace framework, some of them really belong into
> userspace. Also, for future development, it would be really nice to be
> able to just get the data for simulating different allocation
> strategies. Not that I'm planning to do this at any time soon but it's
> just another argument for kmemtrace just relaying the data to userspace.
> 
> Eduard, Mathieu, comments?
> 
> 		Pekka
> 

Hi Pekka,

If what you need to do is to relay the information to userspace as
quickly as possible, LTTng seems to be very well suited for this.

You basically just have to do the following (e.g.) :

(on top of the -lttng tree, code in ltt/probes/kmemtrace.c)

#include <linux/marker.h>
#include "ltt-type-serializer.h"

DEFINE_MARKER(kmemtrace, eventname1, "ip 0x%lX address 0x%lX);

void probe_callback(void *ip, unsigned long address)
{
  struct marker *marker;
  struct serialize_long_long data;

  data.f1 = (unsigned long)ip;
  data.f2 = address;

  /* Send data to trace buffers */
  marker = &GET_MARKER(kmemtrace, eventname1);
  ltt_specialized_trace(marker, marker->single.probe_private,
    &data, serialize_sizeof(data), sizeof(long));
}

And think of any other pre-treatment on the data before sending it to
the ltt_specialized_trace function.

Then getting the data (only metadata and kmemtrace channels) can be done with :

lttctl -C -w /tmp/trace -o channel.all.enable=0 \
                        -o channel.kmemtrace.enable=1 trace

(reproduce behavior)

lttctl -D trace

lttv -m textDump -t /tmp/trace

Please ask if you have any question, ideas and enhancements are welcome.

Mathieu

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68