Good news!<div><br></div><div>After peeking yesterday through the different glibc/SysV implementations (wrappers...) of splice I could find, I realized that those are not actually necessary for what we intend to do: from what I found out, they only redefine/wrap/redeclare the sys_splice system call that's implemented directly in the kernel (in any architecture). Considering we don't need anything more than the default (kernel-space) splice, we can simply rewire the 'splice' declaration we use/have to syscall(__NR_splice,...), just like what was needed for epoll_create1 earlier.<br>
<br>For example : (doing this from memory... I don't have access to the code atm)<br>> in src/common/compat/fcntl.h, inside the linux ifdef, top of the file:</div><div><pre style="margin-right:1em;margin-left:1.6em;padding:2px 2px 2px 0px;background-color:rgb(250,250,250);border:1px solid rgb(218,218,218);width:auto;overflow-x:auto;overflow-y:hidden;color:rgb(72,72,72);font-size:12px">
#ifdef __ANDROID__
#define SPLICE_F_MOVE 0x01
#define SPLICE_F_NONBLOCK 0x02
#define SPLICE_F_MORE 0x04
#define SPLICE_F_GIFT 0x08
static inline ssize_t splice(int fd_in, loff_t *off_in, int fd_out, loff_t *off_out,
size_t len, unsigned int flags)
{
return syscall(__NR_splice,fd_in,loff_in,fd_out,loff_out,len,flags);
}
#endif</pre></div><div><br></div><div>Result? Boom, traces.</div><div><br></div><div>The rest of the project seems to work without any other visible problem, and we can finally obtain traces when asking the tools nicely. In fact, we can obtain traces from any device without even having to modify the kernel/system image at all (except to add module and tracepoints support, which should already be there), which is pretty surprising, but also very useful, considering we can just ship the required binaries anywhere and start tracing.</div>
<div><br></div><div>Once we confirm that the traces are actually good and we haven't broken anything else (we're still not sure liburcu works 100%, we're trying to run all the tests), we'll start sending in results for anyone who might be interested.</div>
<div><br></div><div>- PL<br><br><div class="gmail_quote">On Tue, Mar 12, 2013 at 6:51 PM, Pierre-Luc St-Charles <span dir="ltr"><<a href="mailto:pierre-luc.st-charles@polymtl.ca" target="_blank">pierre-luc.st-charles@polymtl.ca</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey Amit,<div><br></div><div>Here's a small update on my latest findings regarding the broken *splice* / bad Bionic impl problem:</div>
<div><br></div><div>Although it didn't really cross my mind earlier, we had seen this exact problem before, but in another project; we had also investigated a bit at the time, but to no avail. It seems the issue is known by the Android/Bionic community, but it's simply not on their immediate todo list (and hasn't been for at least the past two years -- see <a href="https://code.google.com/p/android/issues/detail?id=11330" target="_blank">https://code.google.com/p/android/issues/detail?id=11330</a>). The solutions we are now considering are: adding splice support (and whatever else might also be needed) to Bionic ourselves (saving BILLIONS of broken lives in the process, IF the changes are accepted), or simply inserting glibc (and building against it) in our device images.</div>
<div><br></div><div>In the next few days, we'll be studying how much time the first solution would actually take (as we're not really experts in that domain at all), but considering the nature/objective of this project, it might be the wisest decision (as it would not require too much swerving away from the default images people have on their devices for LTTng to actually work). If we deem it 'impossible/out of our league', we'll simply try to get a working tool asap using glibc, which should not be very complicated itself (many people have actually successfully done it before).</div>
<div><br></div><div>In any case, I'll keep you updated, and I'll also post any big updates on the mailing list.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>-PL</div></font></span><div class="HOEnZb">
<div class="h5"><div><br><div class="gmail_quote">On Wed, Mar 6, 2013 at 8:04 PM, Pierre-Luc St-Charles <span dir="ltr"><<a href="mailto:pierre-luc.st-charles@polymtl.ca" target="_blank">pierre-luc.st-charles@polymtl.ca</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Great findings!</p>
<p dir="ltr">I'll also try to look into this new problem by next monday; I'm falling a bit behind on my 'investigations', but I'll keep you updated if anything comes up.</p><span><font color="#888888">
<p dir="ltr">-PL</p></font></span><div><div>
<div class="gmail_quote">On Mar 6, 2013 6:09 PM, "Amit Balboul" <<a href="mailto:amit.balboul@shinesec.com" target="_blank">amit.balboul@shinesec.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi All,<div><br></div><div>Regarding the last mail about tracing system calls on Android using LTTng,</div><div>The issue of not receiving enabled event to trace from the kernel can be fixed patching <font face="courier new, monospace">src/bin/lttng-sessiond/kernel.c </font><font face="arial, helvetica, sans-serif">(patch attached)</font></div>
<div>In the function <i><font face="courier new, monospace">kernel_list_events,</font></i> fscanf-ing the events exported by the kernel using the fp must not use the syntax <b>%m</b> as it is obsolete (in Android),</div>
<div>thus the fscanf returns 0 and won't read the kernel events (first condition false).</div><div>Instead I use a pre-allocated buffer for the string and use the <b>%[^;]</b> syntax, and omitting the memory free (attached is the patch).</div>
<div><br></div><div>The resulting output is (sessiond is run with verbose debug prints):<br></div><div><br></div><div><div><font face="courier new, monospace">root@android:/data/lttng/bin # <b>./lttng list -k</b></font></div>
<div><font face="courier new, monospace">DEBUG1 [10239/10290]: Wait for client response (in thread_manage_clients() at main.c:3242)</font></div><div><font face="courier new, monospace">DEBUG1 [10239/10290]: Receiving data from client ... (in thread_manage_clients() at main.c:3287)</font></div>
<div><font face="courier new, monospace">DEBUG1 [10239/10290]: Nothing recv() from client... continuing (in thread_manage_clients() at main.c:3291)</font></div><div><font face="courier new, monospace">DEBUG1 [10239/10290]: Clean command context structure (in clean_command_ctx() at main.c:482)</font></div>
<div><font face="courier new, monospace">DEBUG1 [10239/10290]: Accepting client command ... (in thread_manage_clients() at main.c:3200)</font></div><div><font face="courier new, monospace">DEBUG1 [10239/10290]: Wait for client response (in thread_manage_clients() at main.c:3242)</font></div>
<div><font face="courier new, monospace">DEBUG1 [10239/10290]: Receiving data from client ... (in thread_manage_clients() at main.c:3287)</font></div><div><font face="courier new, monospace">DEBUG1 [10239/10290]: Processing client command 14 (in process_client_msg() at main.c:2227)</font></div>
<div><font face="courier new, monospace">DEBUG1 [10239/10290]: Kernel list events done (61 events) (in kernel_list_events() at kernel.c:653)</font></div><div><font face="courier new, monospace">DEBUG1 [10239/10290]: Sending response (size: 35640, retcode: Success) (in thread_manage_clients() at main.c:3338)</font></div>
<div><font face="courier new, monospace">DEBUG1 [10239/10290]: Clean command context structure (in clean_command_ctx() at main.c:482)</font></div><div><font face="courier new, monospace">DEBUG1 [10239/10290]: Accepting client command ... (in thread_manage_clients() at main.c:3200)</font></div>
<div><font face="courier new, monospace">Kernel events:</font></div><div><font face="courier new, monospace">-------------</font></div><div><font face="courier new, monospace"> block_rq_abort (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> block_rq_requeue (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> block_rq_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> block_rq_insert (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> block_rq_issue (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> block_bio_bounce (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> block_bio_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> block_bio_backmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> block_bio_frontmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> block_bio_queue (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> block_getrq (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> block_sleeprq (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> block_plug (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> block_unplug (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> block_split (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> block_bio_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> block_rq_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> irq_handler_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> irq_handler_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> softirq_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> softirq_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> softirq_raise (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> sched_kthread_stop (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> sched_kthread_stop_ret (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> sched_wakeup (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> sched_wakeup_new (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> sched_migrate_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> sched_process_free (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> sched_process_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> sched_wait_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> sched_process_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> sched_process_fork (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> sched_stat_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> sched_stat_sleep (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> sched_stat_iowait (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> sched_stat_runtime (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> sched_pi_setprio (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> signal_generate (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> signal_deliver (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> signal_overflow_fail (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> signal_lose_info (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> lttng_statedump_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> lttng_statedump_end (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> lttng_statedump_process_state (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> lttng_statedump_file_descriptor (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> lttng_statedump_vm_map (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> lttng_statedump_network_interface (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> lttng_statedump_interrupt (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> timer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> timer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> timer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> timer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> timer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> hrtimer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> hrtimer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> hrtimer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> hrtimer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> hrtimer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"> itimer_state (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div><div><font face="courier new, monospace"> itimer_expire (loglevel: TRACE_EMERG (0)) (type: tracepoint)</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">root@android:/data/lttng/bin #</font></div><div><br></div><div>Now I'm facing the issue of starting the traces:</div>
<div>debugging the sessiond, consumerd and lttng I concluded that the issue is due to <font face="courier new, monospace">splice</font> system call failure.</div><div>even using a MMAP output type channel, the metadata channel is SPLICE type (hard-coded).</div>
<div>Problem persists also when using relayd to pass the traces through the network.</div><div>Ignoring the splice failure means closing of the metadata channel (maybe??) but need not affect the traces channel (right ?)</div>
<div>Attached is a debug log of each thread of the sessiond, right after invoking "<b><font face="courier new, monospace">./lttng start</font></b>". (with extra informative my prints)</div><div><br></div>
<div>Please keep me in touch with any progress in this issue,</div><div>Thank you,</div><div>Amit.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br>
</div><div><br></div></div><div><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Amit Balboul</b> <span dir="ltr"><<a href="mailto:amit.balboul@shinesec.com" target="_blank">amit.balboul@shinesec.com</a>></span><br>
Date: Wed, Feb 27, 2013 at 4:37 PM<br>Subject: Re: [lttng-dev] compiling LTTng for android<br>To: PLSTC <<a href="mailto:b0mb00z.it@gmail.com" target="_blank">b0mb00z.it@gmail.com</a>><br><br><br><div dir="ltr">Hey Pierre,<div>
<br>
</div><div>Some major progress has been achieved, following is the summary:</div><div><br></div><div>1. Modules issue:</div><div>I built (for ARM/Android) and installed latest stable version of busybox on the device.</div>
<div>Seeing the code tries to run (hardcoded) /sbin/modprobe, made a soft link to busybox's modeprobe.</div><div>Also, created /lib/modules/<version+build> directory and pushed the <version+build> directory from /lib/modules on the machine.</div>
<div>Now session daemon modprobes the desired modules successfully, (some are not present though as mentioned before) - no need to use insmod.</div><div><br></div><div>2. epoll_create1 issue:</div><div>
epoll_create1 patch was OK, I accidentally defined EPOLL_CLOEXEC as 0x02000000 instead of 02000000 (fixed).</div><div><br></div><div>The running scenario:</div><div><br></div><div>1. Run <font face="courier new, monospace">./lttng-sessiond -d -vvv</font><br>
</div><div><br></div><div>Then: (commands are in bold, output is in blue for readability)</div><div><br></div><div><div><font face="courier new, monospace">root@android:/data/lttng/install_sysroot/bin # <b>./lttng create s1</b> </font></div>
<div><font face="courier new, monospace" color="#0000ff">Session s1 created.</font></div><div><font face="courier new, monospace" color="#0000ff">Traces will be written in /data/lttng-traces/s1-20130227-161145</font></div>
<div><font face="courier new, monospace">root@android:/data/lttng/install_sysroot/bin # <b>./lttng enable-channel ch -k</b> </font></div><div><font face="courier new, monospace" color="#0000ff">Kernel channel ch enabled for session s1</font></div>
<div><font face="courier new, monospace">root@android:/data/lttng/install_sysroot/bin # <b>./lttng enable-event ev -k -a</b></font></div><div><font face="courier new, monospace" color="#0000ff">All kernel events are enabled in channel channel0</font></div>
<div><font face="courier new, monospace">root@android:/data/lttng/install_sysroot/bin # <b>./lttng list s1</b></font></div><div><font face="courier new, monospace" color="#0000ff">Tracing session s1: [inactive]</font></div>
<div><font face="courier new, monospace" color="#0000ff"> Trace path: /data/lttng-traces/s1-20130227-161145</font></div><div><font face="courier new, monospace" color="#0000ff"><br></font></div><div><font face="courier new, monospace" color="#0000ff">=== Domain: Kernel ===</font></div>
<div><font face="courier new, monospace" color="#0000ff"><br></font></div><div><font face="courier new, monospace" color="#0000ff">Channels:</font></div><div><font face="courier new, monospace" color="#0000ff">-------------</font></div>
<div><font face="courier new, monospace" color="#0000ff">- channel0: [enabled]</font></div><div><font face="courier new, monospace" color="#0000ff"><br></font></div><div><font face="courier new, monospace" color="#0000ff"> Attributes:</font></div>
<div><font face="courier new, monospace" color="#0000ff"> overwrite mode: 0</font></div><div><font face="courier new, monospace" color="#0000ff"> subbufers size: 262144</font></div><div><font face="courier new, monospace" color="#0000ff"> number of subbufers: 4</font></div>
<div><font face="courier new, monospace" color="#0000ff"> switch timer interval: 0</font></div><div><font face="courier new, monospace" color="#0000ff"> read timer interval: 200</font></div><div><font face="courier new, monospace" color="#0000ff"> output: splice()</font></div>
<div><font face="courier new, monospace" color="#0000ff"><br></font></div><div><font face="courier new, monospace" color="#0000ff"> Events:</font></div><div><font face="courier new, monospace" color="#0000ff"> None</font></div>
<div><font face="courier new, monospace" color="#0000ff"><br></font></div><div><font face="courier new, monospace" color="#0000ff">- ch: [enabled]</font></div><div><font face="courier new, monospace" color="#0000ff"><br>
</font></div>
<div><font face="courier new, monospace" color="#0000ff"> Attributes:</font></div><div><font face="courier new, monospace" color="#0000ff"> overwrite mode: 0</font></div><div><font face="courier new, monospace" color="#0000ff"> subbufers size: 262144</font></div>
<div><font face="courier new, monospace" color="#0000ff"> number of subbufers: 4</font></div><div><font face="courier new, monospace" color="#0000ff"> switch timer interval: 0</font></div><div><font face="courier new, monospace" color="#0000ff"> read timer interval: 200</font></div>
<div><font face="courier new, monospace" color="#0000ff"> output: splice()</font></div><div><font face="courier new, monospace" color="#0000ff"><br></font></div><div><font face="courier new, monospace" color="#0000ff"> Events:</font></div>
<div><font face="courier new, monospace" color="#0000ff"> None</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">root@android:/data/lttng/install_sysroot/bin # <b>./lttng list -k </b> </font></div>
<div><font face="courier new, monospace" color="#0000ff">Kernel events:</font></div><div><font face="courier new, monospace" color="#0000ff">-------------</font></div><div><font face="courier new, monospace"><br></font></div>
<div><font face="courier new, monospace">root@android:/data/lttng/install_sysroot/bin # <b>./lttng start</b></font></div><div><font face="courier new, monospace" color="#0000ff">Tracing started for session s1</font></div>
<div><font face="courier new, monospace">root@android:/data/lttng/install_sysroot/bin # <b>./lttng stop </b> </font></div><div><font face="courier new, monospace" color="#0000ff">Waiting for data availability</font></div>
<div><font face="courier new, monospace" color="#0000ff">Tracing stopped for session s1</font></div><div><font face="courier new, monospace">root@android:/data/lttng/install_sysroot/bin # </font></div><div><br></div></div>
<div><u>Now: (this is not surprising because the kernel events list is empty...)</u></div><div>the folder /<font face="courier new, monospace">data/lttng-traces/s1-20130227-161145/kernel</font> contains:</div>
<div><div><font face="courier new, monospace">total 0</font></div><div><font face="courier new, monospace">-rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_0</font></div><div><font face="courier new, monospace">-rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_1</font></div>
<div><font face="courier new, monospace">-rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_0</font></div><div><font face="courier new, monospace">-rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_1</font></div>
<div><font face="courier new, monospace">-rwxrwxrwx 1 0 0 0 Feb 27 14:18 metadata</font></div><div><br></div></div><div><u>By the way:</u></div><div>When running the same scenario but creating the channel with <font face="courier new, monospace"><b>--output mmap</b></font>, still no data but the size of the first two files in the traces directory are 4Kb (I didn't bother to investigate them as I assume it is just the header - not the traces payload).</div>
<div><br></div><div>Another issue: at this point, listing the sessions gives:<br></div><div><div><font face="courier new, monospace">root@android:/data/lttng/install_sysroot/bin # <b>./lttng list s1 </b> </font></div>
<div><font face="courier new, monospace" color="#0000ff">Tracing session s1: [inactive]</font></div><div><font face="courier new, monospace" color="#0000ff"> Trace path: /data/lttng-traces/s1-20130227-161145</font></div>
<div><font face="courier new, monospace" color="#0000ff"><br></font></div><div><font face="courier new, monospace" color="#0000ff">=== Domain: Kernel ===</font></div><div><font face="courier new, monospace" color="#0000ff"><br>
</font></div><div><font face="courier new, monospace" color="#0000ff">Error: No kernel consumer detected</font></div><div><font face="courier new, monospace">99|root@android:/data/lttng/install_sysroot/bin # <b>./lttng list -k </b> </font></div>
<div><font face="courier new, monospace" color="#0000ff">Error: Unable to list kernel events</font></div><div><font face="courier new, monospace" color="#0000ff">Error: Command error</font></div><div><font face="courier new, monospace">1|root@android:/data/lttng/install_sysroot/bin # </font></div>
<div><font face="courier new, monospace"><br></font></div></div><div>So currently I'm facing these issues:</div><div>1. Kernel events list is empty.</div><div>2. Listing the session/ kernel events after creating session, channel, event, start, stop - results in "no kernel consumer detected"</div>
<div><br></div><div>If you have some insights please share them with me..</div><div><br></div><div>Thanks</div><span><font color="#888888"><div><br></div><div>Amit.</div></font></span></div><div><div>
<div class="gmail_extra"><br><br><div class="gmail_quote">
On Tue, Feb 26, 2013 at 7:35 PM, PLSTC <span dir="ltr"><<a href="mailto:b0mb00z.it@gmail.com" target="_blank">b0mb00z.it@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
The modules you can't find are the same I am also missing I believe; I think they are only compiled when certain kernel config defines are met (and those might simply be unavailable under ARM).<div><br></div><div>As for what you pointed in your first mail, I believe modprobe isn't available on ARM, did you mean that some modules were not found when using insmod?</div>
<div><br></div><div>For the epoll_create1 problem, what kind of patch did you apply to add support for the epoll_create1 function?<div><div><br><br><div class="gmail_quote">On Tue, Feb 26, 2013 at 11:29 AM, Amit Balboul <span dir="ltr"><<a href="mailto:amit.balboul@shinesec.com" target="_blank">amit.balboul@shinesec.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div style="font-family:arial,sans-serif;font-size:12.571428298950195px">
Ok I got :</div><div style="font-family:arial,sans-serif;font-size:12.571428298950195px">
lttng-kretprobes.ko</div><div style="font-family:arial,sans-serif;font-size:12.571428298950195px">
lttng-kprobes.ko</div><div style="font-family:arial,sans-serif;font-size:12.571428298950195px">, the other two are missing.</div><div><div><div style="font-family:arial,sans-serif;font-size:12.571428298950195px">
<br></div><br><div class="gmail_quote">
---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Amit Balboul</b> <span dir="ltr"><<a href="mailto:amit.balboul@shinesec.com" target="_blank">amit.balboul@shinesec.com</a>></span><br>Date: Tue, Feb 26, 2013 at 5:49 PM<br>
Subject: Re: [lttng-dev] compiling LTTng for android<br>To: PLSTC <<a href="mailto:b0mb00z.it@gmail.com" target="_blank">b0mb00z.it@gmail.com</a>><br><br><br><div dir="ltr">One more thing:<div><div><br></div><div>The list of modules you gave my is not identical to mine:</div>
<div>a. I pushed the modules from my machine to my folder in the device (not to /system/lib) as you directed me, and insmoded them manually.</div>
<div>b. I pushed the whole tree under /lib/modules/<KERNEL_VER_BUILD>/ which includes the modules.* and the sub directories:</div><div><br></div><div><div><font face="courier new, monospace" size="1">/lib/modules/3.0.31-gd5a18e0/extra$ ll</font></div>
<div><font face="courier new, monospace" size="1">total 2464</font></div><div><font face="courier new, monospace" size="1">drwxr-xr-x 4 root root 4096 Feb 26 15:58 ./</font></div><div><font face="courier new, monospace" size="1">drwxr-xr-x 3 root root 4096 Feb 26 15:58 ../</font></div>
<div><font face="courier new, monospace" size="1">drwxr-xr-x 2 root root 4096 Feb 26 15:58 lib/</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 167184 Feb 26 15:58 lttng-ring-buffer-client-discard.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 167206 Feb 26 15:58 lttng-ring-buffer-client-mmap-discard.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 178586 Feb 26 15:58 lttng-ring-buffer-client-mmap-overwrite.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 178564 Feb 26 15:58 lttng-ring-buffer-client-overwrite.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 137179 Feb 26 15:58 lttng-ring-buffer-metadata-client.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 137233 Feb 26 15:58 lttng-ring-buffer-metadata-mmap-client.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 213451 Feb 26 15:58 lttng-statedump.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 1314235 Feb 26 15:58 lttng-tracer.ko</font></div><div><font face="courier new, monospace" size="1">drwxr-xr-x 2 root root 4096 Feb 26 15:58 probes/</font></div>
<div><font face="courier new, monospace" size="1"><br></font></div><div><div><font face="courier new, monospace" size="1">/lib/modules/3.0.31-gd5a18e0/extra/lib$ ll</font></div><div><font face="courier new, monospace" size="1">total 732</font></div>
<div><font face="courier new, monospace" size="1">drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./</font></div><div><font face="courier new, monospace" size="1">drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../</font></div><div>
<font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 738964 Feb 26 15:58 lttng-lib-ring-buffer.ko</font></div></div><div><font face="courier new, monospace" size="1"><br></font></div><div><div><font face="courier new, monospace" size="1">/lib/modules/3.0.31-gd5a18e0/extra/probes$ ll</font></div>
<div><font face="courier new, monospace" size="1">total 2872</font></div><div><font face="courier new, monospace" size="1">drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./</font></div><div><font face="courier new, monospace" size="1">drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 162872 Feb 26 15:58 lttng-probe-asoc.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 171049 Feb 26 15:58 lttng-probe-block.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 81165 Feb 26 15:58 lttng-probe-compaction.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 333215 Feb 26 15:58 lttng-probe-ext4.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 80413 Feb 26 15:58 lttng-probe-gpio.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 84256 Feb 26 15:58 lttng-probe-irq.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 135453 Feb 26 15:58 lttng-probe-jbd2.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 105399 Feb 26 15:58 lttng-probe-kmem.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 76240 Feb 26 15:58 lttng-probe-lttng.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 88059 Feb 26 15:58 lttng-probe-module.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 154032 Feb 26 15:58 lttng-probe-napi.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 142038 Feb 26 15:58 lttng-probe-net.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 93564 Feb 26 15:58 lttng-probe-power.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 85530 Feb 26 15:58 lttng-probe-regulator.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 138121 Feb 26 15:58 lttng-probe-sched.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 152909 Feb 26 15:58 lttng-probe-scsi.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 107491 Feb 26 15:58 lttng-probe-signal.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 142711 Feb 26 15:58 lttng-probe-skb.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 181983 Feb 26 15:58 lttng-probe-statedump.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 126731 Feb 26 15:58 lttng-probe-timer.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 86591 Feb 26 15:58 lttng-probe-workqueue.ko</font></div><div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 133886 Feb 26 15:58 lttng-probe-writeback.ko</font></div>
<div><font face="courier new, monospace" size="1">-rw-r--r-- 1 root root 30863 Feb 26 15:58 lttng-types.ko</font></div></div><div><br></div><div>Looking at the files shows I miss:<br></div><div><div>lttng-kretprobes.ko</div>
<div>lttng-kprobes.ko</div><div>lttng-ftrace.ko</div><div>lttng-probe-kvm.ko</div></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div>On Tue, Feb 26, 2013 at 3:56 PM, PLSTC <span dir="ltr"><<a href="mailto:b0mb00z.it@gmail.com" target="_blank">b0mb00z.it@gmail.com</a>></span> wrote:<br>
</div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Back on the last email:<div><br>
</div><div>Here's the insmod list I ended up building; you might not have compiled all those modules, depending on your kernel config and on the architecture used, so some might simply be absent from your output directory.</div>
<div><br></div><div><pre style="margin-right:1em;margin-left:1.6em;padding:2px 2px 2px 0px;background-color:rgb(250,250,250);border:1px solid rgb(218,218,218);width:auto;color:rgb(72,72,72);font-size:12px">insmod /system/lib/modules/lttng-types.ko
insmod /system/lib/modules/lttng-kretprobes.ko
insmod /system/lib/modules/lttng-kprobes.ko
insmod /system/lib/modules/lttng-ftrace.ko
insmod /system/lib/modules/lttng-lib-ring-buffer.ko
insmod /system/lib/modules/lttng-statedump.ko
insmod /system/lib/modules/lttng-tracer.ko
insmod /system/lib/modules/lttng-probe-timer.ko
insmod /system/lib/modules/lttng-probe-statedump.ko
insmod /system/lib/modules/lttng-probe-signal.ko
insmod /system/lib/modules/lttng-probe-sched.ko
insmod /system/lib/modules/lttng-probe-lttng.ko
insmod /system/lib/modules/lttng-probe-kvm.ko
insmod /system/lib/modules/lttng-probe-irq.ko
insmod /system/lib/modules/lttng-probe-block.ko
insmod /system/lib/modules/lttng-ring-buffer-metadata-mmap-client.ko
insmod /system/lib/modules/lttng-ring-buffer-metadata-client.ko
insmod /system/lib/modules/lttng-ring-buffer-client-overwrite.ko
insmod /system/lib/modules/lttng-ring-buffer-client-mmap-overwrite.ko
insmod /system/lib/modules/lttng-ring-buffer-client-mmap-discard.ko
insmod /system/lib/modules/lttng-ring-buffer-client-discard.ko</pre><div><br></div><div>We also haven't had time recently to test if the kprobes/kretprobes worked, but they probably do; we decided to focus a bit more on the 'tracepoints' aspect instead.</div>
<div><br></div><div>Anyway, good luck with your exploration, and if you find something interesting, I'd love to know more about it!</div><div><br></div><div>-PL</div><div><div><br><div class="gmail_quote">
On Tue, Feb 26, 2013 at 8:06 AM, PLSTC <span dir="ltr"><<a href="mailto:b0mb00z.it@gmail.com" target="_blank">b0mb00z.it@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><p dir="ltr">Hey Amit,</p>
<p dir="ltr">Good news indeed it seems! </p>
<p dir="ltr">When it comes to pushing the libs/bin, we haven't found any better way than using adb push yet; of course, if all the projects were included directly in Android's external, we would simply flash the device after each build, but currently we're still having trouble generating the correct Android makefiles.</p>
<p dir="ltr">As for the modules, I might still have a dep load list somewhere I could send you in a few hours, I remember fighting to get the order right...</p>
<p dir="ltr">Also, once everything loads, could you tell me what kind of output 'lttng list -k' provides you? Here, we're having trouble listing the 'default' kernel trace points provided by HAVE_TRACEPOINTS, and the returned list is simply empty (which is a bit strange).</p>
<p dir="ltr">-PL</p>
</blockquote></div><br></div></div></div>
</blockquote></div></div></div><br></div>
</div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</div></div></div><br></div></div>
</blockquote></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>