[ltt-dev] problems with ust

[Pierre-Marc Fournier]

>> Sylvain Geneves wrote:
>>>
>>> I've pulled UST from git yesterday, and noticed some errors, and some
>>> are confusing me...
>>>
>>> first of all, i have a problem for compiling my program (when i include
>>> marker.h), it gives the following error :
>>>
>>> In file included from /usr/local/include/ust/kcompat/kcompat.h:64,
>>>                   from /usr/local/include/ust/kernelcompat.h:21,
>>>                   from /usr/local/include/ust/marker.h:31,
>>>                   from task.lbc.C:4:
>>> /usr/local/include/ust/kcompat/jhash.h: In function ‘u32 jhash(const
>>> void*, u32, u32)’:
>>> /usr/local/include/ust/kcompat/jhash.h:47: error: invalid conversion
>>> from ‘const void*’ to ‘const u8*’
>>

Should be fixed in the latest git. I expect the error here was due to 
the usage of strict aliasing rules. This is now disabled in ust because 
the lttng code assumes -fno-strict-aliasing.


>>> Here's what i see in the resulting trace directory:
>>>
>>> http://pastebin.com/raw.php?i=TqJWarkA
>>>
>>> it seems that all traces aren't recorded (some metadata and ust are zero
>>> size), i can't understand why, i must be missing something here... ?
>>
>> -f should fix this.
>>
>>>
>>> also, when using lttv, it says "Cannot open trace : maybe you should
>>> enter in the trace directory to select it ?"
>>> i note that lttv can open a subdirectory (like
>>> /root/.usttraces/californium-20100329172748911948191/8494_5454077942969735172 
>>>
>>> in my example), but obviously all i can see is a subset of what really
>>> happenned...

I just looked at this more closely.

If your program does not fork (nor clone without the CLONE_VM flag), 
it's still possible you are tracing several processes. This would be 
caused by the fact that the command you are passing to usttrace is not 
an elf executable but rather a shell script that does many things like 
run commands, which starts new processes. For example, a shell script 
that uses find, sed, ls etc. will start them in their distinct process. 
The tracer will trace everything. To find out what these processes are, 
you could run the same command with strace -f and see what is going on.

The fact that some of the files are empty means that the daemon was 
unable to connect to the processes at some point. Indeed, you are 
getting printed errors about this. But it is initially able to because 
we see there is at least one file that is not empty. This is unlikely 
due to the fact the process stops existing too quickly. This could 
happen because of an exec() or an exit(), but there is a keep alive 
mechanism (triggered by ustctl -f in the case of the exec) that induces 
a delay in these calls when the ustd did not have the time to connect to 
all buffers yet.

Either there is a bug in this mechanism, or something crashes. Could be 
a bug in your program or in ust. When the traced program crashes, only 
the buffers that were already connected can be recovered. Enabling core 
dumps (ulimit -c) and checking if there are core files in the directory 
after execution could help finding if you have segfaults or similar crashes.

Did you have a look at ustd.log and app.log in the trace directory? They 
could give additional information.

It could help to just try to run the raw executable to usttrace. It 
would diminish the complexity of the operation. But it is supposed to 
work even with lots of processes like here.

Thanks

pmf