[ltt-dev] problems with ust

[Pierre-Marc Fournier]

On 04/15/2010 11:43 AM, Sylvain Geneves wrote:
> On 04/14/2010 06:02 PM, Pierre-Marc Fournier wrote:
>> On 04/14/2010 05:26 AM, Sylvain Geneves wrote:
>>>>
>>>>>>> Here's what i see in the resulting trace directory:
>>>>>>>
>>>>>>> http://pastebin.com/raw.php?i=TqJWarkA
>>>>>>>
>>>>>>> it seems that all traces aren't recorded (some metadata and ust are
>>>>>>> zero
>>>>>>> size), i can't understand why, i must be missing something here... ?
>>>>>>
>>>>>> -f should fix this.
>>>>>>
>>>>>>>
>>>>>>> also, when using lttv, it says "Cannot open trace : maybe you should
>>>>>>> enter in the trace directory to select it ?"
>>>>>>> i note that lttv can open a subdirectory (like
>>>>>>> /root/.usttraces/californium-20100329172748911948191/8494_5454077942969735172
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> in my example), but obviously all i can see is a subset of what
>>>>>>> really
>>>>>>> happenned...
>>>>
>>>> I just looked at this more closely.
>>>>
>>> thanks for your help
>>>
>>>
>>>> If your program does not fork (nor clone without the CLONE_VM flag),
>>>> it's still possible you are tracing several processes. This would be
>>>> caused by the fact that the command you are passing to usttrace is not
>>>> an elf executable but rather a shell script that does many things like
>>>> run commands, which starts new processes. For example, a shell script
>>>> that uses find, sed, ls etc. will start them in their distinct process.
>>>> The tracer will trace everything. To find out what these processes are,
>>>> you could run the same command with strace -f and see what is going on.
>>>>
>>>
>>> the command i launch is actually an executable, not a script. What it
>>> does is quite simple :
>>> - the application registers some events to be processed
>>> - it creates one thread per core on the machine
>>> - each thread executes a loop to execute events
>>> after all events has been processed, each created thread calls
>>> pthread_exit, and the application exits normally.
>>
>> Interesting, I have a very similar program I use for benchmarking ust
>> and it works well.
>>
>>>
>>> the output of strace -f shows that there is no call to fork in there,
>>> but clone is called multiple times.
>>>
>>> that leads me to a question about how to correctly enable the CLONE_VM
>>> flag for ust : is enabling it through the shell environment (export
>>> CLONE_VM=1) sufficient ?
>>> is there another way to do it ?
>>
>> CLONE_VM is a flag you pass to the clone() call. If you don't pass it,
>> clone() acts more like fork() in the sense it creates a new address
>> space for the new task. If you do pass it, the same address space is
>> reused, effectively creating a thread. If you are using pthreads, then
>> pthread is passing CLONE_VM, you can actually see it with strace.
>
> thanks for your explanation, i only use pthread_create, and don't call
> clone directly so it shouldn't be a problem.
>
>>
>> The reason I was suggesting you use strace is to find out what the tons
>> of processes we are seeing are. Are you using the -l option to usttrace?
>> This could be another cause. If so, try to stop using it. It's going to
>> simplify things.
>
> indeed i was using -l flag with usttrace, removing it simplifies a lot
> the output!
> here's what i get when i list the trace directory without the -l option:
>
> $ls ~/.usttraces/californium-20100415164805348661147/
> 14989_5460376113077649648 app.log ustd.log
>
>
> the only error message i see now is the following :
>
> sh: symbol lookup error: /usr/local/lib/libustfork.so: undefined symbol:
> ust_before_fork
>
> the simlink is valid and points to the librairy.
>
> $readelf -a /usr/local/lib/libustfork.so.0.0.0 | grep ust_before_fork
> 00001ba4 00000607 R_386_JUMP_SLOT 00000000 ust_before_fork
> 6: 00000000 0 NOTYPE GLOBAL DEFAULT UND ust_before_fork
> 64: 00000000 0 NOTYPE GLOBAL DEFAULT UND ust_before_fork
>
>
> it seems that the function is declared somewhere, but not defined ?

Since your program does not use fork(), don't use the -f option to 
usttrace, and it should fix this problem.

Thanks for these detailed reports, it's giving me ideas about how to 
make error messages more comprehensible.