[lttng-dev] Valgrind Memory Error in lttng-relayd

Fri Jul 3 15:30:34 EDT 2015

On Fri, Jul 3, 2015 at 1:48 PM, Aravind HT <aravind.ht at gmail.com> wrote:

> Hi,
>
> Thanks for replying.
>
> I have a daemon which i'm using for some CLI application, this forks
> whenever invoked but never exec()'s and am using lttng for tracing this and
> a few other apps.
> I have a script that I run to stress test this daemon; this I think means
> lttng is also being stressed.
> As soon as I run this script, after a few streams are created, I sometimes
> see a soft hang as mentioned in my previous thread
> http://lists.lttng.org/pipermail/lttng-dev/2015-May/024533.html
> or
> I see lttng-relayd coring.
>
> In short, an app that calls quite a bit of fork and exits which is being
> traced with other normal apps that run forever should re-create the problem.
>
> To run valgrind, use the -DVALGRIND when building, your can grep the code
> for what this does.
> Kill the valgrind  proc with -TERM to produce the logs.
>
>
Doh, of course! :-)

> This is the command and the options that I use to run valgrind
>
> valgrind --tool=memcheck --leak-check=full -v --log-file=/tmp/test
> --trace-children=yes  /usr/bin/lttng-relayd -o /var/log/lttng-traces -d
>
>
>
> I have tried the fix detailed in the checkin
> http://git.lttng.org/?p=lttng-tools.git;a=commitdiff;h=cd2ef1ef1d54ced9e4d0d03b865bb7fc6a905f80
>
> and it seems to work for me. This however only solves the relayd core, but
> I still see the "soft hang" happening.
>

> Can some one please tell me how the above fix is missing from the source ?
>

Somehow it appears this was reverted by this commit:
https://github.com/lttng/lttng-tools/commit/1dc0526df43f2b5f86ef451e4c0331445346b15f

I'm trying to reproduce the problem here.

Jérémie

>
> Regards,
>
> Aravind.
>
>
>
>
> On Fri, Jul 3, 2015 at 8:36 PM, Jérémie Galarneau <
> jeremie.galarneau at efficios.com> wrote:
>
>> Hi,
>>
>> Thanks for the detailed report. Can you provide a sequence of steps that
>> leads to this problem?
>> Also, did you modify lttng-relayd at all to be able to run valgrind? I
>> seem to remember valgrind not supporting clone().
>>
>> Thanks,
>> Jérémie
>>
>> On Thu, Jul 2, 2015 at 12:42 PM, Aravind HT <aravind.ht at gmail.com> wrote:
>>
>>> Possible Solution :
>>>
>>> The following changes seem to be missing from the actual code
>>>
>>>
>>> http://git.lttng.org/?p=lttng-tools.git;a=commitdiff;h=cd2ef1ef1d54ced9e4d0d03b865bb7fc6a905f80
>>>
>>>
>>> only the connection.h changes have been absorbed, while the main.c
>>> changes left out.
>>>
>>> On Mon, Jun 29, 2015 at 11:51 PM, Aravind HT <aravind.ht at gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I ran valgrind on lttng-relayd for a soft hang issue that I had
>>>> reported earlier.
>>>> http://lists.lttng.org/pipermail/lttng-dev/2015-May/024533.html
>>>>
>>>> Below are the list of errors that I found.
>>>>
>>>> ==793== Thread 4:
>>>> ==793== Invalid write of size 8
>>>> ==793==    at 0x406038: cds_list_add (list.h:53)
>>>> ==793==    by 0x409442: queue_stream (main.c:1175)
>>>> ==793==    by 0x409AB7: relay_add_stream (main.c:1274)
>>>> ==793==    by 0x40C47A: relay_process_control (main.c:2099)
>>>> ==793==    by 0x40DBB1: relay_thread_worker (main.c:2574)
>>>> ==793==    by 0x5BDAFE2: start_thread (pthread_create.c:312)
>>>> ==793==    by 0x5ED8AFC: clone (clone.S:111)
>>>> ==793==  Address 0x6e09040 is 64 bytes inside a block of size 312 free'd
>>>> ==793==    at 0x4C2898C: free (in
>>>> /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>> ==793==    by 0x418EF8: rcu_destroy_stream (stream.c:33)
>>>> ==793==    by 0x5079C94: ??? (in /usr/lib64/liburcu.so.2.0.0)
>>>> ==793==    by 0x5BDAFE2: start_thread (pthread_create.c:312)
>>>> ==793==    by 0x5ED8AFC: clone (clone.S:111)
>>>> ==793==
>>>>
>>>> Are there any known bugs/fixes for these ?
>>>>
>>>>
>>>> Regards,
>>>> Aravind.
>>>>
>>>>
>>>
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev at lists.lttng.org
>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>
>>>
>>
>>
>> --
>> Jérémie Galarneau
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>
>

-- 
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20150703/d29bf3db/attachment-0001.html>