[lttng-dev] Valgrind Memory Error in lttng-relayd
Jérémie Galarneau
jeremie.galarneau at efficios.com
Mon Jul 6 10:53:38 EDT 2015
Are you running both 32 and 64-bits apps at once?
Jérémie
On Mon, Jul 6, 2015 at 12:10 AM, Aravind HT <aravind.ht at gmail.com> wrote:
> This is how I am creating the session and using
>
>
> lttng-relayd -o /var/tmp/lttng-traces -d
>
> lttng-sessiond --consumerd32-path /usr/lib/lttng/libexec/lttng-consumerd
> --consumerd32-libdir /usr/lib/ --consumerd64-path
> /usr/lib64/lttng/libexec/lttng-consumerd --consumerd64-libdir /usr/lib64/
> -b --no-kernel
>
>
>
> lttng --no-sessiond create mysession --live 2000000 -U net://128.0.0.4
>
> lttng enable-channel mychannel --userspace --buffers-pid -s mysession
> --tracefile-size 2000000 --tracefile-count 2
>
> lttng enable-event --channel mychannel --userspace --all --exclude
> 'ust_libc:*' -s mysession
>
> lttng start mysession
>
> lttng list
>
>
> At present, I am not trying to enable new channels. The above is pretty
> much what I do.
>
>
>
>
> On Sat, Jul 4, 2015 at 1:26 AM, Jérémie Galarneau <
> jeremie.galarneau at efficios.com> wrote:
>
>> Can you share a bit more information about the session configurations in
>> your use case?
>> Are you creating and destroying new sessions, tracing in per-pid or uid
>> mode, enabling new channels, etc.
>>
>> Also, is this in "live" mode or are you just streaming to the relay
>> daemon? (lttng create --set-url net://somewhere)
>>
>> Jérémie
>>
>>
>> On Fri, Jul 3, 2015 at 3:30 PM, Jérémie Galarneau <
>> jeremie.galarneau at efficios.com> wrote:
>>
>>>
>>>
>>> On Fri, Jul 3, 2015 at 1:48 PM, Aravind HT <aravind.ht at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Thanks for replying.
>>>>
>>>> I have a daemon which i'm using for some CLI application, this forks
>>>> whenever invoked but never exec()'s and am using lttng for tracing this and
>>>> a few other apps.
>>>> I have a script that I run to stress test this daemon; this I think
>>>> means lttng is also being stressed.
>>>> As soon as I run this script, after a few streams are created, I
>>>> sometimes see a soft hang as mentioned in my previous thread
>>>> http://lists.lttng.org/pipermail/lttng-dev/2015-May/024533.html
>>>> or
>>>> I see lttng-relayd coring.
>>>>
>>>> In short, an app that calls quite a bit of fork and exits which is
>>>> being traced with other normal apps that run forever should re-create the
>>>> problem.
>>>>
>>>> To run valgrind, use the -DVALGRIND when building, your can grep the
>>>> code for what this does.
>>>> Kill the valgrind proc with -TERM to produce the logs.
>>>>
>>>>
>>> Doh, of course! :-)
>>>
>>>
>>>> This is the command and the options that I use to run valgrind
>>>>
>>>> valgrind --tool=memcheck --leak-check=full -v --log-file=/tmp/test
>>>> --trace-children=yes /usr/bin/lttng-relayd -o /var/log/lttng-traces -d
>>>>
>>>>
>>>>
>>>> I have tried the fix detailed in the checkin
>>>> http://git.lttng.org/?p=lttng-tools.git;a=commitdiff;h=cd2ef1ef1d54ced9e4d0d03b865bb7fc6a905f80
>>>>
>>>> and it seems to work for me. This however only solves the relayd core,
>>>> but I still see the "soft hang" happening.
>>>>
>>>
>>>> Can some one please tell me how the above fix is missing from the
>>>> source ?
>>>>
>>>
>>> Somehow it appears this was reverted by this commit:
>>>
>>> https://github.com/lttng/lttng-tools/commit/1dc0526df43f2b5f86ef451e4c0331445346b15f
>>>
>>> I'm trying to reproduce the problem here.
>>>
>>> Jérémie
>>>
>>>
>>>>
>>>> Regards,
>>>>
>>>> Aravind.
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Jul 3, 2015 at 8:36 PM, Jérémie Galarneau <
>>>> jeremie.galarneau at efficios.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Thanks for the detailed report. Can you provide a sequence of steps
>>>>> that leads to this problem?
>>>>> Also, did you modify lttng-relayd at all to be able to run valgrind? I
>>>>> seem to remember valgrind not supporting clone().
>>>>>
>>>>> Thanks,
>>>>> Jérémie
>>>>>
>>>>> On Thu, Jul 2, 2015 at 12:42 PM, Aravind HT <aravind.ht at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Possible Solution :
>>>>>>
>>>>>> The following changes seem to be missing from the actual code
>>>>>>
>>>>>>
>>>>>> http://git.lttng.org/?p=lttng-tools.git;a=commitdiff;h=cd2ef1ef1d54ced9e4d0d03b865bb7fc6a905f80
>>>>>>
>>>>>>
>>>>>> only the connection.h changes have been absorbed, while the main.c
>>>>>> changes left out.
>>>>>>
>>>>>> On Mon, Jun 29, 2015 at 11:51 PM, Aravind HT <aravind.ht at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I ran valgrind on lttng-relayd for a soft hang issue that I had
>>>>>>> reported earlier.
>>>>>>> http://lists.lttng.org/pipermail/lttng-dev/2015-May/024533.html
>>>>>>>
>>>>>>> Below are the list of errors that I found.
>>>>>>>
>>>>>>> ==793== Thread 4:
>>>>>>> ==793== Invalid write of size 8
>>>>>>> ==793== at 0x406038: cds_list_add (list.h:53)
>>>>>>> ==793== by 0x409442: queue_stream (main.c:1175)
>>>>>>> ==793== by 0x409AB7: relay_add_stream (main.c:1274)
>>>>>>> ==793== by 0x40C47A: relay_process_control (main.c:2099)
>>>>>>> ==793== by 0x40DBB1: relay_thread_worker (main.c:2574)
>>>>>>> ==793== by 0x5BDAFE2: start_thread (pthread_create.c:312)
>>>>>>> ==793== by 0x5ED8AFC: clone (clone.S:111)
>>>>>>> ==793== Address 0x6e09040 is 64 bytes inside a block of size 312
>>>>>>> free'd
>>>>>>> ==793== at 0x4C2898C: free (in
>>>>>>> /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>>>>> ==793== by 0x418EF8: rcu_destroy_stream (stream.c:33)
>>>>>>> ==793== by 0x5079C94: ??? (in /usr/lib64/liburcu.so.2.0.0)
>>>>>>> ==793== by 0x5BDAFE2: start_thread (pthread_create.c:312)
>>>>>>> ==793== by 0x5ED8AFC: clone (clone.S:111)
>>>>>>> ==793==
>>>>>>>
>>>>>>> Are there any known bugs/fixes for these ?
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Aravind.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> lttng-dev mailing list
>>>>>> lttng-dev at lists.lttng.org
>>>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jérémie Galarneau
>>>>> EfficiOS Inc.
>>>>> http://www.efficios.com
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Jérémie Galarneau
>>> EfficiOS Inc.
>>> http://www.efficios.com
>>>
>>
>>
>>
>> --
>> Jérémie Galarneau
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>
>
--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20150706/5039e3bc/attachment-0001.html>
More information about the lttng-dev
mailing list