[lttng-dev] Tracepoints firing without being enabled in LTTng

Mathieu Desnoyers mathieu.desnoyers at efficios.com
Wed Feb 20 09:46:57 EST 2019 

----- On Feb 20, 2019, at 9:10 AM, Mosleh Uddin <mosleh1235 at gmail.com> wrote: 

> Hello,
> Here is what I do to get the consistent behavior:
> When the app is running as a service, lttng list -u does not display any events,
> initially I do have to lttng enable-events and lttng start to see the
> tracepoints in lttng view. However, after lttng start, I can disable all events
> and I can still see the events coming through in lttng view. When I run the app
> normally, lttng list -u displays the tracepoints I place in the app, and all
> functionality of lttng works as expected.

> I apologize, I am in the process of trying to get you the outputs you are asking
> for, however due to confidentiality with the app the outputs of running the
> commands need to be reviewed by others. I will try to get this to you as soon
> as I can.

> I can say the service is being kick started by systemd. Also I believe the app
> is mass-closing file descriptors when it is daemonizing. I tried to dynamically
> load the liblttng-ust-fd.so but the behavior did not change.

Did you load both liblttng-ust-fork.so and liblttng-ust-fd.so with LD_PRELOAD when launching your application ? 

If not using LD_PRELOAD env. var., how did you load them ? 

Thanks, 

Mathieu 

> Thanks,
> Mosleh

> On Tue, Feb 19, 2019 at 5:16 PM Francis Deslauriers < [
> mailto:francis.deslauriers at efficios.com | francis.deslauriers at efficios.com ] >
> wrote:

>> Hi Mosleh,
>> Just to be clear:
>> When you run your app as a service (background), lttng list -u doesn't
>> display any events AND you see events in the live session without
>> having run enable-event and start commands.
>> When you run your app normally, lttng list -u displays events AND you
>> only see events after running enable-event and start commands.
>> Is that correct?

>> Please provide the output of the lttng-sessiond in verbose mode using
>> the following command:
>> > lttng-sessiond -vvv
>> please provide the output of your application as a service with this
>> environment variable: LTTNG_UST_DEBUG=1
>> and please provide us with the service file describing how your
>> service is started.
>> Put all of this on a [ http://pastebin.com/ | pastebin.com ]

>> Is your application mass-closing file descriptors when it's
>> daemonizing? how about using fork() syscall without an exec() after
>> it? If so, have a look at this part of the UST man page:
>> [ https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons |
>> https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons ]

>> Thank you,
>> Francis

>> Le mar. 19 févr. 2019, à 16 h 37, Mosleh Uddin < [ mailto:mosleh1235 at gmail.com |
>> mosleh1235 at gmail.com ] > a écrit :

>> > Hello,

>>> Sorry for the late reply. I was able to figure out the weird behavior from the
>>> application. Turns out if I run my application as a service, lttng no longer
>> > can see any of the UST tracepoints and causes the behavior I was explaining.
>>> When I execute the application normally (./app_name), the functionality is as
>>> expected. Is this the expected behavior? Is there some way I can trace the
>> > userspace application while running it as a service simultaneously?

>> > Thanks for any info,
>> > Mosleh

>>> On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers < [
>>> mailto:francis.deslauriers at efficios.com | francis.deslauriers at efficios.com ] >
>> > wrote:

>> >> Can you start the lttng-sessiond in verbose mode using the following command:
>> >> > lttng-sessiond -vvv
>>>> and then do the lttng create, start of application and lttng view on another
>> >> console.

>>>> Then copy the output of the sessiond on [ https://pastebin.com/ |
>> >> https://pastebin.com/ ] and send us the link.

>> >> Thank you,
>> >> Francis

>>>> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin < [ mailto:mosleh1235 at gmail.com |
>> >> mosleh1235 at gmail.com ] > a écrit :

>> >>> That's correct, I don't enable any events or do: lttng start.
>>>>> Here is some snapshots of starting everything up, followed by starting my
>> >>> application.


>> >>> Here are the outputs for the commands:







>>>>> Note that even though the session is inactive, when a trace event occurs in my
>> >>> application it is being recorded in the appropriate directory:



>>>>> Also maybe important, the system has root only user. Would having the
>>>>> application belonging to a different user and trying to run as root cause these
>>>>> kinds of complications? I did notice when testing on a raspberry pi that
>>>>> running an application as a root user would mean running sessions as root user
>> >>> as well.

>> >>> Thanks,
>> >>> Mosleh

>>>>> On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers < [
>>>>> mailto:francis.deslauriers at efficios.com | francis.deslauriers at efficios.com ] >
>> >>> wrote:

>> >>>> Adding back lttng-dev mailing list.

>> >>>> So you don't even run a lttng-enable-event command?
>> >>>> That's odd.

>> >>>> Can you send us the output of the following commands:
>> >>>> Is it possible that there are other lttng-sessiond daemons running on
>> >>>> the machine?
>> >>>> > ps aux | grep "lttng-sessiond"
>> >>>> Verify if there are other active tracing sessions:
>> >>>> > lttng list
>> >>>> Check the details of the current session:
>> >>>> > lttng status

>> >>>> Thank you,
>> >>>> Francis

>>>>>> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin < [ mailto:mosleh1235 at gmail.com |
>> >>>> mosleh1235 at gmail.com ] > a écrit :

>> >>>> > Hi Francis,

>> >>>> > Thank you for the quick response.
>> >>>> > Just to clear things up, my application is running when I run that command.

>> >>>> > The commands I am running are:
>> >>>> > > lttng-sessiond -d
>> >>>> > > lttng create my-sess --live
>> >>>> > After this command I start my application. If I then do the command:
>> >>>> > > lttng view
>>>>>> > I will see the tracepoints in the application in the terminal even though I have
>> >>>> > not done: lttng start.

>>>>>> > I thought it was strange that I cannot find my application in the: lttng list
>> >>>> > command under any domain.
>>>>>> > Also the reason I start my application after creating a session is because the
>> >>>> > system automatically starts the application using systemd,
>>>>>> > in order to get tracing data at all I have to manually stop the application and
>> >>>> > start it again after creating a session.

>> >>>> > Thanks again for any help,
>> >>>> > Mosleh

>>>>>> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers < [
>>>>>> > mailto:francis.deslauriers at efficios.com | francis.deslauriers at efficios.com ] >
>> >>>> > wrote:

>> >>>> >> Hi Mosleh,
>> >>>> >> It's expected to not see any event when running lttng list -u if no
>> >>>> >> lttng-ust instrumented applications are running.

>> >>>> >> What are the exact lttng commands you are running to see those
>> >>>> >> unexpected events?

>> >>>> >> Thank you,
>> >>>> >> Francis


>>>>>> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin < [ mailto:mosleh1235 at gmail.com |
>> >>>> >> mosleh1235 at gmail.com ] > a écrit :

>> >>>> >> > Hello,

>>>>>> >> > I am having a strange issue with my current application. For some context, I
>>>>>> >> > have integrated LTTng into a Yocto build and put custom trace points in my
>>>>>> >> > embedded system application. Once I load the built image onto my system and
>>>>>> >> > start a session daemon, I try to list all available userspace events (using
>> >>>> >> > lttng list -u), but no events show up.

>>>>>> >> > The strange occurrence is when I create a session (lltng create my-sess) and not
>>>>>> >> > start tracing the trace events will display on the screen regardless (when
>>>>>> >> > starting session in live mode with lttng view). I was wondering if anyone has
>> >>>> >> > any insight on this issue I am having.

>> >>>> >> > Thanks for any assistance,
>> >>>> >> > Mosleh
>> >>>> >> > _______________________________________________
>> >>>> >> > lttng-dev mailing list
>> >>>> >> > [ mailto:lttng-dev at lists.lttng.org | lttng-dev at lists.lttng.org ]
>>>>>> >> > [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>> >>>> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>> >>>> >> --
>> >>>> >> Francis Deslauriers
>> >>>> >> Computer Engineer
>> >>>> >> EfficiOS inc.



>> >>>> --
>> >>>> Francis Deslauriers
>> >>>> Computer Engineer
>> >>>> EfficiOS inc.

>> >>> _______________________________________________
>> >>> lttng-dev mailing list
>> >>> [ mailto:lttng-dev at lists.lttng.org | lttng-dev at lists.lttng.org ]
>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>> >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]



>> >> --
>> >> Francis Deslauriers
>> >> Computer Engineer
>> >> EfficiOS inc.

>> --
>> Francis Deslauriers
>> Computer Engineer
>> EfficiOS inc.

> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20190220/aadcfa11/attachment-0001.html>

More information about the lttng-dev mailing list