[lttng-dev] Tracepoints firing without being enabled in LTTng

Wed Feb 20 09:27:55 EST 2019

Le mer. 20 févr. 2019, à 09 h 10, Mosleh Uddin <mosleh1235 at gmail.com> a écrit :
>
> Hello,
>
> Here is what I do to get the consistent behavior:
> When the app is running as a service, lttng list -u does not display any events, initially I do have to lttng enable-events and lttng start to see the tracepoints in lttng view. However, after lttng start, I can disable all events and I can still see the events coming through in lttng view. When I run the app normally, lttng list -u displays the tracepoints I place in the app, and all functionality of lttng works as expected.
>
> I apologize, I am in the process of trying to get you the outputs you are asking for, however due to confidentiality with the app the outputs of running the commands need to be reviewed by others. I will try to get this to you as soon as I can.
I understand.

Thanks,
Francis
>
> I can say the service is being kick started by systemd. Also I believe the app is mass-closing file descriptors when it is daemonizing. I tried to dynamically load the liblttng-ust-fd.so but the behavior did not change.
>
> Thanks,
> Mosleh
>
> On Tue, Feb 19, 2019 at 5:16 PM Francis Deslauriers <francis.deslauriers at efficios.com> wrote:
>>
>> Hi Mosleh,
>> Just to be clear:
>> When you run your app as a service (background), lttng list -u doesn't
>> display any events AND you see events in the live session without
>> having run enable-event and start commands.
>> When you run your app normally, lttng list -u displays events AND you
>> only see events after running enable-event and start commands.
>> Is that correct?
>>
>> Please provide the output of the lttng-sessiond in verbose mode using
>> the following command:
>> > lttng-sessiond -vvv
>> please provide the output of your application as a service with this
>> environment variable:  LTTNG_UST_DEBUG=1
>> and please provide us with the service file describing how your
>> service is started.
>> Put all of this on a pastebin.com
>>
>> Is your application mass-closing file descriptors when it's
>> daemonizing? how about using fork() syscall without an exec() after
>> it? If so, have a look at this part of the UST man page:
>> https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons
>>
>> Thank you,
>> Francis
>>
>>
>> Le mar. 19 févr. 2019, à 16 h 37, Mosleh Uddin <mosleh1235 at gmail.com> a écrit :
>> >
>> > Hello,
>> >
>> > Sorry for the late reply. I was able to figure out the weird behavior from the application. Turns out if I run my application as a service,  lttng no longer can see any of the UST tracepoints and causes the behavior I was explaining.
>> > When I execute the application normally (./app_name), the functionality is as expected. Is this the expected behavior? Is there some way I can trace the userspace application while running it as a service simultaneously?
>> >
>> > Thanks for any info,
>> > Mosleh
>> >
>> > On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers <francis.deslauriers at efficios.com> wrote:
>> >>
>> >> Can you start the lttng-sessiond in verbose mode using the following command:
>> >> > lttng-sessiond -vvv
>> >> and then do the lttng create, start of application and lttng view on another console.
>> >>
>> >> Then copy the output of the sessiond on https://pastebin.com/ and send us the link.
>> >>
>> >> Thank you,
>> >> Francis
>> >>
>> >> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin <mosleh1235 at gmail.com> a écrit :
>> >>>
>> >>> That's correct, I don't enable any events or do: lttng start.
>> >>> Here is some snapshots of starting everything up, followed by starting my application.
>> >>>
>> >>>
>> >>> Here are the outputs for the commands:
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Note that even though the session is inactive, when a trace event occurs in my application it is being recorded in the appropriate directory:
>> >>>
>> >>>
>> >>>
>> >>> Also maybe important, the system has root only user. Would having the application belonging to a different user and trying to run as root cause these kinds of complications? I did notice when testing on a raspberry pi that running an application as a root user would mean running sessions as root user as well.
>> >>>
>> >>> Thanks,
>> >>> Mosleh
>> >>>
>> >>> On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers <francis.deslauriers at efficios.com> wrote:
>> >>>>
>> >>>> Adding back lttng-dev mailing list.
>> >>>>
>> >>>> So you don't even run a lttng-enable-event command?
>> >>>> That's odd.
>> >>>>
>> >>>> Can you send us the output of the following commands:
>> >>>> Is it possible that there are other lttng-sessiond daemons running on
>> >>>> the machine?
>> >>>> > ps aux | grep "lttng-sessiond"
>> >>>> Verify if there are other active tracing sessions:
>> >>>> > lttng list
>> >>>> Check the details of the current session:
>> >>>> > lttng status
>> >>>>
>> >>>> Thank you,
>> >>>> Francis
>> >>>>
>> >>>> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin <mosleh1235 at gmail.com> a écrit :
>> >>>> >
>> >>>> > Hi Francis,
>> >>>> >
>> >>>> > Thank you for the quick response.
>> >>>> > Just to clear things up, my application is running when I run that command.
>> >>>> >
>> >>>> > The commands I am running are:
>> >>>> > > lttng-sessiond -d
>> >>>> > > lttng create my-sess --live
>> >>>> > After this command I start my application. If I then do the command:
>> >>>> > > lttng view
>> >>>> > I will see the tracepoints in the application in the terminal even though I have not done: lttng start.
>> >>>> >
>> >>>> > I thought it was strange that I cannot find my application in the: lttng list command under any domain.
>> >>>> > Also the reason I start my application after creating a session is because the system automatically starts the application using systemd,
>> >>>> > in order to get tracing data at all I have to manually stop the application and start it again after creating a session.
>> >>>> >
>> >>>> > Thanks again for any help,
>> >>>> > Mosleh
>> >>>> >
>> >>>> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers <francis.deslauriers at efficios.com> wrote:
>> >>>> >>
>> >>>> >> Hi Mosleh,
>> >>>> >> It's expected to not see any event when running lttng list -u if no
>> >>>> >> lttng-ust instrumented applications are running.
>> >>>> >>
>> >>>> >> What are the exact lttng commands you are running to see those
>> >>>> >> unexpected events?
>> >>>> >>
>> >>>> >> Thank you,
>> >>>> >> Francis
>> >>>> >>
>> >>>> >>
>> >>>> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin <mosleh1235 at gmail.com> a écrit :
>> >>>> >> >
>> >>>> >> > Hello,
>> >>>> >> >
>> >>>> >> > I am having a strange issue with my current application. For some context, I have integrated LTTng into a Yocto build and put custom trace points in my embedded system application. Once I load the built image onto my system and start a session daemon, I try to list all available userspace events (using lttng list -u), but no events show up.
>> >>>> >> >
>> >>>> >> > The strange occurrence is when I create a session (lltng create my-sess) and not start tracing the trace events will display on the screen regardless (when starting session in live mode with lttng view). I was wondering if anyone has any insight on this issue I am having.
>> >>>> >> >
>> >>>> >> > Thanks for any assistance,
>> >>>> >> > Mosleh
>> >>>> >> > _______________________________________________
>> >>>> >> > lttng-dev mailing list
>> >>>> >> > lttng-dev at lists.lttng.org
>> >>>> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>> >>>> >>
>> >>>> >> --
>> >>>> >> Francis Deslauriers
>> >>>> >> Computer Engineer
>> >>>> >> EfficiOS inc.
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Francis Deslauriers
>> >>>> Computer Engineer
>> >>>> EfficiOS inc.
>> >>>
>> >>> _______________________________________________
>> >>> lttng-dev mailing list
>> >>> lttng-dev at lists.lttng.org
>> >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>> >>
>> >>
>> >>
>> >> --
>> >> Francis Deslauriers
>> >> Computer Engineer
>> >> EfficiOS inc.
>>
>>
>>
>> --
>> Francis Deslauriers
>> Computer Engineer
>> EfficiOS inc.


-- 
Francis Deslauriers
Computer Engineer
EfficiOS inc.