[lttng-dev] Tracepoints firing without being enabled in LTTng

Tue Feb 19 17:16:39 EST 2019

Hi Mosleh,
Just to be clear:
When you run your app as a service (background), lttng list -u doesn't
display any events AND you see events in the live session without
having run enable-event and start commands.
When you run your app normally, lttng list -u displays events AND you
only see events after running enable-event and start commands.
Is that correct?

Please provide the output of the lttng-sessiond in verbose mode using
the following command:
> lttng-sessiond -vvv
please provide the output of your application as a service with this
environment variable:  LTTNG_UST_DEBUG=1
and please provide us with the service file describing how your
service is started.
Put all of this on a pastebin.com

Is your application mass-closing file descriptors when it's
daemonizing? how about using fork() syscall without an exec() after
it? If so, have a look at this part of the UST man page:
https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons

Thank you,
Francis

Le mar. 19 févr. 2019, à 16 h 37, Mosleh Uddin <mosleh1235 at gmail.com> a écrit :
>
> Hello,
>
> Sorry for the late reply. I was able to figure out the weird behavior from the application. Turns out if I run my application as a service,  lttng no longer can see any of the UST tracepoints and causes the behavior I was explaining.
> When I execute the application normally (./app_name), the functionality is as expected. Is this the expected behavior? Is there some way I can trace the userspace application while running it as a service simultaneously?
>
> Thanks for any info,
> Mosleh
>
> On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers <francis.deslauriers at efficios.com> wrote:
>>
>> Can you start the lttng-sessiond in verbose mode using the following command:
>> > lttng-sessiond -vvv
>> and then do the lttng create, start of application and lttng view on another console.
>>
>> Then copy the output of the sessiond on https://pastebin.com/ and send us the link.
>>
>> Thank you,
>> Francis
>>
>> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin <mosleh1235 at gmail.com> a écrit :
>>>
>>> That's correct, I don't enable any events or do: lttng start.
>>> Here is some snapshots of starting everything up, followed by starting my application.
>>>
>>>
>>> Here are the outputs for the commands:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Note that even though the session is inactive, when a trace event occurs in my application it is being recorded in the appropriate directory:
>>>
>>>
>>>
>>> Also maybe important, the system has root only user. Would having the application belonging to a different user and trying to run as root cause these kinds of complications? I did notice when testing on a raspberry pi that running an application as a root user would mean running sessions as root user as well.
>>>
>>> Thanks,
>>> Mosleh
>>>
>>> On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers <francis.deslauriers at efficios.com> wrote:
>>>>
>>>> Adding back lttng-dev mailing list.
>>>>
>>>> So you don't even run a lttng-enable-event command?
>>>> That's odd.
>>>>
>>>> Can you send us the output of the following commands:
>>>> Is it possible that there are other lttng-sessiond daemons running on
>>>> the machine?
>>>> > ps aux | grep "lttng-sessiond"
>>>> Verify if there are other active tracing sessions:
>>>> > lttng list
>>>> Check the details of the current session:
>>>> > lttng status
>>>>
>>>> Thank you,
>>>> Francis
>>>>
>>>> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin <mosleh1235 at gmail.com> a écrit :
>>>> >
>>>> > Hi Francis,
>>>> >
>>>> > Thank you for the quick response.
>>>> > Just to clear things up, my application is running when I run that command.
>>>> >
>>>> > The commands I am running are:
>>>> > > lttng-sessiond -d
>>>> > > lttng create my-sess --live
>>>> > After this command I start my application. If I then do the command:
>>>> > > lttng view
>>>> > I will see the tracepoints in the application in the terminal even though I have not done: lttng start.
>>>> >
>>>> > I thought it was strange that I cannot find my application in the: lttng list command under any domain.
>>>> > Also the reason I start my application after creating a session is because the system automatically starts the application using systemd,
>>>> > in order to get tracing data at all I have to manually stop the application and start it again after creating a session.
>>>> >
>>>> > Thanks again for any help,
>>>> > Mosleh
>>>> >
>>>> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers <francis.deslauriers at efficios.com> wrote:
>>>> >>
>>>> >> Hi Mosleh,
>>>> >> It's expected to not see any event when running lttng list -u if no
>>>> >> lttng-ust instrumented applications are running.
>>>> >>
>>>> >> What are the exact lttng commands you are running to see those
>>>> >> unexpected events?
>>>> >>
>>>> >> Thank you,
>>>> >> Francis
>>>> >>
>>>> >>
>>>> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin <mosleh1235 at gmail.com> a écrit :
>>>> >> >
>>>> >> > Hello,
>>>> >> >
>>>> >> > I am having a strange issue with my current application. For some context, I have integrated LTTng into a Yocto build and put custom trace points in my embedded system application. Once I load the built image onto my system and start a session daemon, I try to list all available userspace events (using lttng list -u), but no events show up.
>>>> >> >
>>>> >> > The strange occurrence is when I create a session (lltng create my-sess) and not start tracing the trace events will display on the screen regardless (when starting session in live mode with lttng view). I was wondering if anyone has any insight on this issue I am having.
>>>> >> >
>>>> >> > Thanks for any assistance,
>>>> >> > Mosleh
>>>> >> > _______________________________________________
>>>> >> > lttng-dev mailing list
>>>> >> > lttng-dev at lists.lttng.org
>>>> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>> >>
>>>> >> --
>>>> >> Francis Deslauriers
>>>> >> Computer Engineer
>>>> >> EfficiOS inc.
>>>>
>>>>
>>>>
>>>> --
>>>> Francis Deslauriers
>>>> Computer Engineer
>>>> EfficiOS inc.
>>>
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev at lists.lttng.org
>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>>
>>
>> --
>> Francis Deslauriers
>> Computer Engineer
>> EfficiOS inc.

-- 
Francis Deslauriers
Computer Engineer
EfficiOS inc.