[lttng-dev] Tracepoints firing without being enabled in LTTng

Wed Feb 20 13:43:00 EST 2019

Hello,

I see, I was loading the liblttng-ust-fd but not fork. Once loading both of
these with LD_PRELOAD everything seems to be working great. Thank you for
all your assistance.

I was also wondering, it may be impractical to always load these shared
objects because the systems starts the app service on boot. Is there an
lttng alternative to integrate this into the build?  I am not too familiar
with dynamic linking so sorry if this sounds trivial.

Thanks,
Mosleh

On Wed, Feb 20, 2019 at 9:46 AM Mathieu Desnoyers <
mathieu.desnoyers at efficios.com> wrote:

>
>
> ----- On Feb 20, 2019, at 9:10 AM, Mosleh Uddin <mosleh1235 at gmail.com>
> wrote:
>
> Hello,
> Here is what I do to get the consistent behavior:
> When the app is running as a service, lttng list -u does not display any
> events, initially I do have to lttng enable-events and lttng start to see
> the tracepoints in lttng view. However, after lttng start, I can disable
> all events and I can still see the events coming through in lttng view.
> When I run the app normally, lttng list -u displays the tracepoints I place
> in the app, and all functionality of lttng works as expected.
>
> I apologize, I am in the process of trying to get you the outputs you are
> asking for, however due to confidentiality with the app the outputs of
> running the commands need to be reviewed by others. I will try to get this
> to you as soon as I can.
>
> I can say the service is being kick started by systemd. Also I believe the
> app is mass-closing file descriptors when it is daemonizing. I tried to
> dynamically load the liblttng-ust-fd.so but the behavior did not change.
>
>
> Did you load both liblttng-ust-fork.so and liblttng-ust-fd.so with
> LD_PRELOAD when launching your application ?
>
> If not using LD_PRELOAD env. var., how did you load them ?
>
> Thanks,
>
> Mathieu
>
>
> Thanks,
> Mosleh
>
> On Tue, Feb 19, 2019 at 5:16 PM Francis Deslauriers <
> francis.deslauriers at efficios.com> wrote:
>
>> Hi Mosleh,
>> Just to be clear:
>> When you run your app as a service (background), lttng list -u doesn't
>> display any events AND you see events in the live session without
>> having run enable-event and start commands.
>> When you run your app normally, lttng list -u displays events AND you
>> only see events after running enable-event and start commands.
>> Is that correct?
>>
>> Please provide the output of the lttng-sessiond in verbose mode using
>> the following command:
>> > lttng-sessiond -vvv
>> please provide the output of your application as a service with this
>> environment variable:  LTTNG_UST_DEBUG=1
>> and please provide us with the service file describing how your
>> service is started.
>> Put all of this on a pastebin.com
>>
>> Is your application mass-closing file descriptors when it's
>> daemonizing? how about using fork() syscall without an exec() after
>> it? If so, have a look at this part of the UST man page:
>> https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons
>>
>> Thank you,
>> Francis
>>
>>
>> Le mar. 19 févr. 2019, à 16 h 37, Mosleh Uddin <mosleh1235 at gmail.com> a
>> écrit :
>> >
>> > Hello,
>> >
>> > Sorry for the late reply. I was able to figure out the weird behavior
>> from the application. Turns out if I run my application as a service,
>> lttng no longer can see any of the UST tracepoints and causes the behavior
>> I was explaining.
>> > When I execute the application normally (./app_name), the functionality
>> is as expected. Is this the expected behavior? Is there some way I can
>> trace the userspace application while running it as a service
>> simultaneously?
>> >
>> > Thanks for any info,
>> > Mosleh
>> >
>> > On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers <
>> francis.deslauriers at efficios.com> wrote:
>> >>
>> >> Can you start the lttng-sessiond in verbose mode using the following
>> command:
>> >> > lttng-sessiond -vvv
>> >> and then do the lttng create, start of application and lttng view on
>> another console.
>> >>
>> >> Then copy the output of the sessiond on https://pastebin.com/ and
>> send us the link.
>> >>
>> >> Thank you,
>> >> Francis
>> >>
>> >> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin <mosleh1235 at gmail.com>
>> a écrit :
>> >>>
>> >>> That's correct, I don't enable any events or do: lttng start.
>> >>> Here is some snapshots of starting everything up, followed by
>> starting my application.
>> >>>
>> >>>
>> >>> Here are the outputs for the commands:
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Note that even though the session is inactive, when a trace event
>> occurs in my application it is being recorded in the appropriate directory:
>> >>>
>> >>>
>> >>>
>> >>> Also maybe important, the system has root only user. Would having the
>> application belonging to a different user and trying to run as root cause
>> these kinds of complications? I did notice when testing on a raspberry pi
>> that running an application as a root user would mean running sessions as
>> root user as well.
>> >>>
>> >>> Thanks,
>> >>> Mosleh
>> >>>
>> >>> On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers <
>> francis.deslauriers at efficios.com> wrote:
>> >>>>
>> >>>> Adding back lttng-dev mailing list.
>> >>>>
>> >>>> So you don't even run a lttng-enable-event command?
>> >>>> That's odd.
>> >>>>
>> >>>> Can you send us the output of the following commands:
>> >>>> Is it possible that there are other lttng-sessiond daemons running on
>> >>>> the machine?
>> >>>> > ps aux | grep "lttng-sessiond"
>> >>>> Verify if there are other active tracing sessions:
>> >>>> > lttng list
>> >>>> Check the details of the current session:
>> >>>> > lttng status
>> >>>>
>> >>>> Thank you,
>> >>>> Francis
>> >>>>
>> >>>> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin <mosleh1235 at gmail.com>
>> a écrit :
>> >>>> >
>> >>>> > Hi Francis,
>> >>>> >
>> >>>> > Thank you for the quick response.
>> >>>> > Just to clear things up, my application is running when I run that
>> command.
>> >>>> >
>> >>>> > The commands I am running are:
>> >>>> > > lttng-sessiond -d
>> >>>> > > lttng create my-sess --live
>> >>>> > After this command I start my application. If I then do the
>> command:
>> >>>> > > lttng view
>> >>>> > I will see the tracepoints in the application in the terminal even
>> though I have not done: lttng start.
>> >>>> >
>> >>>> > I thought it was strange that I cannot find my application in the:
>> lttng list command under any domain.
>> >>>> > Also the reason I start my application after creating a session is
>> because the system automatically starts the application using systemd,
>> >>>> > in order to get tracing data at all I have to manually stop the
>> application and start it again after creating a session.
>> >>>> >
>> >>>> > Thanks again for any help,
>> >>>> > Mosleh
>> >>>> >
>> >>>> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers <
>> francis.deslauriers at efficios.com> wrote:
>> >>>> >>
>> >>>> >> Hi Mosleh,
>> >>>> >> It's expected to not see any event when running lttng list -u if
>> no
>> >>>> >> lttng-ust instrumented applications are running.
>> >>>> >>
>> >>>> >> What are the exact lttng commands you are running to see those
>> >>>> >> unexpected events?
>> >>>> >>
>> >>>> >> Thank you,
>> >>>> >> Francis
>> >>>> >>
>> >>>> >>
>> >>>> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin <
>> mosleh1235 at gmail.com> a écrit :
>> >>>> >> >
>> >>>> >> > Hello,
>> >>>> >> >
>> >>>> >> > I am having a strange issue with my current application. For
>> some context, I have integrated LTTng into a Yocto build and put custom
>> trace points in my embedded system application. Once I load the built image
>> onto my system and start a session daemon, I try to list all available
>> userspace events (using lttng list -u), but no events show up.
>> >>>> >> >
>> >>>> >> > The strange occurrence is when I create a session (lltng create
>> my-sess) and not start tracing the trace events will display on the screen
>> regardless (when starting session in live mode with lttng view). I was
>> wondering if anyone has any insight on this issue I am having.
>> >>>> >> >
>> >>>> >> > Thanks for any assistance,
>> >>>> >> > Mosleh
>> >>>> >> > _______________________________________________
>> >>>> >> > lttng-dev mailing list
>> >>>> >> > lttng-dev at lists.lttng.org
>> >>>> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>> >>>> >>
>> >>>> >> --
>> >>>> >> Francis Deslauriers
>> >>>> >> Computer Engineer
>> >>>> >> EfficiOS inc.
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Francis Deslauriers
>> >>>> Computer Engineer
>> >>>> EfficiOS inc.
>> >>>
>> >>> _______________________________________________
>> >>> lttng-dev mailing list
>> >>> lttng-dev at lists.lttng.org
>> >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>> >>
>> >>
>> >>
>> >> --
>> >> Francis Deslauriers
>> >> Computer Engineer
>> >> EfficiOS inc.
>>
>>
>>
>> --
>> Francis Deslauriers
>> Computer Engineer
>> EfficiOS inc.
>>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20190220/f75f69ed/attachment.html>