[lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Brian Hutchinson]

On Thu, May 18, 2023 at 3:07 PM Brian Hutchinson <b.hutchman at gmail.com> wrote:
>
> On Thu, May 18, 2023 at 3:03 PM Mathieu Desnoyers
> <mathieu.desnoyers at efficios.com> wrote:
> >
> > On 2023-05-18 14:58, Brian Hutchinson wrote:
> > > On Thu, May 18, 2023 at 11:00 AM Brian Hutchinson <b.hutchman at gmail.com> wrote:
> > >>
> > >> On Thu, May 18, 2023 at 10:45 AM Mathieu Desnoyers
> > >> <mathieu.desnoyers at efficios.com> wrote:
> > >>>
> > >>> On 2023-05-18 10:10, Brian Hutchinson wrote:
> > >>> [...]
> > >>>> I updated my hello world to have a function I'd like to use the
> > >>>> --userspace-probe method on with the very original name of
> > >>>> 'probe_function':
> > >>>>
> > >>>> #include <stdio.h>
> > >>>> #include <lttng/tracef.h>
> > >>>>
> > >>>> void probe_function(int i);
> > >>>>
> > >>>> int main(int argc, char *argv[])
> > >>>> {
> > >>>>      unsigned int i;
> > >>>>      puts("Hello, World!\nPress Enter to continue...");
> > >>>>      /*
> > >>>>       * The following getchar() call only exists for the purpose of this
> > >>>>       * demonstration, to pause the application in order for you to have
> > >>>>       * time to list its tracepoints. You don't need it otherwise.
> > >>>>       */
> > >>>>      getchar();
> > >>>>
> > >>>>      lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
> > >>>>      printf("Number %d, string %s", 23, "hi there!");
> > >>>>
> > >>>>      for (i = 0; i < argc; i++) {
> > >>>>          lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
> > >>>>          printf("Number %d, argv %s", i, argv[i]);
> > >>>>      }
> > >>>>
> > >>>>      puts("Quitting now!");
> > >>>>
> > >>>>      probe_function(i);
> > >>>>
> > >>>>      return 0;
> > >>>> }
> > >>>>
> > >>>> void probe_function(int i) {
> > >>>>
> > >>>>      lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
> > >>>>      printf("Number %d, string %s", i * i, "i^2");
> > >>>>
> > >>>> }
> > >>>>
> > >>>> ... and I get the same error as before when I try to enable the probe:
> > >>>> # lttng enable-event --kernel
> > >>>> --userspace-probe=/usr/local/bin/hello:probe_function
> > >>>> Error: Missing event name(s).
> > >>>
> > >>> As the error states, you are missing the event name. See
> > >>>
> > >>> man 1 lttng-enable-event
> > >>>
> > >>>          lttng [GENERAL OPTIONS] enable-event --kernel
> > >>>                [--probe=SOURCE | --function=SOURCE | --syscall |
> > >>>                 --userspace-probe=SOURCE]
> > >>>                [--filter=EXPR] [--session=SESSION]
> > >>>                [--channel=CHANNEL] EVENT[,EVENT]...
> > >>>
> > >>>
> > >>> You will want something like:
> > >>>
> > >>> lttng enable-event --kernel --userspace-probe=/usr/local/bin/hello:probe_function my_probe_function
> > >>>
> > >>> Where "my_probe_function" is the event name that will appear in the collected traces.
> > >>
> > >> Wow!  I must not have woken up this morning ha, ha.  Thanks for that!
> > >> The event is enabled now.  Hope to actually get tracing data now.
> > >
> > > Well, I guess we just have the app that thwarts all attempts at tracing.
> > >
> > > I did a dynamic probe on several functions that should be getting
> > > called like crazy and again I get no tracing data.
> > >
> > > Tried it with my hello world example above after Mathieu set me
> > > straight on the event syntax and it works.
> > >
> > > I saw this comment in the documentation "As of this version, only USDT
> > > probes that are not surrounded by a reference counter (semaphore) are
> > > supported."
> > >
> > > I don't know that I can say that this function I'm probing isn't
> > > "surrounded" by a reference counter, it's in a large multi-threaded
> > > application so I guess it's possible.
> > >
> > > Sigh, I'm striking out every which way.
> > >
> > > No offense (since this is lttng list - please don't flame me ... I
> > > want/need lttng), but I think I'm going to try just straight kprobes
> > > and uprobes and see if trace compass can show those traces in an
> > > attempt to get "something/anything" working.
> >
> > If you attach to an ELF symbol (function), then there is no USDT in
> > play, so it should not be related to the issue you have.
>
> That is what I was thinking which is why I wanted to try it.
>
> >
> > But if your functions happen to be inlined, then there will be nothing
> > to attach to. Perhaps this is what happens there ?
>
> I don't see any evidence of anything being inlined in this module.  I
> grepped the code to verify.
>
> Back to being stumped/stuck.

I can do trace-cmd stuff and it works.  The hello world above works so
I don't "think" this is a problem but again in full disclosure I'll
mention/ask about it.

Does any of the lttng tools/libs depend on kernel headers?  I ask
because old yocto (Dunfell) built lttng package against a 4.something
kernel and we're running a 5.10.69 kernel that lttng modules were
added to it with the "builtin" script and built that way.

Should probably have yocto build the local kernel too, but kernel is
being built stand alone due to vendor stuff that hasn't been mainlined
yet.

I'm running out of things to think about that could be the issue.

B