[lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Brian Hutchinson]

On Thu, May 18, 2023 at 3:03 PM Mathieu Desnoyers
<mathieu.desnoyers at efficios.com> wrote:
>
> On 2023-05-18 14:58, Brian Hutchinson wrote:
> > On Thu, May 18, 2023 at 11:00 AM Brian Hutchinson <b.hutchman at gmail.com> wrote:
> >>
> >> On Thu, May 18, 2023 at 10:45 AM Mathieu Desnoyers
> >> <mathieu.desnoyers at efficios.com> wrote:
> >>>
> >>> On 2023-05-18 10:10, Brian Hutchinson wrote:
> >>> [...]
> >>>> I updated my hello world to have a function I'd like to use the
> >>>> --userspace-probe method on with the very original name of
> >>>> 'probe_function':
> >>>>
> >>>> #include <stdio.h>
> >>>> #include <lttng/tracef.h>
> >>>>
> >>>> void probe_function(int i);
> >>>>
> >>>> int main(int argc, char *argv[])
> >>>> {
> >>>>      unsigned int i;
> >>>>      puts("Hello, World!\nPress Enter to continue...");
> >>>>      /*
> >>>>       * The following getchar() call only exists for the purpose of this
> >>>>       * demonstration, to pause the application in order for you to have
> >>>>       * time to list its tracepoints. You don't need it otherwise.
> >>>>       */
> >>>>      getchar();
> >>>>
> >>>>      lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
> >>>>      printf("Number %d, string %s", 23, "hi there!");
> >>>>
> >>>>      for (i = 0; i < argc; i++) {
> >>>>          lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
> >>>>          printf("Number %d, argv %s", i, argv[i]);
> >>>>      }
> >>>>
> >>>>      puts("Quitting now!");
> >>>>
> >>>>      probe_function(i);
> >>>>
> >>>>      return 0;
> >>>> }
> >>>>
> >>>> void probe_function(int i) {
> >>>>
> >>>>      lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
> >>>>      printf("Number %d, string %s", i * i, "i^2");
> >>>>
> >>>> }
> >>>>
> >>>> ... and I get the same error as before when I try to enable the probe:
> >>>> # lttng enable-event --kernel
> >>>> --userspace-probe=/usr/local/bin/hello:probe_function
> >>>> Error: Missing event name(s).
> >>>
> >>> As the error states, you are missing the event name. See
> >>>
> >>> man 1 lttng-enable-event
> >>>
> >>>          lttng [GENERAL OPTIONS] enable-event --kernel
> >>>                [--probe=SOURCE | --function=SOURCE | --syscall |
> >>>                 --userspace-probe=SOURCE]
> >>>                [--filter=EXPR] [--session=SESSION]
> >>>                [--channel=CHANNEL] EVENT[,EVENT]...
> >>>
> >>>
> >>> You will want something like:
> >>>
> >>> lttng enable-event --kernel --userspace-probe=/usr/local/bin/hello:probe_function my_probe_function
> >>>
> >>> Where "my_probe_function" is the event name that will appear in the collected traces.
> >>
> >> Wow!  I must not have woken up this morning ha, ha.  Thanks for that!
> >> The event is enabled now.  Hope to actually get tracing data now.
> >
> > Well, I guess we just have the app that thwarts all attempts at tracing.
> >
> > I did a dynamic probe on several functions that should be getting
> > called like crazy and again I get no tracing data.
> >
> > Tried it with my hello world example above after Mathieu set me
> > straight on the event syntax and it works.
> >
> > I saw this comment in the documentation "As of this version, only USDT
> > probes that are not surrounded by a reference counter (semaphore) are
> > supported."
> >
> > I don't know that I can say that this function I'm probing isn't
> > "surrounded" by a reference counter, it's in a large multi-threaded
> > application so I guess it's possible.
> >
> > Sigh, I'm striking out every which way.
> >
> > No offense (since this is lttng list - please don't flame me ... I
> > want/need lttng), but I think I'm going to try just straight kprobes
> > and uprobes and see if trace compass can show those traces in an
> > attempt to get "something/anything" working.
>
> If you attach to an ELF symbol (function), then there is no USDT in
> play, so it should not be related to the issue you have.

That is what I was thinking which is why I wanted to try it.

>
> But if your functions happen to be inlined, then there will be nothing
> to attach to. Perhaps this is what happens there ?

I don't see any evidence of anything being inlined in this module.  I
grepped the code to verify.

Back to being stumped/stuck.

B