[lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Brian Hutchinson]

On Thu, May 18, 2023 at 3:24 PM Mathieu Desnoyers
<mathieu.desnoyers at efficios.com> wrote:
>
> On 2023-05-18 15:20, Brian Hutchinson wrote:
> > On Thu, May 18, 2023 at 3:07 PM Brian Hutchinson <b.hutchman at gmail.com> wrote:
> >>
> >> On Thu, May 18, 2023 at 3:03 PM Mathieu Desnoyers
> >> <mathieu.desnoyers at efficios.com> wrote:
> >>>
> >>> On 2023-05-18 14:58, Brian Hutchinson wrote:
> >>>> On Thu, May 18, 2023 at 11:00 AM Brian Hutchinson <b.hutchman at gmail.com> wrote:
> >>>>>
> >>>>> On Thu, May 18, 2023 at 10:45 AM Mathieu Desnoyers
> >>>>> <mathieu.desnoyers at efficios.com> wrote:
> >>>>>>
> >>>>>> On 2023-05-18 10:10, Brian Hutchinson wrote:
> >>>>>> [...]
> >>>>>>> I updated my hello world to have a function I'd like to use the
> >>>>>>> --userspace-probe method on with the very original name of
> >>>>>>> 'probe_function':
> >>>>>>>
> >>>>>>> #include <stdio.h>
> >>>>>>> #include <lttng/tracef.h>
> >>>>>>>
> >>>>>>> void probe_function(int i);
> >>>>>>>
> >>>>>>> int main(int argc, char *argv[])
> >>>>>>> {
> >>>>>>>       unsigned int i;
> >>>>>>>       puts("Hello, World!\nPress Enter to continue...");
> >>>>>>>       /*
> >>>>>>>        * The following getchar() call only exists for the purpose of this
> >>>>>>>        * demonstration, to pause the application in order for you to have
> >>>>>>>        * time to list its tracepoints. You don't need it otherwise.
> >>>>>>>        */
> >>>>>>>       getchar();
> >>>>>>>
> >>>>>>>       lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
> >>>>>>>       printf("Number %d, string %s", 23, "hi there!");
> >>>>>>>
> >>>>>>>       for (i = 0; i < argc; i++) {
> >>>>>>>           lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
> >>>>>>>           printf("Number %d, argv %s", i, argv[i]);
> >>>>>>>       }
> >>>>>>>
> >>>>>>>       puts("Quitting now!");
> >>>>>>>
> >>>>>>>       probe_function(i);
> >>>>>>>
> >>>>>>>       return 0;
> >>>>>>> }
> >>>>>>>
> >>>>>>> void probe_function(int i) {
> >>>>>>>
> >>>>>>>       lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
> >>>>>>>       printf("Number %d, string %s", i * i, "i^2");
> >>>>>>>
> >>>>>>> }
> >>>>>>>
> >>>>>>> ... and I get the same error as before when I try to enable the probe:
> >>>>>>> # lttng enable-event --kernel
> >>>>>>> --userspace-probe=/usr/local/bin/hello:probe_function
> >>>>>>> Error: Missing event name(s).
> >>>>>>
> >>>>>> As the error states, you are missing the event name. See
> >>>>>>
> >>>>>> man 1 lttng-enable-event
> >>>>>>
> >>>>>>           lttng [GENERAL OPTIONS] enable-event --kernel
> >>>>>>                 [--probe=SOURCE | --function=SOURCE | --syscall |
> >>>>>>                  --userspace-probe=SOURCE]
> >>>>>>                 [--filter=EXPR] [--session=SESSION]
> >>>>>>                 [--channel=CHANNEL] EVENT[,EVENT]...
> >>>>>>
> >>>>>>
> >>>>>> You will want something like:
> >>>>>>
> >>>>>> lttng enable-event --kernel --userspace-probe=/usr/local/bin/hello:probe_function my_probe_function
> >>>>>>
> >>>>>> Where "my_probe_function" is the event name that will appear in the collected traces.
> >>>>>
> >>>>> Wow!  I must not have woken up this morning ha, ha.  Thanks for that!
> >>>>> The event is enabled now.  Hope to actually get tracing data now.
> >>>>
> >>>> Well, I guess we just have the app that thwarts all attempts at tracing.
> >>>>
> >>>> I did a dynamic probe on several functions that should be getting
> >>>> called like crazy and again I get no tracing data.
> >>>>
> >>>> Tried it with my hello world example above after Mathieu set me
> >>>> straight on the event syntax and it works.
> >>>>
> >>>> I saw this comment in the documentation "As of this version, only USDT
> >>>> probes that are not surrounded by a reference counter (semaphore) are
> >>>> supported."
> >>>>
> >>>> I don't know that I can say that this function I'm probing isn't
> >>>> "surrounded" by a reference counter, it's in a large multi-threaded
> >>>> application so I guess it's possible.
> >>>>
> >>>> Sigh, I'm striking out every which way.
> >>>>
> >>>> No offense (since this is lttng list - please don't flame me ... I
> >>>> want/need lttng), but I think I'm going to try just straight kprobes
> >>>> and uprobes and see if trace compass can show those traces in an
> >>>> attempt to get "something/anything" working.
> >>>
> >>> If you attach to an ELF symbol (function), then there is no USDT in
> >>> play, so it should not be related to the issue you have.
> >>
> >> That is what I was thinking which is why I wanted to try it.
> >>
> >>>
> >>> But if your functions happen to be inlined, then there will be nothing
> >>> to attach to. Perhaps this is what happens there ?
> >>
> >> I don't see any evidence of anything being inlined in this module.  I
> >> grepped the code to verify.
> >>
> >> Back to being stumped/stuck.
> >
> > I can do trace-cmd stuff and it works.  The hello world above works so
> > I don't "think" this is a problem but again in full disclosure I'll
> > mention/ask about it.
> >
> > Does any of the lttng tools/libs depend on kernel headers?  I ask
> > because old yocto (Dunfell) built lttng package against a 4.something
> > kernel and we're running a 5.10.69 kernel that lttng modules were
> > added to it with the "builtin" script and built that way.
> >
> > Should probably have yocto build the local kernel too, but kernel is
> > being built stand alone due to vendor stuff that hasn't been mainlined
> > yet.
> >
> > I'm running out of things to think about that could be the issue.
>
> If lttng-modules can trace your smaller test application through
> uprobes, then the problem is likely elsewhere.
>
> Only lttng-modules has dependencies on kernel headers. lttng-tools/ust
> don't depend on kernel headers.
>
> Mathieu

Ok, that's what I thought but to cover all bases just wanted to make sure.

B