[lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Mathieu Desnoyers]

On 2023-05-18 15:20, Brian Hutchinson wrote:
> On Thu, May 18, 2023 at 3:07 PM Brian Hutchinson <b.hutchman at gmail.com> wrote:
>>
>> On Thu, May 18, 2023 at 3:03 PM Mathieu Desnoyers
>> <mathieu.desnoyers at efficios.com> wrote:
>>>
>>> On 2023-05-18 14:58, Brian Hutchinson wrote:
>>>> On Thu, May 18, 2023 at 11:00 AM Brian Hutchinson <b.hutchman at gmail.com> wrote:
>>>>>
>>>>> On Thu, May 18, 2023 at 10:45 AM Mathieu Desnoyers
>>>>> <mathieu.desnoyers at efficios.com> wrote:
>>>>>>
>>>>>> On 2023-05-18 10:10, Brian Hutchinson wrote:
>>>>>> [...]
>>>>>>> I updated my hello world to have a function I'd like to use the
>>>>>>> --userspace-probe method on with the very original name of
>>>>>>> 'probe_function':
>>>>>>>
>>>>>>> #include <stdio.h>
>>>>>>> #include <lttng/tracef.h>
>>>>>>>
>>>>>>> void probe_function(int i);
>>>>>>>
>>>>>>> int main(int argc, char *argv[])
>>>>>>> {
>>>>>>>       unsigned int i;
>>>>>>>       puts("Hello, World!\nPress Enter to continue...");
>>>>>>>       /*
>>>>>>>        * The following getchar() call only exists for the purpose of this
>>>>>>>        * demonstration, to pause the application in order for you to have
>>>>>>>        * time to list its tracepoints. You don't need it otherwise.
>>>>>>>        */
>>>>>>>       getchar();
>>>>>>>
>>>>>>>       lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
>>>>>>>       printf("Number %d, string %s", 23, "hi there!");
>>>>>>>
>>>>>>>       for (i = 0; i < argc; i++) {
>>>>>>>           lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
>>>>>>>           printf("Number %d, argv %s", i, argv[i]);
>>>>>>>       }
>>>>>>>
>>>>>>>       puts("Quitting now!");
>>>>>>>
>>>>>>>       probe_function(i);
>>>>>>>
>>>>>>>       return 0;
>>>>>>> }
>>>>>>>
>>>>>>> void probe_function(int i) {
>>>>>>>
>>>>>>>       lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
>>>>>>>       printf("Number %d, string %s", i * i, "i^2");
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> ... and I get the same error as before when I try to enable the probe:
>>>>>>> # lttng enable-event --kernel
>>>>>>> --userspace-probe=/usr/local/bin/hello:probe_function
>>>>>>> Error: Missing event name(s).
>>>>>>
>>>>>> As the error states, you are missing the event name. See
>>>>>>
>>>>>> man 1 lttng-enable-event
>>>>>>
>>>>>>           lttng [GENERAL OPTIONS] enable-event --kernel
>>>>>>                 [--probe=SOURCE | --function=SOURCE | --syscall |
>>>>>>                  --userspace-probe=SOURCE]
>>>>>>                 [--filter=EXPR] [--session=SESSION]
>>>>>>                 [--channel=CHANNEL] EVENT[,EVENT]...
>>>>>>
>>>>>>
>>>>>> You will want something like:
>>>>>>
>>>>>> lttng enable-event --kernel --userspace-probe=/usr/local/bin/hello:probe_function my_probe_function
>>>>>>
>>>>>> Where "my_probe_function" is the event name that will appear in the collected traces.
>>>>>
>>>>> Wow!  I must not have woken up this morning ha, ha.  Thanks for that!
>>>>> The event is enabled now.  Hope to actually get tracing data now.
>>>>
>>>> Well, I guess we just have the app that thwarts all attempts at tracing.
>>>>
>>>> I did a dynamic probe on several functions that should be getting
>>>> called like crazy and again I get no tracing data.
>>>>
>>>> Tried it with my hello world example above after Mathieu set me
>>>> straight on the event syntax and it works.
>>>>
>>>> I saw this comment in the documentation "As of this version, only USDT
>>>> probes that are not surrounded by a reference counter (semaphore) are
>>>> supported."
>>>>
>>>> I don't know that I can say that this function I'm probing isn't
>>>> "surrounded" by a reference counter, it's in a large multi-threaded
>>>> application so I guess it's possible.
>>>>
>>>> Sigh, I'm striking out every which way.
>>>>
>>>> No offense (since this is lttng list - please don't flame me ... I
>>>> want/need lttng), but I think I'm going to try just straight kprobes
>>>> and uprobes and see if trace compass can show those traces in an
>>>> attempt to get "something/anything" working.
>>>
>>> If you attach to an ELF symbol (function), then there is no USDT in
>>> play, so it should not be related to the issue you have.
>>
>> That is what I was thinking which is why I wanted to try it.
>>
>>>
>>> But if your functions happen to be inlined, then there will be nothing
>>> to attach to. Perhaps this is what happens there ?
>>
>> I don't see any evidence of anything being inlined in this module.  I
>> grepped the code to verify.
>>
>> Back to being stumped/stuck.
> 
> I can do trace-cmd stuff and it works.  The hello world above works so
> I don't "think" this is a problem but again in full disclosure I'll
> mention/ask about it.
> 
> Does any of the lttng tools/libs depend on kernel headers?  I ask
> because old yocto (Dunfell) built lttng package against a 4.something
> kernel and we're running a 5.10.69 kernel that lttng modules were
> added to it with the "builtin" script and built that way.
> 
> Should probably have yocto build the local kernel too, but kernel is
> being built stand alone due to vendor stuff that hasn't been mainlined
> yet.
> 
> I'm running out of things to think about that could be the issue.

If lttng-modules can trace your smaller test application through 
uprobes, then the problem is likely elsewhere.

Only lttng-modules has dependencies on kernel headers. lttng-tools/ust 
don't depend on kernel headers.

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com