[lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Mathieu Desnoyers]

On 2023-05-17 12:37, Brian Hutchinson wrote:
> On Wed, May 17, 2023 at 12:08 PM Mathieu Desnoyers
> <mathieu.desnoyers at efficios.com> wrote:
>>
>> On 2023-05-16 22:11, Brian Hutchinson via lttng-dev wrote:
>>> Hi,
>>>
>>> I'm trying to figure out how to use uprobes with lttng.
>>>
>>> I can't use a normal uprobe for a line number just using the address I
>>> want to probe obtained from objdump?  As in:
>>>
>>> echo 'p /usr/local/bin/my_app:0x2c3a8' >>
>>> /sys/kernel/debug/tracing/uprobe_events
>>>
>>> ... which isn't a function entry, it's just a line of code I want to probe on.
>>>
>>> This link says it has to be elf or sdt:
>>> https://lttng.org/man/1/lttng-enable-event/v2.11/#doc-opt--userspace-probe
>>>
>>> So can I not probe on just a line of code by specifying an address???
>>>
>>> It doesn't look like these methods above will do what I'm wanting to
>>> do.  I've tried to find examples of using enable-event --kernel
>>> --userspace-probe= but there doesn't appear to be many.
>>>
>>
>> There are examples here:
>>
>> https://lttng.org/docs/v2.13/#doc-enabling-disabling-events
>>
>> Indeed inserting a lttng-modules uprobe within functions is not
>> supported at the moment, mainly because we prefer to err towards safety
>> and don't have the validation in place to prevent corrupting the
>> program's instructions if an end user would try to insert a uprobe at an
>> address which is not an instruction boundary.
> 
> Hmm, was really hoping to be able to do dynamic tracing without having
> to modify code.

uprobes with the proper validations about instruction boundaries would 
eventually provide this. Another approach we want to invest time in is 
to integrate libpatch from Olivier Dion into lttng-ust. This would 
provide dynamic instrumentation with the performance of a purely 
userspace tracer.

But those are all things that were never prioritized by any of our 
customers, so they progress at a "back burner" pace.

> 
> I guess if I add a function call to a debug statement or something at
> the point I want to probe then I could use the elf example.

Yes.

> 
>>
>> So we only support inserting uprobe on functions and SDT probes at
>> the moment.
> 
> I've heard of system tap but never used it.  Will have to look into that.
> 
> I really want to get lttng-ust working but I'm getting pushback on the
> time I'm spending trying to get it to work ... and would really like
> to demonstrate something (was hoping kernel events and uprobes)
> quickly to an audience that knows nothing about lttng or full stack
> tracing to gain "buy in" for the effort.

Understood.

The main thing we are missing to help you on the UST front is a console 
log of the _application_ with LTTNG_UST_DEBUG=1. I suspect it is not 
collected in your tests.

Thanks,

Mathieu


> 
> You know, those pesky things called schedules.
> 
> Thanks,
> 
> Brian
> 
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>> Thanks,
>>>
>>> Brian
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev at lists.lttng.org
>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> https://www.efficios.com
>>

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com