Tracing Ceph with LTTng-UST issue

[Kienan Stewart]

Hi Lina,

I have a couple of questions about your tracing setup.

On 1/2/25 5:14 PM, Lina SADI via lttng-dev wrote:
> Hello,
> 
> We are writing to you regarding an issue we encountered while attempting to
> trace Ceph with LTTng-UST. Below are the steps we have followed so far:
> 
>     1.
> 
>     *Compiling Ceph with LTTng Support*:
>     - We modified the debian/rules file to enable LTTng support using the
>        following flags: -DWITH_LTTNG=ON and
>        -DWITH_OSD_INSTRUMENT_FUNCTIONS=ON.
>        - The compilation process took approximately 7 hours on a virtual
>        machine running Debian 12 with a kernel version 6.1.0-27-amd64, 8 GB of
>        RAM, and 2 CPU cores.
>        - The build steps can be found in our GitHub
>        <https://github.com/KedadsaChakib/ceph-configuration/blob/main/build/README.md>
>         repository.
>     2.
> 
>     *Deploying a Ceph Cluster*:
>     - We deployed the cluster manually (1MON,1MGR,2OSDs). Please find the
>        script here
>        <https://github.com/KedadsaChakib/ceph-configuration/blob/main/cluster/cluster-dep-script.sh>
>        .

With which user a running lttng-sessiond?

When you run `lttng list -u`, what user are you using?

Is that user a member of the `tracing` group?

If you are running lttng-sessiond with verbose logs and/or verbose 
consumers - do you see the traced application(s) connecting?

>     3.
> 
>     *Attempting to Trace with LTTng-UST*:
>     - We tried listing the tracepoints using the following command :
> 
>        # lttng list -u
> 
>        - This command did not return any Ceph-related tracepoints:
> 
>        UST events:
>        -------------
>        NONE
> 
>        4.
> 
>     *Preloading liblttng-ust-fork.so*:
>     - We attempted to preload liblttng-ust-fork.so before starting the Ceph
>        daemons (MON, OSD, MGR) as follows:
> 
>        # systemctl edit ceph-mon at debian
> 
>        - We added the following configuration:
> 
>        [Service]
>        Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/liblttng-ust-fork.so
> 
> 
> 

Does adding `LTTNG_UST_DEBUG=1` to the environment variables add 
information to the service logs that are relevant?


>     - After restarting the daemons, we executed lttng list -u, which
>        displayed tracepoints related to the libraries (lttng_ust_lib:*),
>        trace logs (lttng_ust_tracelog:*), and state dump events (
>        lttng_ust_statedump:*), instead of the expected Ceph-related
>        tracepoints.
> 
> *Expected Result*:
> 
>     UST events:
>     -------------
>     PID: 100859 - Name: /path/to/ceph-osd
>         pg:queue_op (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
>         osd:do_osd_op_post (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
>         osd:do_osd_op_pre_unknown (loglevel: TRACE_DEBUG_LINE (13))
> (type: tracepoint)
>         osd:do_osd_op_pre_copy_from (loglevel: TRACE_DEBUG_LINE (13))
> (type: tracepoint)
>         osd:do_osd_op_pre_copy_get (loglevel: TRACE_DEBUG_LINE (13))
> (type: tracepoint)
>         ...
> 
> 
> *Actual Result*:
> 
>     UST events:
>     -------------
>     lttng_ust_lib:*
>     lttng_ust_tracelog:*
>     lttng_ust_statedump:*
> 
> 
> We would greatly appreciate any help  you could provide to resolve this problem.
> 

thanks,
kienan