From kl_sadi at esi.dz Thu Jan 2 17:14:16 2025 From: kl_sadi at esi.dz (Lina SADI) Date: Thu, 2 Jan 2025 23:14:16 +0100 Subject: Tracing Ceph with LTTng-UST issue Message-ID: Hello, We are writing to you regarding an issue we encountered while attempting to trace Ceph with LTTng-UST. Below are the steps we have followed so far: 1. *Compiling Ceph with LTTng Support*: - We modified the debian/rules file to enable LTTng support using the following flags: -DWITH_LTTNG=ON and -DWITH_OSD_INSTRUMENT_FUNCTIONS=ON. - The compilation process took approximately 7 hours on a virtual machine running Debian 12 with a kernel version 6.1.0-27-amd64, 8 GB of RAM, and 2 CPU cores. - The build steps can be found in our GitHub repository. 2. *Deploying a Ceph Cluster*: - We deployed the cluster manually (1MON,1MGR,2OSDs). Please find the script here . 3. *Attempting to Trace with LTTng-UST*: - We tried listing the tracepoints using the following command : # lttng list -u - This command did not return any Ceph-related tracepoints: UST events: ------------- NONE 4. *Preloading liblttng-ust-fork.so*: - We attempted to preload liblttng-ust-fork.so before starting the Ceph daemons (MON, OSD, MGR) as follows: # systemctl edit ceph-mon at debian - We added the following configuration: [Service] Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/liblttng-ust-fork.so - After restarting the daemons, we executed lttng list -u, which displayed tracepoints related to the libraries (lttng_ust_lib:*), trace logs (lttng_ust_tracelog:*), and state dump events ( lttng_ust_statedump:*), instead of the expected Ceph-related tracepoints. *Expected Result*: UST events: ------------- PID: 100859 - Name: /path/to/ceph-osd pg:queue_op (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) osd:do_osd_op_post (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) osd:do_osd_op_pre_unknown (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) osd:do_osd_op_pre_copy_from (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) osd:do_osd_op_pre_copy_get (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) ... *Actual Result*: UST events: ------------- lttng_ust_lib:* lttng_ust_tracelog:* lttng_ust_statedump:* We would greatly appreciate any help you could provide to resolve this problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kstewart at efficios.com Mon Jan 6 11:33:09 2025 From: kstewart at efficios.com (Kienan Stewart) Date: Mon, 6 Jan 2025 11:33:09 -0500 Subject: Tracing Ceph with LTTng-UST issue In-Reply-To: References: Message-ID: <023def9c-ff2a-421c-8e89-b6036fb14668@efficios.com> Hi Lina, I have a couple of questions about your tracing setup. On 1/2/25 5:14 PM, Lina SADI via lttng-dev wrote: > Hello, > > We are writing to you regarding an issue we encountered while attempting to > trace Ceph with LTTng-UST. Below are the steps we have followed so far: > > 1. > > *Compiling Ceph with LTTng Support*: > - We modified the debian/rules file to enable LTTng support using the > following flags: -DWITH_LTTNG=ON and > -DWITH_OSD_INSTRUMENT_FUNCTIONS=ON. > - The compilation process took approximately 7 hours on a virtual > machine running Debian 12 with a kernel version 6.1.0-27-amd64, 8 GB of > RAM, and 2 CPU cores. > - The build steps can be found in our GitHub > > repository. > 2. > > *Deploying a Ceph Cluster*: > - We deployed the cluster manually (1MON,1MGR,2OSDs). Please find the > script here > > . With which user a running lttng-sessiond? When you run `lttng list -u`, what user are you using? Is that user a member of the `tracing` group? If you are running lttng-sessiond with verbose logs and/or verbose consumers - do you see the traced application(s) connecting? > 3. > > *Attempting to Trace with LTTng-UST*: > - We tried listing the tracepoints using the following command : > > # lttng list -u > > - This command did not return any Ceph-related tracepoints: > > UST events: > ------------- > NONE > > 4. > > *Preloading liblttng-ust-fork.so*: > - We attempted to preload liblttng-ust-fork.so before starting the Ceph > daemons (MON, OSD, MGR) as follows: > > # systemctl edit ceph-mon at debian > > - We added the following configuration: > > [Service] > Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/liblttng-ust-fork.so > > > Does adding `LTTNG_UST_DEBUG=1` to the environment variables add information to the service logs that are relevant? > - After restarting the daemons, we executed lttng list -u, which > displayed tracepoints related to the libraries (lttng_ust_lib:*), > trace logs (lttng_ust_tracelog:*), and state dump events ( > lttng_ust_statedump:*), instead of the expected Ceph-related > tracepoints. > > *Expected Result*: > > UST events: > ------------- > PID: 100859 - Name: /path/to/ceph-osd > pg:queue_op (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) > osd:do_osd_op_post (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) > osd:do_osd_op_pre_unknown (loglevel: TRACE_DEBUG_LINE (13)) > (type: tracepoint) > osd:do_osd_op_pre_copy_from (loglevel: TRACE_DEBUG_LINE (13)) > (type: tracepoint) > osd:do_osd_op_pre_copy_get (loglevel: TRACE_DEBUG_LINE (13)) > (type: tracepoint) > ... > > > *Actual Result*: > > UST events: > ------------- > lttng_ust_lib:* > lttng_ust_tracelog:* > lttng_ust_statedump:* > > > We would greatly appreciate any help you could provide to resolve this problem. > thanks, kienan From martin at sr-research.com Wed Jan 15 09:23:00 2025 From: martin at sr-research.com (Martin Hicks) Date: Wed, 15 Jan 2025 09:23:00 -0500 Subject: [PATCH] lttng-modules fix for RHEL 9.5 kernels Message-ID: <20250115142300.vlupsqd2lr5bipfo@martin-pc> This fixes the asoc_snd_*, kfree_skb and udp related tracepoints for the RHEL 9.5 distro kernel. In all cases they've updated to the format of newer kernels. Signed-off-by: Martin Hicks --- include/instrumentation/events/asoc.h | 8 ++++++-- include/instrumentation/events/skb.h | 4 +++- include/instrumentation/events/udp.h | 4 +++- 3 files changed, 12 insertions(+), 4 deletions(-) diff --git a/include/instrumentation/events/asoc.h b/include/instrumentation/events/asoc.h index f6b2c76..31bedfe 100644 --- a/include/instrumentation/events/asoc.h +++ b/include/instrumentation/events/asoc.h @@ -120,7 +120,9 @@ LTTNG_TRACEPOINT_EVENT_INSTANCE_MAP(asoc_snd_soc_preg, snd_soc_preg_read, ) #endif -#if (LTTNG_LINUX_VERSION_CODE >= LTTNG_KERNEL_VERSION(6,9,0)) +#if (LTTNG_LINUX_VERSION_CODE >= LTTNG_KERNEL_VERSION(6,9,0) || \ + LTTNG_RHEL_KERNEL_RANGE(5,14,0,503,21,1, 5,15,0,0,0,0)) + LTTNG_TRACEPOINT_EVENT_CLASS(asoc_snd_soc_dapm_context, TP_PROTO(struct snd_soc_dapm_context *dapm, int val), @@ -187,7 +189,9 @@ LTTNG_TRACEPOINT_EVENT_INSTANCE_MAP(asoc_snd_soc_card, snd_soc_bias_level_done, ) #endif -#if (LTTNG_LINUX_VERSION_CODE >= LTTNG_KERNEL_VERSION(6,9,0)) +#if (LTTNG_LINUX_VERSION_CODE >= LTTNG_KERNEL_VERSION(6,9,0) || \ + LTTNG_RHEL_KERNEL_RANGE(5,14,0,503,21,1, 5,15,0,0,0,0)) + LTTNG_TRACEPOINT_EVENT_CLASS(asoc_snd_soc_dapm_basic, TP_PROTO(struct snd_soc_card *card, int event), diff --git a/include/instrumentation/events/skb.h b/include/instrumentation/events/skb.h index 5556ab6..becfd80 100644 --- a/include/instrumentation/events/skb.h +++ b/include/instrumentation/events/skb.h @@ -40,7 +40,9 @@ LTTNG_TRACEPOINT_ENUM(skb_drop_reason, ) #endif -#if (LTTNG_LINUX_VERSION_CODE >= LTTNG_KERNEL_VERSION(6,11,0)) +#if (LTTNG_LINUX_VERSION_CODE >= LTTNG_KERNEL_VERSION(6,11,0) || \ + LTTNG_RHEL_KERNEL_RANGE(5,14,0,503,21,1, 5,15,0,0,0,0)) + LTTNG_TRACEPOINT_EVENT_MAP(kfree_skb, skb_kfree, diff --git a/include/instrumentation/events/udp.h b/include/instrumentation/events/udp.h index 0a94b46..c228ef6 100644 --- a/include/instrumentation/events/udp.h +++ b/include/instrumentation/events/udp.h @@ -9,7 +9,9 @@ #include #include -#if (LTTNG_LINUX_VERSION_CODE >= LTTNG_KERNEL_VERSION(6,10,0)) +#if (LTTNG_LINUX_VERSION_CODE >= LTTNG_KERNEL_VERSION(6,10,0) || \ + LTTNG_RHEL_KERNEL_RANGE(5,14,0,503,21,1, 5,15,0,0,0,0)) + LTTNG_TRACEPOINT_ENUM(lttng_sk_family, TP_ENUM_VALUES( ctf_enum_value("AF_INET", AF_INET) -- 2.43.5 -- Martin Hicks SR Research Ltd. Fast, Accurate, Reliable Eye Tracking