lttng-dev Digest, Vol 203, Issue 7
Kienan Stewart
kstewart at efficios.com
Thu Mar 20 16:51:42 EDT 2025
Hi Lakshya,
I did some digging around. What you are seeing is the result of the
switching to MAP_POPULATE by default in LTTng-UST 2.12[1] in commit
4d4838b ("Use MAP_POPULATE to reduce pagefault when available").
The purpose of this change is to avoid taking page faults which tracing,
reducing first-event in a page latency.
In the master branch, this feature has been made configurable for users
who don't want to pre-populate the pages and would rather take page
faults while tracing[2].
Here is an example from LTTng master with map populate per possible CPU:
```
export LTTNG_UST_MAP_POPULATE_POLICY=cpu_possible
# Create session, channels, start tracing, and run test app
# top -n1 -b | grep -E '(MiB|COMMAND|lttng)'
MiB Mem : 32768.0 total, 21883.7 free, 1456.0 used, 9428.3
buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 31312.0 avail
Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
301117 debian 20 0 880176 2760 2760 S 0.0 0.0 0:00.04
lttng-sessiond
301118 debian 20 0 43856 1376 1376 S 0.0 0.0 0:00.01
lttng-runas
301133 debian 20 0 718616 263456 263456 S 0.0 0.8 0:00.17
lttng-consumerd
301135 debian 20 0 6996 0 0 S 0.0 0.0 0:00.05
lttng-runas
# cat /proc/$(pgrep lttng-sessiond)/statm
lttng-sessiond: 220044 690 690 345 0 29900 0
# pmap $(pgrep lttng-sessiond) | grep total
total 880176K
# smem -P lttng-sessiond
PID User Command Swap USS PSS
RSS
301118 debian lttng-sessiond --daemonize 0 344 881
2236
301117 debian lttng-sessiond --daemonize 0 5676 6683
9276
301201 debian /usr/bin/python3 /usr/bin/s 0 8636 10086
12936
# /proc/PID/statm for lttng-consumerd
lttng-consumerd: 1749 0 0 129 0 130 0
# pmap lttng-consumerd-pid | grep total
total kB 6996 1700 472
# smem -P lttng-consumerd
PID User Command Swap USS PSS
RSS
301135 debian lttng-consumerd -u --consu 0 280 563
1700
301211 debian /usr/bin/python3 /usr/bin/s 0 10048 11501
14404
301133 debian lttng-consumerd -u --consu 0 262376 263177
265480
# smem -m | grep -i ust
/dev/shm/lttng-ust-wait-8-1000 1 4 4
/dev/shm/shm-ust-consumer-301133 1 260756 260756
```
When using the none policy:
```
# export LTTNG_UST_MAP_POPULATE_POLICY=none
# as above
Running test app UID 0
procs -----------memory---------- ---swap-- -----io---- -system--
------cpu-----
r b swpd free buff cache si so bi bo in cs us sy
id wa st
1 0 0 21875 0 9434 0 0 39 636 1105 2496 0 1
99 0 0
MiB Mem : 32768.0 total, 21875.0 free, 1458.2 used, 9434.7
buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 31309.8 avail
Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
301616 debian 20 0 880176 2756 2756 S 0.0 0.0 0:00.04
lttng-sessiond
301617 debian 20 0 43856 1392 1392 S 0.0 0.0 0:00.01
lttng-runas
301632 debian 20 0 718612 5416 5416 S 0.0 0.0 0:00.17
lttng-consumerd
301634 debian 20 0 6992 0 0 S 0.0 0.0 0:00.05
lttng-runas
lttng-sessiond: 220044 689 689 345 0 29900 0
total 880176K
PID User Command Swap USS PSS
RSS
301617 debian lttng-sessiond --daemonize 0 344 862
2188
301616 debian lttng-sessiond --daemonize 0 5784 6759
9328
301700 debian /usr/bin/python3 /usr/bin/s 0 8632 10079
12928
lttng-consumerd: 1748 0 0 129 0 129 0
total kB 6992 1580 468
PID User Command Swap USS PSS
RSS
301634 debian lttng-consumerd -u --consu 0 276 536
1580
301632 debian lttng-consumerd -u --consu 0 5672 6433
8652
301710 debian /usr/bin/python3 /usr/bin/s 0 9996 11449
14328
/dev/shm/lttng-ust-wait-8-1000 1 4 4
/dev/shm/shm-ust-consumer-301632 1 4048 4048
```
thanks,
kienan
[1]:
https://github.com/lttng/lttng-ust/commit/4d4838bad480d48424bddc686f5ad0089e28ac94
[2]:
https://github.com/lttng/lttng-ust/commit/97572c0438845cee953ebd3e39615f78bfa405a7
On 3/17/25 2:29 AM, Gour DEV wrote:
> Hi, Kienan
>
> Sorry for the late reply.
>
> Looks like in buster the memory is allocated by lttng-consumerd reserved
>
> I buster, the rss is less than the VIRT
> root at localhost:~# COLUMNS=500 top -b -n 1 | grep lttng
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4095 root 20 0 1003188 31256 4660 S 0.0 0.1 0:03.81
> lttng-sessiond
> 4096 root 20 0 44260 796 0 S 0.0 0.0 0:00.01
> lttng-runas
> 4440 root 20 0 5236020 10224 8756 S 0.0 0.0 2:56.25
> lttng-consumerd -- here the VIRT is much more higher than RSS
> 4443 root 20 0 48048 540 0 S 0.0 0.0 0:00.12
> lttng-runas
>
>
>
> In bookworm the VIRT and RES are nearly the same only.
> root at edgecore-40XKE-j2-101-32:~# COLUMNS=500 top -b -n 1 | grep lttng
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4382 root 20 0 1098824 42600 8436 S 0.0 0.1 0:08.87
> lttng-sessiond
> 4403 root 20 0 48928 2116 996 S 0.0 0.0 0:00.00
> lttng-runas
> 5171 root 20 0 9879764 8.9g 8.9g S 0.0 28.7 108:23.53
> lttng-consumerd -- here the VRIT is nearly equal to RSS
> 5173 root 20 0 3680 1028 680 S 0.0 0.0 0:00.88
> lttng-runas
>
>
> Looks like lttng consumerd is allocating and reserving those pages, when
> any instrumented application starts.
>
> I am attaching the lttng status output in the mail, please do tell me if
> you need any more information regarding this.
>
>
> These is how we used to create the lttng channels and enable event which is
> same in both buster and bookworm, (number of channels might differ)
>
> def enable_channel(channels, session, subbuf_size, subbuf_num):
> for c in channels:
> call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size',
> str(subbuf_size), '--num-subbuf', str(subbuf_num),],
> stdout=devnull, stderr=subprocess.STDOUT)
>
>
> def enable_events(traces, session):
> for t in traces:
> if 'log-level-only' in t:
> log_opt = '--loglevel-only=' + t['log-level-only']
> elif 'log-level' in t:
> log_opt = '--loglevel=' + t['log-level']
> else:
> log_opt = ''
>
> else:
> call(['lttng', 'enable-event', '-u', t['name'], '-c', t['channel'],
> '-s', session], stdout=devnull, stderr=subprocess.STDOUT)
>
>
> Thank You.
> Lakshya
>
>
>
>
> On Wed, Mar 12, 2025 at 8:06 PM <lttng-dev-request at lists.lttng.org> wrote:
>
>> Send lttng-dev mailing list submissions to
>> lttng-dev at lists.lttng.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>> or, via email, send a message with subject or body 'help' to
>> lttng-dev-request at lists.lttng.org
>>
>> You can reach the person managing the list at
>> lttng-dev-owner at lists.lttng.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of lttng-dev digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: Memory Consumption High After Upgrading to 2.13 from 2.10
>> (Kienan Stewart)
>> 2. Re: Memory Consumption High After Upgrading to 2.13 from 2.10
>> (Gour DEV)
>> 3. Re: Memory Consumption High After Upgrading to 2.13 from 2.10
>> (Kienan Stewart)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Tue, 11 Mar 2025 14:55:21 -0400
>> From: Kienan Stewart <kstewart at efficios.com>
>> To: Gour DEV <lakshyagour10 at gmail.com>, lttng-dev at lists.lttng.org
>> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10
>> Message-ID: <38dab5ef-f106-4e57-9e36-b4b30015c019 at efficios.com>
>> Content-Type: text/plain; charset=UTF-8; format=flowed
>>
>> Hi Lakshya,
>>
>> On 3/11/25 12:25 PM, Gour DEV wrote:
>> > Hi, Kienan
>> >
>> > here is the requested output
>> >
>> > root at localhost:~# top -b -n 1 | grep lttng
>> > 4841 root 20 0 11.5g 11.0g 11.0g S 5.9 35.4 8:39.93
>> > lttng-c+
>> > 4824 root 20 0 1098824 26456 5380 S 0.0 0.1 0:07.25
>> > lttng-s+
>> > 4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00
>> > lttng-r+
>> > 4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23
>>
>> This top output for `localhost` seems very different than the output for
>> `localhost` in your previous message.
>>
>>
>> > lttng-r+
>> > root at localhost:~# nrpco
>> > bash: nrpco: command not found
>> > root at localhost:~# nproc
>> > 16
>> > root at localhost:~# cat /sys/devices/system/cpu/possible
>> > 0-15
>> >
>>
>> You indicated the bookworm machine has 32 cores, this is showing 16. If
>> you're comparing a 16 core machine to a 32 core machine, it is very
>> normal that the memory usage is higher on the 32 core machine.
>>
>> >
>> > Most of the process are running as asorcs user but some are running
>> as root.
>>
>> So you have two users with instrumented applications.
>>
>>
>> Given the discrepancies in the information provided I'm finding it a bit
>> hard to understand what you're looking at.
>>
>>
>> In general, a channel's shared memory footprint can be estimated with[1]:
>>
>> (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
>> (nUIDs or nPIDs)
>>
>> Note that the sub-buffer sizes you are using get rounded to the nearest
>> larger power of 2. See [2].
>>
>> thanks,
>> kienan
>>
>> [1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes
>> [2]:
>> https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Wed, 12 Mar 2025 14:49:07 +0530
>> From: Gour DEV <lakshyagour10 at gmail.com>
>> To: Kienan Stewart <kstewart at efficios.com>
>> Cc: lttng-dev at lists.lttng.org
>> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10
>> Message-ID:
>> <CAE9Jrzg7qsabhPiO-0=B1DY3bVo-3FYu2tiJR2Bmb=
>> nqOHNZMw at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hi, Kienan
>>
>> I am attaching an screen recording of the behaviour I am seeing in this
>> mail. The behaviour is same irrespective of the device i use, sorry for
>> miscommunication in the npocs output (I assumed it was 32), but other than
>> that all outputs are same (except the hostname as there are multiple
>> devices with same lttng config but this memory cosumption is seen on all
>> the devices).
>>
>> I had few question
>>
>> 1. Does lltng allocated all the memory it needs and mark it as dirty in ram
>> when any process which links/uses lttng-ust runs? (here i tried with one
>> process but it is same for any of my process)
>> 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
>> (nUIDs or nPIDs)
>>
>> How do we calculate uid in the system is it all uids in the system? is it
>> equal to `cat /etc/passwd | wc -l` ?
>>
>> I will put my calculations according to the above estimate based on all the
>> channel i am creating
>>
>> (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal to
>> `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on the
>> start_lttng.py please do correct me if am wrong here.]
>>
>> But since there are only two users which uses lttng i think the correct
>> estimate would be
>> (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB
>>
>> Please do correct me If I am wrong calculations here.
>>
>> Now, there are a few things here, according to my output lttng is using 11G
>> which is much more higher than the what is configured.
>>
>> I am attaching the lttng status and the file which is uses to create the
>> lttng sessions.
>>
>>
>>
>> Thank You.
>>
>>
>>
>> https://drive.google.com/file/d/1tS_ZWEsXDpHZXfWzZHXmWcT0igiIOIaa/view?usp=sharing
>> -- recording of the behaviour which is seen
>>
>> https://drive.google.com/file/d/1PrU31oyEw1n9tKETlUtmNGO50s6ywx7p/view?usp=sharing
>> -- the file which is used to create lttng sessions
>>
>>
>> On Wed, Mar 12, 2025 at 12:25?AM Kienan Stewart <kstewart at efficios.com>
>> wrote:
>>
>>> Hi Lakshya,
>>>
>>> On 3/11/25 12:25 PM, Gour DEV wrote:
>>> > Hi, Kienan
>>> >
>>> > here is the requested output
>>> >
>>> > root at localhost:~# top -b -n 1 | grep lttng
>>> > 4841 root 20 0 11.5g 11.0g 11.0g S 5.9 35.4
>> 8:39.93
>>> > lttng-c+
>>> > 4824 root 20 0 1098824 26456 5380 S 0.0 0.1
>> 0:07.25
>>> > lttng-s+
>>> > 4825 root 20 0 48872 2188 1012 S 0.0 0.0
>> 0:00.00
>>> > lttng-r+
>>> > 4843 root 20 0 3680 1160 816 S 0.0 0.0
>> 0:00.23
>>>
>>> This top output for `localhost` seems very different than the output for
>>> `localhost` in your previous message.
>>>
>>>
>>> > lttng-r+
>>> > root at localhost:~# nrpco
>>> > bash: nrpco: command not found
>>> > root at localhost:~# nproc
>>> > 16
>>> > root at localhost:~# cat /sys/devices/system/cpu/possible
>>> > 0-15
>>> >
>>>
>>> You indicated the bookworm machine has 32 cores, this is showing 16. If
>>> you're comparing a 16 core machine to a 32 core machine, it is very
>>> normal that the memory usage is higher on the 32 core machine.
>>>
>>> >
>>> > Most of the process are running as asorcs user but some are running
>>> as root.
>>>
>>> So you have two users with instrumented applications.
>>>
>>>
>>> Given the discrepancies in the information provided I'm finding it a bit
>>> hard to understand what you're looking at.
>>>
>>>
>>> In general, a channel's shared memory footprint can be estimated with[1]:
>>>
>>> (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
>>> (nUIDs or nPIDs)
>>>
>>> Note that the sub-buffer sizes you are using get rounded to the nearest
>>> larger power of 2. See [2].
>>>
>>> thanks,
>>> kienan
>>>
>>> [1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes
>>> [2]:
>>> https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size
>>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> https://lists.lttng.org/pipermail/lttng-dev/attachments/20250312/57f240d8/attachment-0001.htm
>>>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Wed, 12 Mar 2025 10:36:28 -0400
>> From: Kienan Stewart <kstewart at efficios.com>
>> To: Gour DEV <lakshyagour10 at gmail.com>, lttng-dev at lists.lttng.org
>> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10
>> Message-ID: <0f819583-ea8e-468e-9102-e1410d886a6f at efficios.com>
>> Content-Type: text/plain; charset=UTF-8; format=flowed
>>
>> Hi Lakshya,
>>
>> On 3/12/25 5:03 AM, Gour DEV wrote:
>>> Hi, Kienan
>>>
>>> I am attaching an screen recording of the behaviour I am seeing in this
>>> mail. The behaviour is same irrespective of the device i use, sorry for
>>> miscommunication in the npocs output (I assumed it was 32), but other
>> than
>>> that all outputs are same (except the hostname as there are multiple
>>> devices with same lttng config but this memory cosumption is seen on all
>>> the devices).
>>>
>>> I had few question
>>>
>>> 1. Does lltng allocated all the memory it needs and mark it as dirty in
>> ram
>>> when any process which links/uses lttng-ust runs? (here i tried with one
>>> process but it is same for any of my process)
>>
>> I believe the shared memory for per-CPU data structures is allocated
>> when an instrumented application connects. There is no pre-allocation
>> for each possible UID on the system.
>>
>> You can run your instrumented applications with `LTTNG_UST_DEBUG=1` to
>> see when the connection happens[1].
>>
>>> 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
>>> (nUIDs or nPIDs)
>>>
>>> How do we calculate uid in the system is it all uids in the system? is it
>>> equal to `cat /etc/passwd | wc -l` ?
>>
>> nUIDs is the number of distinct UIDs running instrumented applications.
>>
>>>
>>> I will put my calculations according to the above estimate based on all
>> the
>>> channel i am creating
>>>
>>> (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal to
>>> `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on the
>>> start_lttng.py please do correct me if am wrong here.]
>>>
>>> But since there are only two users which uses lttng i think the correct
>>> estimate would be
>>> (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB
>>
>> The estimate I gave is per-channel.
>>
>> small channel: (0.015625 MiB * 4) * (16 + 1) = 1.0625 MiB per-channel
>> per-UID
>> medium channel: (0.250 MiB * 4) * (16 + 1) = 17.0 MiB per-channel per-UID
>> large channel: (4 MiB * 4) * (16 + 1) = 27 2MiB per-channel per-UID
>>
>> Now, you said you have 0 small channels, 6 medium channels, and 16 large
>> channels in your session. (Note: I see your script differs from these
>> stated channel counts).
>>
>> small: 0 * 1.0625 MiB = 0 MiB per-UID
>> medium: 6 * 17 MiB = 102 MiB per-UID
>> large: 16 * 272 MiB = 4352 MiB per-UID
>>
>> And if you're running instrumented applications with 2 users:
>>
>> small: 0 MiB * 2 = 0 MiB with 2 UIDs
>> medium: 102 MiB * 2 = 204 MiB with 2 UIDs
>> large: 4352 MiB * 2 = 8704 MiB with 2 UIDs
>>
>> Now this is just an estimation for the per-CPU ring buffers only, and
>> you numbers aren't hugely off so without analyzing your specific system
>> it doesn't seem to be that strange to me.
>>
>> If I take the number of channels I see in your script, it becomes:
>>
>> small: 0 MiB with 2 UIDs
>> medium: 136 MiB with 2 UIDs
>> large: 7616 MiB with 2 UIDs
>>
>> total: 7.57 GiB with 2 UIDs
>>
>>>
>>> Please do correct me If I am wrong calculations here.
>>>
>>> Now, there are a few things here, according to my output lttng is using
>> 11G
>>> which is much more higher than the what is configured.
>>>
>>
>> I have no idea what 'service start spyder' is doing. Maybe it's running
>> instrumented applications with an extra user that you didn't expect? I
>> can't help you with that aspect of your system.
>>
>> The above estimated 7.57 GiB with 2 UIDs would be 11.35 GiB with 3 UIDs
>> so maybe?
>>
>> I'd recommend you read your verbose sessiond log so see which
>> applications are connecting and with which UIDs.
>>
>>> I am attaching the lttng status and the file which is uses to create the
>>> lttng sessions.
>>>
>>>
>>>
>>> Thank You.
>>>
>>
>> In any case, the information you have given to date hasn't demonstrated
>> to me in a tangible manner that you are seeing a difference related to
>> the version of LTTng being used.
>>
>> thanks,
>> kienan
>>
>> [1]: https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables
>>
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev at lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>>
>> ------------------------------
>>
>> End of lttng-dev Digest, Vol 203, Issue 7
>> *****************************************
>>
>
More information about the lttng-dev
mailing list