<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Hi Zhenyu,<br></div><div><br data-mce-bogus="1"></div><div>Can you try this fix please ?<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>https://review.lttng.org/c/lttng-ust/+/7530<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>And let me know how it goes.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Thanks,<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Mathieu<br data-mce-bogus="1"></div><div><br></div><div><span id="zwchr" data-marker="__DIVIDER__">----- On Mar 9, 2022, at 11:37 AM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:<br></span></div><div data-marker="__QUOTED_TEXT__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:12pt;color:#000000"><div>When this happpens, is the process holding a single (or very few) shm file references, or references to many<br></div><div>shm files ?<br></div><br><div>I wonder if you end up in a scenario where an application very frequently performs exec(), and therefore<br></div><div>sometimes the exec() will happen in the window between the unix socket file descriptor reception and<br></div><div>call to fcntl FD_CLOEXEC.<br></div><br><div>Thanks,<br></div><br><div>Mathieu<br></div><br><div><span id="zwchr">----- On Mar 8, 2022, at 8:29 PM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:<br></span></div><div><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:'helvetica' , 'arial' , sans-serif;font-size:12pt"><div class="__aliyun_email_body_block"><div><span style="font-family:'tahoma' , 'arial' , 'stheiti' , 'simsun';font-size:14px;color:#000000">Thanks a lot for reply. I do not reply it in bug tracker since I have not gotten a reliable way to reproduce the leak case. </span></div><blockquote style="margin-right:0;margin-top:0;margin-bottom:0"><div><span style="font-family:'tahoma' , 'arial' , 'stheiti' , 'simsun';font-size:14px;color:#000000">------------------------------------------------------------------</span></div><div><span style="font-family:'tahoma' , 'arial' , 'stheiti' , 'simsun';font-size:14px;color:#000000">发件人：Mathieu Desnoyers <mathieu.desnoyers@efficios.com></span></div><div><span style="font-family:'tahoma' , 'arial' , 'stheiti' , 'simsun';font-size:14px;color:#000000">发送时间：2022年3月8日(星期二) 23:26</span></div><div><span style="font-family:'tahoma' , 'arial' , 'stheiti' , 'simsun';font-size:14px;color:#000000">收件人：zhenyu.ren <zhenyu.ren@aliyun.com></span></div><div><span style="font-family:'tahoma' , 'arial' , 'stheiti' , 'simsun';font-size:14px;color:#000000">抄 送：Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>; lttng-dev <lttng-dev@lists.lttng.org></span></div><div><span style="font-family:'tahoma' , 'arial' , 'stheiti' , 'simsun';font-size:14px;color:#000000">主 题：Re: [lttng-dev] 回复： 回复： 回复： shm leak in traced application?</span></div><div><span style="font-family:'tahoma' , 'arial' , 'stheiti' , 'simsun';font-size:14px;color:#000000"><br></span></div><br><br>----- On Mar 8, 2022, at 12:18 AM, lttng-dev lttng-dev@lists.lttng.org wrote:<br><br>> Hi,<br>> In shm_object_table_append_shm()/alloc_shm()， why not calling FD_CLOEXEC fcntl()<br>> to shmfds? I guess this omission leads to shm fds leak.<br><br>Those file descriptors are created when received by ustcomm_recv_fds_unix_sock, and<br>immediately after creation they are set as FD_CLOEXEC.<br><br>We should continue this discussion in the bug tracker as suggested by Jonathan.<br>It would greatly help if you can provide a small reproducer.<br><br>Thanks,<br><br>Mathieu<br><br><br>> Thanks<br>> zhenyu.ren<br><br>>> ------------------------------------------------------------------<br>>> 发件人：Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com><br>>> 发送时间：2022年2月25日(星期五) 22:31<br>>> 收件人：zhenyu.ren <zhenyu.ren@aliyun.com><br>>> 抄 送：lttng-dev <lttng-dev@lists.lttng.org><br>>> 主 题：Re: [lttng-dev] 回复： 回复： shm leak in traced application?<br><br>>> Hi zhenyu.ren,<br><br>>> Please open a bug on our bug tracker and provide a reproducer against the latest<br>>> stable version (2.13.x).<br><br>>> https://bugs.lttng.org/<br><br>>> Please follow the guidelines: https://bugs.lttng.org/#Bug-reporting-guidelines<br><br>>> Cheers<br><br>>> On Fri, Feb 25, 2022 at 12:47:34PM +0800, zhenyu.ren via lttng-dev wrote:<br>>> > Hi, lttng-dev team<br>>>> When lttng-sessiond exits, the ust applications should call<br>>>> lttng_ust_objd_table_owner_cleanup() and clean up all shm resource(unmap and<br>>>> close). Howerver I do find that the ust applications keep opening "all" of the<br>>> > shm fds("/dev/shm/ust-shm-consumer-81132 (deleted)") and do NOT free shm.<br>>>> If we run lttng-sessiond again, ust applications can get a new piece of shm and<br>>>> a new list of shm fds so double shm usages. Then if we kill lttng-sessiond,<br>>>> what the mostlikely happened is ust applications close the new list of shm fds<br>>>> and free new shm resource but keeping old shm still. In other word, we can not<br>>> > free this piece of shm unless we killing ust applications!!!<br>>>> So Is there any possilbe that ust applications failed calling<br>>>> lttng_ust_objd_table_owner_cleanup()? Do you have ever see this problem? Do you<br>>>> have any advice to free the shm without killling ust applications(I tried to<br>>> > dig into kernel shm_open and /dev/shm, but not found any ideas)?<br><br>>> > Thanks in advance<br>>> > zhenyu.ren<br><br><br><br>>> > ------------------------------------------------------------------<br>>> > 发件人：zhenyu.ren via lttng-dev <lttng-dev@lists.lttng.org><br>>> > 发送时间：2022年2月23日(星期三) 23:09<br>>> > 收件人：lttng-dev <lttng-dev@lists.lttng.org><br>>> > 主 题：[lttng-dev] 回复： shm leak in traced application?<br><br>>>> >"I found these items also exist in a traced application which is a long-time<br>>> > >running daemon"<br>>> > Even if lttng-sessiond has been killed!!<br><br>>> > Thanks<br>>> > zhenyu.ren<br>>> > ------------------------------------------------------------------<br>>> > 发件人：zhenyu.ren via lttng-dev <lttng-dev@lists.lttng.org><br>>> > 发送时间：2022年2月23日(星期三) 22:44<br>>> > 收件人：lttng-dev <lttng-dev@lists.lttng.org><br>>> > 主 题：[lttng-dev] shm leak in traced application?<br><br>>> > Hi,<br>>>> There are many items such as "/dev/shm/ust-shm-consumer-81132 (deleted)" exist<br>>>> in lttng-sessiond fd spaces. I know it is the result of shm_open() and<br>>> > shm_unlnik() in create_posix_shm().<br>>>> However, today, I found these items also exist in a traced application which is<br>>>> a long-time running daemon. The most important thing I found is that there<br>>> > seems no reliable way to release share memory.<br>>>> I tried to kill lttng-sessiond but not always release share memory. Sometimes I<br>>>> need to kill the traced application to free share memory....But it is not a<br>>> > good idea to kill these applications.<br>>> > My questions are:<br>>>> 1. Is there any way to release share memory without killing any traced<br>>> > application?<br>>>> 2. Is it normal that many items such as "/dev/shm/ust-shm-consumer-81132<br>>> > (deleted)" exist in the traced application?<br><br>>> > Thanks<br>>> > zhenyu.ren<br><br><br><br>>> > _______________________________________________<br>>> > lttng-dev mailing list<br>>> > lttng-dev@lists.lttng.org<br>>> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev<br><br>>> --<br>>> Jonathan Rajotte-Julien<br>>> EfficiOS<br>> _______________________________________________<br>> lttng-dev mailing list<br>> lttng-dev@lists.lttng.org<br>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev<br>-- <br>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com</blockquote></div><br></blockquote></div><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com</div></div><br></blockquote></div><div><br></div><div data-marker="__SIG_POST__">-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com</div></div></body></html>