<div class="__aliyun_email_body_block"><div  style="clear:both;"><span  style="font-family:Tahoma,Arial,STHeiti,SimSun;font-size:14.0px;color:#000000;">></span><span  style="color:#000000;font-family:arial,helvetica,sans-serif;font-size:16.0px;">When this happpens, is the process holding a single (or very few) shm file references, or references to many </span><span  style="color:#000000;font-family:arial,helvetica,sans-serif;font-size:16.0px;">shm files ?</span></div><div  style="clear:both;"><span  style="color:#000000;font-family:arial,helvetica,sans-serif;font-size:16.0px;"><br ></span></div><div  style="clear:both;"><span  style="color:#000000;font-family:arial,helvetica,sans-serif;"><span  style="font-size:16.0px;">It is holding "all" of shm files' reference , neither a single one nor some few ones.</span></span></div><div  style="clear:both;"><span  style="color:#000000;font-family:arial,helvetica,sans-serif;"><span  style="font-size:16.0px;"><br ></span></span></div><div  style="clear:both;"><span  style="color:#000000;font-family:arial,helvetica,sans-serif;"><span  style="font-size:16.0px;">In fact, yesterday, I tried to fix it as the following and it seems work.</span></span></div><div  style="clear:both;"><span  style="color:#000000;font-family:arial,helvetica,sans-serif;"><span  style="font-size:16.0px;"><br ></span></span></div><div  style="clear:both;"><p  class="p1" style="margin:.0px;font:400 12.0px Menlo;color:#fd8a35;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;"><b >--- a/lttng-ust/libringbuffer/shm.c</b></span></p><p  class="p1" style="margin:.0px;font:400 12.0px Menlo;color:#fd8a35;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;"><b >+++ b/lttng-ust/libringbuffer/shm.c</b></span></p><p  class="p2" style="margin:.0px;font:400 12.0px Menlo;color:#7ddad0;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">@@ -32,7 +32,6 @@</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;"> #include <lttng/align.h></span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;"> #include <limits.h></span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;"> #include <helper.h></span></p><p  class="p4" style="margin:.0px;font:400 12.0px Menlo;color:#fc5870;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">-</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;"> /*</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">  * Ensure we have the required amount of space available by writing 0</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">  * into the entire buffer. Not doing so can trigger SIGBUS when going</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s2" style="font-variant-ligatures:no-common-ligatures;color:#7ddad0;">@@ -122,6 +121,12 @@</span><span  class="s1" style="font-variant-ligatures:no-common-ligatures;"> struct shm_object *_shm_object_table_alloc_shm(struct shm_object_table *table,</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">        /* create shm */</span></p><p  class="p5" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;"> </span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">        shmfd = stream_fd;</span></p><p  class="p6" style="margin:.0px;font:400 12.0px Menlo;color:#84de79;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">+    if (shmfd >= 0) {</span></p><p  class="p6" style="margin:.0px;font:400 12.0px Menlo;color:#84de79;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">+</span><span  class="s3" style="font-variant-ligatures:no-common-ligatures;color:#ffffff;">       </span><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">    ret = fcntl(shmfd, F_SETFD, FD_CLOEXEC);</span></p><p  class="p6" style="margin:.0px;font:400 12.0px Menlo;color:#84de79;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">+</span><span  class="s3" style="font-variant-ligatures:no-common-ligatures;color:#ffffff;">       </span><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">    if (ret < 0) {</span></p><p  class="p6" style="margin:.0px;font:400 12.0px Menlo;color:#84de79;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">+</span><span  class="s3" style="font-variant-ligatures:no-common-ligatures;color:#ffffff;">       </span><span  class="s4" style="font-variant-ligatures:no-common-ligatures;color:#ffffff;background-color:#c9173d;">    </span><span  class="s3" style="font-variant-ligatures:no-common-ligatures;color:#ffffff;">    </span><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">PERROR("fcntl shmfd FD_CLOEXEC");</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s5" style="font-variant-ligatures:no-common-ligatures;color:#84de79;">+</span><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">       </span><span  class="s5" style="font-variant-ligatures:no-common-ligatures;color:#84de79;">    }</span></p><p  class="p6" style="margin:.0px;font:400 12.0px Menlo;color:#84de79;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">+    }</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">        ret = zero_file(shmfd, memory_map_size);</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">        if (ret) {</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">                PERROR("zero_file");</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s2" style="font-variant-ligatures:no-common-ligatures;color:#7ddad0;">@@ -272,15 +277,22 @@</span><span  class="s1" style="font-variant-ligatures:no-common-ligatures;"> struct shm_object *shm_object_table_append_shm(struct shm_object_table *table,</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">        obj->shm_fd = shm_fd;</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">        obj->shm_fd_ownership = 1;</span></p><p  class="p5" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;"> </span></p><p  class="p6" style="margin:.0px;font:400 12.0px Menlo;color:#84de79;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">+    if (shm_fd >= 0) {</span></p><p  class="p6" style="margin:.0px;font:400 12.0px Menlo;color:#84de79;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">+</span><span  class="s3" style="font-variant-ligatures:no-common-ligatures;color:#ffffff;">       </span><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">    ret = fcntl(shm_fd, F_SETFD, FD_CLOEXEC);</span></p><p  class="p6" style="margin:.0px;font:400 12.0px Menlo;color:#84de79;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">+</span><span  class="s3" style="font-variant-ligatures:no-common-ligatures;color:#ffffff;">       </span><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">    if (ret < 0) {</span></p><p  class="p6" style="margin:.0px;font:400 12.0px Menlo;color:#84de79;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">+</span><span  class="s3" style="font-variant-ligatures:no-common-ligatures;color:#ffffff;">       </span><span  class="s4" style="font-variant-ligatures:no-common-ligatures;color:#ffffff;background-color:#c9173d;">    </span><span  class="s3" style="font-variant-ligatures:no-common-ligatures;color:#ffffff;">    </span><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">PERROR("fcntl shmfd FD_CLOEXEC");</span></p><p  class="p6" style="margin:.0px;font:400 12.0px Menlo;color:#84de79;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">+</span><span  class="s3" style="font-variant-ligatures:no-common-ligatures;color:#ffffff;">       </span><span  class="s4" style="font-variant-ligatures:no-common-ligatures;color:#ffffff;background-color:#c9173d;">    </span><span  class="s3" style="font-variant-ligatures:no-common-ligatures;color:#ffffff;">    </span><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">//goto error_fcntl;</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s5" style="font-variant-ligatures:no-common-ligatures;color:#84de79;">+</span><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">       </span><span  class="s5" style="font-variant-ligatures:no-common-ligatures;color:#84de79;">    }</span></p><p  class="p6" style="margin:.0px;font:400 12.0px Menlo;color:#84de79;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">+    }</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">        ret = fcntl(obj->wait_fd[1], F_SETFD, FD_CLOEXEC);</span></p><p  class="p3" style="margin:.0px;font:400 12.0px Menlo;color:#ffffff;background-color:#61181b;"><span  class="s1" style="font-variant-ligatures:no-common-ligatures;">        if (ret < 0) {</span></p></div><div  style="clear:both;"><span  style="color:#000000;font-family:arial,helvetica,sans-serif;"><span  style="font-size:16.0px;"><br ></span></span></div><div  style="clear:both;"><span  style="color:#000000;font-family:arial,helvetica,sans-serif;"><span  style="font-size:16.0px;">    As it shows, wait_fd[1] has been set FD_CLOEXEC by fcntl() but not shm_fd. Why your patch do with wait_fd but not shm_fd? As far as I know, wait_fd is just a pipe and it seems not related to shm resource.</span></span></div><div  style="clear:both;"><br ></div><div  style="clear:both;"><span  style="color:#000000;font-family:arial,helvetica,sans-serif;"><span  style="font-size:16.0px;"><br ></span></span></div><div  style="clear:both;"><span  style="color:#000000;font-family:arial,helvetica,sans-serif;"><span  style="font-size:16.0px;"><br ></span></span></div><div  style="clear:both;"><span  style="color:#000000;font-family:arial,helvetica,sans-serif;"><span  style="font-size:16.0px;"><br ></span></span></div><div  style="clear:both;"><span  style="color:#000000;font-family:arial,helvetica,sans-serif;"><span  style="font-size:16.0px;"><br ></span></span></div><div  style="clear:both;"><span  style="color:#000000;font-family:arial,helvetica,sans-serif;"><span  style="font-size:16.0px;"><br ></span></span></div><blockquote  style="margin-right:0;margin-top:0;margin-bottom:0;"><div  style="clear:both;"><span  style="font-family:Tahoma,Arial,STHeiti,SimSun;font-size:14.0px;color:#000000;">------------------------------------------------------------------</span></div><div  style="clear:both;"><span  style="font-family:Tahoma,Arial,STHeiti,SimSun;font-size:14.0px;color:#000000;">发件人:Mathieu Desnoyers <mathieu.desnoyers@efficios.com></span></div><div  style="clear:both;"><span  style="font-family:Tahoma,Arial,STHeiti,SimSun;font-size:14.0px;color:#000000;">发送时间:2022年3月10日(星期四) 00:46</span></div><div  style="clear:both;"><span  style="font-family:Tahoma,Arial,STHeiti,SimSun;font-size:14.0px;color:#000000;">收件人:zhenyu.ren <zhenyu.ren@aliyun.com></span></div><div  style="clear:both;"><span  style="font-family:Tahoma,Arial,STHeiti,SimSun;font-size:14.0px;color:#000000;">抄 送:Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>; lttng-dev <lttng-dev@lists.lttng.org></span></div><div  style="clear:both;"><span  style="font-family:Tahoma,Arial,STHeiti,SimSun;font-size:14.0px;color:#000000;">主 题:Re: 回复:[lttng-dev] 回复: 回复: 回复: shm leak in traced application?</span></div><div  style="clear:both;"><span  style="font-family:Tahoma,Arial,STHeiti,SimSun;font-size:14.0px;color:#000000;"><br /></span></div><div  class=" __aliyun_node_has_color" style="font-family:arial,helvetica,sans-serif;font-size:12.0pt;color:#000000;"><div >When this happpens, is the process holding a single (or very few) shm file references, or references to many<br ></div><div >shm files ?<br ></div><div ><br ></div><div >I wonder if you end up in a scenario where an application very frequently performs exec(), and therefore<br ></div><div >sometimes the exec() will happen in the window between the unix socket file descriptor reception and<br ></div><div >call to fcntl FD_CLOEXEC.<br ></div><div ><br ></div><div >Thanks,<br ></div><div ><br ></div><div >Mathieu<br ></div><div ><br ></div><div ><span  id="zwchr">----- On Mar 8, 2022, at 8:29 PM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:<br ></span></div><div ><div ><span  class=" __aliyun_node_has_color" style="font-family:tahoma,arial,stheiti,simsun;font-size:14.0px;color:#000000;">Thanks a  lot for reply. I do not reply it in bug tracker since I have not gotten a reliable way to reproduce the leak case. </span></div><div ><span  class=" __aliyun_node_has_color" style="font-family:tahoma,arial,stheiti,simsun;font-size:14.0px;color:#000000;">------------------------------------------------------------------</span></div><div ><span  class=" __aliyun_node_has_color" style="font-family:tahoma,arial,stheiti,simsun;font-size:14.0px;color:#000000;">发件人:Mathieu Desnoyers <mathieu.desnoyers@efficios.com></span></div><div ><span  class=" __aliyun_node_has_color" style="font-family:tahoma,arial,stheiti,simsun;font-size:14.0px;color:#000000;">发送时间:2022年3月8日(星期二) 23:26</span></div><div ><span  class=" __aliyun_node_has_color" style="font-family:tahoma,arial,stheiti,simsun;font-size:14.0px;color:#000000;">收件人:zhenyu.ren <zhenyu.ren@aliyun.com></span></div><div ><span  class=" __aliyun_node_has_color" style="font-family:tahoma,arial,stheiti,simsun;font-size:14.0px;color:#000000;">抄 送:Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>; lttng-dev <lttng-dev@lists.lttng.org></span></div><div ><span  class=" __aliyun_node_has_color" style="font-family:tahoma,arial,stheiti,simsun;font-size:14.0px;color:#000000;">主 题:Re: [lttng-dev] 回复: 回复: 回复: shm leak in traced application?</span></div><div ><span  class=" __aliyun_node_has_color" style="font-family:tahoma,arial,stheiti,simsun;font-size:14.0px;color:#000000;"><br ></span></div><br ><br >----- On Mar 8, 2022, at 12:18 AM, lttng-dev lttng-dev@lists.lttng.org wrote:<br ><br >> Hi,<br >> In shm_object_table_append_shm()/alloc_shm(), why not calling FD_CLOEXEC fcntl()<br >> to shmfds? I guess this omission leads to shm fds leak.<br ><br >Those file descriptors are created when received by ustcomm_recv_fds_unix_sock, and<br >immediately after creation they are set as FD_CLOEXEC.<br ><br >We should continue this discussion in the bug tracker as suggested by Jonathan.<br >It would greatly help if you can provide a small reproducer.<br ><br >Thanks,<br ><br >Mathieu<br ><br ><br >> Thanks<br >> zhenyu.ren<br ><br >>> ------------------------------------------------------------------<br >>> 发件人:Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com><br >>> 发送时间:2022年2月25日(星期五) 22:31<br >>> 收件人:zhenyu.ren <zhenyu.ren@aliyun.com><br >>> 抄 送:lttng-dev <lttng-dev@lists.lttng.org><br >>> 主 题:Re: [lttng-dev] 回复: 回复: shm leak in traced application?<br ><br >>> Hi zhenyu.ren,<br ><br >>> Please open a bug on our bug tracker and provide a reproducer against the latest<br >>> stable version (2.13.x).<br ><br >>> <a  href="https://bugs.lttng.org/" target="_blank">https://bugs.lttng.org/</a><br ><br >>> Please follow the guidelines: <a  href="https://bugs.lttng.org/#Bug-reporting-guidelines" target="_blank">https://bugs.lttng.org/#Bug-reporting-guidelines</a><br ><br >>> Cheers<br ><br >>> On Fri, Feb 25, 2022 at 12:47:34PM +0800, zhenyu.ren via lttng-dev wrote:<br >>> > Hi, lttng-dev team<br >>>> When lttng-sessiond exits, the ust applications should call<br >>>> lttng_ust_objd_table_owner_cleanup() and clean up all shm resource(unmap and<br >>>> close). Howerver I do find that the ust applications keep opening "all" of the<br >>> > shm fds("/dev/shm/ust-shm-consumer-81132 (deleted)") and do NOT free shm.<br >>>> If we run lttng-sessiond again, ust applications can get a new piece of shm and<br >>>> a new list of shm fds so double shm usages. Then if we kill lttng-sessiond,<br >>>> what the mostlikely happened is ust applications close the new list of shm fds<br >>>> and free new shm resource but keeping old shm still. In other word, we can not<br >>> > free this piece of shm unless we killing ust applications!!!<br >>>> So Is there any possilbe that ust applications failed calling<br >>>> lttng_ust_objd_table_owner_cleanup()? Do you have ever see this problem? Do you<br >>>> have any advice to free the shm without killling ust applications(I tried to<br >>> > dig into kernel shm_open and /dev/shm, but not found any ideas)?<br ><br >>> > Thanks in advance<br >>> > zhenyu.ren<br ><br ><br ><br >>> > ------------------------------------------------------------------<br >>> > 发件人:zhenyu.ren via lttng-dev <lttng-dev@lists.lttng.org><br >>> > 发送时间:2022年2月23日(星期三) 23:09<br >>> > 收件人:lttng-dev <lttng-dev@lists.lttng.org><br >>> > 主 题:[lttng-dev] 回复: shm leak in traced application?<br ><br >>>> >"I found these items also exist in a traced application which is a long-time<br >>> > >running daemon"<br >>> > Even if lttng-sessiond has been killed!!<br ><br >>> > Thanks<br >>> > zhenyu.ren<br >>> > ------------------------------------------------------------------<br >>> > 发件人:zhenyu.ren via lttng-dev <lttng-dev@lists.lttng.org><br >>> > 发送时间:2022年2月23日(星期三) 22:44<br >>> > 收件人:lttng-dev <lttng-dev@lists.lttng.org><br >>> > 主 题:[lttng-dev] shm leak in traced application?<br ><br >>> > Hi,<br >>>> There are many items such as "/dev/shm/ust-shm-consumer-81132 (deleted)" exist<br >>>> in lttng-sessiond fd spaces. I know it is the result of shm_open() and<br >>> > shm_unlnik() in create_posix_shm().<br >>>> However, today, I found these items also exist in a traced application which is<br >>>> a long-time running daemon. The most important thing I found is that there<br >>> > seems no reliable way to release share memory.<br >>>> I tried to kill lttng-sessiond but not always release share memory. Sometimes I<br >>>> need to kill the traced application to free share memory....But it is not a<br >>> > good idea to kill these applications.<br >>> > My questions are:<br >>>> 1. Is there any way to release share memory without killing any traced<br >>> > application?<br >>>> 2. Is it normal that many items such as "/dev/shm/ust-shm-consumer-81132<br >>> > (deleted)" exist in the traced application?<br ><br >>> > Thanks<br >>> > zhenyu.ren<br ><br ><br ><br >>> > _______________________________________________<br >>> > lttng-dev mailing list<br >>> > lttng-dev@lists.lttng.org<br >>> > <a  href="https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" target="_blank">https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><br ><br >>> --<br >>> Jonathan Rajotte-Julien<br >>> EfficiOS<br >> _______________________________________________<br >> lttng-dev mailing list<br >> lttng-dev@lists.lttng.org<br >> <a  href="https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" target="_blank">https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><br >-- <br >Mathieu Desnoyers<br >EfficiOS Inc.<br ><a  href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br ></div><div ><br ></div><div >-- <br ></div><div >Mathieu Desnoyers<br >EfficiOS Inc.<br ><a  href="http://www.efficios.com" target="_blank">http://www.efficios.com</a></div></div></blockquote></div>