<div dir="ltr">Hi,<div><br></div><div>It does look like the same problem. I would recommend upgrading and seeing if the issue goes away.</div><div><br></div><div>Thanks,</div><div>Nithya</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 9 November 2017 at 15:39, Yao Guotao <span dir="ltr"><<a href="mailto:yaoguo_tao@163.com" target="_blank">yaoguo_tao@163.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div>Thank you very much for your reply.</div><div><br></div><div>I think I found the key to the problem. But, I'd like you to confirm whether this is the issue.</div><div>My gluster version is 3.8.9, And I found there is a bug in it, and fixed in 3.8.11:</div><div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1431592" target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=1431592</a><br></div><div><br></div><div>A large amount of memory allocated for agf_common_mt_strdup. Everythig else seem to be all right.<br></div><div><br></div><div>My cluster:</div><div><br></div><div>Yesterday Afternoon：（I'm sorry I forgot the specific time.）<br></div><div>[features/locks.www-volume-<wbr>locks - usage-type gf_common_mt_strdup memusage]<br>size=1941483443<br>num_allocs=617382139<br>max_size=1941483443<br>max_num_allocs=617382139<br>total_allocs=661873332<br></div><div><br></div><div>TIme: 2017.11.9 17:15 (Today)</div><div>[features/locks.www-volume-<wbr>locks - usage-type gf_common_mt_strdup memusage]<br>size=792538295<br>num_allocs=752904534<br>max_size=792538295<br>max_num_allocs=752904534<br>total_allocs=800889589<br></div><div><br></div><div>The statedump files are in the attachment.</div><div><br></div><div>Thanks again.<br></div><div><div class="h5"><br><br><div style="zoom:1"></div><div id="m_-7566191955498001367divNeteaseMailCard"></div><br>在 2017-11-09 17:22:03，"Nithya Balachandran" <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>> 写道：<br> <blockquote id="m_-7566191955498001367isReplyContent" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 8 November 2017 at 17:16, Yao Guotao <span dir="ltr"><<a href="mailto:yaoguo_tao@163.com" target="_blank">yaoguo_tao@163.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:Arial"><div>Hi all,</div><div>I'm glad to add glusterfs community.</div><div><br></div><div>I have a glusterfs cluster:</div><div>Nodes: 4 <br></div><div>System: Centos7.1</div><div>Glusterfs: 3.8.9</div><div>Each Node:</div><div>CPU: 48 core</div><div>Mem: 128GB</div><div>Disk: 1*4T</div><div><br></div><div>There is one Distributed Replicated volume. There are ~160 k8s pods as clients connecting to glusterfs. But, the memory of glusterfsd process is too high, gradually increase to 100G every node.</div><div>Then, I reboot the glusterfsd process. But the memory increase during <span class="m_-7566191955498001367gmail-m_-7334872074255202815tran" style="box-sizing:border-box;color:rgb(95,98,102);font-family:Arial,"Microsoft YaHei",微软雅黑,宋体,"Malgun Gothic",Meiryo,sans-serif;font-size:13px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:100;letter-spacing:normal;line-height:19px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(249,251,252)">approximate</span><span style="color:rgb(95,98,102);font-family:Arial,"Microsoft YaHei",微软雅黑,宋体,"Malgun Gothic",Meiryo,sans-serif;font-size:13px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:100;letter-spacing:normal;line-height:19px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(249,251,252);float:none;display:inline"><span class="m_-7566191955498001367gmail-m_-7334872074255202815Apple-converted-space"> </span></span><span class="m_-7566191955498001367gmail-m_-7334872074255202815tran" style="box-sizing:border-box;color:rgb(95,98,102);font-family:Arial,"Microsoft YaHei",微软雅黑,宋体,"Malgun Gothic",Meiryo,sans-serif;font-size:13px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:100;letter-spacing:normal;line-height:19px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(249,251,252)">a</span><span style="color:rgb(95,98,102);font-family:Arial,"Microsoft YaHei",微软雅黑,宋体,"Malgun Gothic",Meiryo,sans-serif;font-size:13px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:100;letter-spacing:normal;line-height:19px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(249,251,252);float:none;display:inline"><span class="m_-7566191955498001367gmail-m_-7334872074255202815Apple-converted-space"> </span>week.</span></div><div>How can I debug the problem?</div><div><br></div></div></blockquote>Hi,<div><br></div><div>Please take statedumps at intervals (a minimum of 2 at intervals of an hour) of a brick process for which you see the memory increasing and send them to us. [1] describes how to take statedumps.</div><div><br></div><div>Regards,</div><div>Nithya</div><div><br></div><div>[1] <a href="http://docs.gluster.org/en/latest/Troubleshooting/statedump/" target="_blank">http://docs.gluster.org/<wbr>en/latest/Troubleshooting/<wbr>statedump/ </a></div><div><br></div><div><a href="http://docs.gluster.org/en/latest/Troubleshooting/statedump/" target="_blank"></a> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:Arial"><div></div><div>Thanks.<br></div></div><br><br><span title="neteasefooter"><p> </p></span><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></blockquote></div><br></div></div>
</blockquote></div></div></div><br><br><span title="neteasefooter"><p> </p></span></blockquote></div><br></div>