<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 9, 2018 at 10:43 AM, Raghavendra Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Thu, Aug 9, 2018 at 10:36 AM, huting3 <span dir="ltr"><<a href="mailto:huting3@corp.netease.com" target="_blank">huting3@corp.netease.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="font-family:Helvetica,Helvetica,微软雅黑,宋体;line-height:1.6">
<div>
<div><div>grep count will ouput nothing, so I grep size, the results are:</div><div><br></div><div>$ grep itable glusterdump.109182.dump.153373<wbr>0324 | grep lru | grep size</div><div>xlator.mount.fuse.itable.lru_s<wbr>ize=191726</div></div></div></div></div></blockquote><div><br></div></span><div>Kernel is holding too many inodes in its cache. What's the data set like? Do you've too many directories? How many files do you have?</div></div></div></div></blockquote><div><br></div><div>Just to be sure, can you give the output of following cmd too:</div><div><br></div><div># grep itable <dump> | grep lru | wc -l</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="h5"><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:Helvetica,Helvetica,微软雅黑,宋体;line-height:1.6"><div><div><br></div><div><div>$ grep itable glusterdump.109182.dump.153373<wbr>0324 | grep active | grep size</div><div>xlator.mount.fuse.itable.activ<wbr>e_size=17</div><div><br></div><div><br></div><a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png" style="font-family:Helvetica,"Microsoft Yahei",微软雅黑;background-color:rgb(255,255,255);max-width:400px;padding-top:15px;padding-bottom:10px;text-decoration:none;outline:none;display:inline!important" target="_blank"><table style="width:100%;max-width:100%;table-layout:fixed;border-collapse:collapse;color:#9b9ea1;font-size:14px;line-height:1.3" cellpadding="0"><tbody style="font-family:'PingFang SC','Hiragino Sans GB','WenQuanYi Micro Hei','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1',verdana!important;word-wrap:break-word;word-break:break-all"><tr><td style="box-sizing:border-box;width:38px" width="38"><img style="vertical-align:middle;width:38px;height:38px;border-radius:50%" src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png" width="38" height="38">
</td><td style="padding-right:30px;padding-left:10px;color:rgb(49,53,59)"><div style="font-size:16px;font-weight:bold;width:100%;white-space:nowrap;overflow:hidden;text-overflow:ellipsis">huting3</div></td><td style="text-align:right;width:72px" width="72"><img style="width:72px;height:20px" src="https://mail-online.nosdn.127.net/qiyelogo/209a2912f40f6683af56bb7caff1cb54.png" width="72" height="20">
</td></tr><tr width="100%" style="width:100%"><td colspan="3" style="padding-top:10px;width:100%"><div style="width:100%;word-wrap:break-word;word-break:break-all">huting3@corp.netease.com</div></td></tr></tbody></table></a></div><div id="m_1321315120604727315m_5141060108186289289ntes-pcmail-signature" style="font-family:'Helvetica','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1'"><div style="font-size:14px;padding:0;margin:0">
</div>
<div style="font-size:12px;color:#b5b9bd">
<span>签名由</span>
<a style="text-decoration:none;color:#4196ff;padding:0 5px" href="http://mail.163.com/dashi/" target="_blank">网易邮箱大师</a>
<span>定制</span>
</div>
</div><br>
</div><div><div class="m_1321315120604727315h5"><div class="m_1321315120604727315m_5141060108186289289J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'' Helvetica ',' Microsoft Yahei ', ' 微软雅黑 ''">
<div style="font-size:12px;line-height:1.6;word-break:break-all;margin-left:10px;margin-right:10px">On <span class="m_1321315120604727315m_5141060108186289289mail-date">08/9/2018 12:36</span>，<a class="m_1321315120604727315m_5141060108186289289mail-to" style="text-decoration:none;color:#2a83f2" href="mailto:rgowdapp@redhat.com" target="_blank">Raghavendra Gowdappa<rgowdapp@redhat.com></a> wrote： </div>
</div>
<blockquote id="m_1321315120604727315m_5141060108186289289ntes-pcmail-quote" style="margin:0;padding:0;font-size:14px;font-family:'' Helvetica ',' Microsoft Yahei ', ' 微软雅黑 ''">
<div dir="ltr"><div><div>Can you get the output of following cmds?<br><br></div># grep itable <statedump> | grep lru | grep count<br><br></div># grep itable <statedump> | grep active | grep count<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 9, 2018 at 9:25 AM, huting3 <span dir="ltr"><<a href="mailto:huting3@corp.netease.com" target="_blank">huting3@corp.netease.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="font-family:Helvetica,Helvetica,微软雅黑,宋体;line-height:1.6">
<div>
<div><span>Yes, I got the dump file and found there are many huge num_allocs just like following:</span></div><div><span><br></span></div><div><span>I found memusage of 4 variable types are extreamly huge.</span></div><div><span><br></span></div><div><span> </span>[protocol/client.gv0-client-0 - usage-type gf_common_mt_char memusage]</div><div>size=47202352</div><div>num_allocs=2030212</div><div>max_size=47203074</div><div>max_num_allocs=2030235</div><div>total_allocs=26892201</div><div><br></div><div><div>[protocol/client.gv0-client-0 - usage-type gf_common_mt_memdup memusage]</div><div>size=24362448</div><div>num_allocs=2030204</div><div>max_size=24367560</div><div>max_num_allocs=2030226</div><div>total_allocs=17830860</div><div><br></div><div>[mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]</div><div>size=2497947552</div><div>num_allocs=4578229</div><div>max_size=2459135680</div><div>max_num_allocs=7123206</div><div>total_allocs=41635232</div><div><br></div><div>[mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]</div><div>size=4038730976</div><div>num_allocs=1</div><div>max_size=4294962264</div><div>max_num_allocs=37</div><div>total_allocs=150049981<a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png" style="font-family:Helvetica,"Microsoft Yahei",微软雅黑;background-color:rgb(255,255,255);max-width:400px;padding-top:15px;padding-bottom:10px;text-decoration:none;outline:none;display:inline!important" target="_blank"></a></div><a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png" style="font-family:Helvetica,"Microsoft Yahei",微软雅黑;background-color:rgb(255,255,255);max-width:400px;padding-top:15px;padding-bottom:10px;text-decoration:none;outline:none;display:inline!important" target="_blank"><p><br></p><p><br></p><table style="width:100%;max-width:100%;table-layout:fixed;border-collapse:collapse;color:#9b9ea1;font-size:14px;line-height:1.3" cellpadding="0"><tbody style="font-family:'PingFang SC','Hiragino Sans GB','WenQuanYi Micro Hei','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1',verdana!important;word-wrap:break-word;word-break:break-all"><tr><td style="box-sizing:border-box;width:38px" width="38"><img style="vertical-align:middle;width:38px;height:38px;border-radius:50%" src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png" width="38" height="38">
</td><td style="padding-right:30px;padding-left:10px;color:rgb(49,53,59)"><div style="font-size:16px;font-weight:bold;width:100%;white-space:nowrap;overflow:hidden;text-overflow:ellipsis">huting3</div></td><td style="text-align:right;width:72px" width="72"><img style="width:72px;height:20px" src="https://mail-online.nosdn.127.net/qiyelogo/209a2912f40f6683af56bb7caff1cb54.png" width="72" height="20">
</td></tr><tr width="100%" style="width:100%"><td colspan="3" style="padding-top:10px;width:100%"><div style="width:100%;word-wrap:break-word;word-break:break-all">huting3@corp.netease.com</div></td></tr></tbody></table></a></div><div id="m_1321315120604727315m_5141060108186289289m_5471995425395774604ntes-pcmail-signature" style="font-family:'Helvetica','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1'"><div style="font-size:14px;padding:0;margin:0">
</div>
<div style="font-size:12px;color:#b5b9bd">
<span>签名由</span>
<a style="text-decoration:none;color:#4196ff;padding:0 5px" href="http://mail.163.com/dashi/" target="_blank">网易邮箱大师</a>
<span>定制</span>
</div>
</div><br>
</div><div><div class="m_1321315120604727315m_5141060108186289289h5"><div class="m_1321315120604727315m_5141060108186289289m_5471995425395774604J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'' Helvetica ',' Microsoft Yahei ', ' 微软雅黑 ''">
<div style="font-size:12px;line-height:1.6;word-break:break-all;margin-left:10px;margin-right:10px">On <span class="m_1321315120604727315m_5141060108186289289m_5471995425395774604mail-date">08/9/2018 11:36</span>，<a class="m_1321315120604727315m_5141060108186289289m_5471995425395774604mail-to" style="text-decoration:none;color:#2a83f2" href="mailto:rgowdapp@redhat.com" target="_blank">Raghavendra Gowdappa<rgowdapp@redhat.com></a> wrote： </div>
</div>
<blockquote id="m_1321315120604727315m_5141060108186289289m_5471995425395774604ntes-pcmail-quote" style="margin:0;padding:0;font-size:14px;font-family:'' Helvetica ',' Microsoft Yahei ', ' 微软雅黑 ''">
<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 9, 2018 at 8:55 AM, huting3 <span dir="ltr"><<a href="mailto:huting3@corp.netease.com" target="_blank">huting3@corp.netease.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="font-family:Helvetica,Helvetica,微软雅黑,宋体;line-height:1.6">
<div><span>Hi expert:</span></div><div><span><br></span></div><div><span>I meet a problem when I use glusterfs. The problem is that the fuse client consumes huge memory when write a lot of files(>million) to the gluster, at last leading to killed by OS oom. The memory the fuse process consumes can up to 100G! I wonder if there are memory leaks in the gluster fuse process, or some other causes.</span></div></div></div></blockquote><div><br></div><div>Can you get statedump of fuse process consuming huge memory?</div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:Helvetica,Helvetica,微软雅黑,宋体;line-height:1.6"><div><span><br></span></div><div>My gluster version is 3.13.2, the gluster volume info is listed as following:</div><div><br></div><div><div>Volume Name: gv0</div><div>Type: Distributed-Replicate</div><div>Volume ID: 4a6f96f8-b3fb-4550-bd19-e1a5df<wbr>fad4d0</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 19 x 3 = 57</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: dl20.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick2: dl21.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick3: dl22.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick4: dl20.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick5: dl21.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick6: dl22.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick7: dl20.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick8: dl21.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick9: dl22.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick10: dl23.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick11: dl24.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick12: dl25.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick13: dl26.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick14: dl27.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick15: dl28.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick16: dl29.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick17: dl30.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick18: dl31.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick19: dl32.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick20: dl33.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick21: dl34.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick22: dl23.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick23: dl24.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick24: dl25.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick25: dl26.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick26: dl27.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick27: dl28.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick28: dl29.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick29: dl30.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick30: dl31.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick31: dl32.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick32: dl33.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick33: dl34.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick34: dl23.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick35: dl24.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick36: dl25.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick37: dl26.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick38: dl27.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick39: dl28.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick40: dl29.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick41: dl30.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick42: dl31.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick43: dl32.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick44: dl33.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick45: dl34.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick46: dl0.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick47: dl1.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick48: dl2.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick49: dl3.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick50: dl5.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick51: dl6.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick52: dl9.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick53: dl10.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick54: dl11.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick55: dl12.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick56: dl13.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick57: dl14.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Options Reconfigured:</div><div>performance.cache-size: 10GB</div><div>performance.parallel-readdir: on</div><div>performance.readdir-ahead: on</div><div>network.inode-lru-limit: 200000</div><div>performance.md-cache-timeout: 600</div><div>performance.cache-invalidation<wbr>: on</div><div>performance.stat-prefetch: on</div><div>features.cache-invalidation-ti<wbr>meout: 600</div><div>features.cache-invalidation: on</div><div>features.inode-quota: off</div><div>features.quota: off</div><div>cluster.quorum-reads: on</div><div>cluster.quorum-count: 2</div><div>cluster.quorum-type: fixed</div><div>transport.address-family: inet</div><div>nfs.disable: on</div><div>performance.client-io-threads: off</div><div>cluster.server-quorum-ratio: 51%</div><div><br></div></div><div><br></div>
<div id="m_1321315120604727315m_5141060108186289289m_5471995425395774604m_-6772762122606358812ntes-pcmail-signature" style="font-family:'Helvetica','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1'">
<div style="font-size:14px;padding:0;margin:0">
<div style="border-bottom:1px solid #ccc;display:inline-block">
<a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png" style="display:block;background:#fff;max-width:400px;padding:15px 0 10px 0;text-decoration:none;outline:none" target="_blank">
<table style="width:100%;max-width:100%;table-layout:fixed;border-collapse:collapse;color:#9b9ea1;font-size:14px;line-height:1.3" cellpadding="0">
<tbody style="font-family:'PingFang SC','Hiragino Sans GB','WenQuanYi Micro Hei','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1',verdana!important;word-wrap:break-word;word-break:break-all">
<tr>
<td style="padding:0;box-sizing:border-box;width:38px" width="38">
<img style="vertical-align:middle;width:38px;height:38px;border-radius:50%" src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png" width="38" height="38">
</td>
<td style="padding:0 30px 0 10px;color:#31353b">
<div style="font-size:16px;font-weight:bold;width:100%;white-space:nowrap;overflow:hidden;text-overflow:ellipsis">huting3</div>
</td>
<td style="text-align:right;width:72px" width="72">
<img style="width:72px;height:20px" src="https://mail-online.nosdn.127.net/qiyelogo/209a2912f40f6683af56bb7caff1cb54.png" width="72" height="20">
</td>
</tr>
<tr width="100%" style="font-size:14px!important;width:100%">
<td colspan="3" style="padding:10px 0 0 0;font-size:14px!important;width:100%">
<div style="width:100%;font-size:14px!important;word-wrap:break-word;word-break:break-all">huting3@corp.netease.com</div>
</td>
</tr>
</tbody>
</table>
</a>
</div>
</div>
<div style="font-size:12px;color:#b5b9bd">
<span>签名由</span>
<a style="text-decoration:none;color:#4196ff;padding:0 5px" href="http://mail.163.com/dashi/" target="_blank">网易邮箱大师</a>
<span>定制</span>
</div>
</div><br>
</div>
</div>
<br>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mail<wbr>man/listinfo/gluster-devel</a><br></blockquote></div><br></div></div>
</blockquote>
</div></div></div>
</div>
</blockquote></div><br></div>
</blockquote>
</div></div></div>
</div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div>