<html>
<head>
<meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
</head>
<body>
<style>
font{
line-height: 1.6;
}
ul,ol{
padding-left: 20px;
list-style-position: inside;
}
</style>
<div style = 'font-family:Helvetica,Helvetica,微软雅黑, 宋体; line-height:1.6;'>
<div>
<div><div><a href="https://drive.google.com/file/d/1ZlttTzt4E56Qtk9j7b4I9GkZC2W3mJgp/view?usp=sharing">https://drive.google.com/file/d/1ZlttTzt4E56Qtk9j7b4I9GkZC2W3mJgp/view?usp=sharing</a></div><div><br></div><div>Hi exports:</div><div> I upload the statedump file to the google drive, could you please help me find what is the cause of gluster fuse comsuming huge memory? Thank you!</div><div><br></div><div><br></div><div><br><a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png" style="font-family: Helvetica, "Microsoft Yahei", 微软雅黑; background-color: rgb(255, 255, 255); max-width: 400px; padding-top: 15px; padding-bottom: 10px; text-decoration: none; outline: none; display: inline !important;"><table cellpadding="0" style="width: 100%; max-width: 100%; table-layout: fixed; border-collapse: collapse;color: #9b9ea1;font-size: 14px;line-height:1.3;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;"><tbody style="font-family: 'PingFang SC', 'Hiragino Sans GB','WenQuanYi Micro Hei', 'Microsoft Yahei', '微软雅黑', verdana !important; word-wrap:break-word; word-break:break-all;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;"><tr><td width="38" style="box-sizing: border-box; width: 38px;"><img width="38" height="38" style="vertical-align:middle; width: 38px; height: 38px; border-radius:50%;" src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png">
</td><td style="padding-right: 30px; padding-left: 10px; color: rgb(49, 53, 59);"><div style="font-size: 16px;font-weight:bold; width:100%; white-space: nowrap; overflow:hidden;text-overflow: ellipsis;">huting3</div></td><td width="72" style="text-align:right; width: 72px;"><img width="72" height="20" style="width: 72px; height: 20px;" src="https://mail-online.nosdn.127.net/qiyelogo/209a2912f40f6683af56bb7caff1cb54.png">
</td></tr><tr width="100%" style="width: 100%;"><td colspan="3" style="padding-top: 10px; width: 100%;"><div style="width: 100%; word-wrap: break-word; word-break: break-all;">huting3@corp.netease.com</div></td></tr></tbody></table></a></div></div><div id="ntes-pcmail-signature" style="font-family:'Helvetica','Microsoft Yahei', '微软雅黑'"><div style="font-size:14px; padding: 0; margin:0;">
</div>
<div style="font-size:12px;color:#b5b9bd;">
<span>签名由</span>
<a style="text-decoration: none;color:#4196ff;padding:0 5px;" href="http://mail.163.com/dashi/">网易邮箱大师</a>
<span>定制</span>
</div>
</div><br>
</div><div class="J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:''Helvetica','Microsoft Yahei', '微软雅黑'';">
<div style="font-size:12px;line-height:1.6;word-break:break-all;margin-left:10px;margin-right:10px">On <span class="mail-date">08/9/2018 13:46</span>，<a class="mail-to" style="text-decoration:none;color:#2a83f2;" href="mailto:huting3@corp.netease.com">huting3<huting3@corp.netease.com></a> wrote： </div>
</div>
<blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0; font-size: 14px; font-family: ''Helvetica','Microsoft Yahei', '微软雅黑'';">
<style>
font{
line-height: 1.6;
}
ul,ol{
padding-left: 20px;
list-style-position: inside;
}
</style>
<div style="font-family:Helvetica,Helvetica,微软雅黑, 宋体; line-height:1.6;">
<div>
<div><span><br></span></div>
<div><span>I upload the dump file as the attchment.</span></div>
<div id="ntes-pcmail-signature" style="font-family:'Helvetica','Microsoft Yahei', '微软雅黑'">
<style type="text/css">
a#ntes-pcmail-signature-default:hover {
text-decoration: underline;
color: #3593db;
cursor: pointer;
}
</style>
<div style="font-size:14px; padding: 0; margin:0;">
<div style="border-bottom:1px solid #ccc;display:inline-block;">
<a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png" style="display:block;background:#fff; max-width: 400px; _width: 400px;padding:15px 0 10px 0;text-decoration: none; outline:none;-webkit-tap-highlight-color:transparent;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
<table cellpadding="0" style="width: 100%; max-width: 100%; table-layout: fixed; border-collapse: collapse;color: #9b9ea1;font-size: 14px;line-height:1.3;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
<tbody style="font-family: 'PingFang SC', 'Hiragino Sans GB','WenQuanYi Micro Hei', 'Microsoft Yahei', '微软雅黑', verdana !important; word-wrap:break-word; word-break:break-all;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
<tr>
<td width="38" style="padding:0; box-sizing: border-box; width: 38px;">
<img width="38" height="38" style="vertical-align:middle; width: 38px; height: 38px; border-radius:50%;" src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png">
</td>
<td style="padding: 0 30px 0 10px; color: #31353b;">
<div style="font-size: 16px;font-weight:bold; width:100%; white-space: nowrap; overflow:hidden;text-overflow: ellipsis;">huting3</div>
</td>
<td width="72" style="text-align:right; width: 72px;">
<img width="72" height="20" style="width: 72px; height: 20px;" src="https://mail-online.nosdn.127.net/qiyelogo/209a2912f40f6683af56bb7caff1cb54.png">
</td>
</tr>
<tr width="100%" style="font-size: 14px !important; width: 100%;">
<td colspan="3" style="padding:10px 0 0 0; font-size:14px !important; width: 100%;">
<div style="width: 100%;font-size: 14px !important;word-wrap:break-word;word-break:break-all;">huting3@corp.netease.com</div>
</td>
</tr>
</tbody>
</table>
</a>
</div>
</div>
<div style="font-size:12px;color:#b5b9bd;">
<span>签名由</span>
<a style="text-decoration: none;color:#4196ff;padding:0 5px;" href="http://mail.163.com/dashi/">网易邮箱大师</a>
<span>定制</span>
</div>
</div><br>
</div><div class="J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:''Helvetica','Microsoft Yahei', '微软雅黑'';">
<div style="font-size:12px;line-height:1.6;word-break:break-all;margin-left:10px;margin-right:10px">On <span class="mail-date">08/9/2018 13:30</span>，<a class="mail-to" style="text-decoration:none;color:#2a83f2;" href="mailto:huting3@corp.netease.com">huting3<huting3@corp.netease.com></a> wrote： </div>
</div>
<blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0; font-size: 14px; font-family: ''Helvetica','Microsoft Yahei', '微软雅黑'';">
<style>
font{
line-height: 1.6;
}
ul,ol{
padding-left: 20px;
list-style-position: inside;
}
</style>
<div style="font-family:Helvetica,Helvetica,微软雅黑, 宋体; line-height:1.6;">
<div>
<div>em, the data set is complicated. There are many big files as well as small files. There is about 50T data in gluster server, so I do not know how many files in the dataset exactly. Can the inode cache consume so huge memory? How can I limit the inode cache?</div><div><br></div><div>ps:</div><div><div>$ grep itable glusterdump.109182.dump.1533730324 | grep lru | wc -l</div><div>191728</div></div><div><br></div><div>When I dump the process info, the fuse process consumed about 30G memory.</div><div><br></div>
<div><span><br></span></div>
<div id="ntes-pcmail-signature" style="font-family:'Helvetica','Microsoft Yahei', '微软雅黑'">
<style type="text/css">
a#ntes-pcmail-signature-default:hover {
text-decoration: underline;
color: #3593db;
cursor: pointer;
}
</style>
<div style="font-size:14px; padding: 0; margin:0;">
<div style="border-bottom:1px solid #ccc;display:inline-block;">
<a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png" style="display:block;background:#fff; max-width: 400px; _width: 400px;padding:15px 0 10px 0;text-decoration: none; outline:none;-webkit-tap-highlight-color:transparent;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
<table cellpadding="0" style="width: 100%; max-width: 100%; table-layout: fixed; border-collapse: collapse;color: #9b9ea1;font-size: 14px;line-height:1.3;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
<tbody style="font-family: 'PingFang SC', 'Hiragino Sans GB','WenQuanYi Micro Hei', 'Microsoft Yahei', '微软雅黑', verdana !important; word-wrap:break-word; word-break:break-all;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
<tr>
<td width="38" style="padding:0; box-sizing: border-box; width: 38px;">
<img width="38" height="38" style="vertical-align:middle; width: 38px; height: 38px; border-radius:50%;" src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png">
</td>
<td style="padding: 0 30px 0 10px; color: #31353b;">
<div style="font-size: 16px;font-weight:bold; width:100%; white-space: nowrap; overflow:hidden;text-overflow: ellipsis;">huting3</div>
</td>
<td width="72" style="text-align:right; width: 72px;">
<img width="72" height="20" style="width: 72px; height: 20px;" src="https://mail-online.nosdn.127.net/qiyelogo/209a2912f40f6683af56bb7caff1cb54.png">
</td>
</tr>
<tr width="100%" style="font-size: 14px !important; width: 100%;">
<td colspan="3" style="padding:10px 0 0 0; font-size:14px !important; width: 100%;">
<div style="width: 100%;font-size: 14px !important;word-wrap:break-word;word-break:break-all;">huting3@corp.netease.com</div>
</td>
</tr>
</tbody>
</table>
</a>
</div>
</div>
<div style="font-size:12px;color:#b5b9bd;">
<span>签名由</span>
<a style="text-decoration: none;color:#4196ff;padding:0 5px;" href="http://mail.163.com/dashi/">网易邮箱大师</a>
<span>定制</span>
</div>
</div><br>
</div><div class="J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:''Helvetica','Microsoft Yahei', '微软雅黑'';">
<div style="font-size:12px;line-height:1.6;word-break:break-all;margin-left:10px;margin-right:10px">On <span class="mail-date">08/9/2018 13:13</span>，<a class="mail-to" style="text-decoration:none;color:#2a83f2;" href="mailto:rgowdapp@redhat.com">Raghavendra Gowdappa<rgowdapp@redhat.com></a> wrote： </div>
</div>
<blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0; font-size: 14px; font-family: ''Helvetica','Microsoft Yahei', '微软雅黑'';">
<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 9, 2018 at 10:36 AM, huting3 <span dir="ltr"><<a href="mailto:huting3@corp.netease.com" target="_blank">huting3@corp.netease.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="font-family:Helvetica,Helvetica,微软雅黑,宋体;line-height:1.6">
<div>
<div><div>grep count will ouput nothing, so I grep size, the results are:</div><div><br></div><div>$ grep itable glusterdump.109182.dump.<wbr>1533730324 | grep lru | grep size</div><div>xlator.mount.fuse.itable.lru_<wbr>size=191726</div></div></div></div></div></blockquote><div><br></div><div>Kernel is holding too many inodes in its cache. What's the data set like? Do you've too many directories? How many files do you have?</div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:Helvetica,Helvetica,微软雅黑,宋体;line-height:1.6"><div><div><br></div><div><div>$ grep itable glusterdump.109182.dump.<wbr>1533730324 | grep active | grep size</div><div>xlator.mount.fuse.itable.<wbr>active_size=17</div><div><br></div><div><br></div><a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png" style="font-family:Helvetica,"Microsoft Yahei",微软雅黑;background-color:rgb(255,255,255);max-width:400px;padding-top:15px;padding-bottom:10px;text-decoration:none;outline:none;display:inline!important" target="_blank"><table style="width:100%;max-width:100%;table-layout:fixed;border-collapse:collapse;color:#9b9ea1;font-size:14px;line-height:1.3" cellpadding="0"><tbody style="font-family:'PingFang SC','Hiragino Sans GB','WenQuanYi Micro Hei','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1',verdana!important;word-wrap:break-word;word-break:break-all"><tr><td style="box-sizing:border-box;width:38px" width="38"><img style="vertical-align:middle;width:38px;height:38px;border-radius:50%" src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png" width="38" height="38">
</td><td style="padding-right:30px;padding-left:10px;color:rgb(49,53,59)"><div style="font-size:16px;font-weight:bold;width:100%;white-space:nowrap;overflow:hidden;text-overflow:ellipsis">huting3</div></td><td style="text-align:right;width:72px" width="72"><img style="width:72px;height:20px" src="https://mail-online.nosdn.127.net/qiyelogo/209a2912f40f6683af56bb7caff1cb54.png" width="72" height="20">
</td></tr><tr width="100%" style="width:100%"><td colspan="3" style="padding-top:10px;width:100%"><div style="width:100%;word-wrap:break-word;word-break:break-all">huting3@corp.netease.com</div></td></tr></tbody></table></a></div><div id="m_5141060108186289289ntes-pcmail-signature" style="font-family:'Helvetica','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1'"><div style="font-size:14px;padding:0;margin:0">
</div>
<div style="font-size:12px;color:#b5b9bd">
<span>签名由</span>
<a style="text-decoration:none;color:#4196ff;padding:0 5px" href="http://mail.163.com/dashi/" target="_blank">网易邮箱大师</a>
<span>定制</span>
</div>
</div><br>
</div><div><div class="h5"><div class="m_5141060108186289289J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'' Helvetica ',' Microsoft Yahei ', ' 微软雅黑 ''">
<div style="font-size:12px;line-height:1.6;word-break:break-all;margin-left:10px;margin-right:10px">On <span class="m_5141060108186289289mail-date">08/9/2018 12:36</span>，<a class="m_5141060108186289289mail-to" style="text-decoration:none;color:#2a83f2" href="mailto:rgowdapp@redhat.com" target="_blank">Raghavendra Gowdappa<rgowdapp@redhat.com></a> wrote： </div>
</div>
<blockquote id="m_5141060108186289289ntes-pcmail-quote" style="margin:0;padding:0;font-size:14px;font-family:'' Helvetica ',' Microsoft Yahei ', ' 微软雅黑 ''">
<div dir="ltr"><div><div>Can you get the output of following cmds?<br><br></div># grep itable <statedump> | grep lru | grep count<br><br></div># grep itable <statedump> | grep active | grep count<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 9, 2018 at 9:25 AM, huting3 <span dir="ltr"><<a href="mailto:huting3@corp.netease.com" target="_blank">huting3@corp.netease.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="font-family:Helvetica,Helvetica,微软雅黑,宋体;line-height:1.6">
<div>
<div><span>Yes, I got the dump file and found there are many huge num_allocs just like following:</span></div><div><span><br></span></div><div><span>I found memusage of 4 variable types are extreamly huge.</span></div><div><span><br></span></div><div><span> </span>[protocol/client.gv0-client-0 - usage-type gf_common_mt_char memusage]</div><div>size=47202352</div><div>num_allocs=2030212</div><div>max_size=47203074</div><div>max_num_allocs=2030235</div><div>total_allocs=26892201</div><div><br></div><div><div>[protocol/client.gv0-client-0 - usage-type gf_common_mt_memdup memusage]</div><div>size=24362448</div><div>num_allocs=2030204</div><div>max_size=24367560</div><div>max_num_allocs=2030226</div><div>total_allocs=17830860</div><div><br></div><div>[mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]</div><div>size=2497947552</div><div>num_allocs=4578229</div><div>max_size=2459135680</div><div>max_num_allocs=7123206</div><div>total_allocs=41635232</div><div><br></div><div>[mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]</div><div>size=4038730976</div><div>num_allocs=1</div><div>max_size=4294962264</div><div>max_num_allocs=37</div><div>total_allocs=150049981<a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png" style="font-family:Helvetica,"Microsoft Yahei",微软雅黑;background-color:rgb(255,255,255);max-width:400px;padding-top:15px;padding-bottom:10px;text-decoration:none;outline:none;display:inline!important" target="_blank"></a></div><a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png" style="font-family:Helvetica,"Microsoft Yahei",微软雅黑;background-color:rgb(255,255,255);max-width:400px;padding-top:15px;padding-bottom:10px;text-decoration:none;outline:none;display:inline!important" target="_blank"><p><br></p><p><br></p><table style="width:100%;max-width:100%;table-layout:fixed;border-collapse:collapse;color:#9b9ea1;font-size:14px;line-height:1.3" cellpadding="0"><tbody style="font-family:'PingFang SC','Hiragino Sans GB','WenQuanYi Micro Hei','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1',verdana!important;word-wrap:break-word;word-break:break-all"><tr><td style="box-sizing:border-box;width:38px" width="38"><img style="vertical-align:middle;width:38px;height:38px;border-radius:50%" src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png" width="38" height="38">
</td><td style="padding-right:30px;padding-left:10px;color:rgb(49,53,59)"><div style="font-size:16px;font-weight:bold;width:100%;white-space:nowrap;overflow:hidden;text-overflow:ellipsis">huting3</div></td><td style="text-align:right;width:72px" width="72"><img style="width:72px;height:20px" src="https://mail-online.nosdn.127.net/qiyelogo/209a2912f40f6683af56bb7caff1cb54.png" width="72" height="20">
</td></tr><tr width="100%" style="width:100%"><td colspan="3" style="padding-top:10px;width:100%"><div style="width:100%;word-wrap:break-word;word-break:break-all">huting3@corp.netease.com</div></td></tr></tbody></table></a></div><div id="m_5141060108186289289m_5471995425395774604ntes-pcmail-signature" style="font-family:'Helvetica','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1'"><div style="font-size:14px;padding:0;margin:0">
</div>
<div style="font-size:12px;color:#b5b9bd">
<span>签名由</span>
<a style="text-decoration:none;color:#4196ff;padding:0 5px" href="http://mail.163.com/dashi/" target="_blank">网易邮箱大师</a>
<span>定制</span>
</div>
</div><br>
</div><div><div class="m_5141060108186289289h5"><div class="m_5141060108186289289m_5471995425395774604J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'' Helvetica ',' Microsoft Yahei ', ' 微软雅黑 ''">
<div style="font-size:12px;line-height:1.6;word-break:break-all;margin-left:10px;margin-right:10px">On <span class="m_5141060108186289289m_5471995425395774604mail-date">08/9/2018 11:36</span>，<a class="m_5141060108186289289m_5471995425395774604mail-to" style="text-decoration:none;color:#2a83f2" href="mailto:rgowdapp@redhat.com" target="_blank">Raghavendra Gowdappa<rgowdapp@redhat.com></a> wrote： </div>
</div>
<blockquote id="m_5141060108186289289m_5471995425395774604ntes-pcmail-quote" style="margin:0;padding:0;font-size:14px;font-family:'' Helvetica ',' Microsoft Yahei ', ' 微软雅黑 ''">
<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 9, 2018 at 8:55 AM, huting3 <span dir="ltr"><<a href="mailto:huting3@corp.netease.com" target="_blank">huting3@corp.netease.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="font-family:Helvetica,Helvetica,微软雅黑,宋体;line-height:1.6">
<div><span>Hi expert:</span></div><div><span><br></span></div><div><span>I meet a problem when I use glusterfs. The problem is that the fuse client consumes huge memory when write a lot of files(>million) to the gluster, at last leading to killed by OS oom. The memory the fuse process consumes can up to 100G! I wonder if there are memory leaks in the gluster fuse process, or some other causes.</span></div></div></div></blockquote><div><br></div><div>Can you get statedump of fuse process consuming huge memory?</div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:Helvetica,Helvetica,微软雅黑,宋体;line-height:1.6"><div><span><br></span></div><div>My gluster version is 3.13.2, the gluster volume info is listed as following:</div><div><br></div><div><div>Volume Name: gv0</div><div>Type: Distributed-Replicate</div><div>Volume ID: 4a6f96f8-b3fb-4550-bd19-e1a5df<wbr>fad4d0</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 19 x 3 = 57</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: dl20.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick2: dl21.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick3: dl22.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick4: dl20.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick5: dl21.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick6: dl22.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick7: dl20.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick8: dl21.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick9: dl22.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick10: dl23.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick11: dl24.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick12: dl25.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick13: dl26.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick14: dl27.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick15: dl28.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick16: dl29.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick17: dl30.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick18: dl31.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick19: dl32.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick20: dl33.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick21: dl34.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick22: dl23.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick23: dl24.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick24: dl25.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick25: dl26.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick26: dl27.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick27: dl28.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick28: dl29.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick29: dl30.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick30: dl31.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick31: dl32.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick32: dl33.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick33: dl34.dg.163.org:/glusterfs_bri<wbr>ck/brick2/gv0</div><div>Brick34: dl23.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick35: dl24.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick36: dl25.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick37: dl26.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick38: dl27.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick39: dl28.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick40: dl29.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick41: dl30.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick42: dl31.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick43: dl32.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick44: dl33.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick45: dl34.dg.163.org:/glusterfs_bri<wbr>ck/brick3/gv0</div><div>Brick46: dl0.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick47: dl1.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick48: dl2.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick49: dl3.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick50: dl5.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick51: dl6.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick52: dl9.dg.163.org:/glusterfs_bric<wbr>k/brick1/gv0</div><div>Brick53: dl10.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick54: dl11.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick55: dl12.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick56: dl13.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Brick57: dl14.dg.163.org:/glusterfs_bri<wbr>ck/brick1/gv0</div><div>Options Reconfigured:</div><div>performance.cache-size: 10GB</div><div>performance.parallel-readdir: on</div><div>performance.readdir-ahead: on</div><div>network.inode-lru-limit: 200000</div><div>performance.md-cache-timeout: 600</div><div>performance.cache-invalidation<wbr>: on</div><div>performance.stat-prefetch: on</div><div>features.cache-invalidation-ti<wbr>meout: 600</div><div>features.cache-invalidation: on</div><div>features.inode-quota: off</div><div>features.quota: off</div><div>cluster.quorum-reads: on</div><div>cluster.quorum-count: 2</div><div>cluster.quorum-type: fixed</div><div>transport.address-family: inet</div><div>nfs.disable: on</div><div>performance.client-io-threads: off</div><div>cluster.server-quorum-ratio: 51%</div><div><br></div></div><div><br></div>
<div id="m_5141060108186289289m_5471995425395774604m_-6772762122606358812ntes-pcmail-signature" style="font-family:'Helvetica','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1'">
<div style="font-size:14px;padding:0;margin:0">
<div style="border-bottom:1px solid #ccc;display:inline-block">
<a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png" style="display:block;background:#fff;max-width:400px;padding:15px 0 10px 0;text-decoration:none;outline:none" target="_blank">
<table style="width:100%;max-width:100%;table-layout:fixed;border-collapse:collapse;color:#9b9ea1;font-size:14px;line-height:1.3" cellpadding="0">
<tbody style="font-family:'PingFang SC','Hiragino Sans GB','WenQuanYi Micro Hei','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1',verdana!important;word-wrap:break-word;word-break:break-all">
<tr>
<td style="padding:0;box-sizing:border-box;width:38px" width="38">
<img style="vertical-align:middle;width:38px;height:38px;border-radius:50%" src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png" width="38" height="38">
</td>
<td style="padding:0 30px 0 10px;color:#31353b">
<div style="font-size:16px;font-weight:bold;width:100%;white-space:nowrap;overflow:hidden;text-overflow:ellipsis">huting3</div>
</td>
<td style="text-align:right;width:72px" width="72">
<img style="width:72px;height:20px" src="https://mail-online.nosdn.127.net/qiyelogo/209a2912f40f6683af56bb7caff1cb54.png" width="72" height="20">
</td>
</tr>
<tr width="100%" style="font-size:14px!important;width:100%">
<td colspan="3" style="padding:10px 0 0 0;font-size:14px!important;width:100%">
<div style="width:100%;font-size:14px!important;word-wrap:break-word;word-break:break-all">huting3@corp.netease.com</div>
</td>
</tr>
</tbody>
</table>
</a>
</div>
</div>
<div style="font-size:12px;color:#b5b9bd">
<span>签名由</span>
<a style="text-decoration:none;color:#4196ff;padding:0 5px" href="http://mail.163.com/dashi/" target="_blank">网易邮箱大师</a>
<span>定制</span>
</div>
</div><br>
</div>
</div>
<br>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mail<wbr>man/listinfo/gluster-devel</a><br></blockquote></div><br></div></div>
</blockquote>
</div></div></div>
</div>
</blockquote></div><br></div>
</blockquote>
</div></div></div>
</div>
</blockquote></div><br></div></div>
</blockquote><!--�-->
</div></blockquote><!--�-->
</div></blockquote><style type="text/css">
a#ntes-pcmail-signature-default:hover {
text-decoration: underline;
color: #3593db;
cursor: pointer;
}
</style><!--�-->
</div>
</body>
</html>