[Gluster-devel] gluster fuse comsumes huge memory

Thu Aug 9 04:36:07 UTC 2018

Can you get the output of following cmds?

# grep itable <statedump> | grep lru | grep count

# grep itable <statedump> | grep active | grep count

On Thu, Aug 9, 2018 at 9:25 AM, huting3 <huting3 at corp.netease.com> wrote:

> Yes, I got the dump file and found there are many huge num_allocs just
> like following:
>
> I found memusage of 4 variable types are extreamly huge.
>
>  [protocol/client.gv0-client-0 - usage-type gf_common_mt_char memusage]
> size=47202352
> num_allocs=2030212
> max_size=47203074
> max_num_allocs=2030235
> total_allocs=26892201
>
> [protocol/client.gv0-client-0 - usage-type gf_common_mt_memdup memusage]
> size=24362448
> num_allocs=2030204
> max_size=24367560
> max_num_allocs=2030226
> total_allocs=17830860
>
> [mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
> size=2497947552
> num_allocs=4578229
> max_size=2459135680
> max_num_allocs=7123206
> total_allocs=41635232
>
> [mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]
> size=4038730976
> num_allocs=1
> max_size=4294962264
> max_num_allocs=37
> total_allocs=150049981
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png>
>
>
>
> huting3
> huting3 at corp.netease.com
>
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png>
> 签名由 网易邮箱大师 <http://mail.163.com/dashi/> 定制
>
> On 08/9/2018 11:36，Raghavendra Gowdappa<rgowdapp at redhat.com>
> <rgowdapp at redhat.com> wrote：
>
>
>
> On Thu, Aug 9, 2018 at 8:55 AM, huting3 <huting3 at corp.netease.com> wrote:
>
>> Hi expert:
>>
>> I meet a problem when I use glusterfs. The problem is that the fuse
>> client consumes huge memory when write a   lot of files(>million) to the
>> gluster, at last leading to killed by OS oom. The memory the fuse process
>> consumes can up to 100G! I wonder if there are memory leaks in the gluster
>> fuse process, or some other causes.
>>
>
> Can you get statedump of fuse process consuming huge memory?
>
>
>> My gluster version is 3.13.2, the gluster volume info is listed as
>> following:
>>
>> Volume Name: gv0
>> Type: Distributed-Replicate
>> Volume ID: 4a6f96f8-b3fb-4550-bd19-e1a5dffad4d0
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 19 x 3 = 57
>> Transport-type: tcp
>> Bricks:
>> Brick1: dl20.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick2: dl21.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick3: dl22.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick4: dl20.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick5: dl21.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick6: dl22.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick7: dl20.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick8: dl21.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick9: dl22.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick10: dl23.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick11: dl24.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick12: dl25.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick13: dl26.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick14: dl27.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick15: dl28.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick16: dl29.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick17: dl30.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick18: dl31.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick19: dl32.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick20: dl33.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick21: dl34.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick22: dl23.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick23: dl24.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick24: dl25.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick25: dl26.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick26: dl27.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick27: dl28.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick28: dl29.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick29: dl30.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick30: dl31.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick31: dl32.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick32: dl33.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick33: dl34.dg.163.org:/glusterfs_brick/brick2/gv0
>> Brick34: dl23.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick35: dl24.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick36: dl25.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick37: dl26.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick38: dl27.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick39: dl28.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick40: dl29.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick41: dl30.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick42: dl31.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick43: dl32.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick44: dl33.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick45: dl34.dg.163.org:/glusterfs_brick/brick3/gv0
>> Brick46: dl0.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick47: dl1.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick48: dl2.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick49: dl3.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick50: dl5.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick51: dl6.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick52: dl9.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick53: dl10.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick54: dl11.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick55: dl12.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick56: dl13.dg.163.org:/glusterfs_brick/brick1/gv0
>> Brick57: dl14.dg.163.org:/glusterfs_brick/brick1/gv0
>> Options Reconfigured:
>> performance.cache-size: 10GB
>> performance.parallel-readdir: on
>> performance.readdir-ahead: on
>> network.inode-lru-limit: 200000
>> performance.md-cache-timeout: 600
>> performance.cache-invalidation: on
>> performance.stat-prefetch: on
>> features.cache-invalidation-timeout: 600
>> features.cache-invalidation: on
>> features.inode-quota: off
>> features.quota: off
>> cluster.quorum-reads: on
>> cluster.quorum-count: 2
>> cluster.quorum-type: fixed
>> transport.address-family: inet
>> nfs.disable: on
>> performance.client-io-threads: off
>> cluster.server-quorum-ratio: 51%
>>
>>
>> huting3
>> huting3 at corp.netease.com
>>
>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png>
>> 签名由 网易邮箱大师 <http://mail.163.com/dashi/> 定制
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20180809/e12abd52/attachment-0001.html>