[Gluster-devel] gluster fuse comsumes huge memory

Raghavendra Gowdappa rgowdapp at redhat.com
Thu Aug 9 05:21:26 UTC 2018 

On Thu, Aug 9, 2018 at 10:43 AM, Raghavendra Gowdappa <rgowdapp at redhat.com>
wrote:

>
>
> On Thu, Aug 9, 2018 at 10:36 AM, huting3 <huting3 at corp.netease.com> wrote:
>
>> grep count will ouput nothing, so I grep size, the results are:
>>
>> $ grep itable glusterdump.109182.dump.1533730324 | grep lru | grep size
>> xlator.mount.fuse.itable.lru_size=191726
>>
>
> Kernel is holding too many inodes in its cache. What's the data set like?
> Do you've too many directories? How many files do you have?
>

Just to be sure, can you give the output of following cmd too:

# grep itable <dump> | grep lru | wc -l


>
>> $ grep itable glusterdump.109182.dump.1533730324 | grep active | grep
>> size
>> xlator.mount.fuse.itable.active_size=17
>>
>>
>> huting3
>> huting3 at corp.netease.com
>>
>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png>
>> 签名由 网易邮箱大师 <http://mail.163.com/dashi/> 定制
>>
>> On 08/9/2018 12:36，Raghavendra Gowdappa<rgowdapp at redhat.com>
>> <rgowdapp at redhat.com> wrote：
>>
>> Can you get the output of following cmds?
>>
>> # grep itable <statedump> | grep lru | grep count
>>
>> # grep itable <statedump> | grep active | grep count
>>
>> On Thu, Aug 9, 2018 at 9:25 AM, huting3 <huting3 at corp.netease.com> wrote:
>>
>>> Yes, I got the dump file and found there are many huge num_allocs just
>>> like following:
>>>
>>> I found memusage of 4 variable types are extreamly huge.
>>>
>>>  [protocol/client.gv0-client-0 - usage-type gf_common_mt_char memusage]
>>> size=47202352
>>> num_allocs=2030212
>>> max_size=47203074
>>> max_num_allocs=2030235
>>> total_allocs=26892201
>>>
>>> [protocol/client.gv0-client-0 - usage-type gf_common_mt_memdup memusage]
>>> size=24362448
>>> num_allocs=2030204
>>> max_size=24367560
>>> max_num_allocs=2030226
>>> total_allocs=17830860
>>>
>>> [mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
>>> size=2497947552
>>> num_allocs=4578229
>>> max_size=2459135680
>>> max_num_allocs=7123206
>>> total_allocs=41635232
>>>
>>> [mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]
>>> size=4038730976
>>> num_allocs=1
>>> max_size=4294962264
>>> max_num_allocs=37
>>> total_allocs=150049981
>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png>
>>>
>>>
>>>
>>> huting3
>>> huting3 at corp.netease.com
>>>
>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png>
>>> 签名由 网易邮箱大师 <http://mail.163.com/dashi/> 定制
>>>
>>> On 08/9/2018 11:36，Raghavendra Gowdappa<rgowdapp at redhat.com>
>>> <rgowdapp at redhat.com> wrote：
>>>
>>>
>>>
>>> On Thu, Aug 9, 2018 at 8:55 AM, huting3 <huting3 at corp.netease.com>
>>> wrote:
>>>
>>>> Hi expert:
>>>>
>>>> I meet a problem when I use glusterfs. The problem is that the fuse
>>>> client consumes huge memory when write a   lot of files(>million) to the
>>>> gluster, at last leading to killed by OS oom. The memory the fuse process
>>>> consumes can up to 100G! I wonder if there are memory leaks in the gluster
>>>> fuse process, or some other causes.
>>>>
>>>
>>> Can you get statedump of fuse process consuming huge memory?
>>>
>>>
>>>> My gluster version is 3.13.2, the gluster volume info is listed as
>>>> following:
>>>>
>>>> Volume Name: gv0
>>>> Type: Distributed-Replicate
>>>> Volume ID: 4a6f96f8-b3fb-4550-bd19-e1a5dffad4d0
>>>> Status: Started
>>>> Snapshot Count: 0
>>>> Number of Bricks: 19 x 3 = 57
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: dl20.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick2: dl21.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick3: dl22.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick4: dl20.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick5: dl21.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick6: dl22.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick7: dl20.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick8: dl21.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick9: dl22.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick10: dl23.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick11: dl24.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick12: dl25.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick13: dl26.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick14: dl27.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick15: dl28.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick16: dl29.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick17: dl30.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick18: dl31.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick19: dl32.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick20: dl33.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick21: dl34.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick22: dl23.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick23: dl24.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick24: dl25.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick25: dl26.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick26: dl27.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick27: dl28.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick28: dl29.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick29: dl30.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick30: dl31.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick31: dl32.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick32: dl33.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick33: dl34.dg.163.org:/glusterfs_brick/brick2/gv0
>>>> Brick34: dl23.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick35: dl24.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick36: dl25.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick37: dl26.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick38: dl27.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick39: dl28.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick40: dl29.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick41: dl30.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick42: dl31.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick43: dl32.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick44: dl33.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick45: dl34.dg.163.org:/glusterfs_brick/brick3/gv0
>>>> Brick46: dl0.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick47: dl1.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick48: dl2.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick49: dl3.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick50: dl5.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick51: dl6.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick52: dl9.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick53: dl10.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick54: dl11.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick55: dl12.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick56: dl13.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Brick57: dl14.dg.163.org:/glusterfs_brick/brick1/gv0
>>>> Options Reconfigured:
>>>> performance.cache-size: 10GB
>>>> performance.parallel-readdir: on
>>>> performance.readdir-ahead: on
>>>> network.inode-lru-limit: 200000
>>>> performance.md-cache-timeout: 600
>>>> performance.cache-invalidation: on
>>>> performance.stat-prefetch: on
>>>> features.cache-invalidation-timeout: 600
>>>> features.cache-invalidation: on
>>>> features.inode-quota: off
>>>> features.quota: off
>>>> cluster.quorum-reads: on
>>>> cluster.quorum-count: 2
>>>> cluster.quorum-type: fixed
>>>> transport.address-family: inet
>>>> nfs.disable: on
>>>> performance.client-io-threads: off
>>>> cluster.server-quorum-ratio: 51%
>>>>
>>>>
>>>> huting3
>>>> huting3 at corp.netease.com
>>>>
>>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png>
>>>> 签名由 网易邮箱大师 <http://mail.163.com/dashi/> 定制
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20180809/a5313f61/attachment-0001.html>

More information about the Gluster-devel mailing list