[Gluster-users] glusterfs brick server use too high memory

[Yao Guotao]

Thank you very much for your reply.


I think I found the key to the problem. But, I'd like you to confirm whether this is the issue.
My gluster version is 3.8.9, And I found there is a bug in it, and fixed in 3.8.11:
https://bugzilla.redhat.com/show_bug.cgi?id=1431592



A large amount of memory allocated for agf_common_mt_strdup. Everythig else seem to be all right.



My cluster:


Yesterday Afternoon：（I'm sorry I forgot the specific time.）

[features/locks.www-volume-locks - usage-type gf_common_mt_strdup memusage]
size=1941483443
num_allocs=617382139
max_size=1941483443
max_num_allocs=617382139
total_allocs=661873332



TIme: 2017.11.9 17:15 (Today)
[features/locks.www-volume-locks - usage-type gf_common_mt_strdup memusage]
size=792538295
num_allocs=752904534
max_size=792538295
max_num_allocs=752904534
total_allocs=800889589



The statedump files are in the attachment.


Thanks again.





在 2017-11-09 17:22:03，"Nithya Balachandran" <nbalacha at redhat.com> 写道：





On 8 November 2017 at 17:16, Yao Guotao <yaoguo_tao at 163.com> wrote:

Hi all,
I'm glad to add glusterfs community.


I have a glusterfs cluster:
Nodes: 4

System: Centos7.1
Glusterfs: 3.8.9
Each Node:
CPU: 48 core
Mem: 128GB
Disk: 1*4T


There is one Distributed Replicated volume. There are ~160 k8s pods as clients connecting to glusterfs. But, the memory of  glusterfsd process is too high, gradually increase to 100G every node.
Then, I reboot the glusterfsd process. But the memory increase during approximate a week.
How can I debug the problem?


Hi,


Please take statedumps at intervals (a minimum of 2 at intervals of an hour) of a brick process for which you see the memory increasing and send them to us.  [1] describes how to take statedumps.


Regards,
Nithya


[1] http://docs.gluster.org/en/latest/Troubleshooting/statedump/ 


 
Thanks.





 


_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171109/a394c932/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: home-www-log.152061.dump.20171108
Type: application/octet-stream
Size: 806070 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171109/a394c932/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: home-www-log.152061.dump.20171109
Type: application/octet-stream
Size: 739337 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171109/a394c932/attachment-0003.obj>