[Gluster-users] a workaround for hight cpu load
Norbert
norbert.huang at qq.com
Wed Mar 23 10:42:40 UTC 2016
微云的文件
perf.data.glusterfsd.self-heal.he...(2.05M)
the attachments is the result of perf when cpu load is 500%~800%, it shows "get xattr operation " cost most cpu time.
------------------ 原始邮件 ------------------
发件人: "Russell Purinton";<russell.purinton at gmail.com>;
发送时间: 2016年3月23日(星期三) 下午4:06
收件人: "Norbert"<norbert.huang at qq.com>;
抄送: "gluster-users"<gluster-users at gluster.org>;
主题: Re: [Gluster-users] a workaround for hight cpu load
I’m only about 90% sure about this, but I think the Self Heal Daemon is a glusterfs process that must communicate with glusterfsd for healing to take place. blocking communications like this is basically breaking the Self Heal Daemon and your files are probably not being healed correctly.
Healing requires high CPU due to the default “diff” algorithm which will actually read the files and send only the differences over the network. You can trade CPU impact for Network Bandwidth by using the ‘full’ algorithm, then it doesn’t need to compute info on the files themselves.
Can anyone confirm if I’m correct about this?
On Mar 22, 2016, at 11:05 PM, Norbert <norbert.huang at qq.com> wrote:
problem: when self-heal is running with many files, it make cpu high load and gluster client cannot write new file and read file.
as described at: http://www.gluster.org/pipermail/gluster-users/2015-November/024228.html
from following tests, I realize that: when close tcp connections between glusterfs and glusterfsd in gluster server,it can reduce cpu load and not affect gluster client to write and read file.
========================= config begin ==================================
gluster 3.5.1
#gluster volume status
Status of volume: tr5
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick 192.168.5.252:/home/gluster351/r15 49157 Y 11899
Brick 192.168.5.76:/home/gluster/r15 49152 Y 26692
NFS Server on localhost N/A N N/A
Self-heal Daemon on localhost N/A Y 11918
NFS Server on 192.168.5.76 N/A N N/A
Self-heal Daemon on 192.168.5.76 N/A Y 26705
Task Status of Volume tr5
------------------------------------------------------------------------------
There are no active volume tasks
gluster client: 192.168.16.207:1019
# mount
192.168.5.252:/tr5 on /home/mariadb/data/tr5 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
========================= config end ==================================
========================= test 1 begin =================================
at 192.168.5.252,run:
iptables -A INPUT -p tcp --dport 49157 -j DROP -s 192.168.5.252
iptables -A INPUT -p tcp --dport 49157 -j DROP -s 192.168.5.76
at 192.168.5.76,run:
iptables -A INPUT -p tcp --dport 49152 -j DROP -s 192.168.5.252
iptables -A INPUT -p tcp --dport 49152 -j DROP -s 192.168.5.76
gluster client can write and read files as normal.
========================= test 1 end ==================================
========================= test 2 begin ==================================
1: at 192.168.5.76, shutdown process: glusterd, glusterfsd, glusterfs.
2: at gluster client, copy 10k files to gluster server.
3: at 192.168.5.252, there are 10k+ link file under directory /home/gluster351/r15/.glusterfs/indices/xattrop.
4: at 192.168.5.76, start gluster service. then self-heal begins.
5: during self-heal, at 192.168.5.252 , glusterfs %CPU 5.7 , glusterfsd %CPU 7.0
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11918 root 20 0 321m 36m 2816 S 7.0 0.5 1:28.07 glusterfs
11899 root 20 0 824m 42m 3164 S 5.7 0.5 20:43.39 glusterfsd
6: during self-heal, at 192.168.5.76 , glusterfs %CPU 0 , glusterfsd %CPU 6.0
7: during self-heal, at 192.168.5.76 , run:
iptables -A INPUT -p tcp --dport 49152 -j DROP -s 192.168.5.252
iptables -A INPUT -p tcp --dport 49152 -j DROP -s 192.168.5.76
after run iptables, at 192.168.5.76 and 192.168.5.252, both glusterfs %CPU 0 , glusterfsd %CPU 0
8: at 192.168.5.252, there are 7000+ link file under directory /home/gluster351/r15/.glusterfs/indices/xattrop.
9: gluster client can write and read files as normal.
========================= test 2 begin ==================================
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160323/953dafe4/attachment.html>
More information about the Gluster-users
mailing list