[Gluster-users] a workaround for hight cpu load

[Russell Purinton]

I’m only about 90% sure about this, but I think the Self Heal Daemon is a glusterfs process that must communicate with glusterfsd for healing to take place.  blocking communications like this is basically breaking the Self Heal Daemon and your files are probably not being healed correctly.

Healing requires high CPU due to the default “diff” algorithm which will actually read the files and send only the differences over the network.   You can trade CPU impact for Network Bandwidth by using the ‘full’ algorithm, then it doesn’t need to compute info on the files themselves. 

Can anyone confirm if I’m correct about this?


> On Mar 22, 2016, at 11:05 PM, Norbert <norbert.huang at qq.com> wrote:
> 
> 
> problem:  when self-heal  is running with  many files, it  make cpu high load and  gluster client cannot write new file and  read file.
> 
> as described at: http://www.gluster.org/pipermail/gluster-users/2015-November/024228.html
> 
> from following tests, I realize that: when  close  tcp  connections between glusterfs and  glusterfsd  in gluster server,it  can reduce  cpu load and not affect  gluster  client to write and read file.
> 
> 
> 
> 
> =========================  config begin ==================================
> gluster  3.5.1
> 
> #gluster volume status
> Status of volume: tr5
> Gluster process                                         Port    Online  Pid
> ------------------------------------------------------------------------------
> Brick 192.168.5.252:/home/gluster351/r15                49157   Y       11899
> Brick 192.168.5.76:/home/gluster/r15             49152   Y       26692
> NFS Server on localhost                                 N/A     N       N/A
> Self-heal Daemon on localhost                           N/A     Y       11918
> NFS Server on 192.168.5.76                              N/A     N       N/A
> Self-heal Daemon on 192.168.5.76                        N/A     Y       26705
>  
> Task Status of Volume tr5
> ------------------------------------------------------------------------------
> There are no active volume tasks
> 
> 
> gluster client:   192.168.16.207:1019
> # mount
> 192.168.5.252:/tr5 on /home/mariadb/data/tr5 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
> 
> =========================  config end ==================================
> 
> 
> ========================= test 1  begin =================================
> at 192.168.5.252,run:
> iptables -A INPUT -p tcp --dport 49157  -j DROP  -s  192.168.5.252
> iptables -A INPUT -p tcp --dport 49157  -j DROP  -s  192.168.5.76
> 
> at 192.168.5.76,run:
> iptables -A INPUT -p tcp --dport 49152  -j DROP  -s   192.168.5.252
> iptables -A INPUT -p tcp --dport 49152  -j DROP  -s   192.168.5.76
> 
> gluster client can write and read files as normal.
> ========================= test 1  end ==================================
> 
> 
> ========================= test 2  begin ==================================
> 1:  at 192.168.5.76, shutdown process: glusterd, glusterfsd, glusterfs.
> 2:  at gluster client, copy  10k files to gluster server.  
> 3:  at 192.168.5.252,  there are  10k+ link file under  directory /home/gluster351/r15/.glusterfs/indices/xattrop.
> 4:  at 192.168.5.76, start  gluster service.  then self-heal  begins.
> 5:  during self-heal, at 192.168.5.252 , glusterfs %CPU 5.7 , glusterfsd    %CPU 7.0
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                          
>  11918 root      20   0  321m  36m 2816 S  7.0  0.5   1:28.07 glusterfs                                                                                                                                        
>   11899 root      20   0  824m  42m 3164 S  5.7  0.5  20:43.39 glusterfsd      
> 
> 6:  during self-heal, at 192.168.5.76 , glusterfs %CPU 0 , glusterfsd    %CPU 6.0
> 7:  during self-heal, at 192.168.5.76 , run:
>       iptables -A INPUT -p tcp --dport 49152  -j DROP  -s   192.168.5.252
>       iptables -A INPUT -p tcp --dport 49152  -j DROP  -s   192.168.5.76
>      
>      after run iptables, at 192.168.5.76  and 192.168.5.252, both glusterfs %CPU 0 , glusterfsd    %CPU 0
> 8:   at 192.168.5.252,  there are  7000+ link file under  directory /home/gluster351/r15/.glusterfs/indices/xattrop.
> 9:   gluster client can write and read files as normal.
> ========================= test 2  begin ==================================
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160323/ae2863b6/attachment.html>