[Gluster-users] High load / hang

[Alun James]

Evening running a Wordpress update (a 6 meg file) via wp-admin seems to cause the gluster process CPU to spike and server Load to go higher than I am comfortable with. I get that due to replication there will be some latency overhead, but I was not expecting a meltdown :) 


Regards, 


A 

----- Original Message -----

From: "Alun James" <ajames at tibus.com> 
To: "Hoggins!" <fuckspam at wheres5.com> 
Cc: gluster-users at gluster.org 
Sent: Tuesday, 12 May, 2015 2:16:02 PM 
Subject: Re: [Gluster-users] High load / hang 


A hard reboot solved it for me too, but I am a little worried on the stability of it. 


The entire wordpress web root is stored on the gluster volume, a day later I attempted to tar/gzip the web directory on the gluster volume and again the load on one particular server sky rocketed, server became unresponsive and kernel hang messages appeared claiming the gzip process was hanging. It seems that if I cause any serious IO on the volume, things seem to snowball. 


There are 3 web nodes with a replicated gluster volume between them all. Each web node also mounts the gluster volume locally using NFS (an attempt to improve performance for lots of small php files). I tried mounting native glusterfs instead of NFS for client mount, but I had the same issues. 



ii glusterfs-client 3.6.2-ubuntu1~trusty3 amd64 clustered file-system (client package) 
ii glusterfs-common 3.6.2-ubuntu1~trusty3 amd64 GlusterFS common libraries and translator modules 
ii glusterfs-server 3.6.2-ubuntu1~trusty3 amd64 clustered file-system (server package) 



Volume Name: my_filestore_vol 
Type: Replicate 
Volume ID: xyz 
Status: Started 
Number of Bricks: 1 x 3 = 3 
Transport-type: tcp 
Bricks: 
Brick1: my-web01:/export/brick0 
Brick2: my-web02:/export/brick0 
Brick3: my-web03:/export/brick0 
Options Reconfigured: 
nfs.drc: off 
diagnostics.brick-log-level: WARNING 



Status of volume: my_filestore_vol 
Gluster process Port Online Pid 
------------------------------------------------------------------------------ 
Brick my-web01:/export/brick0 49152 Y 2138 
Brick my-web02:/export/brick0 49152 Y 21104 
Brick my-web03:/export/brick0 49152 Y 1827 
NFS Server on localhost 2049 Y 2145 
Self-heal Daemon on localhost N/A Y 2152 
NFS Server on my-web03 2049 Y 1834 
Self-heal Daemon on my-web03 N/A Y 1841 
NFS Server on my-web02 2049 Y 21118 
Self-heal Daemon on my-web02 N/A Y 21123 


Task Status of Volume my_filestore_vol 
------------------------------------------------------------------------------ 
There are no active volume tasks 




Regards, 


A 

----- Original Message -----

From: "Hoggins!" <fuckspam at wheres5.com> 
To: gluster-users at gluster.org 
Sent: Friday, 8 May, 2015 1:25:26 PM 
Subject: Re: [Gluster-users] High load / hang 

Well, that's "funny", because the exact same thing happened to me this morning, except that I could hard reboot the machine, and it got up and running normally again. 
But the symptoms you describe are oddly similar, and strangely simultaneous. 


Le 08/05/2015 10:36, Alun James a écrit : 


Hi folks, 


I have a 3 node gluster/web/db cluster running a Wordpress site . This morning one of the nodes is under very high load and the mounted gluster partition is inaccessible. Attempts to reboot that node have failed with the server seemingly hung/blocked. Has anyone else experienced this or can give any pointers in how to diagnose the cause of gluster going awry? 



_______________________________________________ 
Gluster-users mailing list 
Gluster-users at gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users 

_______________________________________________ 
Gluster-users mailing list 
Gluster-users at gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150512/96ff45e0/attachment.html>