[Gluster-users] glusterfs and glusterfsd process utilization extremely high

[Kyle Harris]

Hello,

I have an issue with a 3 node replicated cluster.  My issue started after
reboot a while back.  The top command would show the glusterfs and
glusterfsd processes eating up almost all the resources on an all three
nodes of the cluster.  So much so that it would not run the web sites that
are hosted on it.  The httpd processes would begin to hang.  I finally
decided to tear down the cluster and rebuild it from the ground up.  I did
so and then copied all the data back which took all night due to the amount
of data.  All was well during that entire copy process back to the cluster
with no resource spikes.

I should note that this cluster is home to many Apache/PHP based web
sites.  The problem starts again, however the minute I point traffic back
to the sites on the cluster.  Before pointing traffic to it, all is fine
but as soon as the traffic begins to hit it, the utilization again begins
to spike.  Note that all the sites run just fine when hosted from a
standard EXT4 partition.  I noticed another thread labeled "glusterfsd
process thrashing CPU" where Pranith asks if the user has directories with
lots of files and I do.

Here are some other details of my cluster:
- OS:  CentOS 6.6 with all updates on all 3 nodes as of 11-22-2014
- All 3 nodes have 8 cores with 16 GB of RAM
- Nodes are all formatted with EXT4
- All three nodes also have the files systems mounted on them for use with
Apache.  I have experimented with both NFS and Fuse mounts and it doesn't
seem to make a difference which I use for this particular problem.  I am
currently using Fuse.
- Approximately 135 GB of data.  Some deep directories with many small
files.
- No optimization or changes have been made to the cluster . . . it is
running with default options
- Gluster version 3.6.1-1 installed from RPMs
- Note the issue originally occurred on version 3.5.2 but I updated before
rebuilding it in hopes that would fix it (it didn't)

Can anyone give me guidance on how to tackle this problem?  I am hoping
perhaps Pranith can give some details as to why the question about many
files and how to proceed given my situation.  I know others have commented
about having many small files with regard to performance but when the
processors are not spiked, performance has been acceptable.  Any help would
be greatly appreciated.

-- 
Kyle
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141122/3c1d6f59/attachment.html>