[Gluster-users] Sudden performance drop in gluster

[Pat Haley]

Hi,

Today we suddenly experienced a performance drop in gluster:  e.g. doing 
an "ls" of a directory with about 20 files takes about 5 minutes.  This 
is way beyond (and seem separate from) some previous concerns we had.

Our gluster filesystem is two bricks hosted on a single server. Logging 
onto that server and doing "top" shows a load average of ~30.  In 
general, no process is showing significant CPU usage except an 
occasional flash a 3300% from glusterfsd.  The rest of our system is not 
doing any exceptional data demands on the file system (i.e. we aren't 
suddenly running more jobs than we were yesterday).

Any thoughts on how we can proceed with debugging this will be greatly 
appreciated.

Some additional information:

glusterfs 3.7.11 built on Apr 27 2016 14:09:22
CentOS release 6.8 (Final)


[root at mseas-data2 ~]# gluster volume status data-volume
Status of volume: data-volume
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick mseas-data2:/mnt/brick1               49154 0          Y       5021
Brick mseas-data2:/mnt/brick2               49155 0          Y       5026

Task Status of Volume data-volume
------------------------------------------------------------------------------
Task                 : Rebalance
ID                   : 892d9e3a-b38c-4971-b96a-8e4a496685ba
Status               : completed


[root at mseas-data2 ~]# gluster volume info data-volume

Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
Options Reconfigured:
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off


-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301