[Gluster-users] Runaway Open Files

[Dave Bour]

I've got a 2 peer replicated setup with 2 bricks on each system.

Starting sometime overnight, one system ground to a halt, to the point a
local login was not possible.
Restarted.
A couple hours later I tried to log in via SSH, too many open files.  A
quick 'lsof" showed thousands of open files on glusterfs.

I just restarted the server again...and appears to be doing the same
thing.  Addition of nearly 2500 files in 4 minutes, not 5 minutes after
rebooting.

for d in {1..1000};do a=`date`;b=`lsof|grep gluster|wc -l`;echo $a " "
$b;sleep 60;done
Tue Aug 2 07:56:37 EDT 2011   2924
Tue Aug 2 07:57:37 EDT 2011   3568
Tue Aug 2 07:58:38 EDT 2011   4154
Tue Aug 2 07:59:38 EDT 2011   4718
Tue Aug 2 08:00:38 EDT 2011   5376

It's peer is more what's expected:
for d in {1..1000};do a=`date`;b=`lsof|grep gluster|wc -l`;echo $a " "
$b;sleep 60;done
Tue Aug 2 07:57:18 EDT 2011   193
Tue Aug 2 07:58:18 EDT 2011   192
Tue Aug 2 07:59:18 EDT 2011   193
Tue Aug 2 08:00:19 EDT 2011   193
Tue Aug 2 08:01:19 EDT 2011   191

Its running 3.2.2

Any suggestions to what's happening?
Where to start to troubleshoot.

D.




-- 
Dave Bour
Senior Consultant
Desktop Solution Center
905.381.0077 x501
dcbour at desktopsolutioncenter.ca

Linked In: http://ca.linkedin.com/in/dcbour
Facebook: http://www.facebook.com/dcbour


Providing Complete IT Peace of Mind