[Gluster-users] Gluster 3.3.1 consistency problems listing directory

Thu Feb 14 06:39:54 UTC 2013

Dear all,

We ran into severe problems with our recently installed gluster system, described below

Volume Name: GA01
Type: Distributed-Replicate
Volume ID: ba885c47-3b44-4e7b-b8f9-a4cc0959b7a3
Status: Started
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Bricks:
Brick1: c3
Brick2: c2
Brick3: n3
Brick4: n5
Brick5: n9
Brick6: n13

While I was on leave for 10 days, the machine n3 went down. All seemed fine (noone knew) but after  I came back and restarted the machine, the system went into a very bad state. A typical scenario would be creating a new directory, adding files to it and then doing an ls. Any files that were physically started on the n3/n5 pair would not appear (though if you explicitly listed it eg. ls newdir/evec2gp.py it would appear and you could use it). I tried running heal several times, shutting down all cluster daemons in the system, remounting bricks, remounting the gluster system, etc. I could restart the n3 machine but I can't restart other machines because it's in production. This happens reproducibly, not just with files that were created while n3 was down but even now.

We are running gluster 3.3.1. The underlying bricks are formatted as ext3 (not ext4).  The n machines run SL6.3 and the c machines run Ubuntu 12.04 (there are constraints that require this). Could the use of different OS cause a problem?

At this point, I prepared to copy all files off the gluster volume and completely reformat the underlying file systems and recreate the gluster volumes from scratch. However, I am concerned since we need to rely on the system.

Any suggestions

Thanks

Scott

<html><p><font face = "verdana" size = "0.8" color = "navy">This communication is intended for the addressee only. It is confidential. If you have received this communication in error, please notify us immediately and destroy the original message. You may not copy or disseminate this communication without the permission of the University. Only authorized signatories are competent to enter into agreements on behalf of the University and recipients are thus advised that the content of this message may not be legally binding on the University and may contain the personal views and opinions of the author, which are not necessarily the views and opinions of The University of the Witwatersrand, Johannesburg. All agreements between the University and outsiders are subject to South African Law unless the University agrees in writing to the contrary.</font></p></html>