[Gluster-users] Recovering out of sync nodes from input/output error

Jeff Darcy jdarcy at redhat.com
Thu Apr 12 12:49:45 UTC 2012


On 04/11/2012 07:00 AM, Alex Florescu wrote:
> Simulation follows:
> step 1
> node1:
> iptables -I INPUT 1 -s 10.0.2.15 -j DROP (connectivity loss simulation)
> touch /a/howareyou
> 
> node2:
> touch /a/hello
> 
> step 2
> node1:
> iptables -D INPUT 1 (connectivity recovery)
> ls /a
> ls: cannot access /a: Input/output error
> 
> node2:
> ls /a
> ls: cannot access /a: Input/output error

I was able to reproduce this on my own setup using packages built from git,
which has a bit of a surprise TBH.  I'll look into it, but here are some
observations that might suggest workarounds.

(1) To a first approximation, it should be safe to "merge" directory contents
despite there being a split-brain problem, by healing any file that exists on
only one brick from there to its peer(s).  This contrasts with the case for
file contents, where - as Robert points out - we can't determine the correct
thing to do and would risk overwriting data.  Directory entries differ from
file contents in a small but important way: they're sets, not arrays.  If
something's not in the set, there's no danger that adding it will overwrite
anything.

(2) That said, the case you've created is indistinguishable from the case where
"hello" and "howareyou" used to exist on both bricks and each *deleted* one
while they couldn't communicate.  Unconditionally recreating the files would
effectively undo those deletes, which many would consider an error as serious
as overwriting data.  It would not be valid for such merge behavior to kick in
unconditionally.  At the very least, there should be a configuration option for it.

(3) The reason you continue to get I/O errors is probably that the xattrs on
the *parent directory* still indicate pending operations on both sides.  You
can verify this with the following command on each brick:

	getfattr -d -e hex -n trusted.glusterfs.dht /a

The format of this value is described here:

	http://hekafs.org/index.php/2011/04/glusterfs-extended-attributes/

If the result is non-zero (most likely in the last four-byte integer indicating
a directory-entry operation) then that confirms our theory.  It should be safe
for the self-heal code to clear these counts if (and only if) the directories
are checked and found identical.  In fact, I think we already do this.  Thus,
manual copying of files followed by self-heal on the parent directory should
make the errors go away.  I encourage you to try that while I go look at the code.



More information about the Gluster-users mailing list