[Gluster-users] Fencing FOPs on data-split-brained files

[Jeff Darcy]

On 11/13/2013 06:01 AM, Ravishankar N wrote:
> Currenly in glusterfs, when there is a data splt-brain (only) on a
> file, we disallow the following operations from the mount-point by
> returning EIO to the application: - Writes to the file (truncate, dd,
> echo, cp etc) - Reads to the file (cat) - Reading extended attributes
> (getfattr) [1]

I see that the patch has already been cherry-picked, but it makes me
uneasy.  Setxattr etc. are inode operations, not data, so data
split-brain shouldn't matter.  Symlink is an entry operation according
to the code; seems to me like it should still be inode, but either way
it's not data so the same concern applies.  The problem seems to be that
afr_is_split_brain always checks for *both* data and metadata
split-brain, but in some cases we need an equivalent call that only
checks for one.

> However we do permit the following operations: -creating hardlinks
> -creating symlinks -mv -setattr -chmod -chown --touch -ls -stat

These are all inode/entry ops.  Because some setattr/stat fields do
interact with data operations I'd say they're the *last* ones we should
consider allowing when in data split-brain.  Some of the others might
complicate self-heal, so we could reasonably consider disallowing them
as well, but we should avoid that where feasible.

> While it makes sense to allow `ls` and `stat`, is it okay to  add
> checks in the FOPS to disallow the other operations? Allowing
> creation of links and changing file attributes only seems to
> complicate things before the admin can go to the backend bricks and
> resolve the splitbrain (by deleteing all but the healthy copy of the
> file including hardlinks). More so if the file is renamed before
> addressing the split-brain. Please share your thoughs.

I can't help but wonder whether a different kind of replication, perhaps
based on logging instead of transactions, might avoid some of this extra
complexity.  ;)