[Gluster-users] Fencing FOPs on data-split-brained files

[Ravishankar N]

On 11/16/2013 01:42 AM, Anand Avati wrote:
> Ravi,
> We should not mix up data and entry operation domains, if a file is in 
> data split brain that should not stop a user from rename/link/unlink 
> operations on the file.
>
> Regarding your concern about complications while healing - we should 
> change our "manual fixing" instructions to:
>
> - go to backend, access through gfid path or normal path
> - rmxattr the afr changelogs
> - truncate the file to 0 bytes (like "> filename")
>
> Accessing the path through gfid and truncating to 0 bytes addresses 
> your concerns about hardlinks/renames.
>
> Avati
>
All,

I have tabulated what operations must/ musn't be permitted in case of 
different split brains. Some of the columns are '?' as I am not sure 
what the expected behaviour should be. Could we have this validated?


*File Operation permitted* 	*Type of Split Brain*
*Data SB* 	*Metadata SB* 	*Entry SB*
*
* 	*
* 	*Same entry gfid mismatch SB* 	*Different entries*
write 	No 	Yes (currently no) 	No 	Yes
read 	No 	Yes (currently no) 	No 	Yes
getfattr 	Yes 	No 	No 	Yes
lookup 	? 	? 	No 	Yes
stat/fstat 	? 	? 	No 	Yes
setfattr 	Yes 	No 	No 	Yes
touch 	Yes 	Yes 	No 	Yes
hard link creation 	Yes 	Yes 	No 	Yes
soft link creation 	Yes 	Yes 	Yes 	Yes
rename 	Yes 	Yes 	no 	Yes
chown 	Yes 	Yes 	Currently No 	Yes
chmod 	Yes 	Yes 	Currently No 	Yes
unlink 	Yes 	Yes 	Currently No 	Yes
readdir 	N/A 	N/A 	? 	?


- stat() also reports the file size. If a data split-brained file has 
different sizes, should stat succeed?
- Likewise if metadata split brain is due to different access 
permissions, say one brick has file chmod'ed with 777 and the other 
brick has it with 744, should we allow read/write if the corresponding 
permission bits are *not* conflciting ? ( as of today they aren't allowed)

Also,In the table above, Entry Split brain has 2 cases-
i) where same entry has different gfids
ii) each brick  has different entries for the same directory (which can 
cause deleted files to appear in case of conservative merge).
Should we allow readdir in either case?

Thanks,
Ravi

> On Wed, Nov 13, 2013 at 3:01 AM, Ravishankar N <ravishankar at redhat.com 
> <mailto:ravishankar at redhat.com>> wrote:
>
>     Hi,
>
>     Currenly in glusterfs, when there is a data splt-brain (only) on a
>     file, we disallow the following operations from the mount-point by
>     returning EIO to the application:
>     - Writes to the file (truncate, dd, echo, cp etc)
>     - Reads to the file (cat)
>     - Reading extended attributes (getfattr) [1]
>
>     However we do permit the following operations:
>     -creating hardlinks
>     -creating symlinks
>     -mv
>     -setattr
>     -chmod
>     -chown
>     --touch
>     -ls
>     -stat
>
>     While it makes sense to allow `ls` and `stat`, is it okay to  add
>     checks in the FOPS to disallow the other operations? Allowing
>     creation of links and changing file attributes only seems to
>     complicate things before the admin can go to the backend bricks
>     and resolve the splitbrain (by deleteing all but the healthy copy
>     of the file including hardlinks). More so if the file is renamed
>     before addressing the split-brain.
>     Please share your thoughs.
>
>     Thanks,
>     Ravi
>
>     [1] http://review.gluster.org/#/c/5988/
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>     http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131119/b258ed06/attachment.html>