[Gluster-devel] Automated split-brain resolution

[Ravishankar N]

On 08/09/2014 01:23 AM, Joe Julian wrote:
> Thinking about it more, I'd still rather have this functionality 
> exposed at the client through xattrs. For 5 years I've thought about 
> this, and the more I encounter split-brain, the more I think this is 
> the needed approach.
>

Joe, why do you feel resolving split-brains should be exposed to 
clients? Whatever approach is taken (either a gluster  CLI command or an 
overloaded get/satfattr call, is it not better to have this done at the 
server side?)


> "getfattr -n trusted.glusterfs.stat" returns 
> xml/json/some_madeup_datastructure with the results of stat from each 
> brick
> "getfattr -n trusted.glusterfs.afr" returns the afr matrix
> "setfattr -n trusted.glusterfs.sb-pick -v "server2:/srv/brick1"
>
> That gives us the tools we need to choose what to do with any given 
> split-brain. For large swaths of automated repair, we can use find.
>
> I suppose that last bit could still be implemented through that cli 
> command.
>
>
> On 08/07/2014 01:35 AM, Ravishankar N wrote:
>>
>> Manual resolution of split-brains [1] has been a tedious task 
>> involving understanding and modifying AFR's changelog extended 
>> attributes. To simplify and to an extent automate this task, we are 
>> proposing a new CLI command with which the user can specify  what the 
>> source brick/file is, and automatically heal the files in the 
>> appropriate direction.
>>
>> Command: gluster volume resolve-split-brain <VOLNAME> {<bigger_file>  
>> |  source-brick <brick_name> [<file>] }
>>
>> Breaking up the command into its possible options, we have:
>>
>> a) gluster volume resolve-split-brain <VOLNAME> <bigger_file>
>> When this command is executed, AFR will consider the brick having the 
>> highest file size as the source and heal it to all other bricks 
>> (including all other sources and sinks) in that replica subvolume. If 
>> the file size is same in all the bricks, it does *not* heal the file.
>>
>> b) gluster volume resolve-split-brain <VOLNAME > source-brick 
>> <brick_name > [<file>]
>>
>> When this command is executed, if <file> is specified, AFR heals the 
>> file from the source-brick <brick_name> to all other bricks of that 
>> replica subvolume. For resolving multiple files, the command must be 
>> run iteratively, once per file.
>> If <file> is not specified, AFR heals all the files that have an 
>> entry in .glusterfs/indices/xattrop *and* are in split-brain. As 
>> before, heals happen from source-brick <brick_name> to all other bricks.
>>
>> Future work could also include extending the command to add other 
>> policies like choosing the file having the latest mtime as the 
>> source, integration with trash xlator wherein the files deleted from 
>> the sink are moved to the trash dir etc.
>>
>> Please give feedback on the above.
>>
>> Regards,
>> Ravi
>>
>> [1] https://github.com/gluster/glusterfs/blob/master/doc/split-brain.md
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20140811/f240b8dd/attachment-0001.html>