[Gluster-devel] Automated split-brain resolution

[Pranith Kumar Karampuri]

On 08/12/2014 11:29 AM, Harshavardhana wrote:
>> This is a standard problem where there are split-brains in distributed
>> systems. For example even in git there are cases where it gives up asking
>> users to fix the file i.e. merge conflicts. If the user doesn't want
>> split-brains they should move to replica-3 and enable client-quorum. But if
>> the user made a conscious decision to live with split-brain problems
>> favouring availability/using replica-2, then split-brains do happen and it
>> needs user intervention. All we are trying to do is to make this process a
>> bit painless by coming up with meaningful policies.
>>
> Agreed, split brains do require manual intervention no one argues about that,
> but it shouldn't be quite as tedious as GlusterFS wants it to be.
We are on same page.
>
> I do agree that it is way simpler than perhaps some other distributed
> filesystems but at
> any point we ask some one to write a script to fix our internal
> structure - that is not a
> feature its a bug.
We are not asking them to write a script with this solution.
>
> We all appreciate the effort, but my wish we incorporate some pain
> points which we have
> seen personally over the years and fix it right when we are at it.
>
>> If the user knows his workload is append only and there are split-brains the
>> only command he needs to execute is:
>> 'gluster volume heal <volname> split-brain bigger-file'
>> no grep, no finding file paths, nothing.
>>
> Adding to this - we need to provide additional sanity checks that
> split brains were
> indeed fixed - since this looks quite destructive operation, are you
> planning a rollback
> at any point during this process?
User can find if split-brain is resolved by executing 'gluster volume 
heal <volname> info split-brain'. The file/gfid shouldn't show up there. 
There is no rollback until we integrate it with trash. When we integrate 
it with trash the file that will be over-written/deleted will be moved 
to trash.
>
>> There were also instances where the user knows the brick he/she would like
>> to be the source but he/she is worried that old brick which comes back up
>> would cause split-brains so he/she had to erase the whole brick which was
>> down and bring it back up.
>> Instead we can suggest him/her to use 'gluster volume heal <VOLNAME>
>> split-brain source-brick <brick_name>' after bringing the brick back up so
>> that not all the contents needs to be healed.
>> 1) gluster volume heal <volname> info split-brain should give output in some
>> 'format' giving stat/pending-matrix etc for all the files in split-brain.
>>    - Unfortunately we still don't have a way to provide with file paths
>> without doing 'find' on the bricks.
> Critical setups require fixing split-brain with quick turn around no
> one really has the
> luxury running a find on a large volume. So i still do not understand,
> if a 'find' can do
> a gfid --> inum --> path - how hard it is for Gluster management
> daemon to know this?
> just to provide better tooling?
Not sure. We can figure this out by traversing up the softlinks for 
directories. But for files there is no way to find the parent at the moment.

Pranith
>
> -- Harsha