[Bugs] [Bug 1542380] New: Changes to self-heal logic w.r.t. detecting of split-brains

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1542380

            Bug ID: 1542380
           Summary: Changes to self-heal logic w.r.t. detecting of
                    split-brains
           Product: GlusterFS
           Version: 4.0
         Component: replicate
          Keywords: Triaged
          Assignee: bugs at gluster.org
          Reporter: ravishankar at redhat.com
                CC: bugs at gluster.org, olim at redhat.com
        Depends On: 1539358
            Blocks: 1384983, 1541458



+++ This bug was initially created as a clone of Bug #1539358 +++

Description of problem:
    We currently don't have a roll-back/undoing of post-ops if quorum is not
    met. Though the FOP is still unwound with failure, the xattrs remain on
    the disk.  Due to these partial post-ops and partial heals (healing only
when
    2 bricks are up), we can end up in split-brain purely from the afr
    xattrs point of view i.e each brick is blamed by atleast one of the
    others. These scenarios are hit when there is frequent
    connect/disconnect of the client/shd to the bricks.

--- Additional comment from Worker Ant on 2018-01-28 03:40:54 EST ---

REVIEW: https://review.gluster.org/19349 (afr: don't treat false cases of
split-brain as genuine) posted (#1) for review on master by Ravishankar N

--- Additional comment from Worker Ant on 2018-02-01 09:18:21 EST ---

COMMIT: https://review.gluster.org/19349 committed in master by "Ravishankar N"
<ravishankar at redhat.com> with a commit message- afr: don't treat all cases all
bricks being blamed as split-brain

Problem:
We currently don't have a roll-back/undoing of post-ops if quorum is not
met. Though the FOP is still unwound with failure, the xattrs remain on
the disk.  Due to these partial post-ops and partial heals (healing only when
2 bricks are up), we can end up in split-brain purely from the afr
xattrs point of view i.e each brick is blamed by atleast one of the
others. These scenarios are hit when there is frequent
connect/disconnect of the client/shd to the bricks while I/O or heal
are in progress.

Fix:
Instead of undoing the post-op, pick a source based on the xattr values.
If 2 bricks blame one, the blamed one must be treated as sink.
If there is no majority, all are sources. Once we pick a source,
self-heal will then do the heal instead of erroring out due to
split-brain.

Change-Id: I3d0224b883eb0945785ade0e9697a1c828aec0ae
BUG: 1539358
Signed-off-by: Ravishankar N <ravishankar at redhat.com>


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1384983
[Bug 1384983] split-brain observed with arbiter & replica 3 volume.
https://bugzilla.redhat.com/show_bug.cgi?id=1539358
[Bug 1539358] Changes to self-heal logic w.r.t. detecting of split-brains
https://bugzilla.redhat.com/show_bug.cgi?id=1541458
[Bug 1541458] Changes to self-heal logic w.r.t. detecting of split-brains
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.