[Bugs] [Bug 1420983] New: Automatic split brain resolution must check for all the bricks to be up to avoiding serving of inconsistent data (visible on x3 or more)

Fri Feb 10 04:17:37 UTC 2017

https://bugzilla.redhat.com/show_bug.cgi?id=1420983

            Bug ID: 1420983
           Summary: Automatic split brain resolution must check for all
                    the bricks to be up to avoiding serving of
                    inconsistent data(visible on x3 or more)
           Product: GlusterFS
           Version: 3.9
         Component: replicate
          Severity: urgent
          Assignee: bugs at gluster.org
          Reporter: ravishankar at redhat.com
                CC: bugs at gluster.org, ravishankar at redhat.com
        Depends On: 1417177, 1417522
            Blocks: 1420982

+++ This bug was initially created as a clone of Bug #1417522 +++

+++ This bug was initially created as a clone of Bug #1417177 +++

Description of problem:
======================
Automatic split brain resolution must come into effect only when all the bricks
are up, else we would be serving inconsistent or undesired data as explained
below

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. create a 1x3 volume (clientside quorum is enabled by default) with say b1,
b2 ,b3
also set favorite child policy to say mtime(automatic resolution of splitbrain)

2. fuse mount the volume on three different clients in below fashion
c1: can ping only b1, b2 bricks and not b3
c2: can ping only b2,b3 and not b1
c3: can ping all bricks

3. now create a file say f1 from c3 ==>that means c3 is now Available on all
bricks
4. now append  from c1 say line-c1 and from c2 line-c2 to file f1
 that means b2 will mark b1 pending with line-c2 
            b2 will also mark b2 pending with line-c1

that means b2 has the only good copy

5. Now bring down b2 
6. heal info will now show f1 as in splitbrain as b1 blames b3 and b3 blames b1

Ideally the file should now give IO error for new writes
7. however that means automatic splitbrain resolution will pick this file f1
for resolving.
But that is wrong as the good copy is on b2 which is down.

With the resolving users can now access the file f1 which must not actually be
allowed, as this means the contents on the actual good copy are lost when b2
comes back up, as that is healed because now b1 and b3 blame b2

expected behvior:
1)b2 has the good copy which is down, hence not further writes must be allowed
2) when b2 comes back up, it must be soruce to b1 and b3 instead of healing via
automatic splitbrain and marking b2 as bad copy

Solution:
make sure automatic splitbrain doesnt take effect on afr replica set when even
one  of the bricks are down

Actual results:

Expected results:

Additional info:

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-01-27
07:23:49 EST ---

This bug is automatically being proposed for the current release of Red Hat
Gluster Storage 3 under active development, by setting the release flag
'rhgs‑3.2.0' to '?'. 

If this bug should be proposed for a different release, please manually change
the proposed release flag.

--- Additional comment from Worker Ant on 2017-01-29 23:33:02 EST ---

REVIEW: https://review.gluster.org/16476 (afr: all children of AFR must be up
to resolve s-brain) posted (#1) for review on master by Ravishankar N
(ravishankar at redhat.com)

--- Additional comment from Worker Ant on 2017-02-07 07:07:30 EST ---

REVIEW: https://review.gluster.org/16476 (afr: all children of AFR must be up
to resolve s-brain) posted (#2) for review on master by Ravishankar N
(ravishankar at redhat.com)

--- Additional comment from Worker Ant on 2017-02-08 07:39:52 EST ---

REVIEW: https://review.gluster.org/16476 (afr: all children of AFR must be up
to resolve s-brain) posted (#3) for review on master by Ravishankar N
(ravishankar at redhat.com)

--- Additional comment from Worker Ant on 2017-02-09 20:37:04 EST ---

COMMIT: https://review.gluster.org/16476 committed in master by Pranith Kumar
Karampuri (pkarampu at redhat.com) 
------
commit 0e03336a9362e5717e561f76b0c543e5a197b31b
Author: Ravishankar N <ravishankar at redhat.com>
Date:   Mon Jan 30 09:54:16 2017 +0530

    afr: all children of AFR must be up to resolve s-brain

    Problem:
    The various split-brain resolution policies (favorite-child-policy based,
    CLI based and mount (get/setfattr) based) attempt to resolve split-brain
    even when not all bricks of replica are up. This can be a problem when
    say in a replica 3, the only good copy is down and the other 2 bricks
    are up and blame each other (i.e. split-brain). We end up healing the
    file in such a  case and allow I/O on it.

    Fix:
    A decision on whether the file is in split-brain or not must be taken
    only if we are able to examine the afr xattrs of *all* bricks of a given
    replica.

    Change-Id: Icddb1268b380005799990f5379ef957d84639ef9
    BUG: 1417522
    Signed-off-by: Ravishankar N <ravishankar at redhat.com>
    Reviewed-on: https://review.gluster.org/16476
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu at redhat.com>

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1417177
[Bug 1417177] Split brain resolution must check for all the bricks to be up
to avoiding serving of inconsistent data(visible on x3 or more)
https://bugzilla.redhat.com/show_bug.cgi?id=1417522
[Bug 1417522] Automatic split brain resolution must check for all the
bricks to be up to avoiding serving of inconsistent data(visible on x3 or
more)
https://bugzilla.redhat.com/show_bug.cgi?id=1420982
[Bug 1420982] Automatic split brain resolution must check for all the
bricks to be up to avoiding serving of inconsistent data(visible on x3 or
more)
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.