[Bugs] [Bug 1320020] New: add-brick on a replicate volume could lead to data-loss

Tue Mar 22 07:00:31 UTC 2016

https://bugzilla.redhat.com/show_bug.cgi?id=1320020

            Bug ID: 1320020
           Summary: add-brick on a replicate volume could lead to
                    data-loss
           Product: GlusterFS
           Version: 3.7.9
         Component: replicate
          Priority: high
          Assignee: bugs at gluster.org
          Reporter: atalur at redhat.com
                CC: bugs at gluster.org, sabose at redhat.com,
                    sasundar at redhat.com
        Depends On: 1276203
            Blocks: 1248998, 1258386 (Gluster-HC-1)

+++ This bug was initially created as a clone of Bug #1276203 +++

Description of problem:
On increasing the replica count of a replicate volume (by add-brick command),
self-heal upon failure of a fop on old-brick and success on newly added brick
could lead to reverse heal and hence data loss.

Pending xattrs should be marked indicating the new brick doesn't have the
latest copy of data yet.
Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

--- Additional comment from Vijay Bellur on 2015-10-29 01:18:04 EDT ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when
replica count increases) posted (#1) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2015-10-29 03:40:55 EDT ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when
replica count increases) posted (#2) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 01:18:27 EST ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when
replica count increases) posted (#3) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 01:18:30 EST ---

REVIEW: http://review.gluster.org/12454 (afr :  Enable auto heal when replica
count increases) posted (#3) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-02-03 02:35:54 EST ---

REVIEW: http://review.gluster.org/12454 (afr :  Enable auto heal when replica
count increases) posted (#4) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-02-23 00:29:21 EST ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when
replica count increases) posted (#4) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-02-23 00:29:24 EST ---

REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica
count increases) posted (#5) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-02-23 01:43:38 EST ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when
replica count increases) posted (#5) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-02-23 01:43:41 EST ---

REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica
count increases) posted (#6) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-02-29 00:14:15 EST ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when
replica count increases) posted (#6) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-03-02 02:32:46 EST ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when
replica count increases) posted (#7) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-03-03 03:55:01 EST ---

REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica
count increases) posted (#7) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-03-03 07:27:31 EST ---

REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica
count increases) posted (#8) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-03-14 04:25:47 EDT ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when
replica count increases) posted (#8) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-03-14 04:25:50 EDT ---

REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica
count increases) posted (#9) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-03-16 01:25:46 EDT ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when
replica count increases) posted (#9) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-03-17 02:18:59 EDT ---

REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica
count increases) posted (#10) for review on master by Anuradha Talur
(atalur at redhat.com)

--- Additional comment from Vijay Bellur on 2016-03-21 13:51:11 EDT ---

COMMIT: http://review.gluster.org/12451 committed in master by Atin Mukherjee
(amukherj at redhat.com) 
------
commit 020bc022c342c4c015e29c63399757e36d653a49
Author: Anuradha Talur <atalur at redhat.com>
Date:   Wed Mar 16 10:55:09 2016 +0530

    glusterd / afr : Enable auto heal when replica count increases

    In replicate volumes, when a brick is added to a replicate
    group, heal to the new brick should be triggered.
    Also, the new brick should not be considered as source for
    healing till it is up to date.

    Previously, extended attributes had to be set manually on
    the bricks for this to happen. This patch is part 1 patch
    to automate this process.

    Change-Id: I29958448618372bfde23bf1dac5dd23dba1ad98f
    BUG: 1276203
    Signed-off-by: Anuradha Talur <atalur at redhat.com>
    Reviewed-on: http://review.gluster.org/12451
    Reviewed-by: Atin Mukherjee <amukherj at redhat.com>
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
    Reviewed-by: Ravishankar N <ravishankar at redhat.com>
    Smoke: Gluster Build System <jenkins at build.gluster.com>

--- Additional comment from Vijay Bellur on 2016-03-22 01:37:28 EDT ---

COMMIT: http://review.gluster.org/12454 committed in master by Pranith Kumar
Karampuri (pkarampu at redhat.com) 
------
commit 8eaa3506ead4f11b81b146a9e56575c79f3aad7b
Author: Anuradha Talur <atalur at redhat.com>
Date:   Tue Feb 23 10:56:51 2016 +0530

    afr : Enable auto heal when replica count increases

    This patch is part two change to prevent data loss
    in a replicate volume on doing a add-brick operation.

    Problem: After doing add-brick, there is a chance
    that self heal might happen from the newly added
    brick rather than the source brick, leading to data loss.

    Solution: Mark pending changelogs on afr children for
    the new afr-child so that heal is performed in the
    correct direction.

    Change-Id: I11871e55eef3593aec874f92214a2d97da229b17
    BUG: 1276203
    Signed-off-by: Anuradha Talur <atalur at redhat.com>
    Reviewed-on: http://review.gluster.org/12454
    Smoke: Gluster Build System <jenkins at build.gluster.com>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu at redhat.com>
    Tested-by: Pranith Kumar Karampuri <pkarampu at redhat.com>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1248998
[Bug 1248998] [AFR]: Files not available in the mount point after
converting Distributed volume type to Replicated one.
https://bugzilla.redhat.com/show_bug.cgi?id=1258386
[Bug 1258386] [TRACKER] Gluster Hyperconvergence - Phase 1
https://bugzilla.redhat.com/show_bug.cgi?id=1276203
[Bug 1276203] add-brick on a replicate volume could lead to data-loss
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.