[Bugs] [Bug 1294410] New: Friend update floods can render the cluster incapable of handling other commands

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1294410

            Bug ID: 1294410
           Summary: Friend update floods can render the cluster incapable
                    of handling other commands
           Product: GlusterFS
           Version: 3.7.6
         Component: glusterd
          Assignee: bugs at gluster.org
          Reporter: amukherj at redhat.com
                CC: amukherj at redhat.com, bugs at gluster.org,
                    gluster-bugs at redhat.com, kaushal at redhat.com,
                    sasundar at redhat.com
        Depends On: 1292749
            Blocks: 1291386



+++ This bug was initially created as a clone of Bug #1292749 +++

A flood of glusterd friend updates happen whenever a glusterd restarts and
re-establishes all it's connections.

In a large cluster (100s) nodes, this would go on for several minutes. During
this period the cluster isn't able to respond to commands. Simple local
commands, like `gluster volume list` will take relatively very long time to
complete.

When a large number of nodes come back up simultaneously, say due to a network
problem, this flood can last for a long time, longer than expected.

--- Additional comment from Vijay Bellur on 2015-12-18 04:44:02 EST ---

REVIEW: http://review.gluster.org/12999 (glusterd: reduce friend update flood)
posted (#1) for review on master by Kaushal M (kaushal at redhat.com)

--- Additional comment from Vijay Bellur on 2015-12-22 00:09:18 EST ---

REVIEW: http://review.gluster.org/12999 (glusterd: reduce friend update flood)
posted (#2) for review on master by Kaushal M (kaushal at redhat.com)

--- Additional comment from Vijay Bellur on 2015-12-22 22:52:32 EST ---

COMMIT: http://review.gluster.org/12999 committed in master by Atin Mukherjee
(amukherj at redhat.com) 
------
commit f624abd6885752eeaa8d07101ff00f52af48de26
Author: Kaushal M <kaushal at redhat.com>
Date:   Thu Dec 17 11:13:36 2015 +0530

    glusterd: reduce friend update flood

    When in a befriended state, glusterd would broadcast friend updates to
    all other peers whenver a ACC or LOCAL_ACC event occurred.

    When a downed glusterd came back up and established connections again,
    this lead to a flood of friend updates to happen on the order of N^2 (N
    is the number of peers in the cluster)

    In larger clusters this was problematic, and could lead to very long
    times for the cluster to settle down when a peer came back up. Multiple
    peers coming back up at the same time would compound the problem.

    Broadcasting of friend updates doesn't have much use in places other
    that during a peer probe. Instead of broadcasting friend updates on
    connection re-establishment, updates can just be exchanged between the
    peers involved in the connection.

    This patch changes the glusterd friend state-machine to send updates
    only to the required peer for ACC or LOCAL_ACC events when in befriended
    state. The number of updates sent now is in the order of N.

    For a 10 node cluster, the number of updates reduced by 5 times. When
    creating the 10 node cluster, the updates reduced from ~500 to ~150.
    When a glusterd restarted, the number of exchanges reduced from ~160 to
    ~35.

    BUG: 1292749
    Change-Id: Ib6072090c7069b081d018cdaa3dc878819ab1d18
    Signed-off-by: Kaushal M <kaushal at redhat.com>
    Reviewed-on: http://review.gluster.org/12999
    Reviewed-by: Atin Mukherjee <amukherj at redhat.com>
    Tested-by: NetBSD Build System <jenkins at build.gluster.org>
    Tested-by: Gluster Build System <jenkins at build.gluster.com>


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1292749
[Bug 1292749] Friend update floods can render the cluster incapable of
handling other commands
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.