[Bugs] [Bug 1292749] Friend update floods can render the cluster incapable of handling other commands
bugzilla at redhat.com
bugzilla at redhat.com
Wed Dec 23 03:52:32 UTC 2015
https://bugzilla.redhat.com/show_bug.cgi?id=1292749
--- Comment #3 from Vijay Bellur <vbellur at redhat.com> ---
COMMIT: http://review.gluster.org/12999 committed in master by Atin Mukherjee
(amukherj at redhat.com)
------
commit f624abd6885752eeaa8d07101ff00f52af48de26
Author: Kaushal M <kaushal at redhat.com>
Date: Thu Dec 17 11:13:36 2015 +0530
glusterd: reduce friend update flood
When in a befriended state, glusterd would broadcast friend updates to
all other peers whenver a ACC or LOCAL_ACC event occurred.
When a downed glusterd came back up and established connections again,
this lead to a flood of friend updates to happen on the order of N^2 (N
is the number of peers in the cluster)
In larger clusters this was problematic, and could lead to very long
times for the cluster to settle down when a peer came back up. Multiple
peers coming back up at the same time would compound the problem.
Broadcasting of friend updates doesn't have much use in places other
that during a peer probe. Instead of broadcasting friend updates on
connection re-establishment, updates can just be exchanged between the
peers involved in the connection.
This patch changes the glusterd friend state-machine to send updates
only to the required peer for ACC or LOCAL_ACC events when in befriended
state. The number of updates sent now is in the order of N.
For a 10 node cluster, the number of updates reduced by 5 times. When
creating the 10 node cluster, the updates reduced from ~500 to ~150.
When a glusterd restarted, the number of exchanges reduced from ~160 to
~35.
BUG: 1292749
Change-Id: Ib6072090c7069b081d018cdaa3dc878819ab1d18
Signed-off-by: Kaushal M <kaushal at redhat.com>
Reviewed-on: http://review.gluster.org/12999
Reviewed-by: Atin Mukherjee <amukherj at redhat.com>
Tested-by: NetBSD Build System <jenkins at build.gluster.org>
Tested-by: Gluster Build System <jenkins at build.gluster.com>
--
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=zOEYU7MR5M&a=cc_unsubscribe
More information about the Bugs
mailing list