[Bugs] [Bug 1353814] Bricks are starting when server quorum not met.

Fri Jul 8 05:12:44 UTC 2016

https://bugzilla.redhat.com/show_bug.cgi?id=1353814

Samikshan Bairagya <sbairagy at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|bugs at gluster.org            |sbairagy at redhat.com

--- Comment #1 from Samikshan Bairagya <sbairagy at redhat.com> ---
+++ This bug was initially created as a clone of Bug #1345727 +++

+++ This bug was initially created as a clone of Bug #1340995 +++

Description of problem:
=======================
volume bricks are starting when the server quorum not met. 

Version-Release number of selected component (if applicable):
=============================================================
glusterfs-3.7.9-6.

How reproducible:
=================
Always

Steps to Reproduce:
===================
1. Have three node cluster (n1, n2 and n3)
2. Create a distribute volume using all three node bricks. (3 brick volume )
3. Enable the server side quotum //gluster volume set <vol_name>
cluster.server-quorum-type server
4. stop glusterd in n2 and n3 nodes.
5. Now setup will be in server quorum not met condition //check using volume
status
6. Change the cluster.server-quorum-ratio from default to 95
7. Start glusterd on n2 node
8. Check volume status on n1 and n2 nodes  //you will see the bricks part of n2
node online.

Actual results:
===============
Bricks are starting when server quorum not met.

Expected results:
=================
Bricks should not start when server quorum not met.

Additional info: ( info in /var/log/messages)
=================
On node2:
=========
May 31 00:38:01 dhcp43-216 systemd: Starting GlusterFS, a clustered file-system
server...
May 31 00:38:04 dhcp43-216 etc-glusterfs-glusterd.vol[20626]: [2016-05-31
04:38:04.330963] C [MSGID: 106003]
[glusterd-server-quorum.c:346:glusterd_do_volume_quorum_action] 0-management:
Ser
ver quorum regained for volume Dis. Starting local bricks.
May 31 00:38:04 dhcp43-216 systemd: Started GlusterFS, a clustered file-system
server.

On Node1:
=========
May 31 00:36:01 dhcp43-215 systemd: Starting Session 6710 of user root.
May 31 00:36:54 dhcp43-215 etc-glusterfs-glusterd.vol[12032]: [2016-05-31
04:36:54.296022] C [MSGID: 106002]
[glusterd-server-quorum.c:351:glusterd_do_volume_quorum_action] 0-management:
Ser
ver quorum lost for volume Dis. Stopping local bricks.
May 31 00:37:01 dhcp43-215 systemd: Started Session 6711 of user root.
May 31 00:37:01 dhcp43-215 systemd: Starting Session 6711 of user root.

will provide the console logs

--- Additional comment from Atin Mukherjee on 2016-05-31 02:03:44 EDT ---

This is indeed a bug. Since the volume version has been changed while setting
the server side quorum when N2 comes up it will go for importing the volume
from other nodes. On that code path GlusterD invokes glusterd_start_bricks ()
which never checks for quorum and resulting into starting the bricks.

--- Additional comment from Atin Mukherjee on 2016-06-15 06:30:48 EDT ---

My analysis is incorrect here. Since cluster.server-quorum-ratio is applicable
for all the volumes the volume's version will not get incremented and we will
not eventually hit the code path of importing the volume. We'd need to find out
a way to take a decision whether to start/stop the brick(s) based on when this
global option is synced.

--- Additional comment from Vijay Bellur on 2016-06-17 06:21:21 EDT ---

REVIEW: http://review.gluster.org/14758 (Make sure bricks are not started if
server quorum is not met) posted (#1) for review on master by Samikshan
Bairagya (samikshan at gmail.com)

--- Additional comment from Vijay Bellur on 2016-06-17 08:13:27 EDT ---

REVIEW: http://review.gluster.org/14758 (Make sure bricks are not started if
server quorum is not met) posted (#2) for review on master by Samikshan
Bairagya (samikshan at gmail.com)

--- Additional comment from Vijay Bellur on 2016-06-17 08:18:37 EDT ---

REVIEW: http://review.gluster.org/14758 (Make sure bricks are not started if
server quorum is not met) posted (#3) for review on master by Samikshan
Bairagya (samikshan at gmail.com)

--- Additional comment from Vijay Bellur on 2016-06-22 01:22:17 EDT ---

REVIEW: http://review.gluster.org/14758 (glusterd: Don't start bricks if server
quorum is not met) posted (#4) for review on master by Samikshan Bairagya
(samikshan at gmail.com)

--- Additional comment from Vijay Bellur on 2016-06-23 01:51:23 EDT ---

REVIEW: http://review.gluster.org/14758 (glusterd: Don't start bricks if server
quorum is not met) posted (#5) for review on master by Samikshan Bairagya
(samikshan at gmail.com)

--- Additional comment from Vijay Bellur on 2016-07-04 05:07:53 EDT ---

REVIEW: http://review.gluster.org/14758 (glusterd: Don't start bricks if server
quorum is not met) posted (#6) for review on master by Samikshan Bairagya
(samikshan at gmail.com)

--- Additional comment from Vijay Bellur on 2016-07-05 07:57:02 EDT ---

COMMIT: http://review.gluster.org/14758 committed in master by Jeff Darcy
(jdarcy at redhat.com) 
------
commit 807b9a135d697f175fc9933f1d23fb67b0cc6c7d
Author: Samikshan Bairagya <samikshan at gmail.com>
Date:   Tue Jun 14 10:52:27 2016 +0530

    glusterd: Don't start bricks if server quorum is not met

    Upon glusterd restart if it is observered that the server quorum
    isn't met anymore due to changes to the "server-quorum-ratio"
    global option, the bricks should be stopped if they are running.
    Also if glusterd has been restarted, and if server quorum is not
    applicable for a volume, do not restart the bricks corresponding
    to the volume to make sure that bricks that have been brought
    down purposely, say for maintenance, are not brought up. This
    commit moves this check that was previously inside
    "glusterd_spawn_daemons" to "glusterd_restart_bricks" instead.

    Change-Id: I0a44a2e7cad0739ed7d56d2d67ab58058716de6b
    BUG: 1345727
    Signed-off-by: Samikshan Bairagya <samikshan at gmail.com>
    Reviewed-on: http://review.gluster.org/14758
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    Reviewed-by: Jeff Darcy <jdarcy at redhat.com>

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.