[Gluster-users] Clarification on cluster quorum

Sun Mar 16 01:37:04 UTC 2014

How sure are we that this is having the intended effect?

I have two gluster nodes with bricks which are also ovirt hosts, and the
replicated volume configured as such:

gluster volume info rep1

Volume Name: rep1
Type: Replicate
Volume ID: 5876746d-5977-4fa9-a829-69f44b3364ec
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.0.10.2:/mnt/storage/lv-storage-domain/rep1
Brick2: 10.0.10.3:/mnt/storage/lv-storage-domain/rep1
Options Reconfigured:
diagnostics.client-log-level: ERROR
storage.owner-gid: 36
storage.owner-uid: 36
server.allow-insecure: on
cluster.quorum-type: fixed
nfs.trusted-sync: off
nfs.trusted-write: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.server-quorum-type: server

I also set the bricks to 1 (cluster.quorum-count 1, although its not
showing above) as this option seems related to storage bricks, not gluster
peers.

I peered with a 3rd gluster node which doesn't have any volumes configured.
I've tested this setup, and the storage doesn't go down when one of the
nodes is rebooted.

There still seems to be a high potential for split brain here.

http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options

cluster.quorum-typeMethod used for quorum enforcement. "None" means no
quorum enforcement, which is the historical behavior. "Auto" means quorum
is set to be more than half of the *bricks* in a subvolume, or exactly half
if that includes the first listed brick. "Fixed" means quorum is set to a
value specified by
cluster.quorum-count<http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options#cluster.quorum-count>.
If quorum is not met, then modifing operations such as *write* will fail
with EROFS. This prevents most cases of "split brain" which result from
conflicting writes to different bricks.Nonecluster.quorum-countNumber of
subvolumes that must be present to achieve quorum, as described for
cluster.quorum-type<http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options#cluster.quorum-type>.
This value is not used unless *cluster.quorum-type* is "fixed".0

We don't seem able to configure "allow 50% of the bricks to be active, as
long as 51% of the gluster peers are connected".

I also don't understand why 'subvolumes' are referred to here. Is this old
terminology for 'volumes'?

Thoughts?

*Steve Dainard *
IT Infrastructure Manager
Miovision <http://miovision.com/> | *Rethink Traffic*

*Blog <http://miovision.com/blog>  |  **LinkedIn
<https://www.linkedin.com/company/miovision-technologies>  |  Twitter
<https://twitter.com/miovision>  |  Facebook
<https://www.facebook.com/miovision>*
------------------------------
 Miovision Technologies Inc. | 148 Manitou Drive, Suite 101, Kitchener, ON,
Canada | N2C 1L3
This e-mail may contain information that is privileged or confidential. If
you are not the intended recipient, please delete the e-mail and any
attachments and notify us immediately.

On Mon, Mar 10, 2014 at 7:58 AM, James <purpleidea at gmail.com> wrote:

> A quick search reveals:
>
>         # Sets the quorum percentage for the trusted storage pool.
>         cluster.server-quorum-ratio
>
>         # If set to server, enables the specified volume to
> participate in quorum.
>         cluster.server-quorum-type
>
>         # If quorum-type is "fixed" only allow writes if this many
> bricks or present. Other quorum types will OVERWRITE this value.
>         cluster.quorum-count
>
>         # If value is "fixed" only allow writes if quorum-count bricks
> are present. If value is "auto" only allow writes if more than half of
> bricks, or exactly half including the first, are present.
>         cluster.quorum-type
>
> I took these from my previous "notes" (code) in:
>
> https://github.com/purpleidea/puppet-gluster/blob/master/manifests/volume/property/data.pp#L18
>
> You can get newer values or appropriate values for your version by
> running something like:
>
> gluster volume set help ( i think )
>
> Cheers,
> James
>
>
> On Mon, Mar 10, 2014 at 2:43 AM, Andrew Lau <andrew at andrewklau.com> wrote:
> > Thanks again James!
> >
> > So far so good, I plan to test this a little more in a few days but so
> far
> > it seems the only volume setting I need is:
> > cluster.server-quorum-type: server
> >
> > Default cluster.server-quorum-ratio >50%
> > So 2 is greater than 1.5.. which should allow writes.
> >
> > On Thu, Mar 6, 2014 at 5:00 PM, James <purpleidea at gmail.com> wrote:
> >>
> >> Top posting sorry:
> >>
> >> Yes, you can add a third "arbiter" node, that exists to help with the
> >> quorum issues. AFAICT, all you do is peer is with the cluster (as you
> >> did with the other hosts) but don't add any storage for example.
> >>
> >> Then you set the cluster.quorum* style volume settings that you're
> >> interested. I don't have a list of exactly which ones off the top of
> >> my head, but if you make a list, let me know!
> >>
> >> Cheers,
> >> James
> >>
> >>
> >> On Wed, Mar 5, 2014 at 10:51 PM, Andrew Lau <andrew at andrewklau.com>
> wrote:
> >> > Hi,
> >> >
> >> > I'm looking for an option to add an arbiter node to the gluster
> >> > cluster, but the leads I've been following seem to lead to
> >> > inconclusive results.
> >> >
> >> > The scenario is, a 2 node replicated cluster. What I want to do is
> >> > introduce a fake host/arbiter node which would set the cluster to a 3
> >> > node meaning, we can meet the conditions of allow over 50% to write
> >> > (ie. 2 can write, 1 can not).
> >> >
> >> > elyograg from IRC gave me a few links [1], [2]
> >> > But these appear to be over a year old, and still under review.
> >> >
> >> > Gluster 3.2 volume options (I'm running 3.4, but there doesn't seem to
> >> > be an updated page) [3]
> >> > seem to state the that the cluster quorum is identified by active
> >> > peers. This also backs up the statement in [2] in regards to a patch
> >> > for active volumes rather than cluster peers.
> >> >
> >> > Has anyone gone down this path, or could they confirm any of these
> >> > leads? (ie. does a host w/o any volumes get considered as a peer
> >> > within the cluster)
> >> >
> >> > Thanks,
> >> > Andrew
> >> >
> >> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=914804
> >> > [2] http://review.gluster.org/#/c/4363/
> >> > [3]
> >> >
> http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options#cluster.quorum-type
> >> > _______________________________________________
> >> > Gluster-users mailing list
> >> > Gluster-users at gluster.org
> >> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> >
> >
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140315/6ef30541/attachment.html>