[Gluster-users] Clarification on cluster quorum

Andrew Lau andrew at andrewklau.com
Sun Mar 16 22:34:54 UTC 2014 

If you look at the red hat docs for their RH storage, you will see quite a
few more options in regards to this quorum. Those 3.2 docs were missing
quite a few options. I'm not at my PC right now, but if I recall correctly
there was a brick count option and a percentage based option (along with a
many others)..

In your case, your situation does sound correct because the quorum is met.
If one of the bricks drops your other one which is connected to the other
arbator peer which is in the higher peer count so can continue to write. If
the other were to drop due to split network, then it will be denied writes
and resync when it reconnects.

I remember reading a presentation where they mentioned, gluster is not
aware of any network boundaries. This concerns me, as if you were to have
redundant paths to nodes, could that not lead to issues during a partial
network split?

I'm yet to properly test this theory, but I plan to do so in the coming
weeks..
On Mar 16, 2014 12:37 PM, "Steve Dainard" <sdainard at miovision.com> wrote:

> How sure are we that this is having the intended effect?
>
> I have two gluster nodes with bricks which are also ovirt hosts, and the
> replicated volume configured as such:
>
> gluster volume info rep1
>
> Volume Name: rep1
> Type: Replicate
> Volume ID: 5876746d-5977-4fa9-a829-69f44b3364ec
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: 10.0.10.2:/mnt/storage/lv-storage-domain/rep1
> Brick2: 10.0.10.3:/mnt/storage/lv-storage-domain/rep1
> Options Reconfigured:
> diagnostics.client-log-level: ERROR
> storage.owner-gid: 36
> storage.owner-uid: 36
> server.allow-insecure: on
> cluster.quorum-type: fixed
> nfs.trusted-sync: off
> nfs.trusted-write: on
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: off
> cluster.eager-lock: enable
> network.remote-dio: enable
> cluster.server-quorum-type: server
>
> I also set the bricks to 1 (cluster.quorum-count 1, although its not
> showing above) as this option seems related to storage bricks, not gluster
> peers.
>
> I peered with a 3rd gluster node which doesn't have any volumes
> configured. I've tested this setup, and the storage doesn't go down when
> one of the nodes is rebooted.
>
> There still seems to be a high potential for split brain here.
>
>
> http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options
>
> cluster.quorum-typeMethod used for quorum enforcement. "None" means no
> quorum enforcement, which is the historical behavior. "Auto" means quorum
> is set to be more than half of the *bricks* in a subvolume, or exactly
> half if that includes the first listed brick. "Fixed" means quorum is set
> to a value specified by cluster.quorum-count<http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options#cluster.quorum-count>.
> If quorum is not met, then modifing operations such as *write* will fail
> with EROFS. This prevents most cases of "split brain" which result from
> conflicting writes to different bricks. Nonecluster.quorum-countNumber of
> subvolumes that must be present to achieve quorum, as described for
> cluster.quorum-type<http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options#cluster.quorum-type>.
> This value is not used unless *cluster.quorum-type* is "fixed". 0
>
> We don't seem able to configure "allow 50% of the bricks to be active, as
> long as 51% of the gluster peers are connected".
>
> I also don't understand why 'subvolumes' are referred to here. Is this old
> terminology for 'volumes'?
>
> Thoughts?
>
> *Steve Dainard *
> IT Infrastructure Manager
> Miovision <http://miovision.com/> | *Rethink Traffic*
>
> *Blog <http://miovision.com/blog>  |  **LinkedIn
> <https://www.linkedin.com/company/miovision-technologies>  |  Twitter
> <https://twitter.com/miovision>  |  Facebook
> <https://www.facebook.com/miovision>*
> ------------------------------
>  Miovision Technologies Inc. | 148 Manitou Drive, Suite 101, Kitchener,
> ON, Canada | N2C 1L3
> This e-mail may contain information that is privileged or confidential. If
> you are not the intended recipient, please delete the e-mail and any
> attachments and notify us immediately.
>
>
> On Mon, Mar 10, 2014 at 7:58 AM, James <purpleidea at gmail.com> wrote:
>
>> A quick search reveals:
>>
>>         # Sets the quorum percentage for the trusted storage pool.
>>         cluster.server-quorum-ratio
>>
>>         # If set to server, enables the specified volume to
>> participate in quorum.
>>         cluster.server-quorum-type
>>
>>         # If quorum-type is "fixed" only allow writes if this many
>> bricks or present. Other quorum types will OVERWRITE this value.
>>         cluster.quorum-count
>>
>>         # If value is "fixed" only allow writes if quorum-count bricks
>> are present. If value is "auto" only allow writes if more than half of
>> bricks, or exactly half including the first, are present.
>>         cluster.quorum-type
>>
>> I took these from my previous "notes" (code) in:
>>
>> https://github.com/purpleidea/puppet-gluster/blob/master/manifests/volume/property/data.pp#L18
>>
>> You can get newer values or appropriate values for your version by
>> running something like:
>>
>> gluster volume set help ( i think )
>>
>> Cheers,
>> James
>>
>>
>> On Mon, Mar 10, 2014 at 2:43 AM, Andrew Lau <andrew at andrewklau.com>
>> wrote:
>> > Thanks again James!
>> >
>> > So far so good, I plan to test this a little more in a few days but so
>> far
>> > it seems the only volume setting I need is:
>> > cluster.server-quorum-type: server
>> >
>> > Default cluster.server-quorum-ratio >50%
>> > So 2 is greater than 1.5.. which should allow writes.
>> >
>> > On Thu, Mar 6, 2014 at 5:00 PM, James <purpleidea at gmail.com> wrote:
>> >>
>> >> Top posting sorry:
>> >>
>> >> Yes, you can add a third "arbiter" node, that exists to help with the
>> >> quorum issues. AFAICT, all you do is peer is with the cluster (as you
>> >> did with the other hosts) but don't add any storage for example.
>> >>
>> >> Then you set the cluster.quorum* style volume settings that you're
>> >> interested. I don't have a list of exactly which ones off the top of
>> >> my head, but if you make a list, let me know!
>> >>
>> >> Cheers,
>> >> James
>> >>
>> >>
>> >> On Wed, Mar 5, 2014 at 10:51 PM, Andrew Lau <andrew at andrewklau.com>
>> wrote:
>> >> > Hi,
>> >> >
>> >> > I'm looking for an option to add an arbiter node to the gluster
>> >> > cluster, but the leads I've been following seem to lead to
>> >> > inconclusive results.
>> >> >
>> >> > The scenario is, a 2 node replicated cluster. What I want to do is
>> >> > introduce a fake host/arbiter node which would set the cluster to a 3
>> >> > node meaning, we can meet the conditions of allow over 50% to write
>> >> > (ie. 2 can write, 1 can not).
>> >> >
>> >> > elyograg from IRC gave me a few links [1], [2]
>> >> > But these appear to be over a year old, and still under review.
>> >> >
>> >> > Gluster 3.2 volume options (I'm running 3.4, but there doesn't seem
>> to
>> >> > be an updated page) [3]
>> >> > seem to state the that the cluster quorum is identified by active
>> >> > peers. This also backs up the statement in [2] in regards to a patch
>> >> > for active volumes rather than cluster peers.
>> >> >
>> >> > Has anyone gone down this path, or could they confirm any of these
>> >> > leads? (ie. does a host w/o any volumes get considered as a peer
>> >> > within the cluster)
>> >> >
>> >> > Thanks,
>> >> > Andrew
>> >> >
>> >> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=914804
>> >> > [2] http://review.gluster.org/#/c/4363/
>> >> > [3]
>> >> >
>> http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options#cluster.quorum-type
>> >> > _______________________________________________
>> >> > Gluster-users mailing list
>> >> > Gluster-users at gluster.org
>> >> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>> >
>> >
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140317/9dc07818/attachment.html>

More information about the Gluster-users mailing list