[Gluster-devel] Split-brain present and future in afr

[Jeff Darcy]

> 1. Better protection for split-brain over time.
> 2. Policy based split-brain resolution.
> 3. Provide better availability with client quorum and replica 2.

I would add the following:

(4) Quorum enforcement - any kind - on by default.

(5) Fix the problem of volumes losing quorum because unrelated nodes
    went down (i.e. implement volume-level quorum).

(6) Better tools for users to resolve split brain themselves.

> For 3, we are planning to introduce arbiter bricks that can be used to
> determine quorum. The arbiter bricks will be dummy bricks that host only
> files that will be updated from multiple clients. This will be achieved by
> bringing about variable replication count for configurable class of files
> within a volume.
>  In the case of a replicated volume with one arbiter brick per replica group,
>  certain files that are prone to split-brain will be in 3 bricks (2 data
>  bricks + 1 arbiter brick).  All other files will be present in the regular
>  data bricks. For example, when oVirt VM disks are hosted on a replica 2
>  volume, sanlock is used by oVirt for arbitration. sanloclk lease files will
>  be written by all clients and VM disks are written by only a single client
>  at any given point of time. In this scenario, we can place sanlock lease
>  files on 2 data + 1 arbiter bricks. The VM disk files will only be present
>  on the 2 data bricks. Client quorum is now determined by looking at 3
>  bricks instead of 2 and we have better protection when network split-brains
>  happen.

Constantly filtering requests to use either N or N+1 bricks is going to be
complicated and hard to debug.  Every data-structure allocation or loop
based on replica count will have to be examined, and many will have to be
modified.  That's a *lot* of places.  This also overlaps significantly
with functionality that can be achieved with data classification (i.e.
supporting multiple replica levels within the same volume).  What use case
requires that it be implemented within AFR instead of more generally and
flexibly?