[Gluster-users] Experimenting with thin-arbiter

[Diego Zuccato]

Hello all.

I'm experimenting with thin-arbiter and getting disappointing results.

I have 3 hosts in the trusted pool:
root at nas1:~# gluster --version
glusterfs 9.2
[...]
root at nas1:~# gluster pool list
UUID                                    Hostname        State
d4791fed-3e6d-4f8f-bdb6-4e0043610ead    nas3            Connected
bff398f0-9d1d-4bd0-8a47-0bf481d1d593    nas2            Connected
4607034c-919d-4675-b5fc-14e1cad90214    localhost       Connected

When I try to create a new volume, the first initialization succeeds:
root at nas1:~# gluster v create Bck replica 2 thin-arbiter 1 
nas{1,3}:/bricks/00/Bck nas2:/bricks/arbiter/Bck
volume create: Bck: success: please start the volume to access data

But adding a second brick segfaults the daemon:
root at nas1:~# gluster v add-brick Bck nas{1,3}:/bricks/01/Bck
Connection failed. Please check if gluster daemon is operational.

After erroring out, systemctl status glusterd reports daemon in 
"restarting" state and it eventually restarts. But the new brick is not 
added to the volume, even if trying to re-add it yelds a "brick is 
already part of a volume" error. Seems glusterd crashes between marking 
brick dir as used and recording its data in the config.

If I try to add all the bricks during the creation, glusterd does not 
die but the volume doesn't get created:
root at nas1:~# rm -rf /bricks/{00..07}/Bck && mkdir /bricks/{00..07}/Bck
root at nas1:~# gluster v create Bck replica 2 thin-arbiter 1 
nas{1,3}:/bricks/00/Bck nas{1,3}:/bricks/01/Bck nas{1,3}:/bricks/02/Bck 
nas{1,3}:/bricks/03/Bck nas{1,3}:/bricks/04/Bck nas{1,3}:/bricks/05/Bck 
nas{1,3}:/bricks/06/Bck nas{1,3}:/bricks/07/Bck nas2:/bricks/arbiter/Bck
volume create: Bck: failed: Commit failed on localhost. Please check the 
log file for more details.

Couldn't find anything useful in the logs :(

If I create a "replica 3 arbiter 1" over the same brick directories 
(just adding some directories to keep arbiters separated), it succeeds:
root at nas1:~# gluster v create Bck replica 3 arbiter 1 
nas{1,3}:/bricks/00/Bck nas2:/bricks/arbiter/Bck/00
volume create: Bck: success: please start the volume to access data
root at nas1:~# for T in {01..07}; do gluster v add-brick Bck 
nas{1,3}:/bricks/$T/Bck nas2:/bricks/arbiter/Bck/$T ; done
volume add-brick: success
volume add-brick: success
volume add-brick: success
volume add-brick: success
volume add-brick: success
volume add-brick: success
volume add-brick: success
root at nas1:~# gluster v start Bck
volume start: Bck: success
root at nas1:~# gluster v info Bck

Volume Name: Bck
Type: Distributed-Replicate
Volume ID: 4786e747-8203-42bf-abe8-107a50b238ee
Status: Started
Snapshot Count: 0
Number of Bricks: 8 x (2 + 1) = 24
Transport-type: tcp
Bricks:
Brick1: nas1:/bricks/00/Bck
Brick2: nas3:/bricks/00/Bck
Brick3: nas2:/bricks/arbiter/Bck/00 (arbiter)
Brick4: nas1:/bricks/01/Bck
Brick5: nas3:/bricks/01/Bck
Brick6: nas2:/bricks/arbiter/Bck/01 (arbiter)
Brick7: nas1:/bricks/02/Bck
Brick8: nas3:/bricks/02/Bck
Brick9: nas2:/bricks/arbiter/Bck/02 (arbiter)
Brick10: nas1:/bricks/03/Bck
Brick11: nas3:/bricks/03/Bck
Brick12: nas2:/bricks/arbiter/Bck/03 (arbiter)
Brick13: nas1:/bricks/04/Bck
Brick14: nas3:/bricks/04/Bck
Brick15: nas2:/bricks/arbiter/Bck/04 (arbiter)
Brick16: nas1:/bricks/05/Bck
Brick17: nas3:/bricks/05/Bck
Brick18: nas2:/bricks/arbiter/Bck/05 (arbiter)
Brick19: nas1:/bricks/06/Bck
Brick20: nas3:/bricks/06/Bck
Brick21: nas2:/bricks/arbiter/Bck/06 (arbiter)
Brick22: nas1:/bricks/07/Bck
Brick23: nas3:/bricks/07/Bck
Brick24: nas2:/bricks/arbiter/Bck/07 (arbiter)
Options Reconfigured:
cluster.granular-entry-heal: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

Does thin arbiter support just one replica of bricks?

-- 
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786