[Gluster-users] proper way to temporarily remove brick server from replica cluster to avoid kvm guest disruption

[Todd Pfaff]

[root at vh1 ~]# gluster volume info vol1

Volume Name: vol1
Type: Replicate
Volume ID: dfd681bb-5b68-4831-9863-e13f9f027620
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 4 = 4
Transport-type: tcp
Bricks:
Brick1: vh1:/pool/gluster/brick1/data
Brick2: vh2:/pool/gluster/brick1/data
Brick3: vh3:/pool/gluster/brick1/data
Brick4: vh4:/pool/gluster/brick1/data
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off


[root at vh1 ~]# gluster pool list
UUID                                    Hostname        State
75fc4258-fabd-47c9-8198-bbe6e6a906fb    vh2             Connected
00697e28-96c0-4534-a314-e878070b653d    vh3             Connected
2a9b891b-35d0-496c-bb06-f5dab4feb6bf    vh4             Connected
8ba6fb80-3b13-4379-94cf-22662cbb48a2    vh5             Disconnected
1298d334-3500-4b40-a8bd-cc781f7349d0    vh6             Connected
79a533ac-3d89-44b9-b0ce-823cfec8cf75    vh7             Connected
4141cd74-9c13-404c-a02c-f553fa19bc22    vh8             Connected


On Sat, 5 Mar 2022, Strahil Nikolov wrote:

> Hey Todd,
> 
> can you provide 'gluster volume info <VOLUME>' ?
> 
> Best Regards,
> Strahil Nikolov
>
>       On Sat, Mar 5, 2022 at 18:17, Todd Pfaff
> <pfaff at rhpcs.mcmaster.ca> wrote:
> I have a replica volume created as:
> 
> gluster volume create vol1 replica 4 \
>   host{1,2,3,4}:/mnt/gluster/brick1/data \
>   force
> 
> 
> All hosts host{1,2,3,4} mount this volume as:
> 
> localhost:/vol1 /mnt/gluster/vol1 glusterfs defaults
> 
> 
> Some other hosts are trusted peers but do not contribute bricks, and
> they
> also mount vol1 in the same way:
> 
> localhost:/vol1 /mnt/gluster/vol1 glusterfs defaults
> 
> 
> All hosts run CentOS 7.9, and all are running glusterfs 9.4 or 9.5 from
> centos-release-gluster9-1.0-1.el7.noarch.
> 
> 
> All hosts run kvm guests that use qcow2 files for root filesystems that
> are stored on gluster volume vol1.
> 
> 
> This is all working well, as long as none of host{1,2,3,4} go offline.
> 
> 
> I want to take one of host{1,2,3,4} offline temporarily for
> maintenance.
> I'll refer to this as hostX.
> 
> I understand that hostX will need to be healed when it comes back
> online.
> 
> I would, of course, migrate guests from hostX to another host, in which
> case hostX would then only be participating as a gluster replica brick
> provider and serving gluster client requests.
> 
> What I've experienced is that if I take one of host{1,2,3,4} offline,
> this
> can disrupt some of the VM guests on various other hosts such that
> their
> root filesystems go to read-only.
> 
> What I'm looking for here are suggestions as to how to properly take
> one
> of host{1,2,3,4} offline to avoid such disruption or how to tune the
> libvirt kvm hosts and guests to be sufficiently resilient in the face
> of
> taking one gluster replica node offline.
> 
> Thanks,
> Todd
> ________
> 
> 
> 
> Community Meeting Calendar:
> 
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
> 
> 
>