[Gluster-users] proper way to temporarily remove brick server from replica cluster to avoid kvm guest disruption

[Todd Pfaff]

No, just CentOS, libvirt, kvm, OpenZFS, Gluster, and some command-line 
virsh tools.

Todd

On Sun, 6 Mar 2022, Strahil Nikolov wrote:

> Is this an oVirt cluster ?
> Best Regards,
> Strahil Nikolov
>
>       On Sun, Mar 6, 2022 at 10:06, Strahil Nikolov
> <hunter86_bg at yahoo.com> wrote:
> It seems that only vh1-4 provide bricks, so vh5,6,7,8 can be removed.
> First check why vh5 is offline. Changes to all modes are propagated and
> in this case vh5 is down and won't receive the peer detach commands.
> 
> Once you fix vh5, you can safely 'gluster peer detach' any of the nodes
> that is not in the volume.
> 
> Keep in mind that it's always best practice to have odd number of nodes
> in the TSP (3,5,7,9,etc).
> 
> Best Regards,
> Strahil Nikolov
>
>       On Sun, Mar 6, 2022 at 4:06, Todd Pfaff
> <pfaff at rhpcs.mcmaster.ca> wrote:
> [root at vh1 ~]# gluster volume info vol1
> 
> Volume Name: vol1
> Type: Replicate
> Volume ID: dfd681bb-5b68-4831-9863-e13f9f027620
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 4 = 4
> Transport-type: tcp
> Bricks:
> Brick1: vh1:/pool/gluster/brick1/data
> Brick2: vh2:/pool/gluster/brick1/data
> Brick3: vh3:/pool/gluster/brick1/data
> Brick4: vh4:/pool/gluster/brick1/data
> Options Reconfigured:
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: off
> 
> 
> [root at vh1 ~]# gluster pool list
> UUID                                    Hostname        State
> 75fc4258-fabd-47c9-8198-bbe6e6a906fb    vh2            Connected
> 00697e28-96c0-4534-a314-e878070b653d    vh3            Connected
> 2a9b891b-35d0-496c-bb06-f5dab4feb6bf    vh4            Connected
> 8ba6fb80-3b13-4379-94cf-22662cbb48a2    vh5           
> Disconnected
> 1298d334-3500-4b40-a8bd-cc781f7349d0    vh6            Connected
> 79a533ac-3d89-44b9-b0ce-823cfec8cf75    vh7            Connected
> 4141cd74-9c13-404c-a02c-f553fa19bc22    vh8            Connected
> 
> 
> On Sat, 5 Mar 2022, Strahil Nikolov wrote:
> 
> > Hey Todd,
> >
> > can you provide 'gluster volume info <VOLUME>' ?
> >
> > Best Regards,
> > Strahil Nikolov
> >
> >      On Sat, Mar 5, 2022 at 18:17, Todd Pfaff
> > <pfaff at rhpcs.mcmaster.ca> wrote:
> > I have a replica volume created as:
> >
> > gluster volume create vol1 replica 4 \
> >   host{1,2,3,4}:/mnt/gluster/brick1/data \
> >   force
> >
> >
> > All hosts host{1,2,3,4} mount this volume as:
> >
> > localhost:/vol1 /mnt/gluster/vol1 glusterfs defaults
> >
> >
> > Some other hosts are trusted peers but do not contribute
> bricks, and
> > they
> > also mount vol1 in the same way:
> >
> > localhost:/vol1 /mnt/gluster/vol1 glusterfs defaults
> >
> >
> > All hosts run CentOS 7.9, and all are running glusterfs 9.4 or
> 9.5 from
> > centos-release-gluster9-1.0-1.el7.noarch.
> >
> >
> > All hosts run kvm guests that use qcow2 files for root
> filesystems that
> > are stored on gluster volume vol1.
> >
> >
> > This is all working well, as long as none of host{1,2,3,4} go
> offline.
> >
> >
> > I want to take one of host{1,2,3,4} offline temporarily for
> > maintenance.
> > I'll refer to this as hostX.
> >
> > I understand that hostX will need to be healed when it comes
> back
> > online.
> >
> > I would, of course, migrate guests from hostX to another host,
> in which
> > case hostX would then only be participating as a gluster
> replica brick
> > provider and serving gluster client requests.
> >
> > What I've experienced is that if I take one of host{1,2,3,4}
> offline,
> > this
> > can disrupt some of the VM guests on various other hosts such
> that
> > their
> > root filesystems go to read-only.
> >
> > What I'm looking for here are suggestions as to how to properly
> take
> > one
> > of host{1,2,3,4} offline to avoid such disruption or how to
> tune the
> > libvirt kvm hosts and guests to be sufficiently resilient in
> the face
> > of
> > taking one gluster replica node offline.
> >
> > Thanks,
> > Todd
> > ________
> >
> >
> >
> > Community Meeting Calendar:
> >
> > Schedule -
> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> > Bridge: https://meet.google.com/cpu-eiue-hvk
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
> >
> >
> >
> 
> 
>