[Gluster-users] How to remove dead peer, osrry urgent again :(

[WK]

On 6/10/2017 4:38 PM, Lindsay Mathieson wrote:
> Since my node died on friday I have a dead peer (vna) that needs to be 
> removed.
>
>
> I had major issues this morning that I haven't resolve yet with all 
> VM's going offline when I rebooted a node which I *hope * was due to 
> quorum issues as I now have four peers in the cluster, one dead, three 
> live.
>
>


Lets see:

According to your previous note, you had vna, vnb and vng all replica 3 
in a working cluster.

vna died so you had two 'good' nodes left. All was good.

You replaced vna with vnd but it  is probably not fully healed yet cuz 
you had 3.8T worth of chunks to copy.

So you had two good nodes (vnb and vng) working and you rebooted one of 
them?

If so, yes, based on my experience learning how to deal with failed 
nodes you would get a quorum lock under those circumstances UNLESS you 
turned off the quorums prior to the reboot.

Do you show any split brains?

As an aside, I wonder if a strategy would have been to first replace VNA 
with an arbiter, get the metadata synced up for quorum purposes AND then 
turn the arbiter into full node by catching up the chunks.

Is that even possible?

-bill