[Gluster-users] Messy rpm upgrade

[Zenon Panoussis]

This morning I found my gluster volume broken. Whatever
'gluster volume xxxxx gv0' commands I tried, they timed out.
The logs were not very helpful. Restarting gluster on all
nodes did not help. Actually nothing helped, and I didn't
even know what to look for where. The volume spans over
three nodes of one brick each, running Centos 7 and 8.

Eventually I realised that the 9.2 rpms were released last
night, and yum cron had upgraded my 9.1 to 9.2 on two of
the nodes. The third one was still running 9.1. I stopped
gluster on all of them, downgraded the two nodes back to
9.1, and the problem was solved; the volume came back up
just fine.

Subsequently I stopped gluster on all nodes, upgraded all
of them to 9.2, and restarted; the volume came back up just
fine again.

Conclusion: something in gluster doesn't like version
mismatches. On systems that run automatic updates, gluster
should be excluded and only be upgraded manually at the same
time across the entire cluster.

One question remains: how can a cluster be upgraded without
taking it down? Stopping, upgrading, and restarting one node
at a time doesn't seem to work.