[Gluster-Maintainers] Upgrade issue when new mem type is added in libglusterfs

[Atin Mukherjee]

We have hit a bug 1347250 in downstream (applicable upstream too) where it
was seen that glusterd didnt regenerate the volfiles when it was interimly
brought up with upgrade mode by yum. Log file captured that gsyncd
--version failed to execute and hence glusterd init couldnt proceed till
the volfile regeneration. Since the ret code is not handled here in spec
file users wouldnt come to know about this and going forward this is going
to cause major issues in healing and all and finally it exploits the
possibility of split brains at its best.

Further analysis by Kotresh & Raghavendra Talur reveals that gsyncd failed
here because of the compatibility issue where gsyncd was still not upgraded
where as glusterfs-server was and this failure was mainly because of change
in the mem type enum. We have seen a similar issue for RDMA as well
(probably a year back). So to be very generic this can happen in any
upgrade path from one version to another where new mem type is introduced.
We have seen this from 3.7.8 to 3.7.12 and 3.8. People upgrading from 3.6
to 3.7/3.8 will also experience this issue.

Till we work on this fix, I suggest all the release managers to highlight
this in the release note of the latest releases with the following work
around after yum update:

1. grep -irns "geo-replication module not working as desired"
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log | wc -l

 If the output is non-zero, then go to step 2 else follow the rest of
the steps as per the guide.

2.Check if glusterd instance is running or not by 'ps aux | grep
glusterd', if it is, then stop the glusterd service.

 3. glusterd --xlator-option *.upgrade=on -N

and then proceed ahead with rest of the steps as per the guide.

Thoughts?

P.S : this email is limited to maintainers till we decide on the approach
to highlight this issues to the users


-- 
Atin
Sent from iPhone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/maintainers/attachments/20160709/45be593a/attachment.html>