[Gluster-Maintainers] Upgrade issue when new mem type is added in libglusterfs

[Kaushal M]

On Sat, Jul 9, 2016 at 10:02 PM, Atin Mukherjee <amukherj at redhat.com> wrote:
> We have hit a bug 1347250 in downstream (applicable upstream too) where it
> was seen that glusterd didnt regenerate the volfiles when it was interimly
> brought up with upgrade mode by yum. Log file captured that gsyncd --version
> failed to execute and hence glusterd init couldnt proceed till the volfile
> regeneration. Since the ret code is not handled here in spec file users
> wouldnt come to know about this and going forward this is going to cause
> major issues in healing and all and finally it exploits the possibility of
> split brains at its best.
>
> Further analysis by Kotresh & Raghavendra Talur reveals that gsyncd failed
> here because of the compatibility issue where gsyncd was still not upgraded
> where as glusterfs-server was and this failure was mainly because of change
> in the mem type enum. We have seen a similar issue for RDMA as well
> (probably a year back). So to be very generic this can happen in any upgrade
> path from one version to another where new mem type is introduced. We have
> seen this from 3.7.8 to 3.7.12 and 3.8. People upgrading from 3.6 to 3.7/3.8
> will also experience this issue.
>
> Till we work on this fix, I suggest all the release managers to highlight
> this in the release note of the latest releases with the following work
> around after yum update:
>
> 1. grep -irns "geo-replication module not working as desired"
> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log | wc -l
>
>  If the output is non-zero, then go to step 2 else follow the rest of the
> steps as per the guide.
>
> 2.Check if glusterd instance is running or not by 'ps aux | grep glusterd',
> if it is, then stop the glusterd service.
>
>  3. glusterd --xlator-option *.upgrade=on -N
>
> and then proceed ahead with rest of the steps as per the guide.
>
> Thoughts?

Proper .so versioning of libglusterfs should help with problems like
this. I don't know how to do this though.

But I do have some thoughts to share on using GlusterDs upgrade-mode.

GlusterD depends on the cluster op-version when generating volfiles,
to insert new features/xlators into the volfile graph.
This was done to make sure that the homogeneity of the volfiles is
preserved across the cluster.
This behaviour makes running GlusterD in upgrade mode after a package
upgrade, essentially a noop.
The cluster op-version doesn't change automatically when packages are upgraded,
so the regenerated volfiles in the post-upgrade section are basically
the same as before.
(If something is getting added into volfiles after this, it is
incorrect, and is something I'm yet to check).

The correct time to regenerate the volfiles is after all members of
the cluster have been upgraded and the cluster op-version has been
bumped.
(Bumping op-version doesn't regenerate anything, it is just an
indication that the cluster is now ready to use new features.)

We don't have a direct way to get volfiles regenerated on all members
with a single command yet. We can implement such a command with
relative ease.
For now, volfiles can regenerated by making use of the `volume set`
command, by setting a `user.upgrade` option on a volume.
Options in the `user.` namespace are passed on to hook scripts and not
added into any volfiles, but setting such an option on a volume causes
GlusterD to regenerate volfiles for the volume.

My suggestion would be to stop using glusterd in upgrade mode during
post-upgrade to regenerate volfiles, and document the above way to get
volfiles regenerated across the cluster correctly.
We could do away with upgrade mode itself, but it could be useful for
other things (Though I can't think of any right now).

What do the other maintainers feel about this?

~kaushal

PS: If this discussion is distracting from the original conversation,
I'll start a new thread.

>
> P.S : this email is limited to maintainers till we decide on the approach to
> highlight this issues to the users
>
>
> --
> Atin
> Sent from iPhone
>
> _______________________________________________
> maintainers mailing list
> maintainers at gluster.org
> http://www.gluster.org/mailman/listinfo/maintainers
>