[Gluster-users] Rolling upgrade from 3.6.3 to 3.10.5

[Yong Tseng]

Hi all,

I'm currently in process of upgrading a replicated cluster (1 x 4) from
3.6.3 to 3.10.5. The nodes run CentOS 6. However after upgrading the first
node, the said node fails to connect to other peers (as seen via 'gluster
peer status'), but somehow other non-upgraded peers can still see the
upgraded peer as connected.

Writes to the Gluster volume via local mounts of non-upgraded peers are
replicated to the upgraded peer, but I can't write via the upgraded peer as
the local mount seems forced to read-only.

Launching heal operations from non-upgraded peers will output 'Commit
failed on <upgraded peer IP>. Please check log for details'.

In addition, during upgrade process there were warning messages about my
old vol files renamed with .rpmsave extension. I tried starting Gluster
with my old vol files but the problem persisted. I tried generating new vol
files with 'glusterd --xlator-option "*.upgrade=on" -N', still no avail.

Also I checked the brick log it had several messages about "failed to get
client opversion". I don't know if this is pertinent. Could it be that the
upgraded node cannot connect to older nodes but still can receive
instructions from them?

Below are command outputs; some data are masked.
I'd provide more information if required.
Thanks in advance.

===> 'gluster volume status' ran on non-upgraded peers

Status of volume: gsnfs
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick gs-nfs01:/ftpdata                                 49154   Y       2931
Brick gs-nfs02:/ftpdata                                 49152   Y
29875
Brick gs-nfs03:/ftpdata                                 49153   Y       6987
Brick gs-nfs04:/ftpdata                                 49153   Y
24768
Self-heal Daemon on localhost                           N/A     Y       2938
Self-heal Daemon on gs-nfs04                            N/A     Y
24788
Self-heal Daemon on gs-nfs03                            N/A     Y       7007
Self-heal Daemon on <IP>                      N/A     Y       29866

Task Status of Volume gsnfs
------------------------------------------------------------------------------
There are no active volume tasks



===> 'gluster volume status' on upgraded peer

Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick gs-nfs02:/ftpdata                     49152     0          Y
29875
Self-heal Daemon on localhost               N/A       N/A        Y
29866

Task Status of Volume gsnfs
------------------------------------------------------------------------------
There are no active volume tasks



===> 'gluster peer status' on non-upgraded peer

Number of Peers: 3

Hostname: gs-nfs03
Uuid: 4c1544e6-550d-481a-95af-2a1da32d10ad
State: Peer in Cluster (Connected)

Hostname: <IP>
Uuid: 17d554fd-9181-4b53-9521-55acf69ac35f
State: Peer in Cluster (Connected)
Other names:
gs-nfs02

Hostname: gs-nfs04
Uuid: c6d165e6-d222-414c-b57a-97c64f06c5e9
State: Peer in Cluster (Connected)



===> 'gluster peer status' on upgraded peer

Number of Peers: 3

Hostname: gs-nfs03
Uuid: 4c1544e6-550d-481a-95af-2a1da32d10ad
State: Peer in Cluster (Disconnected)

Hostname: gs-nfs01
Uuid: 90d3ed27-61ac-4ad3-93a9-3c2b68f41ecf
State: Peer in Cluster (Disconnected)
Other names:
<IP>

Hostname: gs-nfs04
Uuid: c6d165e6-d222-414c-b57a-97c64f06c5e9
State: Peer in Cluster (Disconnected)


-- 
- Yong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170825/8b5c0650/attachment.html>