[Gluster-users] Upgrading from Gluster 3.8 to 3.12
Hari Gowtham
hgowtham at redhat.com
Wed Dec 20 06:01:42 UTC 2017
Yes Atin. I'll take a look.
On Wed, Dec 20, 2017 at 11:28 AM, Atin Mukherjee <amukherj at redhat.com> wrote:
> Looks like a bug as I see tier-enabled = 0 is an additional entry in the
> info file in shchhv01. As per the code, this field should be written into
> the glusterd store if the op-version is >= 30706 . What I am guessing is
> since we didn't have the commit 33f8703a1 "glusterd: regenerate volfiles on
> op-version bump up" in 3.8.4 while bumping up the op-version the info and
> volfiles were not regenerated which caused the tier-enabled entry to be
> missing in the info file.
>
> For now, you can copy the info file for the volumes where the mismatch
> happened from shchhv01 to shchhv02 and restart glusterd service on shchhv02.
> That should fix up this temporarily. Unfortunately this step might need to
> be repeated for other nodes as well.
>
> @Hari - Could you help in debugging this further.
>
>
>
> On Wed, Dec 20, 2017 at 10:44 AM, Gustave Dahl <gustave at dahlfamily.net>
> wrote:
>>
>> I was attempting the same on a local sandbox and also have the same
>> problem.
>>
>>
>> Current: 3.8.4
>>
>> Volume Name: shchst01
>> Type: Distributed-Replicate
>> Volume ID: bcd53e52-cde6-4e58-85f9-71d230b7b0d3
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 4 x 3 = 12
>> Transport-type: tcp
>> Bricks:
>> Brick1: shchhv01-sto:/data/brick3/shchst01
>> Brick2: shchhv02-sto:/data/brick3/shchst01
>> Brick3: shchhv03-sto:/data/brick3/shchst01
>> Brick4: shchhv01-sto:/data/brick1/shchst01
>> Brick5: shchhv02-sto:/data/brick1/shchst01
>> Brick6: shchhv03-sto:/data/brick1/shchst01
>> Brick7: shchhv02-sto:/data/brick2/shchst01
>> Brick8: shchhv03-sto:/data/brick2/shchst01
>> Brick9: shchhv04-sto:/data/brick2/shchst01
>> Brick10: shchhv02-sto:/data/brick4/shchst01
>> Brick11: shchhv03-sto:/data/brick4/shchst01
>> Brick12: shchhv04-sto:/data/brick4/shchst01
>> Options Reconfigured:
>> cluster.data-self-heal-algorithm: full
>> features.shard-block-size: 512MB
>> features.shard: enable
>> performance.readdir-ahead: on
>> storage.owner-uid: 9869
>> storage.owner-gid: 9869
>> server.allow-insecure: on
>> performance.quick-read: off
>> performance.read-ahead: off
>> performance.io-cache: off
>> performance.stat-prefetch: off
>> cluster.eager-lock: enable
>> network.remote-dio: enable
>> cluster.quorum-type: auto
>> cluster.server-quorum-type: server
>> cluster.self-heal-daemon: on
>> nfs.disable: on
>> performance.io-thread-count: 64
>> performance.cache-size: 1GB
>>
>> Upgraded shchhv01-sto to 3.12.3, others remain at 3.8.4
>>
>> RESULT
>> =====================
>> Hostname: shchhv01-sto
>> Uuid: f6205edb-a0ea-4247-9594-c4cdc0d05816
>> State: Peer Rejected (Connected)
>>
>> Upgraded Server: shchhv01-sto
>> ==============================
>> [2017-12-20 05:02:44.747313] I [MSGID: 101190]
>> [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with
>> index 1
>> [2017-12-20 05:02:44.747387] I [MSGID: 101190]
>> [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with
>> index 2
>> [2017-12-20 05:02:44.749087] W [rpc-clnt-ping.c:246:rpc_clnt_ping_cbk]
>> 0-management: RPC_CLNT_PING notify failed
>> [2017-12-20 05:02:44.749165] W [rpc-clnt-ping.c:246:rpc_clnt_ping_cbk]
>> 0-management: RPC_CLNT_PING notify failed
>> [2017-12-20 05:02:44.749563] W [rpc-clnt-ping.c:246:rpc_clnt_ping_cbk]
>> 0-management: RPC_CLNT_PING notify failed
>> [2017-12-20 05:02:54.676324] I [MSGID: 106493]
>> [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received
>> RJT
>> from uuid: 546503ae-ba0e-40d4-843f-c5dbac22d272, host: shchhv02-sto, port:
>> 0
>> [2017-12-20 05:02:54.690237] I [MSGID: 106163]
>> [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack]
>> 0-management:
>> using the op-version 30800
>> [2017-12-20 05:02:54.695823] I [MSGID: 106490]
>> [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req]
>> 0-glusterd:
>> Received probe from uuid: 546503ae-ba0e-40d4-843f-c5dbac22d272
>> [2017-12-20 05:02:54.696956] E [MSGID: 106010]
>> [glusterd-utils.c:3370:glusterd_compare_friend_volume] 0-management:
>> Version
>> of Cksums shchst01-sto differ. local cksum = 4218452135, remote cksum =
>> 2747317484 on peer shchhv02-sto
>> [2017-12-20 05:02:54.697796] I [MSGID: 106493]
>> [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd:
>> Responded to shchhv02-sto (0), ret: 0, op_ret: -1
>> [2017-12-20 05:02:55.033822] I [MSGID: 106493]
>> [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received
>> RJT
>> from uuid: 3de22cb5-c1c1-4041-a1e1-eb969afa9b4b, host: shchhv03-sto, port:
>> 0
>> [2017-12-20 05:02:55.038460] I [MSGID: 106163]
>> [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack]
>> 0-management:
>> using the op-version 30800
>> [2017-12-20 05:02:55.040032] I [MSGID: 106490]
>> [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req]
>> 0-glusterd:
>> Received probe from uuid: 3de22cb5-c1c1-4041-a1e1-eb969afa9b4b
>> [2017-12-20 05:02:55.040266] E [MSGID: 106010]
>> [glusterd-utils.c:3370:glusterd_compare_friend_volume] 0-management:
>> Version
>> of Cksums shchst01-sto differ. local cksum = 4218452135, remote cksum =
>> 2747317484 on peer shchhv03-sto
>> [2017-12-20 05:02:55.040405] I [MSGID: 106493]
>> [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd:
>> Responded to shchhv03-sto (0), ret: 0, op_ret: -1
>> [2017-12-20 05:02:55.584854] I [MSGID: 106493]
>> [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received
>> RJT
>> from uuid: 36306e37-d7f0-4fec-9140-0d0f1bd2d2d5, host: shchhv04-sto, port:
>> 0
>> [2017-12-20 05:02:55.595125] I [MSGID: 106163]
>> [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack]
>> 0-management:
>> using the op-version 30800
>> [2017-12-20 05:02:55.600804] I [MSGID: 106490]
>> [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req]
>> 0-glusterd:
>> Received probe from uuid: 36306e37-d7f0-4fec-9140-0d0f1bd2d2d5
>> [2017-12-20 05:02:55.601288] E [MSGID: 106010]
>> [glusterd-utils.c:3370:glusterd_compare_friend_volume] 0-management:
>> Version
>> of Cksums shchst01-sto differ. local cksum = 4218452135, remote cksum =
>> 2747317484 on peer shchhv04-sto
>> [2017-12-20 05:02:55.601497] I [MSGID: 106493]
>> [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd:
>> Responded to shchhv04-sto (0), ret: 0, op_ret: -1
>>
>> Another Server: shchhv02-sto
>> ==============================
>> [2017-12-20 05:02:44.667833] W
>> [glusterd-locks.c:675:glusterd_mgmt_v3_unlock]
>> (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x1de5c)
>> [0x7f75fdc12e5c]
>> -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x27a08)
>> [0x7f75fdc1ca08]
>> -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd07fa)
>> [0x7f75fdcc57fa] ) 0-management: Lock for vol shchst01-sto not held
>> [2017-12-20 05:02:44.667795] I [MSGID: 106004]
>> [glusterd-handler.c:5219:__glusterd_peer_rpc_notify] 0-management: Peer
>> <shchhv01-sto> (<f6205edb-a0ea-4247-9594-c4cdc0d05816>), in state <Peer
>> Rejected>, has disconnected from glusterd.
>> [2017-12-20 05:02:44.667948] W [MSGID: 106118]
>> [glusterd-handler.c:5241:__glusterd_peer_rpc_notify] 0-management: Lock
>> not
>> released for shchst01-sto
>> [2017-12-20 05:02:44.760103] I [MSGID: 106163]
>> [glusterd-handshake.c:1271:__glusterd_mgmt_hndsk_versions_ack]
>> 0-management:
>> using the op-version 30800
>> [2017-12-20 05:02:44.765389] I [MSGID: 106490]
>> [glusterd-handler.c:2608:__glusterd_handle_incoming_friend_req]
>> 0-glusterd:
>> Received probe from uuid: f6205edb-a0ea-4247-9594-c4cdc0d05816
>> [2017-12-20 05:02:54.686185] E [MSGID: 106010]
>> [glusterd-utils.c:2930:glusterd_compare_friend_volume] 0-management:
>> Version
>> of Cksums shchst01 differ. local cksum = 2747317484, remote cksum =
>> 4218452135 on peer shchhv01-sto
>> [2017-12-20 05:02:54.686882] I [MSGID: 106493]
>> [glusterd-handler.c:3852:glusterd_xfer_friend_add_resp] 0-glusterd:
>> Responded to shchhv01-sto (0), ret: 0, op_ret: -1
>> [2017-12-20 05:02:54.717854] I [MSGID: 106493]
>> [glusterd-rpc-ops.c:476:__glusterd_friend_add_cbk] 0-glusterd: Received
>> RJT
>> from uuid: f6205edb-a0ea-4247-9594-c4cdc0d05816, host: shchhv01-sto, port:
>> 0
>>
>> Another Server: shchhv04-sto
>> ==============================
>> [2017-12-20 05:02:44.667620] I [MSGID: 106004]
>> [glusterd-handler.c:5219:__glusterd_peer_rpc_notify] 0-management: Peer
>> <shchhv01-sto> (<f6205edb-a0ea-4247-9594-c4cdc0d05816>), in state <Peer
>> Rejected>, has disconnected from glusterd.
>> [2017-12-20 05:02:44.667808] W
>> [glusterd-locks.c:675:glusterd_mgmt_v3_unlock]
>> (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x1de5c)
>> [0x7f10a33d9e5c]
>> -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x27a08)
>> [0x7f10a33e3a08]
>> -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd07fa)
>> [0x7f10a348c7fa] ) 0-management: Lock for vol shchst01-sto not held
>> [2017-12-20 05:02:44.667827] W [MSGID: 106118]
>> [glusterd-handler.c:5241:__glusterd_peer_rpc_notify] 0-management: Lock
>> not
>> released for shchst01-sto
>> [2017-12-20 05:02:44.760077] I [MSGID: 106163]
>> [glusterd-handshake.c:1271:__glusterd_mgmt_hndsk_versions_ack]
>> 0-management:
>> using the op-version 30800
>> [2017-12-20 05:02:44.768796] I [MSGID: 106490]
>> [glusterd-handler.c:2608:__glusterd_handle_incoming_friend_req]
>> 0-glusterd:
>> Received probe from uuid: f6205edb-a0ea-4247-9594-c4cdc0d05816
>> [2017-12-20 05:02:55.595095] E [MSGID: 106010]
>> [glusterd-utils.c:2930:glusterd_compare_friend_volume] 0-management:
>> Version
>> of Cksums shchst01-sto differ. local cksum = 2747317484, remote cksum =
>> 4218452135 on peer shchhv01-sto
>> [2017-12-20 05:02:55.595273] I [MSGID: 106493]
>> [glusterd-handler.c:3852:glusterd_xfer_friend_add_resp] 0-glusterd:
>> Responded to shchhv01-sto (0), ret: 0, op_ret: -1
>> [2017-12-20 05:02:55.612957] I [MSGID: 106493]
>> [glusterd-rpc-ops.c:476:__glusterd_friend_add_cbk] 0-glusterd: Received
>> RJT
>> from uuid: f6205edb-a0ea-4247-9594-c4cdc0d05816, host: shchhv01-sto, port:
>> 0
>>
>> <vol>/info
>>
>> Upgraded Server: shchst01-sto
>> =========================
>> type=2
>> count=12
>> status=1
>> sub_count=3
>> stripe_count=1
>> replica_count=3
>> disperse_count=0
>> redundancy_count=0
>> version=52
>> transport-type=0
>> volume-id=bcd53e52-cde6-4e58-85f9-71d230b7b0d3
>> username=5a4ae8d8-dbcb-408e-ab73-629255c14ffc
>> password=58652573-0955-4d00-893a-9f42d0f16717
>> op-version=30700
>> client-op-version=30700
>> quota-version=0
>> tier-enabled=0
>> parent_volname=N/A
>> restored_from_snap=00000000-0000-0000-0000-000000000000
>> snap-max-hard-limit=256
>> cluster.data-self-heal-algorithm=full
>> features.shard-block-size=512MB
>> features.shard=enable
>> nfs.disable=on
>> cluster.self-heal-daemon=on
>> cluster.server-quorum-type=server
>> cluster.quorum-type=auto
>> network.remote-dio=enable
>> cluster.eager-lock=enable
>> performance.stat-prefetch=off
>> performance.io-cache=off
>> performance.read-ahead=off
>> performance.quick-read=off
>> server.allow-insecure=on
>> storage.owner-gid=9869
>> storage.owner-uid=9869
>> performance.readdir-ahead=on
>> performance.io-thread-count=64
>> performance.cache-size=1GB
>> brick-0=shchhv01-sto:-data-brick3-shchst01
>> brick-1=shchhv02-sto:-data-brick3-shchst01
>> brick-2=shchhv03-sto:-data-brick3-shchst01
>> brick-3=shchhv01-sto:-data-brick1-shchst01
>> brick-4=shchhv02-sto:-data-brick1-shchst01
>> brick-5=shchhv03-sto:-data-brick1-shchst01
>> brick-6=shchhv02-sto:-data-brick2-shchst01
>> brick-7=shchhv03-sto:-data-brick2-shchst01
>> brick-8=shchhv04-sto:-data-brick2-shchst01
>> brick-9=shchhv02-sto:-data-brick4-shchst01
>> brick-10=shchhv03-sto:-data-brick4-shchst01
>> brick-11=shchhv04-sto:-data-brick4-shchst01
>>
>> Another Server: shchhv02-sto
>> ==============================
>> type=2
>> count=12
>> status=1
>> sub_count=3
>> stripe_count=1
>> replica_count=3
>> disperse_count=0
>> redundancy_count=0
>> version=52
>> transport-type=0
>> volume-id=bcd53e52-cde6-4e58-85f9-71d230b7b0d3
>> username=5a4ae8d8-dbcb-408e-ab73-629255c14ffc
>> password=58652573-0955-4d00-893a-9f42d0f16717
>> op-version=30700
>> client-op-version=30700
>> quota-version=0
>> parent_volname=N/A
>> restored_from_snap=00000000-0000-0000-0000-000000000000
>> snap-max-hard-limit=256
>> cluster.data-self-heal-algorithm=full
>> features.shard-block-size=512MB
>> features.shard=enable
>> performance.readdir-ahead=on
>> storage.owner-uid=9869
>> storage.owner-gid=9869
>> server.allow-insecure=on
>> performance.quick-read=off
>> performance.read-ahead=off
>> performance.io-cache=off
>> performance.stat-prefetch=off
>> cluster.eager-lock=enable
>> network.remote-dio=enable
>> cluster.quorum-type=auto
>> cluster.server-quorum-type=server
>> cluster.self-heal-daemon=on
>> nfs.disable=on
>> performance.io-thread-count=64
>> performance.cache-size=1GB
>> brick-0=shchhv01-sto:-data-brick3-shchst01
>> brick-1=shchhv02-sto:-data-brick3-shchst01
>> brick-2=shchhv03-sto:-data-brick3-shchst01
>> brick-3=shchhv01-sto:-data-brick1-shchst01
>> brick-4=shchhv02-sto:-data-brick1-shchst01
>> brick-5=shchhv03-sto:-data-brick1-shchst01
>> brick-6=shchhv02-sto:-data-brick2-shchst01
>> brick-7=shchhv03-sto:-data-brick2-shchst01
>> brick-8=shchhv04-sto:-data-brick2-shchst01
>> brick-9=shchhv02-sto:-data-brick4-shchst01
>> brick-10=shchhv03-sto:-data-brick4-shchst01
>> brick-11=shchhv04-sto:-data-brick4-shchst01
>>
>> NOTE
>>
>> [root at shchhv01 shchst01]# gluster volume get shchst01 cluster.op-version
>> Warning: Support to get global option value using `volume get <volname>`
>> will be deprecated from next release. Consider using `volume get all`
>> instead for global options
>> Option Value
>>
>> ------ -----
>>
>> cluster.op-version 30800
>>
>> [root at shchhv02 shchst01]# gluster volume get shchst01 cluster.op-version
>> Option Value
>>
>> ------ -----
>>
>> cluster.op-version 30800
>>
>> -----Original Message-----
>> From: gluster-users-bounces at gluster.org
>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Ziemowit Pierzycki
>> Sent: Tuesday, December 19, 2017 3:56 PM
>> To: gluster-users <gluster-users at gluster.org>
>> Subject: Re: [Gluster-users] Upgrading from Gluster 3.8 to 3.12
>>
>> I have not done the upgrade yet. Since this is a production cluster I
>> need
>> to make sure it stays up or schedule some downtime if it doesn't doesn't.
>> Thanks.
>>
>> On Tue, Dec 19, 2017 at 10:11 AM, Atin Mukherjee <amukherj at redhat.com>
>> wrote:
>> >
>> >
>> > On Tue, Dec 19, 2017 at 1:10 AM, Ziemowit Pierzycki
>> > <ziemowit at pierzycki.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> I have a cluster of 10 servers all running Fedora 24 along with
>> >> Gluster 3.8. I'm planning on doing rolling upgrades to Fedora 27
>> >> with Gluster 3.12. I saw the documentation and did some testing but
>> >> I would like to run my plan through some (more?) educated minds.
>> >>
>> >> The current setup is:
>> >>
>> >> Volume Name: vol0
>> >> Distributed-Replicate
>> >> Number of Bricks: 2 x (2 + 1) = 6
>> >> Bricks:
>> >> Brick1: glt01:/vol/vol0
>> >> Brick2: glt02:/vol/vol0
>> >> Brick3: glt05:/vol/vol0 (arbiter)
>> >> Brick4: glt03:/vol/vol0
>> >> Brick5: glt04:/vol/vol0
>> >> Brick6: glt06:/vol/vol0 (arbiter)
>> >>
>> >> Volume Name: vol1
>> >> Distributed-Replicate
>> >> Number of Bricks: 2 x (2 + 1) = 6
>> >> Bricks:
>> >> Brick1: glt07:/vol/vol1
>> >> Brick2: glt08:/vol/vol1
>> >> Brick3: glt05:/vol/vol1 (arbiter)
>> >> Brick4: glt09:/vol/vol1
>> >> Brick5: glt10:/vol/vol1
>> >> Brick6: glt06:/vol/vol1 (arbiter)
>> >>
>> >> After performing the upgrade because of differences in checksums, the
>> >> upgraded nodes will become:
>> >>
>> >> State: Peer Rejected (Connected)
>> >
>> >
>> > Have you upgraded all the nodes? If yes, have you bumped up the
>> > cluster.op-version after upgrading all the nodes? Please follow :
>> > http://docs.gluster.org/en/latest/Upgrade-Guide/op_version/ for more
>> > details on how to bump up the cluster.op-version. In case you have
>> > done all of these and you're seeing a checksum issue then I'm afraid
>> > you have hit a bug. I'd need further details like the checksum
>> > mismatch error from glusterd.log file along with the the exact
>> > volume's info file from /var/lib/glusterd/vols/<volname>/info between
>> > both the peers to debug this further.
>> >
>> >>
>> >> If I start doing the upgrades one at a time, with nodes glt10 to
>> >> glt01 except for the arbiters glt05 and glt06, and then upgrading the
>> >> arbiters last, everything should remain online at all times through
>> >> the process. Correct?
>> >>
>> >> Thanks.
>> >> _______________________________________________
>> >> Gluster-users mailing list
>> >> Gluster-users at gluster.org
>> >> http://lists.gluster.org/mailman/listinfo/gluster-users
>> >
>> >
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
--
Regards,
Hari Gowtham.
More information about the Gluster-users
mailing list