[Gluster-users] Fixing a rejected peer

[Jamie Lawrence]

Hello,

So I'm seeing a rejected peer with 3.12.6. This is with a replica 3 volume.

It actually began as the same problem with a different peer. I noticed with (call it) gluster-2, when I couldn't make a new volume.  I compared /var/lib/glusterd between them, and found that somehow the options in one of the vols differed. (I suspect this was due to attempting to create the volume via the Ovirt GUI; suffice to say I'm not using it for things like this in the future.) So I stopped the daemons and corrected that (gluster-2 had a tiering entry the others didn't).

Started things back up and now gluster-3 is being rejected by the other two. The error is below.

I'm tempted to repeat - down things, copy the checksum the "good" ones agree on, start things; but given that this has turned into a balloon-squeezing exercise, I want to make sure I'm not doing this the wrong way. 

What is the currently accepted best method for fixing this? 

And given that this happened on a nearly brand-new deployment, it worries me a bit that this happened while nothing hinky was going on - I installed Gluster manually, but the rest of the systems management has been via Ovirt. Has anyone else seen issues with this?

Thanks,

-j

- - snip - - 

[2018-03-06 00:14:06.141281] I [MSGID: 106490] [glusterd-handler.c:2891:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 77cdfbba-348c-43fe-ab3d-00621904ea9c
[2018-03-06 00:14:06.145540] I [MSGID: 106493] [glusterd-handler.c:2954:__glusterd_handle_probe_query] 0-glusterd: Responded to sc5-gluster-1, op_ret: 0, op_errno: 0, ret: 0
[2018-03-06 00:14:06.145697] I [MSGID: 106490] [glusterd-handler.c:2891:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: c1877e0d-ccb2-401e-83a6-e4a680af683a
[2018-03-06 00:14:06.145831] I [MSGID: 106493] [glusterd-handler.c:2954:__glusterd_handle_probe_query] 0-glusterd: Responded to sc5-gluster-10g-2, op_ret: 0, op_errno: 0, ret: 0
[2018-03-06 00:14:06.149357] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: c1877e0d-ccb2-401e-83a6-e4a680af683a
[2018-03-06 00:14:06.149631] E [MSGID: 106010] [glusterd-utils.c:3374:glusterd_compare_friend_volume] 0-management: Version of Cksums sc5-ovirt_engine differ. local cksum = 53769889, remote cksum = 2068896937 on peer sc5-gluster-2
[2018-03-06 00:14:06.149774] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sc5-gluster-2 (0), ret: 0, op_ret: -1
[2018-03-06 00:14:06.151393] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 77cdfbba-348c-43fe-ab3d-00621904ea9c
[2018-03-06 00:14:06.152127] E [MSGID: 106010] [glusterd-utils.c:3374:glusterd_compare_friend_volume] 0-management: Version of Cksums sc5-ovirt_engine differ. local cksum = 53769889, remote cksum = 2068896937 on peer sc5-gluster-10g-1
[2018-03-06 00:14:06.152314] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sc5-gluster-10g-1 (0), ret: 0, op_ret: -1
[2018-03-06 00:14:06.164819] I [MSGID: 106143] [glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick /gluster-bricks/sc5_ovirt_engine/sc5_ovirt_engine on port 49152
[2018-03-06 00:14:06.443882] I [MSGID: 106487] [glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req