<div dir="ltr"><div>Logs from newly added node helped me in RCA of the issue.<br><br>Info file on node 10.5.6.17 consist of an additional property "tier-enabled" which is not present in info file from other 3 nodes, hence <br></div><div>when gluster peer probe call is made, in order to maintain consistency across the cluster cksum is compared. In this <br>case as both files are different leading to different cksum, causing state in "State: Peer Rejected (Connected)".<br></div><div><br></div><div>This inconsistency arise due to upgrade you did.<br></div><div><br></div><div>Workaround:<br></div><div>1.Go to node 10.5.6.17 <br>2.Open info file from "/var/lib/glusterd/vols/<vol-name>/info" and remove "tier-enabled=0".<br></div><div>3.Restart glusterd services<br></div><div>4.Peer probe again.<br><br></div><div>Thanks<br></div><div>Gaurav<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 31, 2017 at 3:37 PM, lejeczek <span dir="ltr"><<a href="mailto:peljasz@yahoo.co.uk" target="_blank">peljasz@yahoo.co.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">attached the lot as per your request.<br>
<br>
Would bee really great if you can find the root cause of this and suggest a resolution. Fingers crossed.<br>
thanks, L.<span class=""><br>
<br>
On 31/08/17 05:34, Gaurav Yadav wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
Could you please sendentire content of "/var/lib/glusterd/" directory of the 4th node which is being peer probed, along with command-history and glusterd.logs.<br>
<br>
Thanks<br>
Gaurav<br>
<br></span><span class="">
On Wed, Aug 30, 2017 at 7:10 PM, lejeczek <<a href="mailto:peljasz@yahoo.co.uk" target="_blank">peljasz@yahoo.co.uk</a> <mailto:<a href="mailto:peljasz@yahoo.co.uk" target="_blank">peljasz@yahoo.co.uk</a>>> wrote:<br>
<br>
<br>
<br>
On 30/08/17 07:18, Gaurav Yadav wrote:<br>
<br>
<br>
Could you please send me "info" file which is<br>
placed in "/var/lib/glusterd/vols/<vol-n<wbr>ame>"<br>
directory from all the nodes along with<br>
glusterd.logs and command-history.<br>
<br>
Thanks<br>
Gaurav<br>
<br>
On Tue, Aug 29, 2017 at 7:13 PM, lejeczek<br>
<<a href="mailto:peljasz@yahoo.co.uk" target="_blank">peljasz@yahoo.co.uk</a> <mailto:<a href="mailto:peljasz@yahoo.co.uk" target="_blank">peljasz@yahoo.co.uk</a>><br></span>
<mailto:<a href="mailto:peljasz@yahoo.co.uk" target="_blank">peljasz@yahoo.co.uk</a><div><div class="h5"><br>
<mailto:<a href="mailto:peljasz@yahoo.co.uk" target="_blank">peljasz@yahoo.co.uk</a>>>> wrote:<br>
<br>
hi fellas,<br>
same old same<br>
in log of the probing peer I see:<br>
...<br>
2017-08-29 13:36:16.882196] I [MSGID: 106493]<br>
<br>
[glusterd-handler.c:3020:__glu<wbr>sterd_handle_probe_query]<br>
0-glusterd: Responded to priv.xx.xx.priv.xx.xx.x,<br>
op_ret: 0, op_errno: 0, ret: 0<br>
[2017-08-29 13:36:16.904961] I [MSGID: 106490]<br>
<br>
[glusterd-handler.c:2606:__glu<wbr>sterd_handle_incoming_friend_<wbr>req]<br>
0-glusterd: Received probe from uuid:<br>
2a17edb4-ae68-4b67-916e-e38a20<wbr>87ca28<br>
[2017-08-29 13:36:16.906477] E [MSGID: 106010]<br>
<br>
[glusterd-utils.c:3034:gluster<wbr>d_compare_friend_volume]<br>
0-management: Version of Cksums CO-DATA<br>
differ. local<br>
cksum = 4088157353, remote cksum = 2870780063<br>
on peer<br>
10.5.6.17<br>
[2017-08-29 13:36:16.907187] I [MSGID: 106493]<br>
<br>
[glusterd-handler.c:3866:glust<wbr>erd_xfer_friend_add_resp]<br>
0-glusterd: Responded to 10.5.6.17 (0), ret:<br>
0, op_ret: -1<br>
...<br>
<br>
Why would adding a new peer make cluster jump<br>
to check<br>
checksums on a vol on that newly added peer?<br>
<br>
<br>
really. I mean, no brick even exists on newly added<br>
peer, it's just been probed, why this?:<br>
<br>
[2017-08-30 13:17:51.949430] E [MSGID: 106010]<br>
[glusterd-utils.c:3034:gluster<wbr>d_compare_friend_volume]<br>
0-management: Version of Cksums CO-DATA differ. local<br>
cksum = 4088157353, remote cksum = 2870780063 on peer<br>
10.5.6.17<br>
<br>
10.5.6.17 is a candidate I'm probing from a working<br>
cluster.<br>
Why gluster wants checksums and why checksums would be<br>
different?<br>
Would anybody know what is going on there?<br>
<br>
<br>
Is it why the peer gets rejected?<br>
That peer I'm hoping to add, was a member of the<br>
cluster in the past but I did "usual" wipe of<br>
/var/lib/gluster on candidate peer.<br>
<br>
a hint, solution would be great to hear.<br>
L.<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<wbr>org</a>><br></div></div>
<mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<wbr>org</a><span class=""><br>
<mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<wbr>org</a>>><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
<<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a>><br>
<br>
<<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a><br>
<<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a>>><br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<wbr>org</a>><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
<<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a>><br>
<br>
<br>
</span></blockquote>
<br>
</blockquote></div><br></div></div>