<div dir="ltr">Even though the subject says the issue is with glusterd, I think the question is more applicable on heal/shards. Added the relevant folks to help out.<br></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Dec 10, 2018 at 3:43 PM Chris Drescher <<a href="mailto:info@linuxfabrik.ch">info@linuxfabrik.ch</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="text-align:left;direction:ltr"><div>Let me provide more information.</div><div><br></div><div>We have 3 gluster nodes running with sharding activated.</div><div><br></div><div>Node1: CentOS 7.5 - Glusterfs 3.12.6</div><div>Node2: CentOS 7.5 - Glusterfs 3.12.6</div><div>Node3: CentOS 7.5 - Glusterfs 3.12.6</div><div><br></div><div>Now we updated Node 3 from CentOS 7.5 to 7.6 which caused a reboot.</div><div>Glusterd Version changed from 3.12.6 to 3.12.15</div><div><br></div><div>Node1: CentOS 7.5 - Glusterfs 3.12.6</div><div>Node2: CentOS 7.5 - Glusterfs 3.12.6</div><div>Node3: CentOS 7.6 - Glusterfs 3.12.15</div><div><br></div><div>Afterwards gluster heal daemon keeps resyncing specific shards on bricks on Node1 and Node2. Always the same shards.</div><div><br></div><div>LOGS:</div><div><br></div><div>On upgraded NODE3:</div><div><br></div><div>/var/log/glusterfs/glusterd.log</div><div>[2018-12-10 09:24:42.314624] E [MSGID: 106062] [glusterd-utils.c:10112:glusterd_max_opversion_use_rsp_dict] 0-management: Maximum supported op-version not set in destination dictionary</div><div><br></div><div>tail -f /var/log/glusterfs/glustershd.log</div><div>[2018-12-09 04:28:05.687127] I [MSGID: 108026] [afr-self-heal-common.c:1726:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal on 3f1711c2-de8c-4e8e-be10-a252f5b1b4ad. sources=[0] 2 sinks=1 </div><div><br></div><div><br></div><div>On NODE1:</div><div><br></div><div>tail -f /var/log/glusterfs/glfsheal-data.log</div><div>[2018-12-10 10:00:01.898139] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-data-client-16: Server lk version = 1</div><div>[2018-12-10 10:00:01.898487] I [MSGID: 114057] [client-handshake.c:1478:select_server_supported_programs] 0-data-client-17: Using Program GlusterFS 3.3, Num (1298437), Version (330) </div><div>[2018-12-10 10:00:01.898892] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-data-client-17: Connected to data-client-17, attached to remote volume '/gluster/arb2/data'. </div><div>[2018-12-10 10:00:01.898900] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-data-client-17: Server and Client lk-version numbers are not same, reopening the fds </div><div>[2018-12-10 10:00:01.899007] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-data-client-17: Server lk version = 1</div><div>[2018-12-10 10:00:01.901528] I [MSGID: 108031] [afr-common.c:2376:afr_local_discovery_cbk] 0-data-replicate-3: selecting local read_child data-client-9</div><div>[2018-12-10 10:00:01.901876] I [MSGID: 108031] [afr-common.c:2376:afr_local_discovery_cbk] 0-data-replicate-5: selecting local read_child data-client-15</div><div>[2018-12-10 10:00:01.901978] I [MSGID: 108031] [afr-common.c:2376:afr_local_discovery_cbk] 0-data-replicate-4: selecting local read_child data-client-12</div><div>[2018-12-10 10:00:01.902708] I [MSGID: 108031] [afr-common.c:2376:afr_local_discovery_cbk] 0-data-replicate-2: selecting local read_child data-client-6</div><div>[2018-12-10 10:00:01.902750] I [MSGID: 104041] [glfs-resolve.c:971:__glfs_active_subvol] 0-data: switched to graph 70312d70-6f64-3031-2e6c-696e75786661 (0)</div><div></div><div><br></div><div>Hope that helps!</div><div><br></div><div>On Mon, 2018-12-10 at 09:22 +0100, Chris Drescher wrote:</div><blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:2px solid rgb(114,159,207);padding-left:1ex"><div>Hello everybody</div><div><br></div><div>We are experiencing an urgent issue with glusterd!</div><div>After an upgrade from centos7.5 to 7.6 our grusterfs keeps resyncing specific shards over and over again!</div><div><br></div><div>Is this a known problem?</div><div><br></div><div>This is very urgent! Please help!</div><div><br></div><div>Thanks in advance!</div><div><br></div><div>Kind regards.</div><div><br></div><div>Chris</div>
<pre>_______________________________________________</pre><pre>Gluster-users mailing list</pre><pre><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a></pre><pre><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre></blockquote></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>