<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p><br>
    </p>
    <br>
    <div class="moz-cite-prefix">On 12/10/2018 05:06 PM, Atin Mukherjee
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAGNCGH2RH03ac6G=HnOVW=MgoZj81WE5S7dYueicaBxZM7Vn4g@mail.gmail.com">
      <div dir="ltr">Even though the subject says the issue is with
        glusterd, I think the question is more applicable on
        heal/shards. Added the relevant folks to help out.<br>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr">On Mon, Dec 10, 2018 at 3:43 PM Chris Drescher
          &lt;<a href="mailto:info@linuxfabrik.ch"
            moz-do-not-send="true">info@linuxfabrik.ch</a>&gt; wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div style="text-align:left;direction:ltr">
            <div>Let me provide more information.</div>
            <div><br>
            </div>
            <div>We have 3 gluster nodes running with sharding
              activated.</div>
            <div><br>
            </div>
            <div>Node1: CentOS 7.5 - Glusterfs 3.12.6</div>
            <div>Node2: CentOS 7.5 - Glusterfs 3.12.6</div>
            <div>Node3: CentOS 7.5 - Glusterfs 3.12.6</div>
            <div><br>
            </div>
            <div>Now we updated Node 3 from CentOS 7.5 to 7.6 which
              caused a reboot.</div>
            <div>Glusterd Version changed from 3.12.6 to 3.12.15</div>
            <div><br>
            </div>
            <div>Node1: CentOS 7.5 - Glusterfs 3.12.6</div>
            <div>Node2: CentOS 7.5 - Glusterfs 3.12.6</div>
            <div>Node3: CentOS 7.6 - Glusterfs 3.12.15</div>
            <div><br>
            </div>
            <div>Afterwards gluster heal daemon keeps resyncing specific
              shards on bricks on Node1 and Node2. Always the same
              shards.</div>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <br>
    Your clients (mounts)  might be experiencing disconnects from the
    brick process(es) while the same set of shards are being written to.
    Possibly to the second brick, going from the "sinks=1 " log below.
    Check if that is the case.<br>
    -Ravi<br>
    <blockquote type="cite"
cite="mid:CAGNCGH2RH03ac6G=HnOVW=MgoZj81WE5S7dYueicaBxZM7Vn4g@mail.gmail.com">
      <div class="gmail_quote">
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div style="text-align:left;direction:ltr">
            <div><br>
            </div>
            <div>LOGS:</div>
            <div><br>
            </div>
            <div>On upgraded NODE3:</div>
            <div><br>
            </div>
            <div>/var/log/glusterfs/glusterd.log</div>
            <div>[2018-12-10 09:24:42.314624] E [MSGID: 106062]
              [glusterd-utils.c:10112:glusterd_max_opversion_use_rsp_dict]
              0-management: Maximum supported op-version not set in
              destination dictionary</div>
            <div><br>
            </div>
            <div>tail -f /var/log/glusterfs/glustershd.log</div>
            <div>[2018-12-09 04:28:05.687127] I [MSGID: 108026]
              [afr-self-heal-common.c:1726:afr_log_selfheal]
              0-data-replicate-0: Completed data selfheal on
              3f1711c2-de8c-4e8e-be10-a252f5b1b4ad. sources=[0]
              2  sinks=1 </div>
            <div><br>
            </div>
            <div><br>
            </div>
            <div>On NODE1:</div>
            <div><br>
            </div>
            <div>tail -f /var/log/glusterfs/glfsheal-data.log</div>
            <div>[2018-12-10 10:00:01.898139] I [MSGID: 114035]
              [client-handshake.c:202:client_set_lk_version_cbk]
              0-data-client-16: Server lk version = 1</div>
            <div>[2018-12-10 10:00:01.898487] I [MSGID: 114057]
              [client-handshake.c:1478:select_server_supported_programs]
              0-data-client-17: Using Program GlusterFS 3.3, Num
              (1298437), Version (330)                             </div>
            <div>[2018-12-10 10:00:01.898892] I [MSGID: 114046]
              [client-handshake.c:1231:client_setvolume_cbk]
              0-data-client-17: Connected to data-client-17, attached to
              remote volume '/gluster/arb2/data'.                      </div>
            <div>[2018-12-10 10:00:01.898900] I [MSGID: 114047]
              [client-handshake.c:1242:client_setvolume_cbk]
              0-data-client-17: Server and Client lk-version numbers are
              not same, reopening the fds                              </div>
            <div>[2018-12-10 10:00:01.899007] I [MSGID: 114035]
              [client-handshake.c:202:client_set_lk_version_cbk]
              0-data-client-17: Server lk version = 1</div>
            <div>[2018-12-10 10:00:01.901528] I [MSGID: 108031]
              [afr-common.c:2376:afr_local_discovery_cbk]
              0-data-replicate-3: selecting local read_child
              data-client-9</div>
            <div>[2018-12-10 10:00:01.901876] I [MSGID: 108031]
              [afr-common.c:2376:afr_local_discovery_cbk]
              0-data-replicate-5: selecting local read_child
              data-client-15</div>
            <div>[2018-12-10 10:00:01.901978] I [MSGID: 108031]
              [afr-common.c:2376:afr_local_discovery_cbk]
              0-data-replicate-4: selecting local read_child
              data-client-12</div>
            <div>[2018-12-10 10:00:01.902708] I [MSGID: 108031]
              [afr-common.c:2376:afr_local_discovery_cbk]
              0-data-replicate-2: selecting local read_child
              data-client-6</div>
            <div>[2018-12-10 10:00:01.902750] I [MSGID: 104041]
              [glfs-resolve.c:971:__glfs_active_subvol] 0-data: switched
              to graph 70312d70-6f64-3031-2e6c-696e75786661 (0)</div>
            <div><br>
            </div>
            <div>Hope that helps!</div>
            <div><br>
            </div>
            <div>On Mon, 2018-12-10 at 09:22 +0100, Chris Drescher
              wrote:</div>
            <blockquote type="cite" style="margin:0px 0px 0px
              0.8ex;border-left:2px solid
              rgb(114,159,207);padding-left:1ex">
              <div>Hello everybody</div>
              <div><br>
              </div>
              <div>We are experiencing an urgent issue with glusterd!</div>
              <div>After an upgrade from centos7.5 to 7.6 our grusterfs
                keeps resyncing specific shards over and over again!</div>
              <div><br>
              </div>
              <div>Is this a known problem?</div>
              <div><br>
              </div>
              <div>This is very urgent! Please help!</div>
              <div><br>
              </div>
              <div>Thanks in advance!</div>
              <div><br>
              </div>
              <div>Kind regards.</div>
              <div><br>
              </div>
              <div>Chris</div>
              <pre>_______________________________________________</pre>
              <pre>Gluster-users mailing list</pre>
              <pre><a href="mailto:Gluster-users@gluster.org" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a></pre>
              <pre><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
            </blockquote>
          </div>
          _______________________________________________<br>
          Gluster-users mailing list<br>
          <a href="mailto:Gluster-users@gluster.org" target="_blank"
            moz-do-not-send="true">Gluster-users@gluster.org</a><br>
          <a
            href="https://lists.gluster.org/mailman/listinfo/gluster-users"
            rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
      </div>
    </blockquote>
    <br>
  </body>
</html>