<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 29, 2017 at 7:48 PM, Paolo Margara <span dir="ltr">&lt;<a href="mailto:paolo.margara@polito.it" target="_blank">paolo.margara@polito.it</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF">
    <p>Hi Pranith,</p>
    <p>I&#39;m using this guide
<a class="gmail-m_-465708280187193584moz-txt-link-freetext" href="https://github.com/nixpanic/glusterdocs/blob/f6d48dc17f2cb6ee4680e372520ec3358641b2bc/Upgrade-Guide/upgrade_to_3.8.md" target="_blank">https://github.com/nixpanic/<wbr>glusterdocs/blob/<wbr>f6d48dc17f2cb6ee4680e372520ec3<wbr>358641b2bc/Upgrade-Guide/<wbr>upgrade_to_3.8.md</a></p>
    <p>Definitely my fault, but I think that is better to specify
      somewhere that restarting the service is not enough simply because
      in many other case, with other services, is sufficient.</p></div></blockquote><div>The steps include the following command before installing 3.8 as per the page (<a href="https://github.com/nixpanic/glusterdocs/blob/f6d48dc17f2cb6ee4680e372520ec3358641b2bc/Upgrade-Guide/upgrade_to_3.8.md#online-upgrade-procedure-for-servers">https://github.com/nixpanic/glusterdocs/blob/f6d48dc17f2cb6ee4680e372520ec3358641b2bc/Upgrade-Guide/upgrade_to_3.8.md#online-upgrade-procedure-for-servers</a>)<br></div><div>So I guess we have it covered?<br></div><div><ul><li>Stop all gluster services using the below command or through your favorite way to stop them.<span class="gmail-pl-c"></span></li><li><span class="gmail-pl-c">killall glusterfs glusterfsd glusterd</span><br></li></ul></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
    <p>Now I&#39;m restarting every brick process (and waiting for the heal
      to complete), this is fixing my problem.</p>
    <p>Many thanks for the help.<br>
    </p>
    <p><br>
    </p>
    <p>Greetings,</p>
    <p>    Paolo<br>
    </p><div><div class="gmail-h5">
    <br>
    <div class="gmail-m_-465708280187193584moz-cite-prefix">Il 29/06/2017 13:03, Pranith Kumar
      Karampuri ha scritto:<br>
    </div>
    </div></div><blockquote type="cite"><div><div class="gmail-h5">
      <div dir="ltr">
        <div>Paolo,<br>
        </div>
              Which document did you follow for the upgrade? We can fix
        the documentation if there are any issues.<br>
      </div>
      </div></div><div class="gmail_extra"><div><div class="gmail-h5"><br>
        <div class="gmail_quote">On Thu, Jun 29, 2017 at 2:07 PM,
          Ravishankar N <span dir="ltr">&lt;<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
            <div bgcolor="#FFFFFF"><span>
                <div class="gmail-m_-465708280187193584m_3264995662940995335moz-cite-prefix">On
                  06/29/2017 01:08 PM, Paolo Margara wrote:<br>
                </div>
                <blockquote type="cite">
                  <p>Hi all,</p>
                  <p>for the upgrade I followed this procedure:</p>
                  <ul>
                    <li>put node in maintenance mode (ensure no client
                      are active)</li>
                    <li>yum versionlock delete glusterfs*<br>
                    </li>
                    <li>service glusterd stop</li>
                    <li>yum update</li>
                    <li>systemctl daemon-reload <br>
                    </li>
                    <li>service glusterd start</li>
                    <li>yum versionlock add glusterfs*</li>
                    <li>gluster volume heal vm-images-repo full</li>
                    <li>gluster volume heal vm-images-repo info</li>
                  </ul>
                  <p>on each server every time I ran &#39;gluster --version&#39;
                    to confirm the upgrade, at the end I ran &#39;gluster
                    volume set all cluster.op-version 30800&#39;.</p>
                  <p>Today I&#39;ve tried to manually kill a brick process
                    on a non critical volume, after that into the log I
                    see:</p>
                  <p>[2017-06-29 07:03:50.074388] I [MSGID: 100030]
                    [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfsd:
                    Started running /usr/sbin/glusterfsd version 3.8.12
                    (args: /usr/sbin/glusterfsd -s virtnode-0-1-gluster
                    --volfile-id
                    iso-images-repo.virtnode-0-1-g<wbr>luster.data-glusterfs-brick1b-<wbr>iso-images-repo
                    -p
                    /var/lib/glusterd/vols/iso-ima<wbr>ges-repo/run/virtnode-0-1-glus<wbr>ter-data-glusterfs-brick1b-<wbr>iso-images-repo.pid
                    -S /var/run/gluster/c779852c21e2a<wbr>91eaabbdda3b9127262.socket
                    --brick-name /data/glusterfs/brick1b/iso-im<wbr>ages-repo
                    -l /var/log/glusterfs/bricks/data<wbr>-glusterfs-brick1b-iso-images-<wbr>repo.log
                    --xlator-option *-posix.glusterd-uuid=e93ebee7<wbr>-5d95-4100-a9df-4a3e60134b73
                    --brick-port 49163 --xlator-option
                    iso-images-repo-server.listen-<wbr>port=49163)</p>
                  <p>I&#39;ve checked after the restart and indeed now the
                    directory &#39;entry-changes&#39; is created, but why
                    stopping the glusterd service has not stopped also
                    the brick processes?</p>
                </blockquote>
                <br>
              </span> Just stopping,upgrading and restarting glusterd
              does not restart the brick processes, You would need to
              kill all gluster processes on the node before upgrading. 
              After upgrading, when you restart glusterd, it will
              automatically spawn the rest of the gluster processes on
              that node.<span><br>
                 <br>
                <blockquote type="cite">
                  <p>Now how can I recover from this issue? Restarting
                    all brick processes is enough?</p>
                </blockquote>
              </span> Yes, but ensure there are no pending heals like
              Pranith mentioned. <a class="gmail-m_-465708280187193584m_3264995662940995335moz-txt-link-freetext" href="https://gluster.readthedocs.io/en/latest/Upgrade-Guide/upgrade_to_3.7/" target="_blank">https://gluster.readthedocs.io<wbr>/en/latest/Upgrade-Guide/upgra<wbr>de_to_3.7/</a> 
              lists the steps for upgrade to 3.7 but the steps mentioned
              there are similar for any rolling upgrade.<br>
              <br>
              -Ravi
              <div>
                <div class="gmail-m_-465708280187193584h5"><br>
                  <blockquote type="cite">
                    <p><br>
                    </p>
                    <p>Greetings,</p>
                    <p>    Paolo Margara<br>
                    </p>
                    <br>
                    <div class="gmail-m_-465708280187193584m_3264995662940995335moz-cite-prefix">Il
                      28/06/2017 18:41, Pranith Kumar Karampuri ha
                      scritto:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr"><br>
                        <div class="gmail_extra"><br>
                          <div class="gmail_quote">On Wed, Jun 28, 2017
                            at 9:45 PM, Ravishankar N <span dir="ltr">&lt;<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>&gt;</span>
                            wrote:<br>
                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                              <div class="gmail-m_-465708280187193584m_3264995662940995335HOEnZb">
                                <div class="gmail-m_-465708280187193584m_3264995662940995335h5">On
                                  06/28/2017 06:52 PM, Paolo Margara
                                  wrote:<br>
                                  <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Hi list,<br>
                                    <br>
                                    yesterday I noted the following
                                    lines into the glustershd.log log
                                    file:<br>
                                    <br>
                                    [2017-06-28 11:53:05.000890] W
                                    [MSGID: 108034]<br>
                                    [afr-self-heald.c:479:afr_shd_<wbr>index_sweep]<br>
                                    0-iso-images-repo-replicate-0:
                                    unable to get index-dir on<br>
                                    iso-images-repo-client-0<br>
                                    [2017-06-28 11:53:05.001146] W
                                    [MSGID: 108034]<br>
                                    [afr-self-heald.c:479:afr_shd_<wbr>index_sweep]
                                    0-vm-images-repo-replicate-0:<br>
                                    unable to get index-dir on
                                    vm-images-repo-client-0<br>
                                    [2017-06-28 11:53:06.001141] W
                                    [MSGID: 108034]<br>
                                    [afr-self-heald.c:479:afr_shd_<wbr>index_sweep]
                                    0-hosted-engine-replicate-0:<br>
                                    unable to get index-dir on
                                    hosted-engine-client-0<br>
                                    [2017-06-28 11:53:08.001094] W
                                    [MSGID: 108034]<br>
                                    [afr-self-heald.c:479:afr_shd_<wbr>index_sweep]
                                    0-vm-images-repo-replicate-2:<br>
                                    unable to get index-dir on
                                    vm-images-repo-client-6<br>
                                    [2017-06-28 11:53:08.001170] W
                                    [MSGID: 108034]<br>
                                    [afr-self-heald.c:479:afr_shd_<wbr>index_sweep]
                                    0-vm-images-repo-replicate-1:<br>
                                    unable to get index-dir on
                                    vm-images-repo-client-3<br>
                                    <br>
                                    Digging into the mailing list
                                    archive I&#39;ve found another user with
                                    a<br>
                                    similar issue (the thread was
                                    &#39;[Gluster-users] glustershd: unable
                                    to get<br>
                                    index-dir on myvolume-client-0&#39;),
                                    the solution suggested was to verify<br>
                                    if the 
                                    /&lt;path-to-backend-brick&gt;/.glus<wbr>terfs/indices
                                    directory contains<br>
                                    all these sub directories: &#39;dirty&#39;,
                                    &#39;entry-changes&#39; and &#39;xattrop&#39; and if<br>
                                    some of them does not exists simply
                                    create it with mkdir.<br>
                                    <br>
                                    In my case the &#39;entry-changes&#39;
                                    directory is not present on all the<br>
                                    bricks and on all the servers:<br>
                                    <br>
                                    /data/glusterfs/brick1a/hosted<wbr>-engine/.glusterfs/indices/:<br>
                                    total 0<br>
                                    drw------- 2 root root 55 Jun 28
                                    15:02 dirty<br>
                                    drw------- 2 root root 57 Jun 28
                                    15:02 xattrop<br>
                                    <br>
                                    /data/glusterfs/brick1b/iso-im<wbr>ages-repo/.glusterfs/indices/:<br>
                                    total 0<br>
                                    drw------- 2 root root 55 May 29
                                    14:04 dirty<br>
                                    drw------- 2 root root 57 May 29
                                    14:04 xattrop<br>
                                    <br>
                                    /data/glusterfs/brick2/vm-imag<wbr>es-repo/.glusterfs/indices/:<br>
                                    total 0<br>
                                    drw------- 2 root root 112 Jun 28
                                    15:02 dirty<br>
                                    drw------- 2 root root  66 Jun 28
                                    15:02 xattrop<br>
                                    <br>
                                    /data/glusterfs/brick3/vm-imag<wbr>es-repo/.glusterfs/indices/:<br>
                                    total 0<br>
                                    drw------- 2 root root 64 Jun 28
                                    15:02 dirty<br>
                                    drw------- 2 root root 66 Jun 28
                                    15:02 xattrop<br>
                                    <br>
                                    /data/glusterfs/brick4/vm-imag<wbr>es-repo/.glusterfs/indices/:<br>
                                    total 0<br>
                                    drw------- 2 root root 112 Jun 28
                                    15:02 dirty<br>
                                    drw------- 2 root root  66 Jun 28
                                    15:02 xattrop<br>
                                    <br>
                                    I&#39;ve recently upgraded gluster from
                                    3.7.16 to 3.8.12 with the rolling<br>
                                    upgrade procedure and I haven&#39;t
                                    noted this issue prior of the
                                    update, on<br>
                                    another system upgraded with the
                                    same procedure I haven&#39;t encountered<br>
                                    this problem.<br>
                                    <br>
                                    Currently all VM images appear to be
                                    OK but prior to create the<br>
                                    &#39;entry-changes&#39; I would like to ask
                                    if this is still the correct<br>
                                    procedure to fix this issue<br>
                                  </blockquote>
                                  <br>
                                </div>
                              </div>
                              Did you restart the bricks after the
                              upgrade? That should have created the
                              entry-changes directory. Can you kill the
                              brick and restart it and see if the dir is
                              created? Double check from the brick logs
                              that you&#39;re indeed running 3.12:  &quot;Started
                              running /usr/local/sbin/glusterfsd version
                              3.8.12&quot; should appear when the brick
                              starts.<br>
                            </blockquote>
                            <div><br>
                            </div>
                            <div>Please note that if you are going the
                              route of killing and restarting, you need
                              to do it in the same way you did rolling
                              upgrade. You need to wait for heal to
                              complete before you kill the other nodes.
                              But before you do this, it is better you
                              look at the logs or confirm the steps you
                              used for doing upgrade.<br>
                            </div>
                            <div> </div>
                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br>
                              -Ravi
                              <div class="gmail-m_-465708280187193584m_3264995662940995335HOEnZb">
                                <div class="gmail-m_-465708280187193584m_3264995662940995335h5"><br>
                                  <br>
                                  <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">   and if
                                    this problem could have affected the<br>
                                    heal operations occurred meanwhile.<br>
                                    <br>
                                    Thanks.<br>
                                    <br>
                                    <br>
                                    Greetings,<br>
                                    <br>
                                         Paolo Margara<br>
                                    <br>
                                    ______________________________<wbr>_________________<br>
                                    Gluster-users mailing list<br>
                                    <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
                                    <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
                                  </blockquote>
                                  <br>
                                  <br>
                                  ______________________________<wbr>_________________<br>
                                  Gluster-users mailing list<br>
                                  <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
                                  <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
                                </div>
                              </div>
                            </blockquote>
                          </div>
                          <br>
                          <br clear="all">
                          <br>
                          -- <br>
                          <div class="gmail-m_-465708280187193584m_3264995662940995335gmail_signature">
                            <div dir="ltr">Pranith<br>
                            </div>
                          </div>
                        </div>
                      </div>
                    </blockquote>
                  </blockquote>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        -- <br>
        </div></div><div class="gmail-m_-465708280187193584gmail_signature">
          <div dir="ltr">Pranith<br>
          </div>
        </div>
      </div>
    </blockquote>
  </div>

</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>