<div dir="ltr"><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, 8 Jun 2019 at 01:29, Alan Orth &lt;<a href="mailto:alan.orth@gmail.com">alan.orth@gmail.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Dear Ravi,</div><div><br></div><div>In the last week I have completed a fix-layout and a full INDEX heal on this volume. Now I&#39;ve started a rebalance and I see a few terabytes of data going around on different bricks since yesterday, which I&#39;m sure is good.</div><div><br></div><div>While I wait for the rebalance to finish, I&#39;m wondering if you know what would cause directories to be missing from the FUSE mount point? If I list the directories explicitly I can see their contents, but they do not appear in their parent directories&#39; listing. In the case of duplicated files it is always because the files are not on the correct bricks (according to the Dynamo/Elastic Hash algorithm), and I can fix it by copying the file to the correct brick(s) and removing it from the others (along with their .glusterfs hard links). So what could cause directories to be missing?</div><div><br></div></div></blockquote><div>Hi Alan,</div><div><br></div><div>The directories that don&#39;t show up in the parent directory listing is probably because they do not exist on the hashed subvol. Please check the backend bricks to see if they are missing on any of them.</div><div><br></div><div>Regards,</div><div>Nithya </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div>Thank you,<br></div><div><br></div><div>Thank you,<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 5, 2019 at 1:08 AM Alan Orth &lt;<a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Ravi,</div><div><br></div><div>You&#39;re right that I had mentioned using rsync to copy the brick content to a new host, but in the end I actually decided not to bring it up on a new brick. Instead I added the original brick back into the volume. So the  xattrs and symlinks to .glusterfs on the original brick are fine. I think the problem probably lies with a remove-brick that got interrupted. A few weeks ago during the maintenance I had tried to remove a brick and then after twenty minutes and no obvious progress I stopped it—after that the bricks were still part of the volume.<br></div><div><br></div><div>In the last few days I have run a fix-layout that took 26 hours and finished successfully. Then I started a full index heal and it has healed about 3.3 million files in a few days and I see a clear increase of network traffic from old brick host to new brick host over that time. Once the full index heal completes I will try to do a rebalance.<br></div><div><br></div><div>Thank you,</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 3, 2019 at 7:40 PM Ravishankar N &lt;<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF">
    <p><br>
    </p>
    <div class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531moz-cite-prefix">On 01/06/19 9:37 PM, Alan Orth wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div>Dear Ravi,</div>
        <div><br>
        </div>
        <div>The .glusterfs hardlinks/symlinks should be fine. I&#39;m not
          sure how I could verify them for six bricks and millions of
          files, though... :\<br>
        </div>
      </div>
    </blockquote>
    <p>Hi Alan,</p>
    <p>The reason I asked this is because you had mentioned in one of
      your earlier emails that when you moved content from the old brick
      to the new one, you had skipped the .glusterfs directory. So I was
      assuming that when you added back this new brick to the cluster,
      it might have been missing the .glusterfs entries. If that is the
      cae, one way to verify could be to check using a script if all
      files on the brick have a link-count of at least 2 and all dirs
      have valid symlinks inside .glusterfs pointing to themselves. <br>
    </p>
    <blockquote type="cite">
      <div dir="ltr">
        <div><br>
        </div>
        <div>I had a small success in fixing some issues with duplicated
          files on the FUSE mount point yesterday. I read quite a bit
          about the elastic hashing algorithm that determines which
          files get placed on which bricks based on the hash of their
          filename and the trusted.glusterfs.dht xattr on brick
          directories (thanks to Joe Julian&#39;s blog post and Python
          script for showing how it works¹). With that knowledge I
          looked closer at one of the files that was appearing as
          duplicated on the FUSE mount and found that it was also
          duplicated on more than `replica 2` bricks. For this
          particular file I found two &quot;real&quot; files and several zero-size
          files with trusted.glusterfs.dht.linkto xattrs. Neither of the
          &quot;real&quot; files were on the correct brick as far as the DHT
          layout is concerned, so I copied one of them to the correct
          brick, deleted the others and their hard links, and did a
          `stat` on the file from the FUSE mount point and it fixed
          itself. Yay!<br>
        </div>
        <div><br>
        </div>
        <div>Could this have been caused by a replace-brick that got
          interrupted and didn&#39;t finish re-labeling the xattrs? </div>
      </div>
    </blockquote>
    No, replace-brick only initiates AFR self-heal, which just copies
    the contents from the other brick(s) of the *same* replica pair into
    the replaced brick.  The link-to files are created by DHT when you
    rename a file from the client. If the new name hashes to a
    different  brick, DHT does not move the entire file there. It
    instead creates the link-to file (the one with the dht.linkto
    xattrs) on the hashed subvol. The value of this xattr points to the
    brick where the actual data is there (`getfattr -e text` to see it
    for yourself).  Perhaps you had attempted a rebalance or
    remove-brick earlier and interrupted that?<br>
    <blockquote type="cite">
      <div dir="ltr">
        <div>Should I be thinking of some heuristics to identify and fix
          these issues with a script (incorrect brick placement), or is
          this something a fix layout or repeated volume heals can fix?
          I&#39;ve already completed a whole heal on this particular volume
          this week and it did heal about 1,000,000 files (mostly data
          and metadata, but about 20,000 entry heals as well).<br>
        </div>
        <div><br>
        </div>
      </div>
    </blockquote>
    <p>Maybe you should let the AFR self-heals complete first and then
      attempt a full rebalance to take care of the dht link-to files.
      But  if the files are in millions, it could take quite some time
      to complete.</p>
    Regards,<br>
    Ravi<br>
    <blockquote type="cite">
      <div dir="ltr">
        <div>Thanks for your support,<br>
        </div>
        <div><br>
        </div>
        <div>¹ <a href="https://joejulian.name/post/dht-misses-are-expensive/" target="_blank">https://joejulian.name/post/dht-misses-are-expensive/</a></div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 7:57
          AM Ravishankar N &lt;<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>&gt; wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div bgcolor="#FFFFFF">
            <p><br>
            </p>
            <div class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578moz-cite-prefix">On
              31/05/19 3:20 AM, Alan Orth wrote:<br>
            </div>
            <blockquote type="cite">
              <div dir="ltr">
                <div>Dear Ravi,</div>
                <div><br>
                </div>
                <div>I spent a bit of time inspecting the xattrs on some
                  files and directories on a few bricks for this volume
                  and it looks a bit messy. Even if I could make sense
                  of it for a few and potentially heal them manually,
                  there are millions of files and directories in total
                  so that&#39;s definitely not a scalable solution. After a
                  few missteps with `replace-brick ... commit force` in
                  the last week—one of which on a brick that was
                  dead/offline—as well as some premature `remove-brick`
                  commands, I&#39;m unsure how how to proceed and I&#39;m
                  getting demotivated. It&#39;s scary how quickly things get
                  out of hand in distributed systems...<br>
                </div>
              </div>
            </blockquote>
            Hi Alan,<br>
            The one good thing about gluster is it that the data is
            always available directly on the backed bricks even if your
            volume has inconsistencies at the gluster level. So
            theoretically, if your cluster is FUBAR, you could just
            create a new volume and copy all data onto it via its mount
            from the old volume&#39;s bricks.<br>
            <blockquote type="cite">
              <div dir="ltr">
                <div><br>
                </div>
                <div>
                  <div>I had hoped that bringing the old brick back up
                    would help, but by the time I added it again a few
                    days had passed and all the brick-id&#39;s had changed
                    due to the replace/remove brick commands, not to
                    mention that the trusted.afr.$volume-client-xx
                    values were now probably pointing to the wrong
                    bricks (?).<br>
                  </div>
                </div>
                <div><br>
                </div>
                <div>Anyways, a few hours ago I started a full heal on
                  the volume and I see that there is a sustained
                  100MiB/sec of network traffic going from the old
                  brick&#39;s host to the new one. The completed heals
                  reported in the logs look promising too:</div>
                <div><br>
                </div>
                <div>Old brick host:</div>
                <div><br>
                </div>
                <div># grep &#39;2019-05-30&#39;
                  /var/log/glusterfs/glustershd.log | grep -o -E
                  &#39;Completed (data|metadata|entry) selfheal&#39; | sort |
                  uniq -c<br>
                   281614 Completed data selfheal<br>
                       84 Completed entry selfheal<br>
                   299648 Completed metadata selfheal</div>
                <div><br>
                </div>
                <div>New brick host:<br>
                </div>
                <div><br>
                </div>
                <div># grep &#39;2019-05-30&#39;
                  /var/log/glusterfs/glustershd.log | grep -o -E
                  &#39;Completed (data|metadata|entry) selfheal&#39; | sort |
                  uniq -c<br>
                   198256 Completed data selfheal<br>
                    16829 Completed entry selfheal<br>
                   229664 Completed metadata selfheal</div>
                <div><br>
                </div>
                <div>So that&#39;s good I guess, though I have no idea how
                  long it will take or if it will fix the &quot;missing
                  files&quot; issue on the FUSE mount. I&#39;ve increased
                  cluster.shd-max-threads to 8 to hopefully speed up the
                  heal process.<br>
                </div>
              </div>
            </blockquote>
            The afr xattrs should not cause files to disappear from
            mount. If the xattr names do not match what each AFR subvol
            expects (for eg. in a replica 2 volume,
            trusted.afr.*-client-{0,1} for 1st subvol, client-{2,3} for
            2nd subvol and so on - ) for its children then it won&#39;t heal
            the data, that is all. But in your case I see some
            inconsistencies like one brick having the actual file (<span style="font-family:&quot;courier new&quot;,monospace">licenseserver.cfg)
            </span>and the other having a linkto file (the one with the<span style="font-family:&quot;courier new&quot;,monospace"> dht.linkto
              xattr)</span> <i>in the same replica pair</i>. <br>
            <blockquote type="cite">
              <div dir="ltr">
                <div><br>
                </div>
                <div>I&#39;d be happy for any advice or pointers,<br>
                </div>
              </div>
            </blockquote>
            <p>Did you check if the .glusterfs hardlinks/symlinks exist
              and are in order for all bricks?</p>
            <p>-Ravi<br>
            </p>
            <blockquote type="cite"><br>
              <div class="gmail_quote">
                <div dir="ltr" class="gmail_attr">On Wed, May 29, 2019
                  at 5:20 PM Alan Orth &lt;<a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a>&gt;
                  wrote:<br>
                </div>
                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                  <div dir="ltr">
                    <div>Dear Ravi,</div>
                    <div><br>
                    </div>
                    <div>Thank you for the link to the blog post
                      series—it is very informative and current! If I
                      understand your blog post correctly then I think
                      the answer to your previous question about pending
                      AFRs is: no, there are no pending AFRs. I have
                      identified one file that is a good test case to
                      try to understand what happened after I issued the
                      `gluster volume replace-brick ... commit force` a
                      few days ago and then added the same original
                      brick back to the volume later. This is the
                      current state of the replica 2
                      distribute/replicate volume:<br>
                    </div>
                    <div><br>
                    </div>
                    <div><span style="font-family:&quot;courier new&quot;,monospace">[root@wingu0
                        ~]# gluster volume info apps<br>
                         <br>
                        Volume Name: apps<br>
                        Type: Distributed-Replicate<br>
                        Volume ID: f118d2da-79df-4ee1-919d-53884cd34eda<br>
                        Status: Started<br>
                        Snapshot Count: 0<br>
                        Number of Bricks: 3 x 2 = 6<br>
                        Transport-type: tcp<br>
                        Bricks:<br>
                        Brick1: wingu3:/mnt/gluster/apps<br>
                        Brick2: wingu4:/mnt/gluster/apps<br>
                        Brick3: wingu05:/data/glusterfs/sdb/apps<br>
                        Brick4: wingu06:/data/glusterfs/sdb/apps<br>
                        Brick5: wingu0:/mnt/gluster/apps<br>
                        Brick6: wingu05:/data/glusterfs/sdc/apps<br>
                        Options Reconfigured:<br>
                        diagnostics.client-log-level: DEBUG<br>
                        storage.health-check-interval: 10<br>
                        nfs.disable: on</span></div>
                    <div><span style="font-family:&quot;courier new&quot;,monospace"><br>
                      </span></div>
                    <div>I checked the xattrs of one file that is
                      missing from the volume&#39;s FUSE mount (though I can
                      read it if I access its full path explicitly), but
                      is present in several of the volume&#39;s bricks (some
                      with full size, others empty):<span style="font-family:&quot;courier new&quot;,monospace"><br>
                      </span></div>
                    <div><span style="font-family:&quot;courier new&quot;,monospace"><br>
                      </span></div>
                    <div><span style="font-family:&quot;courier new&quot;,monospace">[root@wingu0
                        ~]# getfattr -d -m. -e hex
                        /mnt/gluster/apps/clcgenomics/clclicsrv/licenseserver.cfg
                      </span></div>
                    <div>
                      <pre class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-code"><span style="font-family:&quot;courier new&quot;,monospace">getfattr: Removing leading &#39;/&#39; from absolute path names
# file: mnt/gluster/apps/clcgenomics/clclicsrv/licenseserver.cfg
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.apps-client-3=0x000000000000000000000000
trusted.afr.apps-client-5=0x000000000000000000000000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x0200000000000000585a396f00046e15
trusted.gfid=0x878003a2fb5243b6a0d14d2f8b4306bd

[root@wingu05 ~]# getfattr -d -m. -e hex /data/glusterfs/sdb/apps/clcgenomics/clclicsrv/licenseserver.cfg
getfattr: Removing leading &#39;/&#39; from absolute path names
# file: data/glusterfs/sdb/apps/clcgenomics/clclicsrv/licenseserver.cfg
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0x878003a2fb5243b6a0d14d2f8b4306bd
trusted.gfid2path.82586deefbc539c3=0x34666437323861612d356462392d343836382d616232662d6564393031636566333561392f6c6963656e73657365727665722e636667
trusted.glusterfs.dht.linkto=0x617070732d7265706c69636174652d3200

[root@wingu05 ~]# getfattr -d -m. -e hex /data/glusterfs/sdc/apps/clcgenomics/clclicsrv/licenseserver.cfg
getfattr: Removing leading &#39;/&#39; from absolute path names
# file: data/glusterfs/sdc/apps/clcgenomics/clclicsrv/licenseserver.cfg
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0x878003a2fb5243b6a0d14d2f8b4306bd
trusted.gfid2path.82586deefbc539c3=0x34666437323861612d356462392d343836382d616232662d6564393031636566333561392f6c6963656e73657365727665722e636667

[root@wingu06 ~]# getfattr -d -m. -e hex /data/glusterfs/sdb/apps/clcgenomics/clclicsrv/licenseserver.cfg
getfattr: Removing leading &#39;/&#39; from absolute path names
# file: data/glusterfs/sdb/apps/clcgenomics/clclicsrv/licenseserver.cfg
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0x878003a2fb5243b6a0d14d2f8b4306bd
trusted.gfid2path.82586deefbc539c3=0x34666437323861612d356462392d343836382d616232662d6564393031636566333561392f6c6963656e73657365727665722e636667
trusted.glusterfs.dht.linkto=0x617070732d7265706c69636174652d3200</span></pre>
                    </div>
                    <div>According to the <font face="courier
                        new,monospace">trusted.afr.apps-client-xx<font face="arial,sans-serif"> xattrs this
                          particular file should be on bricks with id
                          &quot;apps-client-3&quot; and &quot;apps-client-5&quot;. It took
                          me a few hours to realize that the brick-id
                          values are recorded in the volume&#39;s volfiles
                          in /var/lib/glusterd/vols/apps/bricks. After
                          comparing those brick-id values with a volfile
                          backup from before the replace-brick, I
                          realized that the files are simply on the
                          wrong brick now as far as Gluster is
                          concerned. </font></font><font face="courier
                        new,monospace"><font face="arial,sans-serif">This
                          particular file is now on the brick for
                          &quot;apps-client-</font></font>4&quot;. As an
                      experiment I copied this one file to the two
                      bricks listed in the xattrs and I was then able to
                      see the file from the FUSE mount (yay!).<br>
                    </div>
                    <div><br>
                    </div>
                    <div>Other than replacing the brick, removing it,
                      and then adding the old brick on the original
                      server back, there has been no change in the data
                      this entire time. Can I change the brick IDs in
                      the volfiles so they reflect where the data
                      actually is? Or perhaps script something to reset
                      all the xattrs on the files/directories to point
                      to the correct bricks?<br>
                    </div>
                    <div><br>
                    </div>
                    <div>Thank you for any help or pointers,<br>
                    </div>
                  </div>
                  <br>
                  <div class="gmail_quote">
                    <div dir="ltr" class="gmail_attr">On Wed, May 29,
                      2019 at 7:24 AM Ravishankar N &lt;<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>&gt;
                      wrote:<br>
                    </div>
                    <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                      <div bgcolor="#FFFFFF">
                        <p><br>
                        </p>
                        <div class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754moz-cite-prefix">On
                          29/05/19 9:50 AM, Ravishankar N wrote:<br>
                        </div>
                        <blockquote type="cite">
                          <p><br>
                          </p>
                          <div class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754moz-cite-prefix">On
                            29/05/19 3:59 AM, Alan Orth wrote:<br>
                          </div>
                          <blockquote type="cite">
                            <div dir="ltr">
                              <div>Dear Ravishankar,</div>
                              <div><br>
                              </div>
                              <div>I&#39;m not sure if Brick4 had pending
                                AFRs because I don&#39;t know what that
                                means and it&#39;s been a few days so I am
                                not sure I would be able to find that
                                information.</div>
                            </div>
                          </blockquote>
                          When you find some time, have a look at a <a href="http://wp.me/peiBB-6b" target="_blank">blog</a> series I
                          wrote about AFR- I&#39;ve tried to explain what
                          one needs to know to debug replication related
                          issues in it.<br>
                        </blockquote>
                        <p>Made a typo error. The URL for the blog is <a class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754moz-txt-link-freetext" href="https://wp.me/peiBB-6b" target="_blank">https://wp.me/peiBB-6b</a></p>
                        <p>-Ravi<br>
                        </p>
                        <blockquote type="cite">
                          <blockquote type="cite">
                            <div dir="ltr">
                              <div><br>
                              </div>
                              <div>Anyways, after wasting a few days
                                rsyncing the old brick to a new host I
                                decided to just try to add the old brick
                                back into the volume instead of bringing
                                it up on the new host. I created a new
                                brick directory on the old host, moved
                                the old brick&#39;s contents into that new
                                directory (minus the .glusterfs
                                directory), added the new brick to the
                                volume, and then did Vlad&#39;s find/stat
                                trick¹ from the brick to the FUSE mount
                                point.</div>
                              <div><br>
                              </div>
                              <div>The interesting problem I have now is
                                that some files don&#39;t appear in the FUSE
                                mount&#39;s directory listings, but I can
                                actually list them directly and even
                                read them. What could cause that?<br>
                              </div>
                            </div>
                          </blockquote>
                          Not sure, too many variables in the hacks that
                          you did to take a guess. You can check if the
                          contents of the .glusterfs folder are in order
                          on the new brick (example hardlink for files
                          and symlinks for directories are present etc.)
                          .<br>
                          Regards,<br>
                          Ravi<br>
                          <blockquote type="cite">
                            <div dir="ltr">
                              <div><br>
                              </div>
                              <div>Thanks,<br>
                              </div>
                              <div><br>
                              </div>
                              <div>¹ <a href="https://lists.gluster.org/pipermail/gluster-users/2018-February/033584.html" target="_blank">https://lists.gluster.org/pipermail/gluster-users/2018-February/033584.html</a></div>
                            </div>
                            <br>
                            <div class="gmail_quote">
                              <div dir="ltr" class="gmail_attr">On Fri,
                                May 24, 2019 at 4:59 PM Ravishankar N
                                &lt;<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>&gt;
                                wrote:<br>
                              </div>
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                <div bgcolor="#FFFFFF">
                                  <p><br>
                                  </p>
                                  <div class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail-m_-6098182312870478829moz-cite-prefix">On
                                    23/05/19 2:40 AM, Alan Orth wrote:<br>
                                  </div>
                                  <blockquote type="cite">
                                    <div dir="ltr">Dear list,
                                      <div><br>
                                      </div>
                                      <div>I seem to have gotten into a
                                        tricky situation. Today I
                                        brought up a shiny new server
                                        with new disk arrays and
                                        attempted to replace one brick
                                        of a replica 2
                                        distribute/replicate volume on
                                        an older server using the
                                        `replace-brick` command:</div>
                                      <div><br>
                                      </div>
                                      <div># gluster volume
                                        replace-brick homes
                                        wingu0:/mnt/gluster/homes
                                        wingu06:/data/glusterfs/sdb/homes
                                        commit force</div>
                                      <div><br>
                                      </div>
                                      <div>The command was successful
                                        and I see the new brick in the
                                        output of `gluster volume info`.
                                        The problem is that Gluster
                                        doesn&#39;t seem to be migrating the
                                        data, </div>
                                    </div>
                                  </blockquote>
                                  <p>`replace-brick` definitely must
                                    heal (not migrate) the data. In your
                                    case, data must have been healed
                                    from Brick-4 to the replaced
                                    Brick-3. Are there any errors in the
                                    self-heal daemon logs of Brick-4&#39;s
                                    node? Does Brick-4 have pending AFR
                                    xattrs blaming Brick-3? The doc is a
                                    bit out of date. replace-brick
                                    command internally does all the
                                    setfattr steps that are mentioned in
                                    the doc.</p>
                                  <p>-Ravi<br>
                                  </p>
                                  <p><br>
                                  </p>
                                  <blockquote type="cite">
                                    <div dir="ltr">
                                      <div>and now the original brick
                                        that I replaced is no longer
                                        part of the volume (and a few
                                        terabytes of data are just
                                        sitting on the old brick):</div>
                                      <div><br>
                                      </div>
                                      <div># gluster volume info homes |
                                        grep -E &quot;Brick[0-9]:&quot;<br>
                                        Brick1:
                                        wingu4:/mnt/gluster/homes<br>
                                        Brick2:
                                        wingu3:/mnt/gluster/homes<br>
                                        Brick3:
                                        wingu06:/data/glusterfs/sdb/homes<br>
                                        Brick4:
                                        wingu05:/data/glusterfs/sdb/homes<br>
                                        Brick5:
                                        wingu05:/data/glusterfs/sdc/homes<br>
                                        Brick6:
                                        wingu06:/data/glusterfs/sdc/homes</div>
                                      <br>
                                      <div>I see the Gluster docs have a
                                        more complicated procedure for
                                        replacing bricks that involves
                                        getfattr/setfattr¹. How can I
                                        tell Gluster about the old
                                        brick? I see that I have a
                                        backup of the old volfile thanks
                                        to yum&#39;s rpmsave function if
                                        that helps.<br>
                                      </div>
                                      <div><br>
                                      </div>
                                      <div>We are using Gluster 5.6 on
                                        CentOS 7. Thank you for any
                                        advice you can give.<br>
                                      </div>
                                      <div><br>
                                      </div>
                                      <div>¹ <a href="https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#replace-faulty-brick" target="_blank">https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#replace-faulty-brick</a></div>
                                      <div><br>
                                      </div>
                                      <div>-- <br>
                                        <div dir="ltr" class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail-m_-6098182312870478829gmail_signature">Alan
                                          Orth<br>
                                          <a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a><br>
                                          <a href="https://picturingjordan.com" target="_blank">https://picturingjordan.com</a><br>
                                          <a href="https://englishbulgaria.net" target="_blank">https://englishbulgaria.net</a><br>
                                          <a href="https://mjanja.ch" target="_blank">https://mjanja.ch</a><br>
                                          &quot;In heaven all the interesting
                                          people are missing.&quot;
                                          ―Friedrich Nietzsche</div>
                                      </div>
                                    </div>
                                    <br>
                                    <fieldset class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail-m_-6098182312870478829mimeAttachmentHeader"></fieldset>
                                    <pre class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail-m_-6098182312870478829moz-quote-pre">_______________________________________________
Gluster-users mailing list
<a class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail-m_-6098182312870478829moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail-m_-6098182312870478829moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
                                  </blockquote>
                                </div>
                              </blockquote>
                            </div>
                            <br clear="all">
                            <br>
                            -- <br>
                            <div dir="ltr" class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail_signature">Alan
                              Orth<br>
                              <a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a><br>
                              <a href="https://picturingjordan.com" target="_blank">https://picturingjordan.com</a><br>
                              <a href="https://englishbulgaria.net" target="_blank">https://englishbulgaria.net</a><br>
                              <a href="https://mjanja.ch" target="_blank">https://mjanja.ch</a><br>
                              &quot;In heaven all the interesting people are
                              missing.&quot; ―Friedrich Nietzsche</div>
                          </blockquote>
                          <br>
                          <fieldset class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754mimeAttachmentHeader"></fieldset>
                          <pre class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754moz-quote-pre">_______________________________________________
Gluster-users mailing list
<a class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
                        </blockquote>
                      </div>
                    </blockquote>
                  </div>
                  <br clear="all">
                  <br>
                  -- <br>
                  <div dir="ltr" class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail_signature">Alan
                    Orth<br>
                    <a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a><br>
                    <a href="https://picturingjordan.com" target="_blank">https://picturingjordan.com</a><br>
                    <a href="https://englishbulgaria.net" target="_blank">https://englishbulgaria.net</a><br>
                    <a href="https://mjanja.ch" target="_blank">https://mjanja.ch</a><br>
                    &quot;In heaven all the interesting people are missing.&quot;
                    ―Friedrich Nietzsche</div>
                </blockquote>
              </div>
              <br clear="all">
              <br>
              -- <br>
              <div dir="ltr" class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail-m_3345230001103480578gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail_signature">Alan
                Orth<br>
                <a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a><br>
                <a href="https://picturingjordan.com" target="_blank">https://picturingjordan.com</a><br>
                <a href="https://englishbulgaria.net" target="_blank">https://englishbulgaria.net</a><br>
                <a href="https://mjanja.ch" target="_blank">https://mjanja.ch</a><br>
                &quot;In heaven all the interesting people are missing.&quot;
                ―Friedrich Nietzsche</div>
            </blockquote>
          </div>
        </blockquote>
      </div>
      <br clear="all">
      <br>
      -- <br>
      <div dir="ltr" class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail-m_-5220815797528919531gmail_signature">Alan Orth<br>
        <a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a><br>
        <a href="https://picturingjordan.com" target="_blank">https://picturingjordan.com</a><br>
        <a href="https://englishbulgaria.net" target="_blank">https://englishbulgaria.net</a><br>
        <a href="https://mjanja.ch" target="_blank">https://mjanja.ch</a><br>
        &quot;In heaven all the interesting people are missing.&quot; ―Friedrich
        Nietzsche</div>
    </blockquote>
  </div>

</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail-m_-7745432102239921450gmail-m_-4434069740709880052gmail_signature">Alan Orth<br><a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a><br><a href="https://picturingjordan.com" target="_blank">https://picturingjordan.com</a><br><a href="https://englishbulgaria.net" target="_blank">https://englishbulgaria.net</a><br><a href="https://mjanja.ch" target="_blank">https://mjanja.ch</a><br>&quot;In heaven all the interesting people are missing.&quot; ―Friedrich Nietzsche</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail-m_-7745432102239921450gmail_signature">Alan Orth<br><a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a><br><a href="https://picturingjordan.com" target="_blank">https://picturingjordan.com</a><br><a href="https://englishbulgaria.net" target="_blank">https://englishbulgaria.net</a><br><a href="https://mjanja.ch" target="_blank">https://mjanja.ch</a><br>&quot;In heaven all the interesting people are missing.&quot; ―Friedrich Nietzsche</div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div>