<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<div class="moz-cite-prefix">On 29/05/19 3:59 AM, Alan Orth wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAKKdN4VTTmnt_-TbeT7HanMpLM+cMvm3A9_KFxZJ2HKkg3tUVA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Dear Ravishankar,</div>
<div><br>
</div>
<div>I'm not sure if Brick4 had pending AFRs because I don't
know what that means and it's been a few days so I am not sure
I would be able to find that information.</div>
</div>
</blockquote>
When you find some time, have a look at a <a moz-do-not-send="true"
href="wp.me/peiBB-6b">blog</a> series I wrote about AFR- I've
tried to explain what one needs to know to debug replication related
issues in it.<br>
<blockquote type="cite"
cite="mid:CAKKdN4VTTmnt_-TbeT7HanMpLM+cMvm3A9_KFxZJ2HKkg3tUVA@mail.gmail.com">
<div dir="ltr">
<div><br>
</div>
<div>Anyways, after wasting a few days rsyncing the old brick to
a new host I decided to just try to add the old brick back
into the volume instead of bringing it up on the new host. I
created a new brick directory on the old host, moved the old
brick's contents into that new directory (minus the .glusterfs
directory), added the new brick to the volume, and then did
Vlad's find/stat trick¹ from the brick to the FUSE mount
point.</div>
<div><br>
</div>
<div>The interesting problem I have now is that some files don't
appear in the FUSE mount's directory listings, but I can
actually list them directly and even read them. What could
cause that?<br>
</div>
</div>
</blockquote>
Not sure, too many variables in the hacks that you did to take a
guess. You can check if the contents of the .glusterfs folder are in
order on the new brick (example hardlink for files and symlinks for
directories are present etc.) .<br>
Regards,<br>
Ravi<br>
<blockquote type="cite"
cite="mid:CAKKdN4VTTmnt_-TbeT7HanMpLM+cMvm3A9_KFxZJ2HKkg3tUVA@mail.gmail.com">
<div dir="ltr">
<div><br>
</div>
<div>Thanks,<br>
</div>
<div><br>
</div>
<div>¹ <a
href="https://lists.gluster.org/pipermail/gluster-users/2018-February/033584.html"
moz-do-not-send="true">https://lists.gluster.org/pipermail/gluster-users/2018-February/033584.html</a></div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, May 24, 2019 at 4:59
PM Ravishankar N <<a href="mailto:ravishankar@redhat.com"
moz-do-not-send="true">ravishankar@redhat.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><br>
</p>
<div class="gmail-m_-6098182312870478829moz-cite-prefix">On
23/05/19 2:40 AM, Alan Orth wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Dear list,
<div><br>
</div>
<div>I seem to have gotten into a tricky situation.
Today I brought up a shiny new server with new disk
arrays and attempted to replace one brick of a replica
2 distribute/replicate volume on an older server using
the `replace-brick` command:</div>
<div><br>
</div>
<div># gluster volume replace-brick homes
wingu0:/mnt/gluster/homes
wingu06:/data/glusterfs/sdb/homes commit force</div>
<div><br>
</div>
<div>The command was successful and I see the new brick
in the output of `gluster volume info`. The problem is
that Gluster doesn't seem to be migrating the data, </div>
</div>
</blockquote>
<p>`replace-brick` definitely must heal (not migrate) the
data. In your case, data must have been healed from
Brick-4 to the replaced Brick-3. Are there any errors in
the self-heal daemon logs of Brick-4's node? Does Brick-4
have pending AFR xattrs blaming Brick-3? The doc is a bit
out of date. replace-brick command internally does all the
setfattr steps that are mentioned in the doc.</p>
<p>-Ravi<br>
</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div>and now the original brick that I replaced is no
longer part of the volume (and a few terabytes of data
are just sitting on the old brick):</div>
<div><br>
</div>
<div># gluster volume info homes | grep -E "Brick[0-9]:"<br>
Brick1: wingu4:/mnt/gluster/homes<br>
Brick2: wingu3:/mnt/gluster/homes<br>
Brick3: wingu06:/data/glusterfs/sdb/homes<br>
Brick4: wingu05:/data/glusterfs/sdb/homes<br>
Brick5: wingu05:/data/glusterfs/sdc/homes<br>
Brick6: wingu06:/data/glusterfs/sdc/homes</div>
<br>
<div>I see the Gluster docs have a more complicated
procedure for replacing bricks that involves
getfattr/setfattr¹. How can I tell Gluster about the
old brick? I see that I have a backup of the old
volfile thanks to yum's rpmsave function if that
helps.<br>
</div>
<div><br>
</div>
<div>We are using Gluster 5.6 on CentOS 7. Thank you for
any advice you can give.<br>
</div>
<div><br>
</div>
<div>¹ <a
href="https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#replace-faulty-brick"
target="_blank" moz-do-not-send="true">https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#replace-faulty-brick</a></div>
<div><br>
</div>
<div>-- <br>
<div dir="ltr"
class="gmail-m_-6098182312870478829gmail_signature">Alan
Orth<br>
<a href="mailto:alan.orth@gmail.com" target="_blank"
moz-do-not-send="true">alan.orth@gmail.com</a><br>
<a href="https://picturingjordan.com"
target="_blank" moz-do-not-send="true">https://picturingjordan.com</a><br>
<a href="https://englishbulgaria.net"
target="_blank" moz-do-not-send="true">https://englishbulgaria.net</a><br>
<a href="https://mjanja.ch" target="_blank"
moz-do-not-send="true">https://mjanja.ch</a><br>
"In heaven all the interesting people are missing."
―Friedrich Nietzsche</div>
</div>
</div>
<br>
<fieldset
class="gmail-m_-6098182312870478829mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-6098182312870478829moz-quote-pre">_______________________________________________
Gluster-users mailing list
<a class="gmail-m_-6098182312870478829moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a class="gmail-m_-6098182312870478829moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" class="gmail_signature">Alan Orth<br>
<a href="mailto:alan.orth@gmail.com" target="_blank"
moz-do-not-send="true">alan.orth@gmail.com</a><br>
<a href="https://picturingjordan.com" target="_blank"
moz-do-not-send="true">https://picturingjordan.com</a><br>
<a href="https://englishbulgaria.net" target="_blank"
moz-do-not-send="true">https://englishbulgaria.net</a><br>
<a href="https://mjanja.ch" target="_blank"
moz-do-not-send="true">https://mjanja.ch</a><br>
"In heaven all the interesting people are missing." ―Friedrich
Nietzsche</div>
</blockquote>
</body>
</html>