<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 13, 2018 at 3:39 PM, Brian Andrus <span dir="ltr"><<a href="mailto:toomuchit@gmail.com" target="_blank">toomuchit@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>All,</p>
<p>I have a 5x3 Distributed-Replicate filesystem that has a few
entries that do not clean up when being healed.</p>
<p>I had tracked down what they were and since they were really just
temp/expendable files, I moved the directory and recreated what
was needed.</p></div></blockquote><div><br></div><div>.glusterfs directory in the brick root contains metadata about the data stored in bricks. While the directory was moved, could it be possible that metadata in .glusterfs was not cleaned up?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<p>Now those files in the recreated directory cannot be deleted and
they show up in the gluster volume heal <volume> info output
and never go away.</p>
<p>Examples below:</p>
<p><b><i>Brick brick5.internal:/GLUSTER/<wbr>brick1</i></b><b><i><br>
</i></b><b><i><gfid:8ed6a9e9-2820-40bd-8d9d-<wbr>77b7f79c7748>
- Is in split-brain</i></b><b><i><br>
</i></b><b><i><br>
</i></b><b><i>Status: Connected</i></b><b><i><br>
</i></b><b><i>Number of entries: 1</i></b><b><i><br>
</i></b><b><i><br>
</i></b><b><i>Brick brick6.internal:/GLUSTER/<wbr>brick1</i></b><b><i><br>
</i></b><b><i><gfid:8ed6a9e9-2820-40bd-8d9d-<wbr>77b7f79c7748>/resv_state</i></b><b><i><br>
</i></b><b><i><gfid:8ed6a9e9-2820-40bd-8d9d-<wbr>77b7f79c7748>
- Is in split-brain</i></b><b><i><br>
</i></b><b><i><br>
</i></b><b><i><gfid:8ed6a9e9-2820-40bd-8d9d-<wbr>77b7f79c7748>/node_state</i></b><b><i><br>
</i></b><b><i><gfid:8ed6a9e9-2820-40bd-8d9d-<wbr>77b7f79c7748>/job_state.old</i></b><b><i><br>
</i></b><b><i><gfid:8ed6a9e9-2820-40bd-8d9d-<wbr>77b7f79c7748>/node_state.old</i></b><b><i><br>
</i></b><b><i>Status: Connected</i></b><b><i><br>
</i></b><b><i>Number of entries: 5</i></b><b><i><br>
</i></b></p>
<p><br>
</p>
<p>So, how do I clean those up so they aren't showing up anywhere at
all?</p></div></blockquote><div><br></div><div>output of `heal <volname> info` is derived by looking up metadata in .glusterfs and that metadata would need to be cleaned up, if stale, for the heal commands to function properly. You could move/unlink contents that correspond to stale data in .glusterfs/indices/xattrop directory under the brick root to have a clean state.</div><div><br></div><div><br></div><div>Regards,</div><div>Vijay</div></div></div></div>