<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hello Ravi and Gluster Community,<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Am Mo., 24. Juni 2019 um 14:25 Uhr schrieb David Spisla <<a href="mailto:spisla80@gmail.com">spisla80@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">---------- Forwarded message ---------<br>Von: <b class="gmail_sendername" dir="auto">David Spisla</b> <span dir="auto"><<a href="mailto:spisla80@gmail.com" target="_blank">spisla80@gmail.com</a>></span><br>Date: Fr., 21. Juni 2019 um 10:02 Uhr<br>Subject: Re: [Gluster-users] Pending heal status when deleting files which are marked as to be healed<br>To: Ravishankar N <<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>><br></div><br><br><div dir="ltr"><div dir="ltr">Hello Ravi,<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Am Mi., 19. Juni 2019 um 18:06 Uhr schrieb Ravishankar N <<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><br>
</p>
<div class="gmail-m_2164346705491354316m_4301361431205782487gmail-m_-8255022584843533889moz-cite-prefix">On 17/06/19 3:45 PM, David Spisla
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hello Gluster Community,</div>
<div><br>
</div>
<div>my newest observation concerns the self heal daemon:</div>
<div>Scenario: 2 Node Gluster v5.5 Cluster with Replica 2
Volume. Just one brick per node. Access via SMB Client from a
Win10 machine<br>
</div>
<div><br>
</div>
<div>How to reproduce:</div>
<div>I have created a small folder with a lot of small files and
I copied that folder recursively into itself for a few times.
Additionally I copied three big folders with a lot of content
into the root of the volume. <br>
</div>
<div>Note: There was no node down or something else like brick
down, etc.. So the whole volume was accessible.</div>
<div><br>
</div>
<div>Because of the recursively copy action all this copied
files whre listed as to be healed (via gluster heal info).</div>
</div>
</blockquote>
<p>This is odd. How did you conclude that writing to the volume
(i.e. recursive copy) was the reason for the files to be needing
heal? Did you check if there were any gluster messages about
disconnects in the smb client logs?<br></p></div></blockquote><div>There was no disconnection, I am sure. But at all I am not really sure whats the cause of this problem. <br></div></div></div></div></div></blockquote><div>I reproduce it. Now I don't think that recursive copy is the reason. I copied several small files in the volume (capacity 1GB) unless it is full (see steps to reproduce below). I didn't set RO to the file. There was never a disconnection.<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_quote"><div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div> Now I set some of the effected files ReadOnly (they get
WORMed because worm-file-level is enabled). After this I tried
to delete the parent folder of that files.</div>
<div><br>
</div>
<div>Expected: All files should be healed</div>
<div>Actually: All files, which are Read-Only, are not healed.
heal info shows permanently that this files has to be healed.
<br>
</div>
</div>
</blockquote>
Does disabling read-only let the files to be healed?<br></div></blockquote><div>I have to ty this. <br></div></div></div></div></div></blockquote><div>I tried it out and it had no efffect. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_quote"><div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>glustershd log throws error and brick log (with level
DEBUG) permanently throws a lot of messages which I don't
understand. See the attached file which contains all
informations, also heal info and volume info, beside the logs<br>
</div>
<div><br>
</div>
<div>Maybe some of you know whats going on there? Since we can
reproduce this scenario, we can give more debug information if
needed.</div>
</div>
</blockquote>
<p>Is it possible to script the list of steps to reproduce this
issue? <br></p></div></blockquote><div>I will do that and post it here. Although I will collect more data when it happens</div></div></div></div></div></blockquote><div>Steps to reproduce:</div><div><br></div><div>1. Copy several small files into a volume (here: 1GB capacity)</div><div>2. Copy until the volume is nearly full (70-80% or more)</div><div>3. Now self-heal is listing files to be healed</div><div>4. Move or delete all of this files or a just a part.</div><div>5. The files won't be healed and stay in the heal info list.</div><div><br></div><div>In my case I copied until the volume was 100% full (storage.reserve was 1%). I delete some of the files, to get a level of 98%. I wait for a while but nothing happens. After this I stopped and started the volume. Files are now healed.</div><div>Attached there is the glustershd.log where you can see that performing entry.self-heal (2019-06-24 10:04:02.007328) could not be finished for pgfid:7e4fa649-434a-4bb7-a1c2-258818d76076 until the volume was stopped and started again. After starting again entry.self-heal could be finished for that pgfid (at 2019-06-24 12:38:38.689632). The pgfid refers to the files which were listed to be healed:<br></div><div><br></div><div>fs-davids-c2-n1:~ # gluster vo heal archive1 info<br>Brick fs-davids-c2-n1:/gluster/brick1/glusterbrick<br>/archive1/data/fff/gg - Kopie.txt <br>/archive1/data/fff <br>/archive1/data/fff/gg - Kopie - Kopie.txt <br>/archive1/data/fff/gg - Kopie - Kopie (2).txt <br>Status: Connected<br>Number of entries: 4<br></div><div><br></div><div>All of this files has the same pgfid:</div><div><br></div><div>fs-davids-c2-n1:~ # getfattr -e hex -d -m "" '/gluster/brick1/glusterbrick/archive1/data/fff/'* | grep trusted.pgfid<br>getfattr: Removing leading '/' from absolute path names<br>trusted.pgfid.7e4fa649-434a-4bb7-a1c2-258818d76076=0x00000001<br>trusted.pgfid.7e4fa649-434a-4bb7-a1c2-258818d76076=0x00000001<br>trusted.pgfid.7e4fa649-434a-4bb7-a1c2-258818d76076=0x00000001</div><div><br></div><div>Summary: The pending heal problem seems to occur if a volume is nearly full or completely full.<br></div><div><br></div><div>Regards</div><div>David Spisla<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>Regards</div><div>David<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
</p>
<p>Regards,</p>
<p>Ravi<br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Regards</div>
<div>David Spisla<br>
</div>
<div><br>
</div>
</div>
<br>
<fieldset class="gmail-m_2164346705491354316m_4301361431205782487gmail-m_-8255022584843533889mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_2164346705491354316m_4301361431205782487gmail-m_-8255022584843533889moz-quote-pre">_______________________________________________
Gluster-users mailing list
<a class="gmail-m_2164346705491354316m_4301361431205782487gmail-m_-8255022584843533889moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a class="gmail-m_2164346705491354316m_4301361431205782487gmail-m_-8255022584843533889moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
</div>
</blockquote></div></div>
</div></div>
</blockquote></div></div></div></div></div></div></div></div></div></div>