<div dir="ltr"><div dir="ltr">Brett,<div><br></div></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 5, 2019 at 3:54 AM Brett Holcomb <<a href="mailto:biholcomb@l1049h.com">biholcomb@l1049h.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I wrote a script to search the output of gluster volume heal
projects info, picks the brick I gave it and then deletes any of
the files listed that actually exist in .glusterfs/dir1/dir2. I
did this on the first host which had 85 pending and that cleared
them up so I'll do it via ssh on the other two servers.</p>
<p>Hopefully that will clear it up and glusterfs will be happy
again.</p>
<p></p></div></blockquote><div>If things are fine now, consider posting those scripts as patch to glusterfs, or post in your own github account, so in future we can refer others to use same scripts when in trouble. Thanks.</div><div><br></div><div>-Amar</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>Thanks everyone for the help.</p>
<p><br>
</p>
<div class="gmail-m_-3663921215443053074moz-cite-prefix">On 12/31/18 4:39 AM, Davide Obbi wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">cluster.quorum-type auto<br>
cluster.quorum-count (null)<br>
cluster.server-quorum-type off<br>
cluster.server-quorum-ratio 0<br>
cluster.quorum-reads no<br>
<br>
Where exacty do I remove the gfid entries from - the .glusterfs
<br>
<div>
directory? --> yes can't remember exactly where but try to
do a find in the brick paths with the gfid it should return
something<br>
</div>
<br>
Where do I put the cluster.heal-timeout option - which file?
--> gluster volume set volumename option value <br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Mon, Dec 31, 2018 at 10:34 AM Brett Holcomb
<<a href="mailto:biholcomb@l1049h.com" target="_blank">biholcomb@l1049h.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">That
is probably the case as a lot of files were deleted some time
ago.<br>
<br>
I'm on version 5.2 but was on 3.12 until about a week ago.<br>
<br>
Here is the quorum info. I'm running a distributed replicated
volumes <br>
in 2 x 3 = 6<br>
<br>
cluster.quorum-type auto<br>
cluster.quorum-count (null)<br>
cluster.server-quorum-type off<br>
cluster.server-quorum-ratio 0<br>
cluster.quorum-reads no<br>
<br>
Where exacty do I remove the gfid entries from - the
.glusterfs <br>
directory? Do I just delete all the directories can files
under this <br>
directory?<br>
<br>
Where do I put the cluster.heal-timeout option - which file?<br>
<br>
I think you've hit on the cause of the issue. Thinking back
we've had <br>
some extended power outages and due to a misconfiguration in
the swap <br>
file device name a couple of the nodes did not come up and I
didn't <br>
catch it for a while so maybe the deletes occured then.<br>
<br>
Thank you.<br>
<br>
On 12/31/18 2:58 AM, Davide Obbi wrote:<br>
> if the long GFID does not correspond to any file it could
mean the <br>
> file has been deleted by the client mounting the volume.
I think this <br>
> is caused when the delete was issued and the number of
active bricks <br>
> were not reaching quorum majority or a second brick was
taken down <br>
> while another was down or did not finish the selfheal,
the latter more <br>
> likely.<br>
> It would be interesting to see:<br>
> - what version of glusterfs you running, it happened to
me with 3.12<br>
> - volume quorum rules: "gluster volume get vol all | grep
quorum"<br>
><br>
> To clean it up if i remember correctly it should be
possible to delete <br>
> the gfid entries from the brick mounts on the glusterfs
server nodes <br>
> reporting the files to heal.<br>
><br>
> As a side note you might want to consider changing the
selfheal <br>
> timeout to more agressive schedule in
cluster.heal-timeout option<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" class="gmail-m_-3663921215443053074gmail_signature"><span style="display:block;font-size:11pt;font-family:"Century Gothic";color:rgb(0,53,128)">
<div style="color:rgb(0,53,128);font-family:Arial,Helvetica,sans-serif;font-weight:bold;font-size:15px">
<div>Davide Obbi</div>
<div style="font-weight:normal;font-size:13px;color:rgb(0,174,239)">System
Administrator<br>
<br>
</div>
<div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)">Booking.com
B.V.<br>
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands</div>
<div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><span style="color:rgb(0,174,239)">Direct </span>+31207031558<br>
</div>
<div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)">
<div style="font-weight:bold;font-size:16px;color:rgb(0,53,128)"><a href="https://www.booking.com/" style="color:rgb(0,127,255);background-image:initial;background-position:initial;background-repeat:initial" target="_blank"><img src="https://bstatic.com/static/img/siglogo.jpg" alt="Booking.com" title="Booking.com"></a></div>
<div><span style="font-size:11px">Empowering people to
experience the world since 1996</span></div>
<div><span style="font-size:11px">43 languages, 214+
offices worldwide, 141,000+ global destinations, 29
million reported listings <br>
</span><span style="font-size:11px">Subsidiary of
Booking Holdings Inc. (NASDAQ: BKNG)</span></div>
</div>
</div>
</span></div>
</blockquote>
</div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div></div>