<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>I wrote a script to search the output of gluster volume heal
projects info, picks the brick I gave it and then deletes any of
the files listed that actually exist in .glusterfs/dir1/dir2. I
did this on the first host which had 85 pending and that cleared
them up so I'll do it via ssh on the other two servers.</p>
<p>Hopefully that will clear it up and glusterfs will be happy
again.</p>
<p>Thanks everyone for the help.</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 12/31/18 4:39 AM, Davide Obbi wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAH2S_oTB4=M7N+5sP8SH5njzFghaMmyB7HO1s10cwgjkULqymw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">cluster.quorum-type auto<br>
cluster.quorum-count (null)<br>
cluster.server-quorum-type off<br>
cluster.server-quorum-ratio 0<br>
cluster.quorum-reads no<br>
<br>
Where exacty do I remove the gfid entries from - the .glusterfs
<br>
<div>
directory? --> yes can't remember exactly where but try to
do a find in the brick paths with the gfid it should return
something<br>
</div>
<br>
Where do I put the cluster.heal-timeout option - which file?
--> gluster volume set volumename option value <br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Mon, Dec 31, 2018 at 10:34 AM Brett Holcomb
<<a href="mailto:biholcomb@l1049h.com"
moz-do-not-send="true">biholcomb@l1049h.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">That
is probably the case as a lot of files were deleted some time
ago.<br>
<br>
I'm on version 5.2 but was on 3.12 until about a week ago.<br>
<br>
Here is the quorum info. I'm running a distributed replicated
volumes <br>
in 2 x 3 = 6<br>
<br>
cluster.quorum-type auto<br>
cluster.quorum-count (null)<br>
cluster.server-quorum-type off<br>
cluster.server-quorum-ratio 0<br>
cluster.quorum-reads no<br>
<br>
Where exacty do I remove the gfid entries from - the
.glusterfs <br>
directory? Do I just delete all the directories can files
under this <br>
directory?<br>
<br>
Where do I put the cluster.heal-timeout option - which file?<br>
<br>
I think you've hit on the cause of the issue. Thinking back
we've had <br>
some extended power outages and due to a misconfiguration in
the swap <br>
file device name a couple of the nodes did not come up and I
didn't <br>
catch it for a while so maybe the deletes occured then.<br>
<br>
Thank you.<br>
<br>
On 12/31/18 2:58 AM, Davide Obbi wrote:<br>
> if the long GFID does not correspond to any file it could
mean the <br>
> file has been deleted by the client mounting the volume.
I think this <br>
> is caused when the delete was issued and the number of
active bricks <br>
> were not reaching quorum majority or a second brick was
taken down <br>
> while another was down or did not finish the selfheal,
the latter more <br>
> likely.<br>
> It would be interesting to see:<br>
> - what version of glusterfs you running, it happened to
me with 3.12<br>
> - volume quorum rules: "gluster volume get vol all | grep
quorum"<br>
><br>
> To clean it up if i remember correctly it should be
possible to delete <br>
> the gfid entries from the brick mounts on the glusterfs
server nodes <br>
> reporting the files to heal.<br>
><br>
> As a side note you might want to consider changing the
selfheal <br>
> timeout to more agressive schedule in
cluster.heal-timeout option<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank"
moz-do-not-send="true">Gluster-users@gluster.org</a><br>
<a
href="https://lists.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" class="gmail_signature"><span
style="display:block;font-size:11pt;font-family:Century
Gothic;color:rgb(0,53,128)">
<div
style="color:rgb(0,53,128);font-family:Arial,Helvetica,sans-serif;font-weight:bold;font-size:15px">
<div>Davide Obbi</div>
<div
style="font-weight:normal;font-size:13px;color:rgb(0,174,239)">System
Administrator<br>
<br>
</div>
<div
style="font-weight:normal;font-size:13px;color:rgb(102,102,102)">Booking.com
B.V.<br>
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands</div>
<div
style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><span
style="color:rgb(0,174,239)">Direct </span>+31207031558<br>
</div>
<div
style="font-weight:normal;font-size:13px;color:rgb(102,102,102)">
<div
style="font-weight:bold;font-size:16px;color:rgb(0,53,128)"><a
href="https://www.booking.com/"
style="color:rgb(0,127,255);background-image:initial;background-position:initial;background-repeat:initial"
target="_blank" moz-do-not-send="true"><img
src="https://bstatic.com/static/img/siglogo.jpg"
alt="Booking.com" title="Booking.com"
moz-do-not-send="true"></a></div>
<div><span style="font-size:11px">Empowering people to
experience the world since 1996</span></div>
<div><span style="font-size:11px">43 languages, 214+
offices worldwide, 141,000+ global destinations, 29
million reported listings <br>
</span><span style="font-size:11px">Subsidiary of
Booking Holdings Inc. (NASDAQ: BKNG)</span></div>
</div>
</div>
</span></div>
</blockquote>
</body>
</html>