<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Great, can you raise a bug for the issue so that it is easier to
keep track (plus you'll be notified if the patch is posted) of it?
The general guidelines are @
<a class="moz-txt-link-freetext" href="https://gluster.readthedocs.io/en/latest/Contributors-Guide/Bug-Reporting-Guidelines">https://gluster.readthedocs.io/en/latest/Contributors-Guide/Bug-Reporting-Guidelines</a>
but you just need to provide whatever you described in this email
thread in the bug:<br>
<br>
i.e. volume info, heal info, getfattr and stat output of the file in
question.<br>
<br>
Thanks!<br>
Ravi<br>
<br>
<div class="moz-cite-prefix">On 08/28/2017 07:49 PM, mabi wrote:<br>
</div>
<blockquote type="cite"
cite="mid:2Hp1IwJOgbOoBT_2W6OEbZ7bl7-QzMTFUdrPEWR3flgeRP_Y5Wi2KeQR3SXYidwldXDqROxSJHuqa09te1z9s66kn2aexHt9pYQ-mhxQIwk=@protonmail.ch">
<div>Thank you for the command. I ran it on all my nodes and now
finally the the self-heal daemon does not report any files to be
healed. Hopefully this scenario can get handled properly in
newer versions of GlusterFS.<br>
</div>
<div class="protonmail_signature_block
protonmail_signature_block-empty">
<div class="protonmail_signature_block-user
protonmail_signature_block-empty">
<div><br>
</div>
</div>
<div class="protonmail_signature_block-proton
protonmail_signature_block-empty"><br>
</div>
</div>
<div><br>
</div>
<blockquote type="cite" class="protonmail_quote">
<div>-------- Original Message --------<br>
</div>
<div>Subject: Re: [Gluster-users] self-heal not working<br>
</div>
<div>Local Time: August 28, 2017 10:41 AM<br>
</div>
<div>UTC Time: August 28, 2017 8:41 AM<br>
</div>
<div>From: <a class="moz-txt-link-abbreviated" href="mailto:ravishankar@redhat.com">ravishankar@redhat.com</a><br>
</div>
<div>To: mabi <a class="moz-txt-link-rfc2396E" href="mailto:mabi@protonmail.ch"><mabi@protonmail.ch></a><br>
</div>
<div>Ben Turner <a class="moz-txt-link-rfc2396E" href="mailto:bturner@redhat.com"><bturner@redhat.com></a>, Gluster Users
<a class="moz-txt-link-rfc2396E" href="mailto:gluster-users@gluster.org"><gluster-users@gluster.org></a><br>
</div>
<div><br>
</div>
<div> <br>
</div>
<p><br>
</p>
<div><br>
</div>
<div class="moz-cite-prefix">On 08/28/2017 01:29 PM, mabi wrote:<br>
</div>
<blockquote type="cite">
<div>Excuse me for my naive questions but how do I reset the
afr.dirty xattr on the file to be healed? and do I need to
do that through a FUSE mount? or simply on every bricks
directly? <br>
</div>
<div class="protonmail_signature_block
protonmail_signature_block-empty">
<div class="protonmail_signature_block-user
protonmail_signature_block-empty">
<div><br>
</div>
</div>
<div class="protonmail_signature_block-proton
protonmail_signature_block-empty"><br>
</div>
</div>
</blockquote>
<div>Directly on the bricks: `setfattr -n trusted.afr.dirty -v
0x000000000000000000000000
/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png`<br>
</div>
<div> -Ravi<br>
</div>
<div> <br>
</div>
<blockquote type="cite">
<div><br>
</div>
<blockquote class="protonmail_quote" type="cite">
<div>-------- Original Message --------<br>
</div>
<div>Subject: Re: [Gluster-users] self-heal not working<br>
</div>
<div>Local Time: August 28, 2017 5:58 AM<br>
</div>
<div>UTC Time: August 28, 2017 3:58 AM<br>
</div>
<div>From: <a class="moz-txt-link-abbreviated"
href="mailto:ravishankar@redhat.com"
moz-do-not-send="true">ravishankar@redhat.com</a><br>
</div>
<div>To: Ben Turner <a class="moz-txt-link-rfc2396E"
href="mailto:bturner@redhat.com" moz-do-not-send="true"><bturner@redhat.com></a>,
mabi <a class="moz-txt-link-rfc2396E"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true"><mabi@protonmail.ch></a><br>
</div>
<div>Gluster Users <a class="moz-txt-link-rfc2396E"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true"><gluster-users@gluster.org></a><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>On 08/28/2017 01:57 AM, Ben Turner wrote:<br>
</div>
<div>> ----- Original Message -----<br>
</div>
<div>>> From: "mabi" <a class="moz-txt-link-rfc2396E"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true"><mabi@protonmail.ch></a><br>
</div>
<div>>> To: "Ravishankar N" <a
class="moz-txt-link-rfc2396E"
href="mailto:ravishankar@redhat.com"
moz-do-not-send="true"><ravishankar@redhat.com></a><br>
</div>
<div>>> Cc: "Ben Turner" <a
class="moz-txt-link-rfc2396E"
href="mailto:bturner@redhat.com" moz-do-not-send="true"><bturner@redhat.com></a>,
"Gluster Users" <a class="moz-txt-link-rfc2396E"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true"><gluster-users@gluster.org></a><br>
</div>
<div>>> Sent: Sunday, August 27, 2017 3:15:33 PM<br>
</div>
<div>>> Subject: Re: [Gluster-users] self-heal not
working<br>
</div>
<div>>><br>
</div>
<div>>> Thanks Ravi for your analysis. So as far as I
understand nothing to worry<br>
</div>
<div>>> about but my question now would be: how do I
get rid of this file from the<br>
</div>
<div>>> heal info?<br>
</div>
<div>> Correct me if I am wrong but clearing this is just
a matter of resetting the afr.dirty xattr? @Ravi - Is this
correct?<br>
</div>
<div><br>
</div>
<div>Yes resetting the xattr and launching index heal or
running heal-info <br>
</div>
<div>command should serve as a workaround.<br>
</div>
<div>-Ravi<br>
</div>
<div><br>
</div>
<div>><br>
</div>
<div>> -b<br>
</div>
<div>><br>
</div>
<div>>>> -------- Original Message --------<br>
</div>
<div>>>> Subject: Re: [Gluster-users] self-heal not
working<br>
</div>
<div>>>> Local Time: August 27, 2017 3:45 PM<br>
</div>
<div>>>> UTC Time: August 27, 2017 1:45 PM<br>
</div>
<div>>>> From: <a class="moz-txt-link-abbreviated"
href="mailto:ravishankar@redhat.com"
moz-do-not-send="true">ravishankar@redhat.com</a><br>
</div>
<div>>>> To: mabi <a class="moz-txt-link-rfc2396E"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true"><mabi@protonmail.ch></a><br>
</div>
<div>>>> Ben Turner <a
class="moz-txt-link-rfc2396E"
href="mailto:bturner@redhat.com" moz-do-not-send="true"><bturner@redhat.com></a>,
Gluster Users <a class="moz-txt-link-rfc2396E"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true"><gluster-users@gluster.org></a><br>
</div>
<div>>>><br>
</div>
<div>>>> Yes, the shds did pick up the file for
healing (I saw messages like " got<br>
</div>
<div>>>> entry:
1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea") but no error
afterwards.<br>
</div>
<div>>>><br>
</div>
<div>>>> Anyway I reproduced it by manually setting
the afr.dirty bit for a zero<br>
</div>
<div>>>> byte file on all 3 bricks. Since there are
no afr pending xattrs<br>
</div>
<div>>>> indicating good/bad copies and all files
are zero bytes, the data<br>
</div>
<div>>>> self-heal algorithm just picks the file
with the latest ctime as source.<br>
</div>
<div>>>> In your case that was the arbiter brick.
In the code, there is a check to<br>
</div>
<div>>>> prevent data heals if arbiter is the
source. So heal was not happening and<br>
</div>
<div>>>> the entries were not removed from
heal-info output.<br>
</div>
<div>>>><br>
</div>
<div>>>> Perhaps we should add a check in the code
to just remove the entries from<br>
</div>
<div>>>> heal-info if size is zero bytes in all
bricks.<br>
</div>
<div>>>><br>
</div>
<div>>>> -Ravi<br>
</div>
<div>>>><br>
</div>
<div>>>> On 08/25/2017 06:33 PM, mabi wrote:<br>
</div>
<div>>>><br>
</div>
<div>>>>> Hi Ravi,<br>
</div>
<div>>>>><br>
</div>
<div>>>>> Did you get a chance to have a look at
the log files I have attached in my<br>
</div>
<div>>>>> last mail?<br>
</div>
<div>>>>><br>
</div>
<div>>>>> Best,<br>
</div>
<div>>>>> Mabi<br>
</div>
<div>>>>><br>
</div>
<div>>>>>> -------- Original Message --------<br>
</div>
<div>>>>>> Subject: Re: [Gluster-users]
self-heal not working<br>
</div>
<div>>>>>> Local Time: August 24, 2017 12:08
PM<br>
</div>
<div>>>>>> UTC Time: August 24, 2017 10:08 AM<br>
</div>
<div>>>>>> From: <a
class="moz-txt-link-abbreviated"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true">mabi@protonmail.ch</a><br>
</div>
<div>>>>>> To: Ravishankar N<br>
</div>
<div>>>>>> [<a class="moz-txt-link-rfc2396E"
href="mailto:ravishankar@redhat.com"
moz-do-not-send="true"><ravishankar@redhat.com></a>](<a
class="moz-txt-link-freetext"
href="mailto:ravishankar@redhat.com"
moz-do-not-send="true">mailto:ravishankar@redhat.com</a>)<br>
</div>
<div>>>>>> Ben Turner [<a
class="moz-txt-link-rfc2396E"
href="mailto:bturner@redhat.com" moz-do-not-send="true"><bturner@redhat.com></a>](<a
class="moz-txt-link-freetext"
href="mailto:bturner@redhat.com" moz-do-not-send="true">mailto:bturner@redhat.com</a>),
Gluster<br>
</div>
<div>>>>>> Users [<a
class="moz-txt-link-rfc2396E"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true"><gluster-users@gluster.org></a>](<a
class="moz-txt-link-freetext"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true">mailto:gluster-users@gluster.org</a>)<br>
</div>
<div>>>>>><br>
</div>
<div>>>>>> Thanks for confirming the command.
I have now enabled DEBUG<br>
</div>
<div>>>>>> client-log-level, run a heal and
then attached the glustershd log files<br>
</div>
<div>>>>>> of all 3 nodes in this mail.<br>
</div>
<div>>>>>><br>
</div>
<div>>>>>> The volume concerned is called
myvol-pro, the other 3 volumes have no<br>
</div>
<div>>>>>> problem so far.<br>
</div>
<div>>>>>><br>
</div>
<div>>>>>> Also note that in the mean time it
looks like the file has been deleted<br>
</div>
<div>>>>>> by the user and as such the heal
info command does not show the file<br>
</div>
<div>>>>>> name anymore but just is GFID
which is:<br>
</div>
<div>>>>>><br>
</div>
<div>>>>>>
gfid:1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea<br>
</div>
<div>>>>>><br>
</div>
<div>>>>>> Hope that helps for debugging this
issue.<br>
</div>
<div>>>>>><br>
</div>
<div>>>>>>> -------- Original Message
--------<br>
</div>
<div>>>>>>> Subject: Re: [Gluster-users]
self-heal not working<br>
</div>
<div>>>>>>> Local Time: August 24, 2017
5:58 AM<br>
</div>
<div>>>>>>> UTC Time: August 24, 2017 3:58
AM<br>
</div>
<div>>>>>>> From: <a
class="moz-txt-link-abbreviated"
href="mailto:ravishankar@redhat.com"
moz-do-not-send="true">ravishankar@redhat.com</a><br>
</div>
<div>>>>>>> To: mabi [<a
class="moz-txt-link-rfc2396E"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true"><mabi@protonmail.ch></a>](<a
class="moz-txt-link-freetext"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true">mailto:mabi@protonmail.ch</a>)<br>
</div>
<div>>>>>>> Ben Turner [<a
class="moz-txt-link-rfc2396E"
href="mailto:bturner@redhat.com" moz-do-not-send="true"><bturner@redhat.com></a>](<a
class="moz-txt-link-freetext"
href="mailto:bturner@redhat.com" moz-do-not-send="true">mailto:bturner@redhat.com</a>),
Gluster<br>
</div>
<div>>>>>>> Users [<a
class="moz-txt-link-rfc2396E"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true"><gluster-users@gluster.org></a>](<a
class="moz-txt-link-freetext"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true">mailto:gluster-users@gluster.org</a>)<br>
</div>
<div>>>>>>><br>
</div>
<div>>>>>>> Unlikely. In your case only
the afr.dirty is set, not the<br>
</div>
<div>>>>>>> afr.volname-client-xx xattr.<br>
</div>
<div>>>>>>><br>
</div>
<div>>>>>>> `gluster volume set myvolume
diagnostics.client-log-level DEBUG` is<br>
</div>
<div>>>>>>> right.<br>
</div>
<div>>>>>>><br>
</div>
<div>>>>>>> On 08/23/2017 10:31 PM, mabi
wrote:<br>
</div>
<div>>>>>>><br>
</div>
<div>>>>>>>> I just saw the following
bug which was fixed in 3.8.15:<br>
</div>
<div>>>>>>>><br>
</div>
<div>>>>>>>> <a
class="moz-txt-link-freetext"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1471613"
moz-do-not-send="true">https://bugzilla.redhat.com/show_bug.cgi?id=1471613</a><br>
</div>
<div>>>>>>>><br>
</div>
<div>>>>>>>> Is it possible that the
problem I described in this post is related to<br>
</div>
<div>>>>>>>> that bug?<br>
</div>
<div>>>>>>>><br>
</div>
<div>>>>>>>>> -------- Original
Message --------<br>
</div>
<div>>>>>>>>> Subject: Re:
[Gluster-users] self-heal not working<br>
</div>
<div>>>>>>>>> Local Time: August 22,
2017 11:51 AM<br>
</div>
<div>>>>>>>>> UTC Time: August 22,
2017 9:51 AM<br>
</div>
<div>>>>>>>>> From: <a
class="moz-txt-link-abbreviated"
href="mailto:ravishankar@redhat.com"
moz-do-not-send="true">ravishankar@redhat.com</a><br>
</div>
<div>>>>>>>>> To: mabi [<a
class="moz-txt-link-rfc2396E"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true"><mabi@protonmail.ch></a>](<a
class="moz-txt-link-freetext"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true">mailto:mabi@protonmail.ch</a>)<br>
</div>
<div>>>>>>>>> Ben Turner [<a
class="moz-txt-link-rfc2396E"
href="mailto:bturner@redhat.com" moz-do-not-send="true"><bturner@redhat.com></a>](<a
class="moz-txt-link-freetext"
href="mailto:bturner@redhat.com" moz-do-not-send="true">mailto:bturner@redhat.com</a>),
Gluster<br>
</div>
<div>>>>>>>>> Users [<a
class="moz-txt-link-rfc2396E"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true"><gluster-users@gluster.org></a>](<a
class="moz-txt-link-freetext"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true">mailto:gluster-users@gluster.org</a>)<br>
</div>
<div>>>>>>>>><br>
</div>
<div>>>>>>>>> On 08/22/2017 02:30
PM, mabi wrote:<br>
</div>
<div>>>>>>>>><br>
</div>
<div>>>>>>>>>> Thanks for the
additional hints, I have the following 2 questions<br>
</div>
<div>>>>>>>>>> first:<br>
</div>
<div>>>>>>>>>><br>
</div>
<div>>>>>>>>>> - In order to
launch the index heal is the following command correct:<br>
</div>
<div>>>>>>>>>> gluster volume
heal myvolume<br>
</div>
<div>>>>>>>>> Yes<br>
</div>
<div>>>>>>>>><br>
</div>
<div>>>>>>>>>> - If I run a
"volume start force" will it have any short disruptions<br>
</div>
<div>>>>>>>>>> on my clients
which mount the volume through FUSE? If yes, how long?<br>
</div>
<div>>>>>>>>>> This is a
production system that"s why I am asking.<br>
</div>
<div>>>>>>>>> No. You can actually
create a test volume on your personal linux box<br>
</div>
<div>>>>>>>>> to try these kinds of
things without needing multiple machines. This<br>
</div>
<div>>>>>>>>> is how we develop and
test our patches :)<br>
</div>
<div>>>>>>>>> "gluster volume create
testvol replica 3 /home/mabi/bricks/brick{1..3}<br>
</div>
<div>>>>>>>>> force` and so on.<br>
</div>
<div>>>>>>>>><br>
</div>
<div>>>>>>>>> HTH,<br>
</div>
<div>>>>>>>>> Ravi<br>
</div>
<div>>>>>>>>><br>
</div>
<div>>>>>>>>>>> --------
Original Message --------<br>
</div>
<div>>>>>>>>>>> Subject: Re:
[Gluster-users] self-heal not working<br>
</div>
<div>>>>>>>>>>> Local Time:
August 22, 2017 6:26 AM<br>
</div>
<div>>>>>>>>>>> UTC Time:
August 22, 2017 4:26 AM<br>
</div>
<div>>>>>>>>>>> From: <a
class="moz-txt-link-abbreviated"
href="mailto:ravishankar@redhat.com"
moz-do-not-send="true">ravishankar@redhat.com</a><br>
</div>
<div>>>>>>>>>>> To: mabi [<a
class="moz-txt-link-rfc2396E"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true"><mabi@protonmail.ch></a>](<a
class="moz-txt-link-freetext"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true">mailto:mabi@protonmail.ch</a>),
Ben<br>
</div>
<div>>>>>>>>>>> Turner [<a
class="moz-txt-link-rfc2396E"
href="mailto:bturner@redhat.com" moz-do-not-send="true"><bturner@redhat.com></a>](<a
class="moz-txt-link-freetext"
href="mailto:bturner@redhat.com" moz-do-not-send="true">mailto:bturner@redhat.com</a>)<br>
</div>
<div>>>>>>>>>>> Gluster Users<br>
</div>
<div>>>>>>>>>>> [<a
class="moz-txt-link-rfc2396E"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true"><gluster-users@gluster.org></a>](<a
class="moz-txt-link-freetext"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true">mailto:gluster-users@gluster.org</a>)<br>
</div>
<div>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>> Explore the
following:<br>
</div>
<div>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>> - Launch index
heal and look at the glustershd logs of all bricks<br>
</div>
<div>>>>>>>>>>> for possible
errors<br>
</div>
<div>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>> - See if the
glustershd in each node is connected to all bricks.<br>
</div>
<div>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>> - If not try
to restart shd by `volume start force`<br>
</div>
<div>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>> - Launch index
heal again and try.<br>
</div>
<div>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>> - Try
debugging the shd log by setting client-log-level to DEBUG<br>
</div>
<div>>>>>>>>>>> temporarily.<br>
</div>
<div>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>> On 08/22/2017
03:19 AM, mabi wrote:<br>
</div>
<div>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>> Sure, it
doesn"t look like a split brain based on the output:<br>
</div>
<div>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>> Brick
node1.domain.tld:/data/myvolume/brick<br>
</div>
<div>>>>>>>>>>>> Status:
Connected<br>
</div>
<div>>>>>>>>>>>> Number of
entries in split-brain: 0<br>
</div>
<div>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>> Brick
node2.domain.tld:/data/myvolume/brick<br>
</div>
<div>>>>>>>>>>>> Status:
Connected<br>
</div>
<div>>>>>>>>>>>> Number of
entries in split-brain: 0<br>
</div>
<div>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>> Brick
node3.domain.tld:/srv/glusterfs/myvolume/brick<br>
</div>
<div>>>>>>>>>>>> Status:
Connected<br>
</div>
<div>>>>>>>>>>>> Number of
entries in split-brain: 0<br>
</div>
<div>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>
-------- Original Message --------<br>
</div>
<div>>>>>>>>>>>>>
Subject: Re: [Gluster-users] self-heal not working<br>
</div>
<div>>>>>>>>>>>>> Local
Time: August 21, 2017 11:35 PM<br>
</div>
<div>>>>>>>>>>>>> UTC
Time: August 21, 2017 9:35 PM<br>
</div>
<div>>>>>>>>>>>>> From:
<a class="moz-txt-link-abbreviated"
href="mailto:bturner@redhat.com" moz-do-not-send="true">bturner@redhat.com</a><br>
</div>
<div>>>>>>>>>>>>> To:
mabi [<a class="moz-txt-link-rfc2396E"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true"><mabi@protonmail.ch></a>](<a
class="moz-txt-link-freetext"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true">mailto:mabi@protonmail.ch</a>)<br>
</div>
<div>>>>>>>>>>>>>
Gluster Users<br>
</div>
<div>>>>>>>>>>>>> [<a
class="moz-txt-link-rfc2396E"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true"><gluster-users@gluster.org></a>](<a
class="moz-txt-link-freetext"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true">mailto:gluster-users@gluster.org</a>)<br>
</div>
<div>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>> Can
you also provide:<br>
</div>
<div>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>
gluster v heal <my vol> info split-brain<br>
</div>
<div>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>> If it
is split brain just delete the incorrect file from the
brick<br>
</div>
<div>>>>>>>>>>>>> and
run heal again. I haven"t tried this with arbiter but I<br>
</div>
<div>>>>>>>>>>>>> assume
the process is the same.<br>
</div>
<div>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>> -b<br>
</div>
<div>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>> -----
Original Message -----<br>
</div>
<div>>>>>>>>>>>>>>
From: "mabi" [<a class="moz-txt-link-rfc2396E"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true"><mabi@protonmail.ch></a>](<a
class="moz-txt-link-freetext"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true">mailto:mabi@protonmail.ch</a>)<br>
</div>
<div>>>>>>>>>>>>>>
To: "Ben Turner"<br>
</div>
<div>>>>>>>>>>>>>> [<a
class="moz-txt-link-rfc2396E"
href="mailto:bturner@redhat.com" moz-do-not-send="true"><bturner@redhat.com></a>](<a
class="moz-txt-link-freetext"
href="mailto:bturner@redhat.com" moz-do-not-send="true">mailto:bturner@redhat.com</a>)<br>
</div>
<div>>>>>>>>>>>>>>
Cc: "Gluster Users"<br>
</div>
<div>>>>>>>>>>>>>> [<a
class="moz-txt-link-rfc2396E"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true"><gluster-users@gluster.org></a>](<a
class="moz-txt-link-freetext"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true">mailto:gluster-users@gluster.org</a>)<br>
</div>
<div>>>>>>>>>>>>>>
Sent: Monday, August 21, 2017 4:55:59 PM<br>
</div>
<div>>>>>>>>>>>>>>
Subject: Re: [Gluster-users] self-heal not working<br>
</div>
<div>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>> Hi
Ben,<br>
</div>
<div>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>> So
it is really a 0 kBytes file everywhere (all nodes
including<br>
</div>
<div>>>>>>>>>>>>>>
the arbiter<br>
</div>
<div>>>>>>>>>>>>>>
and from the client).<br>
</div>
<div>>>>>>>>>>>>>>
Here below you will find the output you requested.
Hopefully that<br>
</div>
<div>>>>>>>>>>>>>>
will help<br>
</div>
<div>>>>>>>>>>>>>> to
find out why this specific file is not healing... Let me
know<br>
</div>
<div>>>>>>>>>>>>>> if
you need<br>
</div>
<div>>>>>>>>>>>>>>
any more information. Btw node3 is my arbiter node.<br>
</div>
<div>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>
NODE1:<br>
</div>
<div>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>
STAT:<br>
</div>
<div>>>>>>>>>>>>>>
File:<br>
</div>
<div>>>>>>>>>>>>>>
‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’<br>
</div>
<div>>>>>>>>>>>>>>
Size: 0 Blocks: 38 IO Block: 131072 regular empty file<br>
</div>
<div>>>>>>>>>>>>>>
Device: 24h/36d Inode: 10033884 Links: 2<br>
</div>
<div>>>>>>>>>>>>>>
Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: (
33/www-data)<br>
</div>
<div>>>>>>>>>>>>>>
Access: 2017-08-14 17:04:55.530681000 +0200<br>
</div>
<div>>>>>>>>>>>>>>
Modify: 2017-08-14 17:11:46.407404779 +0200<br>
</div>
<div>>>>>>>>>>>>>>
Change: 2017-08-14 17:11:46.407404779 +0200<br>
</div>
<div>>>>>>>>>>>>>>
Birth: -<br>
</div>
<div>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>
GETFATTR:<br>
</div>
<div>>>>>>>>>>>>>>
trusted.afr.dirty=0sAAAAAQAAAAAAAAAA<br>
</div>
<div>>>>>>>>>>>>>>
trusted.bit-rot.version=0sAgAAAAAAAABZhuknAAlJAg==<br>
</div>
<div>>>>>>>>>>>>>>
trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==<br>
</div>
<div>>>>>>>>>>>>>>
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOyo=<br>
</div>
<div>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>
NODE2:<br>
</div>
<div>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>
STAT:<br>
</div>
<div>>>>>>>>>>>>>>
File:<br>
</div>
<div>>>>>>>>>>>>>>
‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’<br>
</div>
<div>>>>>>>>>>>>>>
Size: 0 Blocks: 38 IO Block: 131072 regular empty file<br>
</div>
<div>>>>>>>>>>>>>>
Device: 26h/38d Inode: 10031330 Links: 2<br>
</div>
<div>>>>>>>>>>>>>>
Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: (
33/www-data)<br>
</div>
<div>>>>>>>>>>>>>>
Access: 2017-08-14 17:04:55.530681000 +0200<br>
</div>
<div>>>>>>>>>>>>>>
Modify: 2017-08-14 17:11:46.403704181 +0200<br>
</div>
<div>>>>>>>>>>>>>>
Change: 2017-08-14 17:11:46.403704181 +0200<br>
</div>
<div>>>>>>>>>>>>>>
Birth: -<br>
</div>
<div>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>
GETFATTR:<br>
</div>
<div>>>>>>>>>>>>>>
trusted.afr.dirty=0sAAAAAQAAAAAAAAAA<br>
</div>
<div>>>>>>>>>>>>>>
trusted.bit-rot.version=0sAgAAAAAAAABZhu6wAA8Hpw==<br>
</div>
<div>>>>>>>>>>>>>>
trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==<br>
</div>
<div>>>>>>>>>>>>>>
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOVE=<br>
</div>
<div>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>
NODE3:<br>
</div>
<div>>>>>>>>>>>>>>
STAT:<br>
</div>
<div>>>>>>>>>>>>>>
File:<br>
</div>
<div>>>>>>>>>>>>>>
/srv/glusterfs/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png<br>
</div>
<div>>>>>>>>>>>>>>
Size: 0 Blocks: 0 IO Block: 4096 regular empty file<br>
</div>
<div>>>>>>>>>>>>>>
Device: ca11h/51729d Inode: 405208959 Links: 2<br>
</div>
<div>>>>>>>>>>>>>>
Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: (
33/www-data)<br>
</div>
<div>>>>>>>>>>>>>>
Access: 2017-08-14 17:04:55.530681000 +0200<br>
</div>
<div>>>>>>>>>>>>>>
Modify: 2017-08-14 17:04:55.530681000 +0200<br>
</div>
<div>>>>>>>>>>>>>>
Change: 2017-08-14 17:11:46.604380051 +0200<br>
</div>
<div>>>>>>>>>>>>>>
Birth: -<br>
</div>
<div>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>
GETFATTR:<br>
</div>
<div>>>>>>>>>>>>>>
trusted.afr.dirty=0sAAAAAQAAAAAAAAAA<br>
</div>
<div>>>>>>>>>>>>>>
trusted.bit-rot.version=0sAgAAAAAAAABZe6ejAAKPAg==<br>
</div>
<div>>>>>>>>>>>>>>
trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==<br>
</div>
<div>>>>>>>>>>>>>>
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOc4=<br>
</div>
<div>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>
CLIENT GLUSTER MOUNT:<br>
</div>
<div>>>>>>>>>>>>>>
STAT:<br>
</div>
<div>>>>>>>>>>>>>>
File:<br>
</div>
<div>>>>>>>>>>>>>>
"/mnt/myvolume/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png"<br>
</div>
<div>>>>>>>>>>>>>>
Size: 0 Blocks: 0 IO Block: 131072 regular empty file<br>
</div>
<div>>>>>>>>>>>>>>
Device: 1eh/30d Inode: 11897049013408443114 Links: 1<br>
</div>
<div>>>>>>>>>>>>>>
Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: (
33/www-data)<br>
</div>
<div>>>>>>>>>>>>>>
Access: 2017-08-14 17:04:55.530681000 +0200<br>
</div>
<div>>>>>>>>>>>>>>
Modify: 2017-08-14 17:11:46.407404779 +0200<br>
</div>
<div>>>>>>>>>>>>>>
Change: 2017-08-14 17:11:46.407404779 +0200<br>
</div>
<div>>>>>>>>>>>>>>
Birth: -<br>
</div>
<div>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>
-------- Original Message --------<br>
</div>
<div>>>>>>>>>>>>>>>
Subject: Re: [Gluster-users] self-heal not working<br>
</div>
<div>>>>>>>>>>>>>>>
Local Time: August 21, 2017 9:34 PM<br>
</div>
<div>>>>>>>>>>>>>>>
UTC Time: August 21, 2017 7:34 PM<br>
</div>
<div>>>>>>>>>>>>>>>
From: <a class="moz-txt-link-abbreviated"
href="mailto:bturner@redhat.com" moz-do-not-send="true">bturner@redhat.com</a><br>
</div>
<div>>>>>>>>>>>>>>>
To: mabi [<a class="moz-txt-link-rfc2396E"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true"><mabi@protonmail.ch></a>](<a
class="moz-txt-link-freetext"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true">mailto:mabi@protonmail.ch</a>)<br>
</div>
<div>>>>>>>>>>>>>>>
Gluster Users<br>
</div>
<div>>>>>>>>>>>>>>>
[<a class="moz-txt-link-rfc2396E"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true"><gluster-users@gluster.org></a>](<a
class="moz-txt-link-freetext"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true">mailto:gluster-users@gluster.org</a>)<br>
</div>
<div>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>
----- Original Message -----<br>
</div>
<div>>>>>>>>>>>>>>>>
From: "mabi" [<a class="moz-txt-link-rfc2396E"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true"><mabi@protonmail.ch></a>](<a
class="moz-txt-link-freetext"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true">mailto:mabi@protonmail.ch</a>)<br>
</div>
<div>>>>>>>>>>>>>>>>
To: "Gluster Users"<br>
</div>
<div>>>>>>>>>>>>>>>>
[<a class="moz-txt-link-rfc2396E"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true"><gluster-users@gluster.org></a>](<a
class="moz-txt-link-freetext"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true">mailto:gluster-users@gluster.org</a>)<br>
</div>
<div>>>>>>>>>>>>>>>>
Sent: Monday, August 21, 2017 9:28:24 AM<br>
</div>
<div>>>>>>>>>>>>>>>>
Subject: [Gluster-users] self-heal not working<br>
</div>
<div>>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>>
Hi,<br>
</div>
<div>>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>>
I have a replicat 2 with arbiter GlusterFS 3.8.11 cluster
and<br>
</div>
<div>>>>>>>>>>>>>>>>
there is<br>
</div>
<div>>>>>>>>>>>>>>>>
currently one file listed to be healed as you can see
below<br>
</div>
<div>>>>>>>>>>>>>>>>
but never gets<br>
</div>
<div>>>>>>>>>>>>>>>>
healed by the self-heal daemon:<br>
</div>
<div>>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>>
Brick node1.domain.tld:/data/myvolume/brick<br>
</div>
<div>>>>>>>>>>>>>>>>
/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png<br>
</div>
<div>>>>>>>>>>>>>>>>
Status: Connected<br>
</div>
<div>>>>>>>>>>>>>>>>
Number of entries: 1<br>
</div>
<div>>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>>
Brick node2.domain.tld:/data/myvolume/brick<br>
</div>
<div>>>>>>>>>>>>>>>>
/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png<br>
</div>
<div>>>>>>>>>>>>>>>>
Status: Connected<br>
</div>
<div>>>>>>>>>>>>>>>>
Number of entries: 1<br>
</div>
<div>>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>>
Brick node3.domain.tld:/srv/glusterfs/myvolume/brick<br>
</div>
<div>>>>>>>>>>>>>>>>
/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png<br>
</div>
<div>>>>>>>>>>>>>>>>
Status: Connected<br>
</div>
<div>>>>>>>>>>>>>>>>
Number of entries: 1<br>
</div>
<div>>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>>
As once recommended on this mailing list I have mounted
that<br>
</div>
<div>>>>>>>>>>>>>>>>
glusterfs<br>
</div>
<div>>>>>>>>>>>>>>>>
volume<br>
</div>
<div>>>>>>>>>>>>>>>>
temporarily through fuse/glusterfs and ran a "stat" on
that<br>
</div>
<div>>>>>>>>>>>>>>>>
file which is<br>
</div>
<div>>>>>>>>>>>>>>>>
listed above but nothing happened.<br>
</div>
<div>>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>>
The file itself is available on all 3 nodes/bricks but on
the<br>
</div>
<div>>>>>>>>>>>>>>>>
last node it<br>
</div>
<div>>>>>>>>>>>>>>>>
has a different date. By the way this file is 0 kBytes
big. Is<br>
</div>
<div>>>>>>>>>>>>>>>>
that maybe<br>
</div>
<div>>>>>>>>>>>>>>>>
the reason why the self-heal does not work?<br>
</div>
<div>>>>>>>>>>>>>>>
Is the file actually 0 bytes or is it just 0 bytes on the<br>
</div>
<div>>>>>>>>>>>>>>>
arbiter(0 bytes<br>
</div>
<div>>>>>>>>>>>>>>>
are expected on the arbiter, it just stores metadata)? Can
you<br>
</div>
<div>>>>>>>>>>>>>>>
send us the<br>
</div>
<div>>>>>>>>>>>>>>>
output from stat on all 3 nodes:<br>
</div>
<div>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>
$ stat <file on back end brick><br>
</div>
<div>>>>>>>>>>>>>>>
$ getfattr -d -m - <file on back end brick><br>
</div>
<div>>>>>>>>>>>>>>>
$ stat <file from gluster mount><br>
</div>
<div>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>
Lets see what things look like on the back end, it should
tell<br>
</div>
<div>>>>>>>>>>>>>>>
us why<br>
</div>
<div>>>>>>>>>>>>>>>
healing is failing.<br>
</div>
<div>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>
-b<br>
</div>
<div>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>>
And how can I now make this file to heal?<br>
</div>
<div>>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>>
Thanks,<br>
</div>
<div>>>>>>>>>>>>>>>>
Mabi<br>
</div>
<div>>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>>>>>>
_______________________________________________<br>
</div>
<div>>>>>>>>>>>>>>>>
Gluster-users mailing list<br>
</div>
<div>>>>>>>>>>>>>>>>
<a class="moz-txt-link-abbreviated"
href="mailto:Gluster-users@gluster.org"
moz-do-not-send="true">Gluster-users@gluster.org</a><br>
</div>
<div>>>>>>>>>>>>>>>>
<a class="moz-txt-link-freetext"
href="http://lists.gluster.org/mailman/listinfo/gluster-users"
moz-do-not-send="true">http://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</div>
<div>>>>>>>>>>>>
_______________________________________________<br>
</div>
<div>>>>>>>>>>>>
Gluster-users mailing list<br>
</div>
<div>>>>>>>>>>>> <a
class="moz-txt-link-abbreviated"
href="mailto:Gluster-users@gluster.org"
moz-do-not-send="true">Gluster-users@gluster.org</a><br>
</div>
<div>>>>>>>>>>>><br>
</div>
<div>>>>>>>>>>>> <a
class="moz-txt-link-freetext"
href="http://lists.gluster.org/mailman/listinfo/gluster-users"
moz-do-not-send="true">http://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</div>
</blockquote>
<div><br>
</div>
</blockquote>
</blockquote>
<div><br>
</div>
</blockquote>
<br>
</body>
</html>