<div dir="ltr"><div><div><div><div><div><div>Hey<span name="Richard Neuboeck" class="gmail-gD"> Richard,<br><br></span></div><span name="Richard Neuboeck" class="gmail-gD">Could you share the following informations please?<br></span></div><span name="Richard Neuboeck" class="gmail-gD">1. gluster volume info <volname><br></span></div><span name="Richard Neuboeck" class="gmail-gD">2. getfattr output of that file from all the bricks<br> getfattr -d -e hex -m . <brickpath/filepath><br></span></div><span name="Richard Neuboeck" class="gmail-gD">3. glustershd & glfsheal logs<br><br></span></div><span name="Richard Neuboeck" class="gmail-gD">Regards,<br></span></div><span name="Richard Neuboeck" class="gmail-gD">Karthik<br></span></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 26, 2017 at 10:21 AM, Amar Tumballi <span dir="ltr"><<a href="mailto:atumball@redhat.com" target="_blank">atumball@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">On a side note, try recently released health report tool, and see if it does diagnose any issues in setup. Currently you may have to run it in all the three machines.<div dir="auto"><br></div><div dir="auto"><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 26-Oct-2017 6:50 AM, "Amar Tumballi" <<a href="mailto:atumball@redhat.com" target="_blank">atumball@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">Thanks for this report. This week many of the developers are at Gluster Summit in Prague, will be checking this and respond next week. Hope that's fine.<div dir="auto"><br></div><div dir="auto">Thanks,</div><div dir="auto">Amar</div><div dir="auto"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 25-Oct-2017 3:07 PM, "Richard Neuboeck" <<a href="mailto:hawk@tbi.univie.ac.at" target="_blank">hawk@tbi.univie.ac.at</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Gluster Gurus,<br>
<br>
I'm using a gluster volume as home for our users. The volume is<br>
replica 3, running on CentOS 7, gluster version 3.10<br>
(3.10.6-1.el7.x86_64). Clients are running Fedora 26 and also<br>
gluster 3.10 (3.10.6-3.fc26.x86_64).<br>
<br>
During the data backup I got an I/O error on one file. Manually<br>
checking for this file on a client confirms this:<br>
<br>
ls -l<br>
romanoch/.mozilla/firefox/vzzq<wbr>qxrm.default-1396429081309/ses<wbr>sionstore-backups/<br>
ls: cannot access<br>
'romanoch/.mozilla/firefox/vzz<wbr>qqxrm.default-1396429081309/se<wbr>ssionstore-backups/<a href="http://recovery.ba">recovery.ba</a><wbr>klz4':<br>
Input/output error<br>
total 2015<br>
-rw-------. 1 romanoch tbi 998211 Sep 15 18:44 previous.js<br>
-rw-------. 1 romanoch tbi 65222 Oct 17 17:57 previous.jsonlz4<br>
-rw-------. 1 romanoch tbi 149161 Oct 1 13:46 recovery.bak<br>
-?????????? ? ? ? ? ? recovery.baklz4<br>
<br>
Out of curiosity I checked all the bricks for this file. It's<br>
present there. Making a checksum shows that the file is different on<br>
one of the three replica servers.<br>
<br>
Querying healing information shows that the file should be healed:<br>
# gluster volume heal home info<br>
Brick sphere-six:/srv/gluster_home/b<wbr>rick<br>
/romanoch/.mozilla/firefox/vzz<wbr>qqxrm.default-1396429081309/se<wbr>ssionstore-backups/<a href="http://recovery.ba">recovery.ba</a><wbr>klz4<br>
<br>
Status: Connected<br>
Number of entries: 1<br>
<br>
Brick sphere-five:/srv/gluster_home/<wbr>brick<br>
/romanoch/.mozilla/firefox/vzz<wbr>qqxrm.default-1396429081309/se<wbr>ssionstore-backups/<a href="http://recovery.ba">recovery.ba</a><wbr>klz4<br>
<br>
Status: Connected<br>
Number of entries: 1<br>
<br>
Brick sphere-four:/srv/gluster_home/<wbr>brick<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
Manually triggering heal doesn't report an error but also does not<br>
heal the file.<br>
# gluster volume heal home<br>
Launching heal operation to perform index self heal on volume home<br>
has been successful<br>
<br>
Same with a full heal<br>
# gluster volume heal home full<br>
Launching heal operation to perform full self heal on volume home<br>
has been successful<br>
<br>
According to the split brain query that's not the problem:<br>
# gluster volume heal home info split-brain<br>
Brick sphere-six:/srv/gluster_home/b<wbr>rick<br>
Status: Connected<br>
Number of entries in split-brain: 0<br>
<br>
Brick sphere-five:/srv/gluster_home/<wbr>brick<br>
Status: Connected<br>
Number of entries in split-brain: 0<br>
<br>
Brick sphere-four:/srv/gluster_home/<wbr>brick<br>
Status: Connected<br>
Number of entries in split-brain: 0<br>
<br>
<br>
I have no idea why this situation arose in the first place and also<br>
no idea as how to solve this problem. I would highly appreciate any<br>
helpful feedback I can get.<br>
<br>
The only mention in the logs matching this file is a rename operation:<br>
/var/log/glusterfs/bricks/srv-<wbr>gluster_home-brick.log:[2017-1<wbr>0-23<br>
09:19:11.561661] I [MSGID: 115061]<br>
[server-rpc-fops.c:1022:server<wbr>_rename_cbk] 0-home-server: 5266153:<br>
RENAME<br>
/romanoch/.mozilla/firefox/vzz<wbr>qqxrm.default-1396429081309/se<wbr>ssionstore-backups/recovery.js<wbr>onlz4<br>
(48e9eea6-cda6-4e53-bb4a-72059<wbr>debf4c2/recovery.jsonlz4) -><br>
/romanoch/.mozilla/firefox/vzz<wbr>qqxrm.default-1396429081309/se<wbr>ssionstore-backups/<a href="http://recovery.ba">recovery.ba</a><wbr>klz4<br>
(48e9eea6-cda6-4e53-bb4a-72059<wbr>debf4c2/recovery.baklz4), client:<br>
romulus.tbi.univie.ac.at-11894<wbr>-2017/10/18-07:06:07:206366-ho<wbr>me-client-3-0-0,<br>
error-xlator: home-posix [No data available]<br>
<br>
I enabled directory quotas the same day this problem showed up but<br>
I'm not sure how quotas could have an effect like this (maybe unless<br>
the limit is reached but that's also not the case).<br>
<br>
Thanks again if anyone as an idea.<br>
Cheers<br>
Richard<br>
--<br>
/dev/null<br>
<br>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></blockquote></div></div>
</blockquote></div></div>
</div></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>