<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal>I am running: glusterfs 3.5.9 built on Mar 28 2016 07:10:17<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Other volume info:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Type: Distributed-Replicate<o:p></o:p></p><p class=MsoNormal>Number of Bricks: 8 x 3 = 24<o:p></o:p></p><p class=MsoNormal>Transport-type: tcp<o:p></o:p></p><p class=MsoNormal>Options Reconfigured:<o:p></o:p></p><p class=MsoNormal>performance.cache-refresh-timeout: 30<o:p></o:p></p><p class=MsoNormal>performance.cache-size: 768MB<o:p></o:p></p><p class=MsoNormal>cluster.quorum-type: auto<o:p></o:p></p><p class=MsoNormal>cluster.server-quorum-type: server<o:p></o:p></p><p class=MsoNormal>cluster.server-quorum-ratio: 51<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>When I try to manipulate a file (def/ghi.gz) on the mounted glusterfs folder (abc) I get an Errno 5 input/output error. Most of the files work, but there are lots that have this same problem.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I visited each brick in my volume to see what the extended file attributes are for this file.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>On my_volume-replicate-0 there is an empty file with the filename. When I run “ls -al” it looks like this:<o:p></o:p></p><p class=MsoNormal>---------T 2 root root 0 Mar 1 14:56 ghi.gz<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>On the first two bricks (bricks 0 and 1) of my_volume-replicate-0 when I run “getfattr -d -m. -e hex ghi.gz” I get the following results:<o:p></o:p></p><p class=MsoNormal># file: ghi.gz<o:p></o:p></p><p class=MsoNormal>trusted.afr.my_volume-client-0=0x000000000000000000000000<o:p></o:p></p><p class=MsoNormal>trusted.afr.my_volume-client-1=0x000000000000000000000000<o:p></o:p></p><p class=MsoNormal>trusted.afr.my_volume-client-2=0x000000020000000200000000<o:p></o:p></p><p class=MsoNormal>trusted.gfid=0xabb0369b05844390add6ea72ce7e107a<o:p></o:p></p><p class=MsoNormal>trusted.glusterfs.dht.linkto=0x686f7374696e672d7265706c69636174652d3400<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The link to looks like the following when I use text encoding instead of hex encoding:<o:p></o:p></p><p class=MsoNormal>trusted.glusterfs.dht.linkto="my_volume-replicate-4"<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The third brick (brick 2) of my_volume-replicate-0 has these extended attributes:<o:p></o:p></p><p class=MsoNormal># file: ghi.gz<o:p></o:p></p><p class=MsoNormal>trusted.gfid=0xc5c99fe21c3f4582b48e6f69ff76e33b<o:p></o:p></p><p class=MsoNormal>trusted.glusterfs.dht.linkto=0x686f7374696e672d7265706c69636174652d3400<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>So the third brick has a DIFFERENT trusted.gfid.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The first two bricks have trusted.afr.my_volume-client-2=0x000000020000000200000000. Does that mean that the first two bricks think that the third brick (brick 2) has differences?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>All three bricks are linking to my_volume-replicate-4.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>All three bricks (bricks 12, 13, and 14) of my_volume-replicate-4 all have the actual file with these extended attributes:<o:p></o:p></p><p class=MsoNormal># file: ghi.gz<o:p></o:p></p><p class=MsoNormal>trusted.afr.my_volume-client-12=0x000000000000000000000000<o:p></o:p></p><p class=MsoNormal>trusted.afr.my_volume-client-13=0x000000000000000000000000<o:p></o:p></p><p class=MsoNormal>trusted.afr.my_volume-client-14=0x000000000000000000000000<o:p></o:p></p><p class=MsoNormal>trusted.gfid=0xabb0369b05844390add6ea72ce7e107a<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>So, my_volume-replicate-4’s trusted.gfid matches bricks 0 and 1 of my_volume-replicate-0. And they all have 0x000000000000000000000000 for all three trusted.afr.my_volume-client-## attribute. I assume this means that the file is the same on all three bricks of my_volume-replicate-4.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>No other bricks in the system have the ghi.gz file on them.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>When I go to .glusterfs/indices/xattrop of bricks 0 and 1 there is a file there named abb0369b-0584-4390-add6-ea72ce7e107a. This means that this file id is in need of healing, correct? There is NOT a file named abb0369b-0584-4390-add6-ea72ce7e107a on brick 2.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>When I run “gluster volume heal my_volume info heal-failed” it lists <gfid:abb0369b-0584-4390-add6-ea72ce7e107a> four times. I have tried to do a full heal and a rebalance of the system, but it does not fix this problem.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>How do I fix this problem? Is there an easy way that I can fix all of the files with the problem in bulk?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thank you very much for any insights or help you may have!!<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Dave<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>