<div dir="ltr"><div>CCing Ravi (arbiter) , Poornima and Raghavendra (parallel readdir)</div><div><br></div>Hi Joshua,<div><br></div><div>I had a quick look at the files you sent across. To summarize the issue, you see empty linkto files on the mount point. </div><div><br></div><div>From the logs I see that parallel readdir is enabled for this volume:</div><div><br></div><div><div>performance.readdir-ahead: on</div><div>performance.parallel-readdir: on</div></div><div><br></div><div>There was a similar issue reported earlier [1] and fixed as part of [2]. Which version of gluster are you running? Poornima and Raghavendra, can you take a look and see if it is the same issue?</div><div><br></div><div><br></div><div><br></div><div>An interesting observation - the value of the trusted.glusterfs.dht.linkto xattr is different on the one of the bricks of the replica set for some of the files(Ravi, any idea why ?)</div><div><div><br></div><div>---------T 2 root root 0 May 18 16:49 /arbiterAA01/gvAA01/brick1/vaultAA01/support-couplers/CPL-SBS01 (CPL-SBS01)/D_VOL-b002-i1873-cd.md5</div><div><br></div><div>root@PB-WA-AA-00-A:/# getfattr -e hex -m . -d /arbiterAA01/gvAA01/brick1/vaultAA01/support-couplers/CPL-SBS01\ \(CPL-SBS01\)/D_VOL-b002-i1873-cd.md5</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: arbiterAA01/gvAA01/brick1/vaultAA01/support-couplers/CPL-SBS01 (CPL-SBS01)/D_VOL-b002-i1873-cd.md5</div><div>trusted.gfid=0xfc31c380697a4c70b6af1af35822e519</div><div>trusted.glusterfs.dht.linkto=0x6776414130312d726561646469722d61686561642d3100</div><div><br></div><div>This value is "gvAA01-readdir-ahead-1"</div><div><br></div><div><br></div><div><br></div><div>---------T 2 root root 0 May 18 16:49 /brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01 (CPL-SBS01)/D_VOL-b002-i1873-cd.md5</div><div><br></div><div>root@PB-WA-AA-01-B:/# getfattr -e hex -m . -d /brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01\ \(CPL-SBS01\)/D_VOL-b002-i1873-cd.md5</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01 (CPL-SBS01)/D_VOL-b002-i1873-cd.md5</div><div>trusted.gfid=0xfc31c380697a4c70b6af1af35822e519</div><div>trusted.glusterfs.dht.linkto=0x6776414130312d726561646469722d61686561642d3100</div><div><br></div><div>This value is "gvAA01-readdir-ahead-1"<br></div><div><br></div><div><br></div><div><br></div><div>---------T 2 root root 0 May 30 14:25 /brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01 (CPL-SBS01)/D_VOL-b002-i1873-cd.md5</div><div><br></div><div>root@PB-WA-AA-02-B:/# getfattr -e hex -m . -d /brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01\ \(CPL-SBS01\)/D_VOL-b002-i1873-cd.md5</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: brick1/gvAA01/brick/vaultAA01/support-couplers/CPL-SBS01 (CPL-SBS01)/D_VOL-b002-i1873-cd.md5</div><div>trusted.afr.dirty=0x000000000000000000000000</div><div>trusted.bit-rot.version=0x020000000000000058df0ab7000710ee</div><div>trusted.gfid=0xfc31c380697a4c70b6af1af35822e519</div><div>trusted.glusterfs.dht.linkto=0x6776414130312d7265706c69636174652d3100</div><div><br></div></div><div>This value is "gvAA01-replicate-1"</div><div><br></div><div><br></div><div>[1] <a href="http://lists.gluster.org/pipermail/gluster-users/2017-March/030254.html">http://lists.gluster.org/pipermail/gluster-users/2017-March/030254.html</a></div><div>[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1435942">https://bugzilla.redhat.com/show_bug.cgi?id=1435942</a></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 30 May 2017 at 14:20, Joshua Coyle <span dir="ltr"><<a href="mailto:joshua.coyle@probax.io" target="_blank">joshua.coyle@probax.io</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="m_4927507232016425578WordSection1"><span class="">
<p class="MsoNormal"><span lang="EN-AU"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-AU"><u></u> <u></u></span></p>
<div>
<table class="m_4927507232016425578MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="432" style="width:4.5in">
<tbody>
<tr>
<td style="padding:15.0pt 0in 9.0pt 0in">
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><img width="147" height="37" style="width:1.5312in;height:.3854in" id="m_4927507232016425578_x0000_i1026" src="cid:image001.jpg@01D2D957.C533E3A0" alt="Probax"></span><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><u></u><u></u></span></p>
</td>
</tr>
<tr>
<td style="padding:0in 0in 0in 27.0pt">
<p class="MsoNormal" style="line-height:15.0pt"><b><span style="font-size:15.0pt;font-family:"Arial",sans-serif;color:#fd6b0d">Joshua Coyle</span></b><b><span style="font-size:15.0pt;font-family:"Arial",sans-serif;color:#fd6b0d"><u></u><u></u></span></b></p>
</td>
</tr>
<tr>
<td style="padding:2.25pt 0in 0in 27.0pt"></td>
</tr>
<tr>
<td width="414" style="width:310.5pt;padding:9.0pt 0in 0in 13.5pt">
<table class="m_4927507232016425578MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="414" style="width:310.5pt;background:#092f87">
<tbody>
<tr>
<td width="196" valign="top" style="width:147.0pt;padding:12.0pt 0in 5.25pt 0in">
<table class="m_4927507232016425578MsoNormalTable" border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td valign="top" style="padding:0in 14.25pt 5.25pt 14.25pt">
<p class="MsoNormal" style="line-height:13.5pt"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">P</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
<td valign="top" style="padding:0in 0in 5.25pt 0in">
<p class="MsoNormal" style="line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">1300 885 117</span><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:0in 14.25pt 5.25pt 14.25pt">
<p class="MsoNormal" style="line-height:13.5pt"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">E</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
<td valign="top" style="padding:0in 0in 5.25pt 0in">
<p class="MsoNormal" style="line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><a href="mailto:joshua.coyle@probax.io" target="_blank"><span style="color:white;text-decoration:none">joshua.coyle@probax.io</span></a></span><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:0in 14.25pt 5.25pt 14.25pt">
<p class="MsoNormal" style="line-height:13.5pt"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">W</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
<td valign="top" style="padding:0in 0in 5.25pt 0in">
<p class="MsoNormal" style="line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><a href="http://www.probax.io/" target="_blank"><span style="color:white;text-decoration:none">probax.io</span></a></span><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
</td>
<td width="218" valign="top" style="width:163.5pt;padding:12.0pt 0in 5.25pt 0in">
<table class="m_4927507232016425578MsoNormalTable" border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td valign="top" style="padding:0in 14.25pt 5.25pt 14.25pt">
<p class="MsoNormal" style="line-height:13.5pt"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">A</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
<td valign="top" style="padding:0in 0in 5.25pt 0in">
<p class="MsoNormal" style="line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">QV1 Perth,</span><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:0in 0in 5.25pt 0in"></td>
<td valign="top" style="padding:0in 0in 5.25pt 0in">
<p class="MsoNormal" style="line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">Level 33/250 St Georges Tce,</span><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:0in 0in 5.25pt 0in"></td>
<td valign="top" style="padding:0in 0in 5.25pt 0in">
<p class="MsoNormal" style="line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">Perth WA 6000</span><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in 0in 0in 13.5pt">
<table class="m_4927507232016425578MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="414" style="width:310.5pt;background:#092f87;border:none;border-top:solid #4763a5 1.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:11.25pt 0in 11.25pt 28.5pt">
<p class="MsoNormal" style="line-height:13.5pt"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white">smart safe local cloud</span></b><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></b></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><span lang="EN-AU"><u></u> <u></u></span></p>
</span><div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Joshua Coyle <br>
<b>Sent:</b> Tuesday, May 30, 2017 3:27 PM<br>
<b>To:</b> '<a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>' <<a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>><br>
<b>Subject:</b> ATTN: nbalacha IRC - Gluster - BlackoutWNCT requested info for 0byte file issue<u></u><u></u></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span lang="EN-AU">Hey nbalacha,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-AU"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-AU">Please find the requested information attached for the issue being discussed in IRC.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-AU"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-AU">If you need any further info, please let me know.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-AU"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-AU">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-AU"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-AU">BlackoutWNCT<u></u><u></u></span></p>
<table class="m_4927507232016425578MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="432" style="width:4.5in">
<tbody>
<tr>
<td style="padding:15.0pt 0in 9.0pt 0in">
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><img border="0" width="147" height="37" style="width:1.5312in;height:.3854in" id="m_4927507232016425578Picture_x0020_2" src="cid:image001.jpg@01D2D957.C533E3A0" alt="Probax"></span><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><u></u><u></u></span></p>
</td>
</tr>
<tr>
<td style="padding:0in 0in 0in 27.0pt">
<p class="MsoNormal" style="line-height:15.0pt"><b><span style="font-size:15.0pt;font-family:"Arial",sans-serif;color:#fd6b0d">Joshua Coyle<u></u><u></u></span></b></p>
</td>
</tr>
<tr>
<td style="padding:2.25pt 0in 0in 27.0pt"></td>
</tr>
<tr>
<td width="414" style="width:310.5pt;padding:9.0pt 0in 0in 13.5pt">
<table class="m_4927507232016425578MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="414" style="width:310.5pt;background:#092f87">
<tbody>
<tr>
<td width="196" valign="top" style="width:147.0pt;padding:12.0pt 0in 5.25pt 0in">
<table class="m_4927507232016425578MsoNormalTable" border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td valign="top" style="padding:0in 14.25pt 5.25pt 14.25pt">
<p class="MsoNormal" style="line-height:13.5pt"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">P</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
<td valign="top" style="padding:0in 0in 5.25pt 0in">
<p class="MsoNormal" style="line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">1300 885 117<u></u><u></u></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:0in 14.25pt 5.25pt 14.25pt">
<p class="MsoNormal" style="line-height:13.5pt"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">E</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
<td valign="top" style="padding:0in 0in 5.25pt 0in">
<p class="MsoNormal" style="line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><a href="mailto:joshua.coyle@probax.io" target="_blank"><span style="color:white;text-decoration:none">joshua.coyle@probax.io</span></a><u></u><u></u></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:0in 14.25pt 5.25pt 14.25pt">
<p class="MsoNormal" style="line-height:13.5pt"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">W</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
<td valign="top" style="padding:0in 0in 5.25pt 0in">
<p class="MsoNormal" style="line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><a href="http://www.probax.io/" target="_blank"><span style="color:white;text-decoration:none">probax.io</span></a><u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
</td>
<td width="218" valign="top" style="width:163.5pt;padding:12.0pt 0in 5.25pt 0in">
<table class="m_4927507232016425578MsoNormalTable" border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td valign="top" style="padding:0in 14.25pt 5.25pt 14.25pt">
<p class="MsoNormal" style="line-height:13.5pt"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">A</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white"><u></u><u></u></span></p>
</td>
<td valign="top" style="padding:0in 0in 5.25pt 0in">
<p class="MsoNormal" style="line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">QV1 Perth,<u></u><u></u></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:0in 0in 5.25pt 0in"></td>
<td valign="top" style="padding:0in 0in 5.25pt 0in">
<p class="MsoNormal" style="line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">Level 33/250 St Georges Tce,<u></u><u></u></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:0in 0in 5.25pt 0in"></td>
<td valign="top" style="padding:0in 0in 5.25pt 0in">
<p class="MsoNormal" style="line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">Perth WA 6000<u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in 0in 0in 13.5pt">
<table class="m_4927507232016425578MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="414" style="width:310.5pt;background:#092f87;border:none;border-top:solid #4763a5 1.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:11.25pt 0in 11.25pt 28.5pt">
<p class="MsoNormal" style="line-height:13.5pt"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white">smart safe local cloud<u></u><u></u></span></b></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span lang="EN-AU"><u></u> <u></u></span></p>
</div></div></div>
</div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>