<div dir="ltr">Hi,<div>I'm Andrea's collegue. I'd like to add that we have no trusted.afr xattr on the root folder where those files are located and every file seems to be clean on each brick.</div><div>You can find another example file's xattr here: <a href="https://nopaste.me/view/3c2014ac">https://nopaste.me/view/3c2014ac</a></div><div>Here a listing: <a href="https://nopaste.me/view/eb4430a2">https://nopaste.me/view/eb4430a2</a></div><div>This behavior causes the directory which contains those files undeletable (we had to clear them up on brick level, clearing all the hardlinks too).</div><div>This issue is visible on fuse mounted volumes while it's not noticeable when mounted in NFS through ganesha.</div><div><br></div><div>Thanks a lot.</div><div><br></div><div>Luca Gervasi</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, 10 Mar 2017 at 17:41 Andrea Fogazzi <<a href="mailto:foggy@fogazzi.com">foggy@fogazzi.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="gmail_msg">
<div id="m_-7359613590814742918divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr" class="gmail_msg">
<div id="m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<p class="gmail_msg">Hi community,</p>
<p class="gmail_msg">we ran an extensive issue on our installation of gluster 3.10, which we did upgraded from 3.8.8 (it's a distribute+replicate, 5 nodes, 3 bricks in replica 2+1 quorum); recently we noticed a frequent issue where files get duplicated on the some of the directories;
this is visible on the fuse mount points (RW), but not on the NFS/Ganesha (RO) mount points.</p>
<p class="gmail_msg"><br class="gmail_msg">
</p>
<p class="gmail_msg">A sample of an ll output:</p>
<p class="gmail_msg"><br class="gmail_msg">
</p>
<p class="gmail_msg"></p>
<div class="gmail_msg">---------T 1 48 web_rw 0 Mar 10 11:57 paginazione.shtml</div>
<div class="gmail_msg">-rw-rw-r-- 1 48 web_rw 272 Feb 18 22:00 paginazione.shtml</div>
<div id="m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<br class="gmail_msg">
</div>
As you can see, the file is listed twice, but only one of the two is good (the name is identical, we verified that no spurious/hidden characters are present in the name); the issue maybe is related on how we uploaded the files on the file system, via incremental
rsync on the fuse mount.</div>
<div id="m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<br class="gmail_msg">
</div>
<div id="m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
Do anyone have suggestion on how it can happen, how to solve existing duplication or how to prevent to happen anymore.</div>
<div id="m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<br class="gmail_msg">
</div>
<div id="m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
Thanks in advance.</div>
<div id="m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
Best regards,</div>
<div id="m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
andrea</div>
<div id="m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<br class="gmail_msg">
</div>
<div id="m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<div class="gmail_msg">Options Reconfigured:</div>
<div class="gmail_msg">performance.cache-invalidation: true</div>
<div class="gmail_msg">cluster.favorite-child-policy: mtime</div>
<div class="gmail_msg">features.cache-invalidation: 1</div>
<div class="gmail_msg">network.inode-lru-limit: 90000</div>
<div class="gmail_msg">performance.cache-size: 1024MB</div>
<div class="gmail_msg">storage.linux-aio: on</div>
<div class="gmail_msg">nfs.outstanding-rpc-limit: 64</div>
<div class="gmail_msg">storage.build-pgfid: on</div>
<div class="gmail_msg">cluster.server-quorum-type: server</div>
<div class="gmail_msg">cluster.self-heal-daemon: enable</div>
<div class="gmail_msg">performance.nfs.io-cache: on</div>
<div class="gmail_msg">performance.client-io-threads: on</div>
<div class="gmail_msg">performance.nfs.stat-prefetch: on</div>
<div class="gmail_msg">performance.nfs.io-threads: on</div>
<div class="gmail_msg">diagnostics.latency-measurement: on</div>
<div class="gmail_msg">diagnostics.count-fop-hits: on</div>
<div class="gmail_msg">performance.md-cache-timeout: 1</div>
<div class="gmail_msg">performance.io-thread-count: 16</div>
<div class="gmail_msg">performance.high-prio-threads: 32</div>
<div class="gmail_msg">performance.normal-prio-threads: 32</div>
<div class="gmail_msg">performance.low-prio-threads: 32</div>
<div class="gmail_msg">performance.least-prio-threads: 1</div>
<div class="gmail_msg">nfs.acl: off</div>
<div class="gmail_msg">nfs.rpc-auth-unix: off</div>
<div class="gmail_msg">diagnostics.client-log-level: ERROR</div>
<div class="gmail_msg">diagnostics.brick-log-level: ERROR</div>
<div class="gmail_msg">cluster.lookup-unhashed: auto</div>
<div class="gmail_msg">performance.nfs.quick-read: on</div>
<div class="gmail_msg">performance.nfs.read-ahead: on</div>
<div class="gmail_msg">cluster.quorum-type: auto</div>
<div class="gmail_msg">cluster.locking-scheme: granular</div>
<div class="gmail_msg">cluster.data-self-heal-algorithm: full</div>
<div class="gmail_msg">transport.address-family: inet</div>
<div class="gmail_msg">performance.readdir-ahead: on</div>
<div class="gmail_msg">nfs.disable: on</div>
<div class="gmail_msg">cluster.lookup-optimize: on</div>
<div class="gmail_msg">cluster.readdir-optimize: on</div>
<div class="gmail_msg">performance.read-ahead: off</div>
<div class="gmail_msg">performance.write-behind-window-size: 1MB</div>
<div class="gmail_msg">client.event-threads: 4</div>
<div class="gmail_msg">server.event-threads: 16</div>
<div class="gmail_msg">cluster.granular-entry-heal: enable</div>
<div class="gmail_msg">performance.parallel-readdir: on</div>
<div class="gmail_msg">cluster.server-quorum-ratio: 51</div>
</div></div></div><div dir="ltr" class="gmail_msg"><div id="m_-7359613590814742918divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr" class="gmail_msg">
<div id="m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<br class="gmail_msg">
</div>
<div id="m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<p class="gmail_msg"><br class="gmail_msg">
</p>
<p class="gmail_msg"><br class="gmail_msg">
</p>
<div id="m_-7359613590814742918Signature" class="gmail_msg"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px" class="gmail_msg">Andrea Fogazzi</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px" class="gmail_msg">
<br class="gmail_msg">
<div class="gmail_msg"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px" class="gmail_msg"></span></div>
</div>
</div>
</div></div></blockquote></div>