<div dir="ltr">Hello Ravishankar,<div>we had to change the directory as we fixed that one, so please check these links which refers to a new (broken) path:</div><div><br></div><div><div><a href="https://nopaste.me/view/1ee13a63">https://nopaste.me/view/1ee13a63</a> LS debug log</div><div><a href="https://nopaste.me/view/80ac1e13">https://nopaste.me/view/80ac1e13</a> getfattr</div><div><a href="https://nopaste.me/view/eafb0b44">https://nopaste.me/view/eafb0b44</a> volume status</div></div><div><br></div><div>Thanks in advance.</div><div><br></div><div>Luca & Andrea</div><div><br></div><br><div class="gmail_quote"><div dir="ltr">On Sat, 11 Mar 2017 at 02:01 Ravishankar N <<a href="mailto:ravishankar@redhat.com">ravishankar@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000" class="gmail_msg">
<div class="m_-2770762843680964896moz-cite-prefix gmail_msg">On 03/10/2017 10:32 PM, Luca Gervasi
wrote:<br class="gmail_msg">
</div>
<blockquote type="cite" class="gmail_msg">
<div dir="ltr" class="gmail_msg">Hi,
<div class="gmail_msg">I'm Andrea's collegue. I'd like to add that we have no
trusted.afr xattr on the root folder</div>
</div>
</blockquote></div><div bgcolor="#FFFFFF" text="#000000" class="gmail_msg">
Just to confirm, this would be 'includes2013' right?</div><div bgcolor="#FFFFFF" text="#000000" class="gmail_msg"><br class="gmail_msg">
<blockquote type="cite" class="gmail_msg">
<div dir="ltr" class="gmail_msg">
<div class="gmail_msg"> where those files are located and every file seems to be
clean on each brick.</div>
<div class="gmail_msg">You can find another example file's xattr here: <a href="https://nopaste.me/view/3c2014ac" class="gmail_msg" target="_blank">https://nopaste.me/view/3c2014ac</a></div>
<div class="gmail_msg">Here a listing: <a href="https://nopaste.me/view/eb4430a2" class="gmail_msg" target="_blank">https://nopaste.me/view/eb4430a2</a></div>
<div class="gmail_msg">This behavior causes the directory which contains those
files undeletable (we had to clear them up on brick level,
clearing all the hardlinks too).</div>
<div class="gmail_msg">This issue is visible on fuse mounted volumes while it's
not noticeable when mounted in NFS through ganesha.</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
</div>
</blockquote></div><div bgcolor="#FFFFFF" text="#000000" class="gmail_msg">
Could you provide the complete output of `gluster volume info`? I
want to find out which bricks constitute a replica pair.<br class="gmail_msg">
<br class="gmail_msg">
Also could you change the diagnostics.client-log-level to DEBUG
temporarily, do an `ls <directory where you see duplicate
entries>` on the fuse mount and share the corresponding mount
log?<br class="gmail_msg">
Thanks,<br class="gmail_msg">
Ravi<br class="gmail_msg">
<br class="gmail_msg">
<blockquote type="cite" class="gmail_msg"></blockquote></div><div bgcolor="#FFFFFF" text="#000000" class="gmail_msg"><blockquote type="cite" class="gmail_msg">
<div dir="ltr" class="gmail_msg">
<div class="gmail_msg">Thanks a lot.</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">Luca Gervasi</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
</div>
<br class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div dir="ltr" class="gmail_msg">On Fri, 10 Mar 2017 at 17:41 Andrea Fogazzi <<a href="mailto:foggy@fogazzi.com" class="gmail_msg" target="_blank">foggy@fogazzi.com</a>>
wrote:<br class="gmail_msg">
</div>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="gmail_msg">
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr" class="gmail_msg">
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<p class="gmail_msg">Hi community,</p>
<p class="gmail_msg">we ran an extensive issue on
our installation of gluster 3.10, which we did
upgraded from 3.8.8 (it's a distribute+replicate, 5
nodes, 3 bricks in replica 2+1 quorum); recently we
noticed a frequent issue where files get duplicated on
the some of the directories; this is visible on the
fuse mount points (RW), but not on the NFS/Ganesha
(RO) mount points.</p>
<p class="gmail_msg"><br class="gmail_msg">
</p>
<p class="gmail_msg">A sample of an ll output:</p>
<p class="gmail_msg"><br class="gmail_msg">
</p>
<div class="gmail_msg">---------T 1 48 web_rw 0 Mar
10 11:57 paginazione.shtml</div>
<div class="gmail_msg">-rw-rw-r-- 1 48 web_rw 272 Feb
18 22:00 paginazione.shtml</div>
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<br class="gmail_msg">
</div>
As you can see, the file is listed twice, but only one
of the two is good (the name is identical, we verified
that no spurious/hidden characters are present in the
name); the issue maybe is related on how we uploaded the
files on the file system, via incremental rsync on the
fuse mount.</div>
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<br class="gmail_msg">
</div>
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
Do anyone have suggestion on how it can happen, how to
solve existing duplication or how to prevent to happen
anymore.</div>
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<br class="gmail_msg">
</div>
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
Thanks in advance.</div>
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
Best regards,</div>
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
andrea</div>
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<br class="gmail_msg">
</div>
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<div class="gmail_msg">Options Reconfigured:</div>
<div class="gmail_msg">performance.cache-invalidation:
true</div>
<div class="gmail_msg">cluster.favorite-child-policy:
mtime</div>
<div class="gmail_msg">features.cache-invalidation: 1</div>
<div class="gmail_msg">network.inode-lru-limit: 90000</div>
<div class="gmail_msg">performance.cache-size: 1024MB</div>
<div class="gmail_msg">storage.linux-aio: on</div>
<div class="gmail_msg">nfs.outstanding-rpc-limit: 64</div>
<div class="gmail_msg">storage.build-pgfid: on</div>
<div class="gmail_msg">cluster.server-quorum-type:
server</div>
<div class="gmail_msg">cluster.self-heal-daemon: enable</div>
<div class="gmail_msg">performance.nfs.io-cache: on</div>
<div class="gmail_msg">performance.client-io-threads: on</div>
<div class="gmail_msg">performance.nfs.stat-prefetch: on</div>
<div class="gmail_msg">performance.nfs.io-threads: on</div>
<div class="gmail_msg">diagnostics.latency-measurement:
on</div>
<div class="gmail_msg">diagnostics.count-fop-hits: on</div>
<div class="gmail_msg">performance.md-cache-timeout: 1</div>
<div class="gmail_msg">performance.io-thread-count: 16</div>
<div class="gmail_msg">performance.high-prio-threads: 32</div>
<div class="gmail_msg">performance.normal-prio-threads:
32</div>
<div class="gmail_msg">performance.low-prio-threads: 32</div>
<div class="gmail_msg">performance.least-prio-threads: 1</div>
<div class="gmail_msg">nfs.acl: off</div>
<div class="gmail_msg">nfs.rpc-auth-unix: off</div>
<div class="gmail_msg">diagnostics.client-log-level:
ERROR</div>
<div class="gmail_msg">diagnostics.brick-log-level:
ERROR</div>
<div class="gmail_msg">cluster.lookup-unhashed: auto</div>
<div class="gmail_msg">performance.nfs.quick-read: on</div>
<div class="gmail_msg">performance.nfs.read-ahead: on</div>
<div class="gmail_msg">cluster.quorum-type: auto</div>
<div class="gmail_msg">cluster.locking-scheme: granular</div>
<div class="gmail_msg">cluster.data-self-heal-algorithm:
full</div>
<div class="gmail_msg">transport.address-family: inet</div>
<div class="gmail_msg">performance.readdir-ahead: on</div>
<div class="gmail_msg">nfs.disable: on</div>
<div class="gmail_msg">cluster.lookup-optimize: on</div>
<div class="gmail_msg">cluster.readdir-optimize: on</div>
<div class="gmail_msg">performance.read-ahead: off</div>
<div class="gmail_msg">performance.write-behind-window-size:
1MB</div>
<div class="gmail_msg">client.event-threads: 4</div>
<div class="gmail_msg">server.event-threads: 16</div>
<div class="gmail_msg">cluster.granular-entry-heal:
enable</div>
<div class="gmail_msg">performance.parallel-readdir: on</div>
<div class="gmail_msg">cluster.server-quorum-ratio: 51</div>
</div>
</div>
</div>
<div dir="ltr" class="gmail_msg">
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr" class="gmail_msg">
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<br class="gmail_msg">
</div>
<div id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" class="gmail_msg">
<p class="gmail_msg"><br class="gmail_msg">
</p>
<p class="gmail_msg"><br class="gmail_msg">
</p>
<div id="m_-2770762843680964896m_-7359613590814742918Signature" class="gmail_msg"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px" class="gmail_msg">Andrea Fogazzi</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px" class="gmail_msg">
<br class="gmail_msg">
<div class="gmail_msg"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px" class="gmail_msg"></span></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="gmail_msg">
<fieldset class="m_-2770762843680964896mimeAttachmentHeader gmail_msg"></fieldset>
<br class="gmail_msg">
</blockquote></div><div bgcolor="#FFFFFF" text="#000000" class="gmail_msg"><blockquote type="cite" class="gmail_msg"><pre class="gmail_msg">_______________________________________________
Gluster-users mailing list
<a class="m_-2770762843680964896moz-txt-link-abbreviated gmail_msg" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a class="m_-2770762843680964896moz-txt-link-freetext gmail_msg" href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<p class="gmail_msg"><br class="gmail_msg">
</p>
</div>
</blockquote></div></div>