<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 03/13/2017 03:08 PM, Luca Gervasi
wrote:<br>
</div>
<blockquote
cite="mid:CAPJi5KN+q27oRwngWMGaDri0F+SqFZemfYXSPjFjEyNDm00mBQ@mail.gmail.com"
type="cite">
<div dir="ltr">Hello Ravishankar,
<div>we had to change the directory as we fixed that one, so
please check these links which refers to a new (broken) path:</div>
<div><br>
</div>
<div>
<div><a moz-do-not-send="true"
href="https://nopaste.me/view/1ee13a63">https://nopaste.me/view/1ee13a63</a>
LS debug log</div>
<div><a moz-do-not-send="true"
href="https://nopaste.me/view/80ac1e13">https://nopaste.me/view/80ac1e13</a>
getfattr</div>
<div><a moz-do-not-send="true"
href="https://nopaste.me/view/eafb0b44">https://nopaste.me/view/eafb0b44</a>
volume status</div>
</div>
<div><br>
</div>
</div>
</blockquote>
Thanks, could you check if setting performance.parallel-readdir to
'off' solves the issue? If yes, do you mind raising a bug and
letting us know the BZ ID?<br>
Please note that parallel-readdir option is still experimental.<br>
-Ravi<br>
<br>
<blockquote
cite="mid:CAPJi5KN+q27oRwngWMGaDri0F+SqFZemfYXSPjFjEyNDm00mBQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Thanks in advance.</div>
<div><br>
</div>
<div>Luca & Andrea</div>
<div><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Sat, 11 Mar 2017 at 02:01 Ravishankar N <<a
moz-do-not-send="true"
href="mailto:ravishankar@redhat.com">ravishankar@redhat.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="gmail_msg">
<div class="m_-2770762843680964896moz-cite-prefix
gmail_msg">On 03/10/2017 10:32 PM, Luca Gervasi wrote:<br
class="gmail_msg">
</div>
<blockquote type="cite" class="gmail_msg">
<div dir="ltr" class="gmail_msg">Hi,
<div class="gmail_msg">I'm Andrea's collegue. I'd like
to add that we have no trusted.afr xattr on the root
folder</div>
</div>
</blockquote>
</div>
<div bgcolor="#FFFFFF" text="#000000" class="gmail_msg">
Just to confirm, this would be 'includes2013' right?</div>
<div bgcolor="#FFFFFF" text="#000000" class="gmail_msg"><br
class="gmail_msg">
<blockquote type="cite" class="gmail_msg">
<div dir="ltr" class="gmail_msg">
<div class="gmail_msg"> where those files are located
and every file seems to be clean on each brick.</div>
<div class="gmail_msg">You can find another example
file's xattr here: <a moz-do-not-send="true"
href="https://nopaste.me/view/3c2014ac"
class="gmail_msg" target="_blank">https://nopaste.me/view/3c2014ac</a></div>
<div class="gmail_msg">Here a listing: <a
moz-do-not-send="true"
href="https://nopaste.me/view/eb4430a2"
class="gmail_msg" target="_blank">https://nopaste.me/view/eb4430a2</a></div>
<div class="gmail_msg">This behavior causes the
directory which contains those files undeletable (we
had to clear them up on brick level, clearing all
the hardlinks too).</div>
<div class="gmail_msg">This issue is visible on fuse
mounted volumes while it's not noticeable when
mounted in NFS through ganesha.</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
</div>
</blockquote>
</div>
<div bgcolor="#FFFFFF" text="#000000" class="gmail_msg">
Could you provide the complete output of `gluster volume
info`? I want to find out which bricks constitute a
replica pair.<br class="gmail_msg">
<br class="gmail_msg">
Also could you change the diagnostics.client-log-level to
DEBUG temporarily, do an `ls <directory where you see
duplicate entries>` on the fuse mount and share the
corresponding mount log?<br class="gmail_msg">
Thanks,<br class="gmail_msg">
Ravi<br class="gmail_msg">
<br class="gmail_msg">
</div>
<div bgcolor="#FFFFFF" text="#000000" class="gmail_msg">
<blockquote type="cite" class="gmail_msg">
<div dir="ltr" class="gmail_msg">
<div class="gmail_msg">Thanks a lot.</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">Luca Gervasi</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
</div>
<br class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div dir="ltr" class="gmail_msg">On Fri, 10 Mar 2017
at 17:41 Andrea Fogazzi <<a
moz-do-not-send="true"
href="mailto:foggy@fogazzi.com" class="gmail_msg"
target="_blank">foggy@fogazzi.com</a>> wrote:<br
class="gmail_msg">
</div>
<blockquote class="gmail_quote gmail_msg"
style="margin:0 0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr" class="gmail_msg">
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
dir="ltr" class="gmail_msg">
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
class="gmail_msg">
<p class="gmail_msg">Hi community,</p>
<p class="gmail_msg">we ran an extensive issue
on our installation of gluster 3.10, which
we did upgraded from 3.8.8 (it's
a distribute+replicate, 5 nodes, 3 bricks in
replica 2+1 quorum); recently we noticed a
frequent issue where files get duplicated on
the some of the directories; this is
visible on the fuse mount points (RW), but
not on the NFS/Ganesha (RO) mount points.</p>
<p class="gmail_msg"><br class="gmail_msg">
</p>
<p class="gmail_msg">A sample of an ll output:</p>
<p class="gmail_msg"><br class="gmail_msg">
</p>
<div class="gmail_msg">---------T 1 48 web_rw
0 Mar 10 11:57 paginazione.shtml</div>
<div class="gmail_msg">-rw-rw-r-- 1 48 web_rw
272 Feb 18 22:00 paginazione.shtml</div>
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
class="gmail_msg"> <br class="gmail_msg">
</div>
As you can see, the file is listed twice, but
only one of the two is good (the name is
identical, we verified that no spurious/hidden
characters are present in the name); the issue
maybe is related on how we uploaded the files
on the file system, via incremental rsync on
the fuse mount.</div>
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
class="gmail_msg"> <br class="gmail_msg">
</div>
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
class="gmail_msg"> Do anyone have suggestion
on how it can happen, how to solve existing
duplication or how to prevent to happen
anymore.</div>
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
class="gmail_msg"> <br class="gmail_msg">
</div>
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
class="gmail_msg"> Thanks in advance.</div>
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
class="gmail_msg"> Best regards,</div>
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
class="gmail_msg"> andrea</div>
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
class="gmail_msg"> <br class="gmail_msg">
</div>
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
class="gmail_msg">
<div class="gmail_msg">Options Reconfigured:</div>
<div class="gmail_msg">performance.cache-invalidation:
true</div>
<div class="gmail_msg">cluster.favorite-child-policy:
mtime</div>
<div class="gmail_msg">features.cache-invalidation:
1</div>
<div class="gmail_msg">network.inode-lru-limit:
90000</div>
<div class="gmail_msg">performance.cache-size:
1024MB</div>
<div class="gmail_msg">storage.linux-aio: on</div>
<div class="gmail_msg">nfs.outstanding-rpc-limit:
64</div>
<div class="gmail_msg">storage.build-pgfid: on</div>
<div class="gmail_msg">cluster.server-quorum-type:
server</div>
<div class="gmail_msg">cluster.self-heal-daemon:
enable</div>
<div class="gmail_msg">performance.nfs.io-cache:
on</div>
<div class="gmail_msg">performance.client-io-threads:
on</div>
<div class="gmail_msg">performance.nfs.stat-prefetch:
on</div>
<div class="gmail_msg">performance.nfs.io-threads:
on</div>
<div class="gmail_msg">diagnostics.latency-measurement:
on</div>
<div class="gmail_msg">diagnostics.count-fop-hits:
on</div>
<div class="gmail_msg">performance.md-cache-timeout:
1</div>
<div class="gmail_msg">performance.io-thread-count:
16</div>
<div class="gmail_msg">performance.high-prio-threads:
32</div>
<div class="gmail_msg">performance.normal-prio-threads:
32</div>
<div class="gmail_msg">performance.low-prio-threads:
32</div>
<div class="gmail_msg">performance.least-prio-threads:
1</div>
<div class="gmail_msg">nfs.acl: off</div>
<div class="gmail_msg">nfs.rpc-auth-unix: off</div>
<div class="gmail_msg">diagnostics.client-log-level:
ERROR</div>
<div class="gmail_msg">diagnostics.brick-log-level:
ERROR</div>
<div class="gmail_msg">cluster.lookup-unhashed:
auto</div>
<div class="gmail_msg">performance.nfs.quick-read:
on</div>
<div class="gmail_msg">performance.nfs.read-ahead:
on</div>
<div class="gmail_msg">cluster.quorum-type:
auto</div>
<div class="gmail_msg">cluster.locking-scheme:
granular</div>
<div class="gmail_msg">cluster.data-self-heal-algorithm:
full</div>
<div class="gmail_msg">transport.address-family:
inet</div>
<div class="gmail_msg">performance.readdir-ahead:
on</div>
<div class="gmail_msg">nfs.disable: on</div>
<div class="gmail_msg">cluster.lookup-optimize:
on</div>
<div class="gmail_msg">cluster.readdir-optimize:
on</div>
<div class="gmail_msg">performance.read-ahead:
off</div>
<div class="gmail_msg">performance.write-behind-window-size:
1MB</div>
<div class="gmail_msg">client.event-threads: 4</div>
<div class="gmail_msg">server.event-threads:
16</div>
<div class="gmail_msg">cluster.granular-entry-heal:
enable</div>
<div class="gmail_msg">performance.parallel-readdir:
on</div>
<div class="gmail_msg">cluster.server-quorum-ratio:
51</div>
</div>
</div>
</div>
<div dir="ltr" class="gmail_msg">
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
dir="ltr" class="gmail_msg">
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
class="gmail_msg"> <br class="gmail_msg">
</div>
<div
id="m_-2770762843680964896m_-7359613590814742918divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
class="gmail_msg">
<p class="gmail_msg"><br class="gmail_msg">
</p>
<p class="gmail_msg"><br class="gmail_msg">
</p>
<div
id="m_-2770762843680964896m_-7359613590814742918Signature"
class="gmail_msg"><span
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"
class="gmail_msg">Andrea Fogazzi</span><br
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"
class="gmail_msg">
<br class="gmail_msg">
<div class="gmail_msg"><span
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"
class="gmail_msg"></span></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="gmail_msg">
<fieldset
class="m_-2770762843680964896mimeAttachmentHeader
gmail_msg"></fieldset>
<br class="gmail_msg">
</blockquote>
</div>
<div bgcolor="#FFFFFF" text="#000000" class="gmail_msg">
<blockquote type="cite" class="gmail_msg">
<pre class="gmail_msg">_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="m_-2770762843680964896moz-txt-link-abbreviated gmail_msg" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="m_-2770762843680964896moz-txt-link-freetext gmail_msg" href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<p class="gmail_msg"><br class="gmail_msg">
</p>
</div>
</blockquote>
</div>
</div>
</blockquote>
<p><br>
</p>
</body>
</html>