<div dir="ltr">Thanks for the report Artem,<div><br></div><div>Looks like the issue is about cache warming up. Specially, I suspect rsync doing a 'readdir(), stat(), file operations' loop, where as when a find or ls is issued, we get 'readdirp()' request, which contains the stat information along with entries, which also makes sure cache is up-to-date (at md-cache layer).</div><div><br></div><div>Note that this is just a off-the memory hypothesis, We surely need to analyse and debug more thoroughly for a proper explanation. Some one in my team would look at it soon.</div><div><br></div><div>Regards,</div><div>Amar </div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 5, 2018 at 7:25 AM, Vlad Kopylov <span dir="ltr"><<a href="mailto:vladkopy@gmail.com" target="_blank">vladkopy@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">You mounting it to the local bricks?<br>
<br>
struggling with same performance issues<br>
try using this volume setting<br>
<a href="http://lists.gluster.org/pipermail/gluster-users/2018-January/033397.html" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>pipermail/gluster-users/2018-<wbr>January/033397.html</a><br>
performance.stat-prefetch: on might be it<br>
<br>
seems like when it gets to cache it is fast - those stat fetch which<br>
seem to come from .gluster are slow<br>
<div><div class="h5"><br>
On Sun, Feb 4, 2018 at 3:45 AM, Artem Russakovskii <<a href="mailto:archon810@gmail.com">archon810@gmail.com</a>> wrote:<br>
> An update, and a very interesting one!<br>
><br>
> After I started stracing rsync, all I could see was lstat calls, quite slow<br>
> ones, over and over, which is expected.<br>
><br>
> For example: lstat("uploads/2016/10/<wbr>nexus2cee_DSC05339_thumb-<wbr>161x107.jpg",<br>
> {st_mode=S_IFREG|0664, st_size=4043, ...}) = 0<br>
><br>
> I googled around and found<br>
> <a href="https://gist.github.com/nh2/1836415489e2132cf85ed3832105fcc1" rel="noreferrer" target="_blank">https://gist.github.com/nh2/<wbr>1836415489e2132cf85ed3832105fc<wbr>c1</a>, which is<br>
> seeing this exact issue with gluster, rsync and xfs.<br>
><br>
> Here's the craziest finding so far. If while rsync is running (or right<br>
> before), I run /bin/ls or find on the same gluster dirs, it immediately<br>
> speeds up rsync by a factor of 100 or maybe even 1000. It's absolutely<br>
> insane.<br>
><br>
> I'm stracing the rsync run, and the slow lstat calls flood in at an<br>
> incredible speed as soon as ls or find run. Several hundred of files per<br>
> minute (excruciatingly slow) becomes thousands or even tens of thousands of<br>
> files a second.<br>
><br>
> What do you make of this?<br>
><br>
><br>
</div></div>> ______________________________<wbr>_________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</div>