<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Thanks Joe,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> Just to clarify, I’m seeing 8 seconds to run ls -l in a dir containing 2 files. I mentioned that the _<i>parent</i>_ dir contains 123k items, in case it was relevant. Although it seems that the
fact we are hitting the dir with many requests seems to be the key factor.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Aaron<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="mso-fareast-language:EN-GB"> Joe Julian [mailto:joe@julianfamily.org]
<br>
<b>Sent:</b> 29 November 2017 16:16<br>
<b>To:</b> gluster-users@gluster.org; Aaron Roberts <aroberts@domicilium.com>; gluster-users@gluster.org<br>
<b>Subject:</b> Re: [Gluster-users] ls performance on directories with small number of items<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:EN-GB">The -l flag is causing a metadata lookup for every file in the directory. The way the ls command does that it's with
individual fstat calls to each directory entry. That's a lot of tiny network round trips with fops that don't even fill a standard frame thus each frame has a high percentage of overhead for tcp. Add to that the replica check to ensure you're not getting stale
data and you have another round trip for each file. Your 123k directory entries require several frames of getdirent and over 492k frames for the individual fstat calls. That's roughly 16us per frame.<br>
<br>
Can you eliminate the fstat calls? If you only get the directory listing that should be significantly better. To prove this, do "echo *". You will instantly see your 123k entries.<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:EN-GB">On November 27, 2017 5:18:56 AM PST, Aaron Roberts <<a href="mailto:aroberts@domicilium.com">aroberts@domicilium.com</a>> wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"> I have a situation where an apache web server is trying to locate the IndexDocument for a directory on a gluster volume. This URL is being hit roughly 20 times per second. There is only 1 file in this directory. However,
the parent directory does have a large number of items (+123,000 files and dirs) and we are performing operations to move these files into 2 levels of subdirs.<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal">We are seeing very slow response times (around 8 seconds) in apache and also when trying to ls on this dir. Before we started the migrations to move files on the large parent dir into 2 sub levels, we weren’t aware of a problem.<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal">[root@web-02 images]# time ls -l dir1/get/ | wc -l<o:p></o:p></p>
<p class="MsoNormal">2<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal">real 0m8.114s<o:p></o:p></p>
<p class="MsoNormal">user 0m0.002s<o:p></o:p></p>
<p class="MsoNormal">sys 0m0.014s<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal">Other directories with only 1 item return very quickly (<1 sec).<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal">[root@Web-01 images]# time ls -l dir1/tmp1/ | wc -l<o:p></o:p></p>
<p class="MsoNormal">2<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal">real 0m0.014s<o:p></o:p></p>
<p class="MsoNormal">user 0m0.003s<o:p></o:p></p>
<p class="MsoNormal">sys 0m0.006s<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal">I’m just trying to understand what would slow down this operation so much. Is it the high frequency of attempts to read the directory (apache hits to dir1/get/) ? Do the move operations on items in the parent directory have any impact?<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal">Some background info:<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal">[root@web-02 images]# gluster --version<o:p></o:p></p>
<p class="MsoNormal">glusterfs 3.7.20 built on Jan 30 2017 15:39:29<o:p></o:p></p>
<p class="MsoNormal">Repository revision: git://git.gluster.com/glusterfs.git<o:p></o:p></p>
<p class="MsoNormal">Copyright (c) 2006-2011 Gluster Inc. <<a href="http://www.gluster.com">http://www.gluster.com</a>><o:p></o:p></p>
<p class="MsoNormal">GlusterFS comes with ABSOLUTELY NO WARRANTY.<o:p></o:p></p>
<p class="MsoNormal">You may redistribute copies of GlusterFS under the terms of the GNU General Public License.<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal">[root@web-02 images]# gluster vol info<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal">Volume Name: web_vol1<o:p></o:p></p>
<p class="MsoNormal">Type: Replicate<o:p></o:p></p>
<p class="MsoNormal">Volume ID: 0d63de20-c9c2-4931-b4a3-6aed5ae28057<o:p></o:p></p>
<p class="MsoNormal">Status: Started<o:p></o:p></p>
<p class="MsoNormal">Number of Bricks: 1 x 2 = 2<o:p></o:p></p>
<p class="MsoNormal">Transport-type: tcp<o:p></o:p></p>
<p class="MsoNormal">Bricks:<o:p></o:p></p>
<p class="MsoNormal">Brick1: web-01:/export/brick1/web_vol1_brick1<o:p></o:p></p>
<p class="MsoNormal">Brick2: web-02:/export/brick1/web_vol1_brick1<o:p></o:p></p>
<p class="MsoNormal">Options Reconfigured:<o:p></o:p></p>
<p class="MsoNormal">performance.readdir-ahead: on<o:p></o:p></p>
<p class="MsoNormal">performance.io-thread-count: 32<o:p></o:p></p>
<p class="MsoNormal">performance.cache-size: 512MB<o:p></o:p></p>
<p> <o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal">Any insight would be gratefully received.<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal"> Aaron<o:p></o:p></p>
<p> <o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:EN-GB"><br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.<o:p></o:p></span></p>
</div>
</body>
</html>