<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8" /></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Also note, Sam's example is comparing apples and orchards. Feeding one person from an orchard is not as efficient as feeding one person an apple, but if you're feeding 10000 people... <br>
<br>
Also in question with the NFS example, how long until that chown was flushed? How long until another client could see those changes? That is ignoring the biggie, what happens when the NFS server goes down?<br><br><div class="gmail_quote">On November 27, 2017 2:49:23 PM PST, Sam McLeod <mailinglists@smcleod.net> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi Aaron,<div class=""><br class="" /></div><div class="">We also find that Gluster is perhaps, not the most performant when performing actions on directories containing large numbers of files.</div><div class="">For example, with a single NFS server on the client side a recursive chown on (many!) files took about 18 seconds, our simple two replica gluster servers took over 15 minutes.</div><div class="">Having said that, while I'm new to the gluster world, things seem to be progressing quite quickly in regards to attempts to improve performance.</div><div class=""><br class="" /></div><div class="">I noticed you're running a _very_ old version of Gluster, I'd first suggest upgrading to the latest stable (3.12.x) and FYI 3.13 is to be release shortly.</div><div class=""><br class="" /></div><div class="">I'd also recommend ensuring the following setting is enabled:</div><div class=""><br class="" /></div><div class="">performance.stat-prefetch</div><div class=""><br class="" /></div><div class="">Further to this, additional information about the cluster / volume typology and configuration would help others assist you (but I still think you should upgrade!).<br class="" /><div class="">
<div dir="auto" style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="" />--<br class="" />Sam McLeod<br class="" /><a href="https://smcleod.net" class="">https://smcleod.net</a><br class="" />https://twitter.com/s_mcleod</div></div></div></div>
</div>
<div><br class="" /><blockquote type="cite" class=""><div class="">On 28 Nov 2017, at 12:18 am, Aaron Roberts <<a href="mailto:aroberts@domicilium.com" class="">aroberts@domicilium.com</a>> wrote:</div><br class="Apple-interchange-newline" /><div class=""><div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hi,<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> I have a situation where an apache web server is trying to locate the IndexDocument for a directory on a gluster volume. This URL is being hit roughly 20 times per second. There is only 1 file in this directory. However, the parent directory does have a large number of items (+123,000 files and dirs) and we are performing operations to move these files into 2 levels of subdirs.<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">We are seeing very slow response times (around 8 seconds) in apache and also when trying to ls on this dir. Before we started the migrations to move files on the large parent dir into 2 sub levels, we weren’t aware of a problem.<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[root@web-02 images]# time ls -l dir1/get/ | wc -l<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">2<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">real 0m8.114s<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">user 0m0.002s<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">sys 0m0.014s<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Other directories with only 1 item return very quickly (<1 sec).<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[root@Web-01 images]# time ls -l dir1/tmp1/ | wc -l<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">2<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">real 0m0.014s<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">user 0m0.003s<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">sys 0m0.006s<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I’m just trying to understand what would slow down this operation so much. Is it the high frequency of attempts to read the directory (apache hits to dir1/get/) ? Do the move operations on items in the parent directory have any impact?<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Some background info:<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[root@web-02 images]# gluster --version<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">glusterfs 3.7.20 built on Jan 30 2017 15:39:29<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Repository revision:<span class="Apple-converted-space"> </span><a href="git://git.gluster.com/glusterfs.git" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">git://git.gluster.com/glusterfs.git</a><p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Copyright (c) 2006-2011 Gluster Inc. <<a href="http://www.gluster.com/" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">http://www.gluster.com</a>><p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">GlusterFS comes with ABSOLUTELY NO WARRANTY.<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">You may redistribute copies of GlusterFS under the terms of the GNU General Public License.<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[root@web-02 images]# gluster vol info<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Volume Name: web_vol1<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Type: Replicate<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Volume ID: 0d63de20-c9c2-4931-b4a3-6aed5ae28057<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Status: Started<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Number of Bricks: 1 x 2 = 2<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Transport-type: tcp<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Bricks:<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Brick1: web-01:/export/brick1/web_vol1_brick1<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Brick2: web-02:/export/brick1/web_vol1_brick1<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Options Reconfigured:<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">performance.readdir-ahead: on<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><a href="http://performance.io/" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">performance.io</a>-thread-count: 32<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">performance.cache-size: 512MB<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Any insight would be gratefully received.<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks,<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Aaron<p class=""></p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><p class=""> </p></div></div><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="" /><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Gluster-users mailing list</span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="" /><a href="mailto:Gluster-users@gluster.org" style="color: rgb(149, 79, 114); text-decoration: underline; font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Gluster-users@gluster.org</a><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="" /><a href="http://lists.gluster.org/mailman/listinfo/gluster-users" style="color: rgb(149, 79, 114); text-decoration: underline; font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.gluster.org/mailman/listinfo/gluster-users</a></div></blockquote></div><br class="" /></div></blockquote></div><br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.</body></html>