<div dir="ltr">another relevant option is setting cluster.lookup-optimize on.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 31, 2018 at 3:22 PM, Davide Obbi <span dir="ltr"><<a href="mailto:davide.obbi@booking.com" target="_blank">davide.obbi@booking.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">#gluster vol set VOLNAME group nl-cache --> didn't know there are groups of options, after this command i got set the following:<div><div>performance.nl-cache-timeout: 600</div><div>performance.nl-cache: on</div><div>performance.parallel-readdir: on</div><div>performance.io-thread-count: 64</div><div>network.inode-lru-limit: 200000</div></div><div><br></div><div>to note that i had network.inode-lru-limit set to max and got reduced to 200000</div><div><br></div><div>then i added</div><div>performance.nl-cache-positive-<wbr>entry: on<br></div><div><br></div><div>The volume options:</div><div><div>Options Reconfigured:</div><div>performance.nl-cache-timeout: 600</div><div>performance.nl-cache: on</div><div>performance.nl-cache-positive-<wbr>entry: on</div><div>performance.parallel-readdir: on</div><div>performance.io-thread-count: 64</div><div>network.inode-lru-limit: 200000</div><div>nfs.disable: on</div><div>transport.address-family: inet</div><div>performance.readdir-ahead: on</div><div>features.cache-invalidation-<wbr>timeout: 600</div><div>features.cache-invalidation: on</div><div>performance.md-cache-timeout: 600</div><div>performance.stat-prefetch: on</div><div>performance.cache-<wbr>invalidation: on</div><div>performance.cache-size: 10GB</div><div>network.ping-timeout: 5</div><div>diagnostics.client-log-level: WARNING</div><div>diagnostics.brick-log-level: WARNING</div><div>features.quota: off</div><div>features.inode-quota: off</div><div>performance.quick-read: on</div></div><div><br></div><div>untar completed in 08mins 30secs</div><div><br></div><div>increasing network.inode-lru-limit to 1048576 untar completed in the same time</div><div>I have attached the gluster profile results of the last test, with network.inode-lru-limit to 1048576 </div><div><br></div><div>I guess the next test will be creating more bricks for the same volume to have a 2x3. Since i do not see bottlenecks at the disk level and i have limited hw ATM i will just carve out the bricks from LVs from the same 1 disk VG.</div><div> <br></div><div>Also i have tried to look for a complete list of options/description unsuccessfully can you point at one?</div><div><br></div><div>thanks</div><span class="HOEnZb"><font color="#888888"><div>Davide</div></font></span></div><div class="HOEnZb"><div class="h5"><br><div class="gmail_quote"><div dir="ltr">On Thu, Aug 30, 2018 at 5:47 PM Poornima Gurusiddaiah <<a href="mailto:pgurusid@redhat.com" target="_blank">pgurusid@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div>To enable nl-cache please use group option instead of single volume set:</div><div dir="auto"><br></div><div dir="auto">#gluster vol set VOLNAME group nl-cache</div><div dir="auto"><br></div><div dir="auto">This sets few other things including time out, invalidation etc.</div><div dir="auto"><br></div><div dir="auto">For enabling the option Raghavendra mentioned, you ll have to execute it explicitly, as it's not part of group option yet:</div><div dir="auto"><br></div><div dir="auto">#gluster vol set VOLNAME <span style="font-family:sans-serif">performance.nl-cache-<wbr>positive-</span><span style="font-family:sans-serif">entry on</span></div><div dir="auto"><br></div><div dir="auto"><span style="font-family:sans-serif">Also from the past experience, setting the below option has helped in performance:</span></div><div dir="auto"><span style="font-family:sans-serif"><br></span></div><div dir="auto"><span style="font-family:sans-serif"># gluster vol set VOLNAME network.inode-lru-limit 200000</span><span style="font-family:sans-serif"><br></span></div><div dir="auto"><span style="font-family:sans-serif"><br></span></div><div dir="auto"><span style="font-family:sans-serif">Regards,</span></div><div dir="auto"><span style="font-family:sans-serif">Poornima</span></div><div dir="auto"><br><br><div class="gmail_quote" dir="auto"><div dir="ltr">On Thu, Aug 30, 2018, 8:49 PM Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 30, 2018 at 8:38 PM, Davide Obbi <span dir="ltr"><<a href="mailto:davide.obbi@booking.com" rel="noreferrer" target="_blank">davide.obbi@booking.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">yes "performance.parallel-readdir on and 1x3 replica </div></blockquote><div><br></div><div>That's surprising. I thought performance.parallel-readdir will help only when distribute count is fairly high. This is something worth investigating further.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="m_8349659897420060266m_-1334993495116340112m_-6477413448204478553m_8836997128040504259m_-3075493856321534392gmail-HOEnZb"><div class="m_8349659897420060266m_-1334993495116340112m_-6477413448204478553m_8836997128040504259m_-3075493856321534392gmail-h5"><br><div class="gmail_quote"><div dir="ltr">On Thu, Aug 30, 2018 at 5:00 PM Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" rel="noreferrer" target="_blank">rgowdapp@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 30, 2018 at 8:08 PM, Davide Obbi <span dir="ltr"><<a href="mailto:davide.obbi@booking.com" rel="noreferrer" target="_blank">davide.obbi@booking.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks Amar,<div><br></div><div>i have enabled the negative lookups cache on the volume:</div></div></blockquote></div></div></div></blockquote></div></div></div></blockquote><div><br></div><div>I think enabling nl-cache-positive-entry might help for untarring or git clone into glusterfs. Its disabled by default. can you let us know the results?<br></div><div> <br>Option: performance.nl-cache-positive-<wbr>entry<br>Default Value: (null)<br>Description: enable/disable storing of entries that were lookedup and found to be present in the volume, thus lookup on non existent file is served from the cache<br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="m_8349659897420060266m_-1334993495116340112m_-6477413448204478553m_8836997128040504259m_-3075493856321534392gmail-HOEnZb"><div class="m_8349659897420060266m_-1334993495116340112m_-6477413448204478553m_8836997128040504259m_-3075493856321534392gmail-h5"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>To deflate a tar archive (not compressed) of 1.3GB it takes aprox 9mins which can be considered a slight improvement from the previous 12-15 however still not fast enough compared to local disk. The tar is present on the gluster share/volume and deflated inside the same folder structure.</div></div></blockquote><div><br></div><div>I am assuming this is with parallel-readdir enabled, right?</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Running the operation twice (without removing the already deflated files) also did not reduce the time spent.</div><div><br></div><div>Running the operation with the tar archive on local disk made no difference</div><div><br></div><div>What really made a huge difference while git cloning was setting "performance.parallel-readdir on". During the phase "Receiving objects" , as i enabled the xlator it bumped up from 3/4MBs to 27MBs</div></div></blockquote><div><br></div><div>What is the distribute count? Is it 1x3 replica?</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>So in conclusion i'm trying to make the untar operation working at an acceptable level, not expecting local disks speed but at least being within the 4mins</div><div><br></div><div>I have attached the profiles collected at the end of the untar operations with the archive on the mount and outside</div><div><br></div><div>thanks</div><div>Davide</div><div><br></div></div><div class="m_8349659897420060266m_-1334993495116340112m_-6477413448204478553m_8836997128040504259m_-3075493856321534392gmail-m_1149015607859417944m_-3148236163896454807HOEnZb"><div class="m_8349659897420060266m_-1334993495116340112m_-6477413448204478553m_8836997128040504259m_-3075493856321534392gmail-m_1149015607859417944m_-3148236163896454807h5"><br><div class="gmail_quote"><div dir="ltr">On Tue, Aug 28, 2018 at 8:41 AM Amar Tumballi <<a href="mailto:atumball@redhat.com" rel="noreferrer" target="_blank">atumball@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">One of the observation we had with git clone like work load was, nl-cache (negative-lookup cache), helps here.<div><br></div><div>Try 'gluster volume set $volume-name nl-cache enable'.</div><div><br></div><div>Also sharing the 'profile info' during this performance observations also helps us to narrow down the situation.</div><div><br></div><div>More on how to capture profile info @ <a href="https://hackmd.io/PhhT5jPdQIKxzfeLQmnjJQ?view" rel="noreferrer" target="_blank">https://hackmd.io/<wbr>PhhT5jPdQIKxzfeLQmnjJQ?view</a></div><div><br></div><div>-Amar</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 23, 2018 at 7:11 PM, Davide Obbi <span dir="ltr"><<a href="mailto:davide.obbi@booking.com" rel="noreferrer" target="_blank">davide.obbi@booking.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello,<br><br>did anyone ever managed to achieve reasonable waiting time while performing metadata intensive operations such as git clone, untar etc...? Is this possible workload or will never be in scope for glusterfs?<div><br>I'd like to know, if possible, what would be the options that affect such volume performances.<br>Albeit i managed to achieve decent git status/git grep operations, 3 and 30 secs, the git clone and untarring a file from/to the same share take ages. for a git repo of aprox 6GB.<br><br><div>I'm running a test environment with 3 way replica 128GB RAM and 24 cores are 2.40GHz, one internal SSD dedicated to the volume brick and 10Gb network</div><div><br><div>The options set so far that affects volume performances are:<br> 48 performance.readdir-ahead: on <br> 49 features.cache-invalidation-<wbr>timeout: 600 <br> 50 features.cache-invalidation: on <br> 51 performance.md-cache-timeout: 600 <br> 52 performance.stat-prefetch: on <br> 53 performance.cache-<wbr>invalidation: on <br> 54 performance.parallel-readdir: on <br> 55 network.inode-lru-limit: 900000 <br> 56 performance.io-thread-count: 32 <br> 57 performance.cache-size: 10GB</div></div></div></div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" rel="noreferrer" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_8349659897420060266m_-1334993495116340112m_-6477413448204478553m_8836997128040504259m_-3075493856321534392gmail-m_1149015607859417944m_-3148236163896454807m_2054578596796752721m_6135096272534325129m_8587282507989722834m_-159724758825858482gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</div>
</blockquote></div><br clear="all"><div><br></div></div></div><span class="m_8349659897420060266m_-1334993495116340112m_-6477413448204478553m_8836997128040504259m_-3075493856321534392gmail-m_1149015607859417944m_-3148236163896454807HOEnZb"><font color="#888888">-- <br><div dir="ltr" class="m_8349659897420060266m_-1334993495116340112m_-6477413448204478553m_8836997128040504259m_-3075493856321534392gmail-m_1149015607859417944m_-3148236163896454807m_2054578596796752721m_6135096272534325129m_8587282507989722834gmail_signature"><span style="display:block;font-size:11pt;font-family:Century Gothic;color:rgb(0,53,128)"><div style="color:rgb(0,53,128);font-family:Arial,Helvetica,sans-serif;font-weight:bold;font-size:15px"><div>Davide Obbi</div><div style="font-weight:normal;font-size:13px;color:rgb(0,174,239)">System Administrator<br><br></div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)">Booking.com B.V.<br><a href="https://maps.google.com/?q=Vijzelstraat+66&entry=gmail&source=g" rel="noreferrer" target="_blank">Vijzelstraat 66</a>-80 Amsterdam 1017HL Netherlands</div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><span style="color:rgb(0,174,239)">Direct </span>+31207031558<br></div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><div style="font-weight:bold;font-size:16px;color:rgb(0,53,128)"><a href="https://www.booking.com/" style="color:rgb(0,127,255);background-image:initial;background-position:initial;background-repeat:initial" rel="noreferrer" target="_blank"><img src="https://bstatic.com/static/img/siglogo.jpg" alt="Booking.com" title="Booking.com"></a></div><span style="font-size:11px">The world's #1 accommodation site <br>43 languages, 198+ offices worldwide, 120,000+ global destinations, 1,550,000+ room nights booked every day <br>No booking fees, best price always guaranteed <br></span><span style="font-size:11px">Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)</span><span style="font-size:11px"><br></span></div></div></span></div>
</font></span><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" rel="noreferrer" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_8349659897420060266m_-1334993495116340112m_-6477413448204478553m_8836997128040504259m_-3075493856321534392gmail-m_1149015607859417944gmail_signature"><span style="display:block;font-size:11pt;font-family:Century Gothic;color:rgb(0,53,128)"><div style="color:rgb(0,53,128);font-family:Arial,Helvetica,sans-serif;font-weight:bold;font-size:15px"><div>Davide Obbi</div><div style="font-weight:normal;font-size:13px;color:rgb(0,174,239)">System Administrator<br><br></div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)">Booking.com B.V.<br><a href="https://maps.google.com/?q=Vijzelstraat+66&entry=gmail&source=g" rel="noreferrer" target="_blank">Vijzelstraat 66</a>-80 Amsterdam 1017HL Netherlands</div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><span style="color:rgb(0,174,239)">Direct </span>+31207031558<br></div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><div style="font-weight:bold;font-size:16px;color:rgb(0,53,128)"><a href="https://www.booking.com/" style="color:rgb(0,127,255);background-image:initial;background-position:initial;background-repeat:initial" rel="noreferrer" target="_blank"><img src="https://bstatic.com/static/img/siglogo.jpg" alt="Booking.com" title="Booking.com"></a></div><span style="font-size:11px">The world's #1 accommodation site <br>43 languages, 198+ offices worldwide, 120,000+ global destinations, 1,550,000+ room nights booked every day <br>No booking fees, best price always guaranteed <br></span><span style="font-size:11px">Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)</span><span style="font-size:11px"><br></span></div></div></span></div>
</div></div></blockquote></div><br></div></div></div>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" rel="noreferrer" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a></blockquote></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_8349659897420060266m_-1334993495116340112m_-6477413448204478553gmail_signature" data-smartmail="gmail_signature"><span style="display:block;font-size:11.0pt;font-family:Century Gothic;color:#003580"><div style="color:rgb(0,53,128);font-family:Arial,Helvetica,sans-serif;font-weight:bold;font-size:15px"><div>Davide Obbi</div><div style="font-weight:normal;font-size:13px;color:rgb(0,174,239)">System Administrator<br><br></div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)">Booking.com B.V.<br>Vijzelstraat 66-80 Amsterdam 1017HL Netherlands</div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><span style="color:rgb(0,174,239)">Direct </span>+31207031558<br></div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><div style="font-weight:bold;font-size:16px;color:rgb(0,53,128)"><a href="https://www.booking.com/" style="color:rgb(0,127,255);background-image:initial;background-position:initial;background-repeat:initial" target="_blank"><img src="https://bstatic.com/static/img/siglogo.jpg" alt="Booking.com" title="Booking.com"></a></div><span style="font-size:11px">The world's #1 accommodation site <br>43 languages, 198+ offices worldwide, 120,000+ global destinations, 1,550,000+ room nights booked every day <br>No booking fees, best price always guaranteed <br></span><span style="font-size:11px">Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)</span><span style="font-size:11px"><br></span></div></div></span></div>
</div></div></blockquote></div><br></div>