<div dir="ltr"><div><div>Hi Vijay<br><br></div>Thanks, It would be great if someone can go through the configuration options. Is there any reference document where all these options are described in detail?<br><br></div><div>I was mainly worried about very slow lookup so only did du on a certain file which has a lot of small files (200K). The lookup time improved dramatically. I didn't do any proper benchmarking. <br></div><div><br></div><div>Gluster 3.8 without any optimization <br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" id="gmail-docs-internal-guid-484eb0d2-3608-54e1-50d0-e071c87e5625"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">time du -ksh binno/</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">3.7G</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">binno/</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">real</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">117m45.733s</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">user</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0m1.635s</span></p><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">sys </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0m6.430s</span><br> <br></div><div>Gluster 3.11 with optimization<br><br>time du -ksh binno/<br>3.7G binno/<br><br>real 2m5.595s<br>user 0m0.767s<br>sys 0m4.437s<br><br><br></div><div>I have also enabled profile <br><br></div><div>Before update<br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" id="gmail-docs-internal-guid-484eb0d2-360b-7354-8ead-8e3b9b4100a9"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Fop </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Call Count</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Avg-Latency</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Min-Latency</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Max-Latency</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">--- </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">----------</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">-----------</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">-----------</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">-----------</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">STAT </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">153 </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">90.72 us </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">5.00 us </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">666.00 us</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">STATFS </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">3 </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">677.67 us </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">620.00 us </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">709.00 us</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">OPENDIR </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">149 </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">1213.81 us </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">519.00 us</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">28777.00 us</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">LOOKUP </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">552 </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">8493.01 us </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">3.00 us</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">79689.00 us</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">READDIRP </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">3518 </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">5351.76 us </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">11.00 us 341877.00 us</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">FORGET </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">10050351 </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 us </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 us </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 us</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">RELEASE </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">9062130 </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 us </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 us </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 us</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">RELEASEDIR </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">5395 </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 us </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 us </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 us</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt">After update</p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt">Interval 8 Stats:<br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br> --------- ----------- ----------- ----------- ------------ ----<br> 0.00 0.00 us 0.00 us 0.00 us 2 RELEASEDIR<br> 0.08 118.00 us 113.00 us 123.00 us 2 STATFS<br> 0.13 190.00 us 189.00 us 191.00 us 2 LOOKUP<br> 0.29 422.00 us 422.00 us 422.00 us 2 OPENDIR<br> 99.49 28539.60 us 1698.00 us 48655.00 us 10 READDIRP<br> 0.00 0.00 us 0.00 us 0.00 us 5217 UPCALL<br> 0.00 0.00 us 0.00 us 0.00 us 5217 CI_FORGET<br><br> Duration: 22 seconds<br> Data Read: 0 bytes<br>Data Written: 0 bytes<br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt">I am not sure about profiling result as I don't understand it correctly.</p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt">Thanks</p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt">Kashif<br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p><br></div><div><br><br></div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 11, 2017 at 4:22 PM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Kashif,<br>
<br>
Thank you for your feedback! Do you have some data on the nature of performance improvement observed with 3.11 in the new setup?<br>
<br>
Adding Raghavendra and Poornima for validation of configuration and help with identifying why certain files disappeared from the mount point after enabling readdir-optimize.<br>
<br>
Regards,<br>
Vijay<div><div class="h5"><br>
<br>
On 07/11/2017 11:06 AM, mohammad kashif wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Hi Vijay and Experts<br>
<br>
I didn't want to experiment with my production setup so started a<br>
parallel system with two server and around 80TB storage. First<br>
configured with gluster 3.8 and had the same lookup performance issue.<br>
Then upgraded to 3.11 as you suggested and it made huge improvement in<br>
lookup time. I also did some more optimization as suggested in other<br>
threads.<br>
Now I am going to update my production server. I am planning to use<br>
following optimization option, it would be very useful if you can point<br>
out any inconsistency or suggest some other options. My production setup<br>
has 5 servers consisting of 400TB storage and around 80 million files<br>
of varying lengths.<br>
<br>
Options Reconfigured:<br>
server.event-threads: 4<br>
client.event-threads: 4<br>
cluster.lookup-optimize: on<br>
cluster.readdir-optimize: off<br>
performance.client-io-threads: on<br>
performance.cache-size: 1GB<br>
performance.parallel-readdir: on<br>
performance.md-cache-timeout: 600<br>
performance.cache-invalidation<wbr>: on<br>
performance.stat-prefetch: on<br>
features.cache-invalidation-ti<wbr>meout: 600<br>
features.cache-invalidation: on<br>
nfs.disable: on<br>
performance.readdir-ahead: on<br>
transport.address-family: inet<br>
auth.allow: 163.1.136.*<br>
diagnostics.latency-measuremen<wbr>t: on<br>
diagnostics.count-fop-hits: on<br>
<br>
I found that setting cluster.readdir-optimize to 'on' made some files<br>
disappear from client !<br>
<br>
Thanks<br>
<br>
Kashif<br>
<br>
<br>
<br>
On Sun, Jun 18, 2017 at 4:57 PM, Vijay Bellur <<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a><br></div></div><span class="">
<mailto:<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>>> wrote:<br>
<br>
Hi Mohammad,<br>
<br>
A lot of time is being spent in addressing metadata calls as<br>
expected. Can you consider testing out with 3.11 with md-cache [1]<br>
and readdirp [2] improvements?<br>
<br>
Adding Poornima and Raghavendra who worked on these enhancements to<br>
help out further.<br>
<br>
Thanks,<br>
Vijay<br>
<br>
[1] <a href="https://gluster.readthedocs.io/en/latest/release-notes/3.9.0/" rel="noreferrer" target="_blank">https://gluster.readthedocs.io<wbr>/en/latest/release-notes/3.9.<wbr>0/</a><br>
<<a href="https://gluster.readthedocs.io/en/latest/release-notes/3.9.0/" rel="noreferrer" target="_blank">https://gluster.readthedocs.i<wbr>o/en/latest/release-notes/3.9.<wbr>0/</a>><br>
<br>
[2] <a href="https://github.com/gluster/glusterfs/issues/166" rel="noreferrer" target="_blank">https://github.com/gluster/glu<wbr>sterfs/issues/166</a><br>
<<a href="https://github.com/gluster/glusterfs/issues/166" rel="noreferrer" target="_blank">https://github.com/gluster/gl<wbr>usterfs/issues/166</a>><br>
<br>
On Fri, Jun 16, 2017 at 2:49 PM, mohammad kashif<br></span><span class="">
<<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a> <mailto:<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>><wbr>> wrote:<br>
<br>
Hi Vijay<br>
<br>
Did you manage to look into the gluster profile logs ?<br>
<br>
Thanks<br>
<br>
Kashif<br>
<br>
On Mon, Jun 12, 2017 at 11:40 AM, mohammad kashif<br></span><span class="">
<<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a> <mailto:<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>><wbr>> wrote:<br>
<br>
Hi Vijay<br>
<br>
I have enabled client profiling and used this script<br>
<a href="https://github.com/bengland2/gluster-profile-analysis/blob/master/gvp-client.sh" rel="noreferrer" target="_blank">https://github.com/bengland2/g<wbr>luster-profile-analysis/blob/m<wbr>aster/gvp-client.sh</a><br>
<<a href="https://github.com/bengland2/gluster-profile-analysis/blob/master/gvp-client.sh" rel="noreferrer" target="_blank">https://github.com/bengland2/<wbr>gluster-profile-analysis/blob/<wbr>master/gvp-client.sh</a>><br>
to extract data. I am attaching output files. I don't have<br>
any reference data to compare with my output. Hopefully you<br>
can make some sense out of it.<br>
<br>
On Sat, Jun 10, 2017 at 10:47 AM, Vijay Bellur<br></span><span class="">
<<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a> <mailto:<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>>> wrote:<br>
<br>
Would it be possible for you to turn on client profiling<br>
and then run du? Instructions for turning on client<br>
profiling can be found at [1]. Providing the client<br>
profile information can help us figure out where the<br>
latency could be stemming from.<br>
<br>
Regards,<br>
Vijay<br>
<br>
[1] <a href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Performance%20Testing/#client-side-profiling" rel="noreferrer" target="_blank">https://gluster.readthedocs.io<wbr>/en/latest/Administrator%20Gui<wbr>de/Performance%20Testing/#<wbr>client-side-profiling</a><br>
<<a href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Performance%20Testing/#client-side-profiling" rel="noreferrer" target="_blank">https://gluster.readthedocs.i<wbr>o/en/latest/Administrator%20Gu<wbr>ide/Performance%20Testing/#<wbr>client-side-profiling</a>><br>
<br>
On Fri, Jun 9, 2017 at 7:22 PM, mohammad kashif<br></span>
<<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a> <mailto:<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>><wbr>><div><div class="h5"><br>
wrote:<br>
<br>
Hi Vijay<br>
<br>
Thanks for your quick response. I am using gluster<br>
3.8.11 on Centos 7 servers<br>
glusterfs-3.8.11-1.el7.x86_64<br>
<br>
clients are centos 6 but I tested with a centos 7<br>
client as well and results didn't change<br>
<br>
gluster volume info Volume Name: atlasglust<br>
Type: Distribute<br>
Volume ID: fbf0ebb8-deab-4388-9d8a-f72261<wbr>8a624b<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 5<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: pplxgluster01.x.y.z:/glusterat<wbr>las/brick001/gv0<br>
Brick2: pplxgluster02..x.y.z:/glustera<wbr>tlas/brick002/gv0<br>
Brick3: pplxgluster03.x.y.z:/glusterat<wbr>las/brick003/gv0<br>
Brick4: pplxgluster04.x.y.z:/glusterat<wbr>las/brick004/gv0<br>
Brick5: pplxgluster05.x.y.z:/glusterat<wbr>las/brick005/gv0<br>
Options Reconfigured:<br>
nfs.disable: on<br>
performance.readdir-ahead: on<br>
transport.address-family: inet<br>
auth.allow: x.y.z<br>
<br>
I am not using directory quota.<br>
<br>
Please let me know if you require some more info<br>
<br>
Thanks<br>
<br>
Kashif<br>
<br>
<br>
<br>
On Fri, Jun 9, 2017 at 2:34 PM, Vijay Bellur<br></div></div><span class="">
<<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a> <mailto:<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>>> wrote:<br>
<br>
Can you please provide more details about your<br>
volume configuration and the version of gluster<br>
that you are using?<br>
<br>
Regards,<br>
Vijay<br>
<br>
On Fri, Jun 9, 2017 at 5:35 PM, mohammad kashif<br>
<<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a><br></span><div><div class="h5">
<mailto:<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>><wbr>> wrote:<br>
<br>
Hi<br>
<br>
I have just moved our 400 TB HPC storage<br>
from lustre to gluster. It is part of a<br>
research institute and users have very small<br>
files to big files ( few KB to 20GB) . Our<br>
setup consists of 5 servers, each with 96TB<br>
RAID 6 disks. All servers are connected<br>
through 10G ethernet but not all clients.<br>
Gluster volumes are distributed without any<br>
replication. There are approximately 80<br>
million files in file system.<br>
I am mounting using glusterfs on clients.<br>
<br>
I have copied everything from lustre to<br>
gluster but old file system exist so I can<br>
compare.<br>
<br>
The problem, I am facing is extremely slow<br>
du on even a small directory. Also the time<br>
taken is substantially different each time.<br>
I tried du from same client on a particular<br>
directory twice and got these results.<br>
<br>
time du -sh /data/aa/bb/cc<br>
3.7G /data/aa/bb/cc<br>
real 7m29.243s<br>
user 0m1.448s<br>
sys 0m7.067s<br>
<br>
time du -sh /data/aa/bb/cc<br>
3.7G /data/aa/bb/cc<br>
real 16m43.735s<br>
user 0m1.097s<br>
sys 0m5.802s<br>
<br>
16m and 7m is too long for a 3.7 G<br>
directory. I must mention that the directory<br>
contains huge number of files (208736)<br>
<br>
but running du on same directory on old data<br>
gives this result<br>
<br>
time du -sh /olddata/aa/bb/cc<br>
4.0G /olddata/aa/bb/cc<br>
real 3m1.255s<br>
user 0m0.755s<br>
sys 0m38.099s<br>
<br>
much better if I run same command again<br>
<br>
time du -sh /olddata/aa/bb/cc<br>
4.0G /olddata/aa/bb/cc<br>
real 0m8.309s<br>
user 0m0.313s<br>
sys 0m7.755s<br>
<br>
Is there anything I can do to improve this<br>
performance? I would also like hear from<br>
some one who is running same kind of setup.<br>
<br>
Thanks<br>
<br>
Kashif<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br></div></div>
<mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<wbr>org</a>><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
<<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a>><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
</blockquote>
<br>
</blockquote></div><br></div>