<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 11, 2017 at 11:39 AM, Jo Goossens <span dir="ltr"><<a href="mailto:jo.goossens@hosted-power.com" target="_blank">jo.goossens@hosted-power.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
<p>Hello Joe,</p><p> </p><p> </p><p>I just did a mount like this (added the bold):</p><p> </p><div>mount -t glusterfs -o <b>attribute-timeout=600,entry-<wbr>timeout=600,negative-timeout=<wbr>600,fopen-keep-cache</b>,use-<wbr>readdirp=no,log-level=WARNING,<wbr>log-file=/var/log/glusterxxx.<wbr>log 192.168.140.41:/www /var/www</div><div> </div><p>Results:</p><p> </p><div>root@app1:~/smallfile-master# ./smallfile_cli.py --top /var/www/test --host-set 192.168.140.41 --threads 8 --files 5000 --file-size 64 --record-size 64</div><div><div class="gmail-h5"><div>smallfile version 3.0</div><div> hosts in test : ['192.168.140.41']</div><div> top test directory(s) : ['/var/www/test']</div><div> operation : cleanup</div><div> files/thread : 5000</div><div> threads : 8</div><div> record size (KB, 0 = maximum) : 64</div><div> file size (KB) : 64</div><div> file size distribution : fixed</div><div> files per dir : 100</div><div> dirs per dir : 10</div><div> threads share directories? : N</div><div> filename prefix :</div><div> filename suffix :</div><div> hash file number into dir.? : N</div><div> fsync after modify? : N</div><div> pause between files (microsec) : 0</div><div> finish all requests? : Y</div><div> stonewall? : Y</div><div> measure response times? : N</div><div> verify read? : Y</div><div> verbose? : False</div><div> log to stderr? : False</div><div> ext.attr.size : 0</div><div> ext.attr.count : 0</div><div> permute host directories? : N</div><div> remote program directory : /root/smallfile-master</div><div> network thread sync. dir. : /var/www/test/network_shared</div><div>starting all threads by creating starting gate file /var/www/test/network_shared/<wbr>starting_gate.tmp</div></div></div><div>host = 192.168.140.41,thr = 00,elapsed = 1.232004,files = 5000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 01,elapsed = 1.148738,files = 5000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 02,elapsed = 1.130913,files = 5000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 03,elapsed = 1.183088,files = 5000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 04,elapsed = 1.220752,files = 5000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 05,elapsed = 1.228039,files = 5000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 06,elapsed = 1.216787,files = 5000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 07,elapsed = 1.229036,files = 5000,records = 0,status = ok</div><span class="gmail-"><div>total threads = 8</div><div>total files = 40000</div><div>100.00% of requested files processed, minimum is 70.00</div></span><div>1.232004 sec elapsed time</div><div>32467.428972 files/sec</div><div> </div><p> </p><div>root@app1:~/smallfile-master# ./smallfile_cli.py --top /var/www/test --host-set 192.168.140.41 --threads 8 --files 50000 --file-size 64 --record-size 64</div><span class="gmail-"><div>smallfile version 3.0</div><div> hosts in test : ['192.168.140.41']</div><div> top test directory(s) : ['/var/www/test']</div><div> operation : cleanup</div></span><div> files/thread : 50000</div><span class="gmail-"><div> threads : 8</div><div> record size (KB, 0 = maximum) : 64</div><div> file size (KB) : 64</div><div> file size distribution : fixed</div><div> files per dir : 100</div><div> dirs per dir : 10</div><div> threads share directories? : N</div><div> filename prefix :</div><div> filename suffix :</div><div> hash file number into dir.? : N</div><div> fsync after modify? : N</div><div> pause between files (microsec) : 0</div><div> finish all requests? : Y</div><div> stonewall? : Y</div><div> measure response times? : N</div><div> verify read? : Y</div><div> verbose? : False</div><div> log to stderr? : False</div><div> ext.attr.size : 0</div><div> ext.attr.count : 0</div><div> permute host directories? : N</div><div> remote program directory : /root/smallfile-master</div><div> network thread sync. dir. : /var/www/test/network_shared</div><div>starting all threads by creating starting gate file /var/www/test/network_shared/<wbr>starting_gate.tmp</div></span><div>host = 192.168.140.41,thr = 00,elapsed = 4.242312,files = 50000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 01,elapsed = 4.250831,files = 50000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 02,elapsed = 3.771269,files = 50000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 03,elapsed = 4.060653,files = 50000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 04,elapsed = 3.880653,files = 50000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 05,elapsed = 3.847107,files = 50000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 06,elapsed = 3.895537,files = 50000,records = 0,status = ok</div><div>host = 192.168.140.41,thr = 07,elapsed = 3.966394,files = 50000,records = 0,status = ok</div><div>total threads = 8</div><div>total files = 400000</div><span class="gmail-"><div>100.00% of requested files processed, minimum is 70.00</div></span><div>4.250831 sec elapsed time</div><div>94099.245073 files/sec</div><div>root@app1:~/smallfile-master#</div><div> </div><p> </p><p>As you can see it's now crazy fast, I think close to or faster than nfs !! What the hell!??!</p><p> </p><p>I'm so exited I already post. Any suggestions for those parameters? I will do additional testing over here , because this is ridiculous. That woud mean defaults or no good at all...</p><p> </p></div></blockquote><div><br></div><div>Would it be possible to profile the client [1] with defaults and the set of options used now? That could help in understanding the performance delta better.</div><div><br></div><div>Thanks,</div><div>Vijay</div><div><br></div><div>[1] <a href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Performance%20Testing/#client-side-profiling">https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Performance%20Testing/#client-side-profiling</a> </div><div> </div></div></div></div>