<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 28, 2017 at 3:44 PM, Szymon Miotk <span dir="ltr"><<a href="mailto:szymon.miotk@gmail.com" target="_blank">szymon.miotk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear Gluster community,<br>
<br>
I have problems with tuning Gluster for small files performance on SSD.<br>
<br>
My usage scenario is, as I've learned, worst possible scenario, but<br>
it's not up to me to change it:<br>
- small 1KB files<br>
- at least 20M of those<br>
- approx. 10 files/directory<br>
- mostly writes<br>
- average speed 1000 files/sec with peaks up to 10K files/sec.<br>
<br>
I'm doing something wrong, because I cannot get past performance metrics<br>
4K files/sec for distributed volume (2 bricks)<br>
2K files/sec for replicated volume (2 bricks).<br>
I've been experimenting with various XFS formatting and mounting<br>
options and with Gluster tuning (md-cache, lookup optimize, thread,<br>
writeback, tiering), but no luck.<br>
<br>
I've learned that it's not disk IO that is the bottleneck (direct<br>
tests on mounted XFS partition show waaay better results, like 100K<br>
files/sec).<br>
<br>
As I've learned from<br>
<a href="http://blog.gluster.org/2014/03/experiments-using-ssds-with-gluster/" rel="noreferrer" target="_blank">http://blog.gluster.org/2014/<wbr>03/experiments-using-ssds-<wbr>with-gluster/</a><br>
it's possible to get 24K files/sec performance (and that was three years ago).<br>
<br></blockquote><div><br></div><div>How many clients are you running? Considering Gluster is a distributed solution, the performance should be measured as aggregate of all the clients. <br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Test I'm using, run on one server (2 x Xeons, 256 GB RAM, 10GbE network):<br>
smallfile_cli.py --operation create --threads 32 --file-size 1 --files<br>
15625 --top /mnt/testdir/test<br>
<br>
Setup:<br>
2 servers with 2 Xeons each, 256GB RAM, 8 x 800GB SSD drives in RAID6,<br>
10GbE network<br>
Ubuntu 14.04<br>
Gluster 3.7.3<br>
<br>
Do you have any hints where I should start investigating for the bottleneck?<br>
<br></blockquote><div><br></div><div>Currently just the fuse mount may be the bottleneck, but I recommend running multiple clients (from different machines) doing these operations in parallel to get the best results out of Gluster.<br></div><div> <br></div><div>-Amar<br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Best regards,<br>
Szymon<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</div></div>