[Gluster-users] Small files performance on SSD

[Szymon Miotk]

Hello,

I have problems with tuning Gluster for optimal small files performance.
My usage scenario is, as I've learned, worst possible scenario, but
it's not up to me to change it:
- small 1KB files
- at least 20M of those
- approx. 10 files/directory
- mostly writes
- average speed 1000 files/sec with peaks up to 10K files/sec.

I'm doing something wrong, because I cannot get past performance metrics
4K files/sec for distributed volume (2 bricks)
2K files/sec for replicated volume (2 bricks).
I've been experimenting with various XFS formatting and mounting
options and with Gluster tuning, but no luck.
I've learned that it's not disk IO that is the bottleneck (direct
tests on mounted XFS partition show waaay better results, like 100K
files/sec).

As I've learned from
http://blog.gluster.org/2014/03/experiments-using-ssds-with-gluster/
it's possible to get 24K files/sec performance (and that was three years ago).

Test I'm using, run on one server (2 x Xeons, 256 GB RAM, 10GbE network):
smallfile_cli.py --operation create --threads 32 --file-size 1 --files
15625 --top /mnt/testdir/test

Setup:
2 servers with 2 Xeons each, 256GB RAM, 8 x 800GB SSD drives in RAID6,
10GbE network
Ubuntu 14.04
Gluster 3.7.3

Do you have any hints where I should start investigating for the bottleneck?

Best regards,
Szymon