[Gluster-users] Performance optimization tips Gluster 3.3? (small files / directory listings)

[Brian Candler]

On Fri, Jun 08, 2012 at 05:46:42PM +0100, Brian Candler wrote:
> The VM boots with io='native' and bus='virtio', but performance is still
> very poor:
> 
>     ubuntu at lucidtest:~$ dd if=/dev/zero of=/var/tmp/test.zeros bs=1024k count=100
>     100+0 records in
>     100+0 records out
>     104857600 bytes (105 MB) copied, 17.4095 s, 6.0 MB/s
> 
> This will need some further work.

And for comparison, it's not the replication which is causing the delay,
because I get very similar performance if I copy the image to a distributed
volume instead.

This is where the VM is running on dev-storage1 but the distributed image
happens to reside on dev-storage2:

ubuntu at lucidtest:~$ dd if=/dev/zero of=/var/tmp/test.zeros2 bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 14.5182 s, 7.2 MB/s

And this is after live-migrating the VM to dev-storage2:

ubuntu at lucidtest:~$ dd if=/dev/zero of=/var/tmp/test.zeros3 bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.17285 s, 25.1 MB/s

Clearly network latency has a part to play - this is 10GE on CAT6 (yes I
know that's a poor choice for latency, but they're the NICs I happened to
have spare) Given that the dd is writing large blocks, I'd hope that large
ranges of blocks get flushed to disk too. 

Of course, 25.1 MB/s is not exactly stellar either.

Maybe using a qcow2 (growable) image is part of the problem - I'll need to
convert to raw and retest.

Regards,

Brian.