[Gluster-users] Slow on writing

[Jeff Darcy]

On 08/16/2013 11:21 PM, Alexey Shalin wrote:
> I wrote small script :
> #!/bin/bash
>
> for i in {1..1000}; do
> size=$((RANDOM%5+1))
> dd if=/dev/zero of=/storage/test/bigfile${i} count=1024 bs=${size}k
> done
>
> This script creates files with different size on volume
>
> here is output:
> 2097152 bytes (2.1 MB) copied, 0.120632 s, 17.4 MB/s
> 1024+0 records in
> 1024+0 records out
> 1048576 bytes (1.0 MB) copied, 0.14548 s, 7.2 MB/s
> 1024+0 records in
> 1024+0 records out

It looks like you're doing small writes (1-6KB) from a single thread.  That 
means network latency is going to be your primary limiting factor.  20MB/s at 
4KB is 5000 IOPS or 0.2ms per network round trip.  You don't say what kind of 
network you're using, but if it's Plain Old GigE that doesn't seem too 
surprising.  BTW, the NFS numbers are likely to be better because the NFS 
client does more caching and you're not writing enough to fill memory, so 
you're actually getting less durability than in the native-protocol test and 
therefore the numbers aren't directly comparable.

I suggest trying larger block sizes and higher I/O thread counts (with iozone 
you can do this in a single command instead of a script).  You should see a 
pretty marked improvement.