[Gluster-users] Write performance in a replicated/distributed setup with KVM?

[Harald Hannelius]

On Fri, 2 Mar 2012, Brian Candler wrote:

> On Fri, Mar 02, 2012 at 02:41:30PM +0200, Harald Hannelius wrote:
>>> So next is back to the four-node setup you had before. I would expect that
>>> to perform about the same.
>>
>> So would I expect too. But;
>>
>> # time dd if=/dev/zero bs=1M count=20000 of=/gluster/testfile
>> 20000+0 records in
>> 20000+0 records out
>> 20971520000 bytes (21 GB) copied, 1058.22 s, 19.8 MB/s
>>
>> real	17m38.357s
>> user	0m0.040s
>> sys	0m12.501s
>
> Right, so we know:
>
> - replic of aethra and alcippe is fast
> - distrib/replic across all four nodes is slow
>
> So chopping further, what about:
>
> - replic of adraste and helen?

The pattern for me starts to look like this;

   max-write-speed ~= <link speed>/nodes.

Volume Name: test
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: adraste:/data/single
Brick2: helen:/data/single

# time dd if=/dev/zero bs=1M count=10000 
of=/mnt/testfile
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 195.816 s, 53.5 MB/s

real	3m15.821s
user	0m0.016s
sys	0m8.169s


> This would show whether one of these nodes is at fault.
>
>> At least I got double figure readings this time. Sometimes I get
>> write speeds of 5-6 MB/s.
>
> Well, I'm a bit lost when you start talking about VMs. Is this a production
> environment, and you are doing these dd/cp tests *in addition* to the
> production load of VM traffic?  Or are you doing tests on an unloaded
> system?

I have some systems running in the background yes. They are not really 
production machines.

> Note: mail servers have a nasty habit of doing fsync() all the time, for
> every single received message.

It looks like openldap's slapadd uses some kind of sync as well. The 
load-average on the KVM-host was up at 9.00 while slapadd was running.

> Tools which might be useful to observe the production load:
>
>  iostat 1
>  # shows the count of I/O requests and KB read/written per second

iotop is handy too.

>  btrace /dev/sdb | grep ' [DC] '
>  # shows the actual I/O operations dispatched (D) and completed (C)
>  # to the drive
>
> There are also gluster-layer tools but I've not tried them:
> http://download.gluster.com/pub/gluster/glusterfs/3.2/Documentation/AG/html/chap-Gluster_Administration_Guide-Monitor_Workload.html
>
> Regards,
>
> Brian.
>
>

-- 

Harald Hannelius | harald.hannelius/a\arcada.fi | +358 50 594 1020