[Gluster-users] Write performance in a replicated/distributed setup with KVM?
harald.hannelius at arcada.fi
Fri Mar 2 13:33:19 UTC 2012
On Fri, 2 Mar 2012, Brian Candler wrote:
> On Fri, Mar 02, 2012 at 02:41:30PM +0200, Harald Hannelius wrote:
>>> So next is back to the four-node setup you had before. I would expect that
>>> to perform about the same.
>> So would I expect too. But;
>> # time dd if=/dev/zero bs=1M count=20000 of=/gluster/testfile
>> 20000+0 records in
>> 20000+0 records out
>> 20971520000 bytes (21 GB) copied, 1058.22 s, 19.8 MB/s
>> real 17m38.357s
>> user 0m0.040s
>> sys 0m12.501s
> Right, so we know:
> - replic of aethra and alcippe is fast
> - distrib/replic across all four nodes is slow
> So chopping further, what about:
> - replic of adraste and helen?
The pattern for me starts to look like this;
max-write-speed ~= <link speed>/nodes.
Volume Name: test
Number of Bricks: 2
# time dd if=/dev/zero bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 195.816 s, 53.5 MB/s
> This would show whether one of these nodes is at fault.
>> At least I got double figure readings this time. Sometimes I get
>> write speeds of 5-6 MB/s.
> Well, I'm a bit lost when you start talking about VMs. Is this a production
> environment, and you are doing these dd/cp tests *in addition* to the
> production load of VM traffic? Or are you doing tests on an unloaded
I have some systems running in the background yes. They are not really
> Note: mail servers have a nasty habit of doing fsync() all the time, for
> every single received message.
It looks like openldap's slapadd uses some kind of sync as well. The
load-average on the KVM-host was up at 9.00 while slapadd was running.
> Tools which might be useful to observe the production load:
> iostat 1
> # shows the count of I/O requests and KB read/written per second
iotop is handy too.
> btrace /dev/sdb | grep ' [DC] '
> # shows the actual I/O operations dispatched (D) and completed (C)
> # to the drive
> There are also gluster-layer tools but I've not tried them:
Harald Hannelius | harald.hannelius/a\arcada.fi | +358 50 594 1020
More information about the Gluster-users