[Gluster-users] Write performance in a replicated/distributed setup with KVM?

[Brian Candler]

On Fri, Mar 02, 2012 at 02:41:30PM +0200, Harald Hannelius wrote:
> >So next is back to the four-node setup you had before. I would expect that
> >to perform about the same.
> 
> So would I expect too. But;
> 
> # time dd if=/dev/zero bs=1M count=20000 of=/gluster/testfile
> 20000+0 records in
> 20000+0 records out
> 20971520000 bytes (21 GB) copied, 1058.22 s, 19.8 MB/s
> 
> real	17m38.357s
> user	0m0.040s
> sys	0m12.501s

Right, so we know:

- replic of aethra and alcippe is fast
- distrib/replic across all four nodes is slow

So chopping further, what about:

- replic of adraste and helen?

This would show whether one of these nodes is at fault.

> At least I got double figure readings this time. Sometimes I get
> write speeds of 5-6 MB/s.

Well, I'm a bit lost when you start talking about VMs. Is this a production
environment, and you are doing these dd/cp tests *in addition* to the
production load of VM traffic?  Or are you doing tests on an unloaded
system?

Note: mail servers have a nasty habit of doing fsync() all the time, for
every single received message.

Tools which might be useful to observe the production load:

  iostat 1
  # shows the count of I/O requests and KB read/written per second

  btrace /dev/sdb | grep ' [DC] '
  # shows the actual I/O operations dispatched (D) and completed (C)
  # to the drive

There are also gluster-layer tools but I've not tried them:
http://download.gluster.com/pub/gluster/glusterfs/3.2/Documentation/AG/html/chap-Gluster_Administration_Guide-Monitor_Workload.html

Regards,

Brian.