[Gluster-users] about HA infrastructure for hypervisors

[Brian Candler]

On Wed, Jun 27, 2012 at 03:07:21PM -0500, Nathan Stratton wrote:
> >I've made a test setup like this, but unfortunately I haven't yet been able
> >to get half-decent performance out of glusterfs 3.3 as a KVM backend.  It
> >may work better if you use local disk for the VM images, and within the VM
> >mount the glusterfs volume for application data.
> 
> What is considered half-decent? I have a 8 cluster
> distribute+replicate setup and I am getting about 65MB/s and about
> 1.5K IOPS. Considering that I am only using a single two disk SAS
> strip in each host I think that is not bad.

For a 16-disk array, your IOPS is not bad.  But are you actually storing a
VM image on it, and then doing lots of I/O within that VM (as opposed to
mounting the volume form within the VM)?  If so, can you specify your exact
configuration, including OS and kernel versions?

I did my tests on two quad-core/8GB nodes, 12 disks in each (md RAID10),
running ubuntu 12.04, and 10GE RJ45 direct connection.  The disk arrays
locally perform at 350MB/s for streaming writes.

But doing a dd if=/dev/zero bs=1024k within a VM, whose image was mounted on
glusterfs, I was getting only 6-25MB/s.

http://gluster.org/pipermail/gluster-users/2012-June/010553.html
http://gluster.org/pipermail/gluster-users/2012-June/010560.html
http://gluster.org/pipermail/gluster-users/2012-June/010570.html

I get much better performance on locally-attached storage with O_DIRECT (kvm
option "cache=none"), but have been unable to get O_DIRECT to work with
glusterfs.

After a kernel upgrade (to a 3.4+ kernel which supports O_DIRECT for fuse),
and using the mount option direct-io-mode=enable, the VM simply wouldn't
boot:

http://gluster.org/pipermail/gluster-users/2012-June/010572.html
http://gluster.org/pipermail/gluster-users/2012-June/010573.html

Hence I'm keen to learn the recipe for good performance with glusterfs
storing VM images, as if it exists, it doesn't seem to be well documented at
all.

Regards,

Brian.