[Gluster-users] "Granular locking" - does this need to be enabled in 3.3.0 ?
Pranith Kumar Karampuri
pkarampu at redhat.com
Mon Jul 23 09:02:06 UTC 2012
Raised the following bug to track the issue: https://bugzilla.redhat.com/show_bug.cgi?id=842254
----- Original Message -----
From: Anand Avati <anand.avati at gmail.com>
To: Jake Grimmett <jog at mrc-lmb.cam.ac.uk>
Cc: Pranith Kumar Karampuri <pkarampu at redhat.com>, gluster-users at gluster.org
Sent: Thu, 19 Jul 2012 11:13:36 -0400 (EDT)
Subject: Re: [Gluster-users] "Granular locking" - does this need to be enabled in 3.3.0 ?
On Thu, Jul 19, 2012 at 2:14 AM, Jake Grimmett <jog at mrc-lmb.cam.ac.uk>wrote:
> Dear Pranith /Anand ,
> Update on our progress with using KVM & Gluster:
> We built a two server (Dell R710) cluster, each box has...
> 5 x 500 GB SATA RAID5 array (software raid)
> an Intel 10GB ethernet HBA.
> One box has 8GB RAM, the other 48GB
> both have 2 x E5520 Xeon
> Centos 6.3 installed
> Gluster 3.3 installed from the rpm files on the gluster site
> 1) create a replicated gluster volume (on top of xfs)
> 2) setup qemu/kvm with a gluster volume (mounts localhost:/gluster-vol)
> 3) sanlock configured (this is evil!)
> 4) build a virtual machines with 30GB qcow2 image, 1GB RAM
> 5) clone this VM into 4 machines
> 6) check that live migration works (OK)
> Start basic test cycle:
> a) migrate all machines to host #1, then reboot host #2
> b) watch logs for self-heal to complete
> c) migrate VM's to host #2, reboot host #1
> d) check logs for self heal
> The above cycle can be repeated numerous times, and completes without
> error, provided that no (or little) load is on the VM.
> If I give the VM's a work load, such by running "bonnie++" on each VM,
> things start to break.
> 1) it becomes almost impossible to log in to each VM
> 2) the kernel on each VM starts giving timeout errors
> i.e. "echo 0 > /proc/sys/kernel/hung_task_**timeout_secs"
> 3) top / uptime on the hosts shows load average of up to 24
> 4) dd write speed (block size 1K) to gluster is around 3MB/s on the host
> While I agree that running bonnie++ on four VM's is possibly unfair, there
> are load spikes on quiet machines (yum updates etc). I suspect that the I/O
> of one VM starts blocking that of another VM, and the pressure builds up
> rapidly on gluster - which does not seem to cope well under pressure.
> Possibly this is the access pattern / block size of qcow2 disks?
> I'm (slightly) disappointed.
> Though it doesn't corrupt data, the I/O performance is < 1% of my
> hardwares capability. Hopefully work on buffering and other tuning will fix
> this ? Or maybe the work mentioned getting qemu talking directly to gluster
> will fix this?
Do you mean that the I/O is bad when you are performing the migration? Or
bad in general? If it is bad in general the qemu driver should help. Also
try presenting each VM a FUSE mount point of its own (we have seen that
help improve the overall system IOPs)
If it is slow performance only during failover/failback, we probably need
to do some more internal QoS tuning to de-prioritize self-heal traffic from
preempting VM traffic for resources.
More information about the Gluster-users