[Gluster-users] Migrating a VM makes its gluster storage inaccessible

[Paul Boven]

Hi again,

First the good news: I found a way to make live migrations work again.

As quoted below, libvirt changes the ownership of the guest image, 
unless it detects that the image is on a shared filesystem. After 
looking at the code for libvirt, they have code to detect NFS, GFS2 and 
SMB/CIFS, but not Gluster. As libvirt does not detect that the storage 
is on a shared file system, the originating host will perform a chown 
back to root:root at the end of a successful migration, whereas the 
destination host will do a chown to libvirt-qemu:kvm. This is in fact a 
race condition, so the difference in behaviour between 3.4.0 and 3.4.1 
could be down to timing differences.

Workaround: stop your guests, then stop libvirt, and edit 
/etc/libvirt/qemu.conf - this contains a commented out entry 
'dynamic_ownership=1', which is the default. Change this to 0, and 
remove the comment. Then do a chown to libvirt-qemu:kvm for all your 
stopped images. Then you can start the service libvirt-bin again, and 
bring up the guests. Repeat on the other half of your cluster, and test 
a live migration. For me, they work again.

The workaround seems fine with me, but now you have to take care of 
properly setting the ownership of a guest image yourself (presumably 
only once when you create it).

Other possible solutions:

JoeJulian suggested using libgfapi, giving libvirt direct access without 
having to go through the filesystem. This is the preferred setup for 
libvirt+gluster and should also result in better I/O performance. I 
haven't tested this yet, but it's high on my to-do list.

Submit a patch to libvirt so it can detect that the filesystem is 
Gluster. statfs() will only show 'FUSE", but we could then use getxattr 
to see if there is a gluster-specific attribute set (suggested by 
kkeithley). This could be trusted.glusterfs.volume-id, e.g.

Regards, Paul Boven.


On 01/28/2014 01:57 PM, Paul Boven wrote:
> Hi everyone,
>
> On the libvirt Wiki, I found the text below which might well apply to
> our live-migration issue:
>
> "The directory used for storing disk images has to be mounted from
> shared storage on both hosts. Otherwise, the domain may lose access to
> its disk images during migration because source libvirtd may change the
> owner, permissions, and SELinux labels on the disk images once it
> successfully migrates the domain to its destination. Libvirt avoids
> doing such things if it detects that the disk images are mounted from a
> shared storage. "
>
> So perhaps libvirtd fails to recognize that it is on shared storage, and
> it is the originating libvirt that throws a wrench in the wheels by
> changing the ownership?
>
> http://wiki.libvirt.org/page/Migration_fails_because_disk_image_cannot_be_found
>
>
> Regards, Paul Boven.


-- 
Paul Boven <boven at jive.nl> +31 (0)521-596547
Unix/Linux/Networking specialist
Joint Institute for VLBI in Europe - www.jive.nl
VLBI - It's a fringe science