[Gluster-users] Migrating a VM makes its gluster storage inaccessible
Paul Boven
boven at jive.nl
Wed Jan 22 16:35:21 UTC 2014
Hi Paul, everyone,
Thanks for your reply. The networking setup for these is very simple,
and does not involve NAT, only statically assigned addresses.
Each host has a 1G on a private space address which is set up as a
bridge interface, which contains the IP address of the host itself, and
its guest VMs. The hosts are also connected together through a 10G link
in a separate private space /31. Dnsmasq and NAT are not in use.
Is there any way in which I could debug whether this is a networking issue?
This is the network configuration file /etc/network/interfaces:
# The primary network interface
auto p2p1
iface p2p1 inet manual
# Virtual bridge interface on p2p1 (control net)
auto br0
iface br0 inet static
address 10.0.0.10
netmask 255.255.255.0
gateway 10.0.0.1
dns-nameservers 10.0.0.100 10.0.0.101
dns-search example.com
bridge_ports p2p1
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off
# 10G cluster interconnect
auto p1p1
iface p1p1 inet static
address 10.0.1.0
netmask 255.255.255.254
mtu 9000
Regards, Paul Boven.
On 01/22/2014 05:20 PM, Paul Robert Marino wrote:
> are you doing any thin like NATing the VM's on the physical host, or
> do you have any Iptables forward rules on the physical host.
> if so you may have a connection tracking issue.
> there are a couple of ways you can fix that if thats the case the
> easiest of which is to install conntrackd on the physical hosts and
> configure it to sync directly into the live connection tracking table;
> however it does limit your scaling capabilities for your fail over
> zones.
> The second is not to do that any more.
>
>
>
> On Wed, Jan 22, 2014 at 9:38 AM, Paul Boven <boven at jive.nl> wrote:
>> Hi Josh, everyone,
>>
>> I've just tried the server.allow-insecure option, and it makes no
>> difference.
>>
>> You can find a summary and the logfiles at this URL:
>>
>> http://epboven.home.xs4all.nl/gluster-migrate.html
>>
>> The migration itself happens at 14:00:00, with the first write access
>> attempt by the migrated guest at 14:00:25 which results in the 'permission
>> denied' errors in the gluster.log. Some highlights from gluster.log:
>>
>> [2014-01-22 14:00:00.779741] D
>> [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-gv0-replicate-0:
>> /kvmtest.raw: failed to get the gfid from dict
>>
>> [2014-01-22 14:00:00.780458] D
>> [afr-common.c:1380:afr_lookup_select_read_child] 0-gv0-replicate-0: Source
>> selected as 1 for /kvmtest.raw
>>
>> [2014-01-22 14:00:25.176181] W [client-rpc-fops.c:471:client3_3_open_cbk]
>> 0-gv0-client-1: remote operation failed: Permission denied. Path:
>> /kvmtest.raw (f7ed9edd-c6bd-4e86-b448-1d98bb38314b)
>>
>> [2014-01-22 14:00:25.176322] W [fuse-bridge.c:2167:fuse_writev_cbk]
>> 0-glusterfs-fuse: 2494829: WRITE => -1 (Permission denied)
>>
>> Regards, Paul Boven.
>>
>>
>> On 01/21/2014 05:35 PM, Josh Boon wrote:
>>>
>>> Hey Paul,
>>>
>>>
>>> Have you tried server.allow-insecure: on as a volume option? If that
>>> doesn't work we'll need the logs for both bricks.
>>>
>>> Best,
>>> Josh
>>>
>>> ----- Original Message -----
>>> From: "Paul Boven" <boven at jive.nl>
>>> To: gluster-users at gluster.org
>>> Sent: Tuesday, January 21, 2014 11:12:03 AM
>>> Subject: Re: [Gluster-users] Migrating a VM makes its gluster storage
>>> inaccessible
>>>
>>> Hi Josh, everyone,
>>>
>>> Glad you're trying to help, so no need to apologize at all.
>>>
>>> mount output:
>>> /dev/sdb1 on /export/brick0 type xfs (rw)
>>>
>>> localhost:/gv0 on /gluster type fuse.glusterfs
>>> (rw,default_permissions,allow_other,max_read=131072)
>>>
>>> gluster volume info all:
>>> Volume Name: gv0
>>> Type: Replicate
>>> Volume ID: ee77a036-50c7-4a41-b10d-cc0703769df9
>>> Status: Started
>>> Number of Bricks: 1 x 2 = 2
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: 10.88.4.0:/export/brick0/sdb1
>>> Brick2: 10.88.4.1:/export/brick0/sdb1
>>> Options Reconfigured:
>>> diagnostics.client-log-level: INFO
>>> diagnostics.brick-log-level: INFO
>>>
>>> Regards, Paul Boven.
>>>
>>>
>>>
>>>
>>> On 01/21/2014 05:02 PM, Josh Boon wrote:
>>>>
>>>> Hey Paul,
>>>>
>>>> Definitely looks to be gluster. Sorry about the wrong guess on UID/GID.
>>>> What's the output of "mount" and "gluster volume info all"?
>>>>
>>>> Best,
>>>> Josh
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: "Paul Boven" <boven at jive.nl>
>>>> To: gluster-users at gluster.org
>>>> Sent: Tuesday, January 21, 2014 10:56:34 AM
>>>> Subject: Re: [Gluster-users] Migrating a VM makes its gluster storage
>>>> inaccessible
>>>>
>>>> Hi Josh,
>>>>
>>>> I've taken great care that /etc/passwd and /etc/group are the same on
>>>> both machines. When the problem occurs, even root gets 'permission
>>>> denied' when trying to read /gluster/guest.raw. So my first reaction was
>>>> that it cannot be a uid problem.
>>>>
>>>> In the normal situation, the storage for a running guest is owned by
>>>> libvirt-qemu:kvm. When I shut a guest down (virsh destroy), the
>>>> ownership changes to root:root on both cluster servers.
>>>>
>>>> During a migration (that fails), the ownership also ends up as root:root
>>>> on both, which I hadn't noticed before. Filemode is 0644.
>>>>
>>>> On the originating server, root can still read /gluster/guest.raw,
>>>> whereas on the destination, this gives me 'permission denied'.
>>>>
>>>> The qemu logfile for the guest doesn't show much interesting
>>>> information, merely 'shutting down' on the originating server, and the
>>>> startup on de destination server. Libvirt/qemu does not seem to be aware
>>>> of the situation that the guest ends up in. I'll post the gluster logs
>>>> somewhere, too.
>>>>
>>>> From the destination server:
>>>>
>>>> LC_ALL=C
>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
>>>> /usr/bin/kvm -name kvmtest -S -M pc-i440fx-1.4 -m 1024 -smp
>>>> 1,sockets=1,cores=1,threads=1 -uuid 97db2d3f-c8e4-31de-9f89-848356b20da5
>>>> -nographic -no-user-config -nodefaults -chardev
>>>>
>>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/kvmtest.monitor,server,nowait
>>>> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
>>>> -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
>>>>
>>>> file=/gluster/kvmtest.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none
>>>> -device
>>>>
>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>>>> -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device
>>>>
>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:01:01:11,bus=pci.0,addr=0x3
>>>> -chardev pty,id=charserial0 -device
>>>> isa-serial,chardev=charserial0,id=serial0 -incoming tcp:0.0.0.0:49166
>>>> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>>>> W: kvm binary is deprecated, please use qemu-system-x86_64 instead
>>>> char device redirected to /dev/pts/4 (label charserial0)
>>>>
>>>> Regards, Paul Boven.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 01/21/2014 04:22 PM, Josh Boon wrote:
>>>>>
>>>>>
>>>>> Paul,
>>>>>
>>>>> Sounds like a potential uid/gid problem. Would you be able to update
>>>>> with the logs from cd /var/log/libvirt/qemu/ for the guest from both source
>>>>> and destination? Also the gluster logs for the volume would be awesome.
>>>>>
>>>>>
>>>>> Best,
>>>>> Josh
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Paul Boven" <boven at jive.nl>
>>>>> To: gluster-users at gluster.org
>>>>> Sent: Tuesday, January 21, 2014 9:36:06 AM
>>>>> Subject: Re: [Gluster-users] Migrating a VM makes its gluster storage
>>>>> inaccessible
>>>>>
>>>>> Hi James,
>>>>>
>>>>> Thanks for the quick reply.
>>>>>
>>>>> We are only using the fuse mounted paths at the moment. So libvirt/qemu
>>>>> simply know of these files as /gluster/guest.raw, and the guests are not
>>>>> aware of libgluster.
>>>>>
>>>>> Some version numbers:
>>>>>
>>>>> Kernel: Ubuntu 3.8.0-35-generic (13.10, Raring)
>>>>> Glusterfs: 3.4.1-ubuntu1~raring1
>>>>> qemu: 1.4.0+dfsg-1expubuntu4
>>>>> libvirt0: 1.0.2-0ubuntu11.13.04.4
>>>>> The gluster bricks are on xfs.
>>>>>
>>>>> Regards, Paul Boven.
>>>>>
>>>>>
>>>>> On 01/21/2014 03:25 PM, James wrote:
>>>>>>
>>>>>> Are you using the qemu gluster:// storage or are you using a fuse
>>>>>> mounted file path?
>>>>>>
>>>>>> I would actually expect it to work with either, however I haven't had
>>>>>> a chance to test this yet.
>>>>>>
>>>>>> It's probably also useful if you post your qemu versions...
>>>>>>
>>>>>> James
>>>>>>
>>>>>> On Tue, Jan 21, 2014 at 9:15 AM, Paul Boven <boven at jive.nl> wrote:
>>>>>>>
>>>>>>> Hi everyone
>>>>>>>
>>>>>>> We've been running glusterfs-3.4.0 on Ubuntu 13.04, using semiosis'
>>>>>>> packages. We're using kvm (libvrt) to host guest installs, and thanks
>>>>>>> to
>>>>>>> gluster and libvirt, we can live-migrate guests between the two hosts.
>>>>>>>
>>>>>>> Recently I ran an apt-get update/upgrade to stay up-to-date with
>>>>>>> security
>>>>>>> patches, and this also upgraded our glusterfs to the 3.4.1 version of
>>>>>>> the
>>>>>>> packages.
>>>>>>>
>>>>>>> Since this upgrade (which updated the gluster packages, but also the
>>>>>>> Ubuntu
>>>>>>> kernel package), kvm live migration fails in a most unusual manner.
>>>>>>> The live
>>>>>>> migration itself succeeds, but on the receiving machine, the
>>>>>>> vm-storage for
>>>>>>> that machine becomes inaccessible. Which in turn causes the guest OS
>>>>>>> to no
>>>>>>> longer be able to read or write its filesystem, with of course fairly
>>>>>>> disastrous consequences for such a guest.
>>>>>>>
>>>>>>> So before a migration, everything is running smoothly. The two cluster
>>>>>>> nodes
>>>>>>> are 'cl0' and 'cl1', and we do the migration like this:
>>>>>>>
>>>>>>> virsh migrate --live --persistent --undefinesource <guest>
>>>>>>> qemu+tls://cl1/system
>>>>>>>
>>>>>>> The migration itself works, but soon as you do the migration, the
>>>>>>> /gluster/guest.raw file (which holds the filesystem for the guest)
>>>>>>> becomes
>>>>>>> completely inaccessible: trying to read it (e.g. with dd or md5sum)
>>>>>>> results
>>>>>>> in a 'permission denied' on the destination cluster node, whereas the
>>>>>>> file
>>>>>>> is still perfectly fine on the machine that the migration originated
>>>>>>> from.
>>>>>>>
>>>>>>> As soon as I stop the guest (virsh destroy), the /gluster/guest.raw
>>>>>>> file
>>>>>>> becomes readable again and I can start up the guest on either server
>>>>>>> without
>>>>>>> further issues. It does not affect any of the other files in
>>>>>>> /gluster/.
>>>>>>>
>>>>>>> The problem seems to be in the gluster or fuse part, because once this
>>>>>>> error
>>>>>>> condition is triggered, the /gluster/guest.raw cannot be read by any
>>>>>>> application on the destination server. This situation is 100%
>>>>>>> reproducible,
>>>>>>> every attempted live migration fails in this way.
>>>>>>>
>>>>>>> Has anyone else experienced this? Is this a known or new bug?
>>>>>>>
>>>>>>> We've done some troubleshooting already in the irc channel (thanks to
>>>>>>> everyone for their help) but haven't found the smoking gun yet. I
>>>>>>> would
>>>>>>> appreciate any help in debugging and resolving this.
>>>>>>>
>>>>>>> Regards, Paul Boven.
>>>>>>> --
>>>>>>> Paul Boven <boven at jive.nl> +31 (0)521-596547
>>>>>>> Unix/Linux/Networking specialist
>>>>>>> Joint Institute for VLBI in Europe - www.jive.nl
>>>>>>> VLBI - It's a fringe science
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>> --
>> Paul Boven <boven at jive.nl> +31 (0)521-596547
>> Unix/Linux/Networking specialist
>> Joint Institute for VLBI in Europe - www.jive.nl
>> VLBI - It's a fringe science
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
--
Paul Boven <boven at jive.nl> +31 (0)521-596547
Unix/Linux/Networking specialist
Joint Institute for VLBI in Europe - www.jive.nl
VLBI - It's a fringe science
More information about the Gluster-users
mailing list