[Gluster-users] Migrating a VM makes its gluster storage inaccessible

Paul Robert Marino prmarino1 at gmail.com
Wed Jan 22 19:02:46 UTC 2014 

well the first thing I see that stands out to me is 10.0.1.0/31 is
dubious and can cause strange behavior I usually do /32's with static
routes or a /29 instead.
I also would check the output of  "ip route" on both hosts just to
make sure you don't have any routing issues.
further more I often don't put an IP on the bridged interface and
instead have a dedicated physically seperate management interface
because it ensures that the physical hosts routing table doesn't
pollute the VM's routing which it may do otherwise.

also I have an other suspicion did you use the IP addresses of the
crossover as your gluster peer
if so that may be your problem. The VM may be connecting to the local
crossover IP.
and unless tou have forwarding enabled in sysctl "net.ipv4.ip_forward
= 1" that would cause a problem like the one you described.
The IP for the peer should be the external IP address for the box.


If you want to use a crossover for Gluster traffic here is a cool trick.
assign all of the interfaces on the box as a /32 then you can set up
static prioritized routes so the first thing they hit in the routing
table is a static host route to their neighbor over the crossover
connection then the standard route for the rest of the subnet.
or if you want to get real fancy you can even configure a policy route
just to control how just the Gluster traffic gets transported.


On Wed, Jan 22, 2014 at 11:35 AM, Paul Boven <boven at jive.nl> wrote:
> Hi Paul, everyone,
>
>
> Thanks for your reply. The networking setup for these is very simple, and
> does not involve NAT, only statically assigned addresses.
>
> Each host has a 1G on a private space address which is set up as a bridge
> interface, which contains the IP address of the host itself, and its guest
> VMs. The hosts are also connected together through a 10G link
> in a separate private space /31. Dnsmasq and NAT are not in use.
>
> Is there any way in which I could debug whether this is a networking issue?
>
> This is the network configuration file /etc/network/interfaces:
>
> # The primary network interface
> auto p2p1
> iface p2p1 inet manual
>
> # Virtual bridge interface on p2p1 (control net)
> auto br0
> iface br0 inet static
>         address 10.0.0.10
>         netmask 255.255.255.0
>         gateway 10.0.0.1
>         dns-nameservers 10.0.0.100 10.0.0.101
>         dns-search example.com
>         bridge_ports p2p1
>         bridge_fd 9
>         bridge_hello 2
>         bridge_maxage 12
>         bridge_stp off
>
> # 10G cluster interconnect
> auto p1p1
> iface p1p1 inet static
>         address 10.0.1.0
>         netmask 255.255.255.254
>         mtu 9000
>
> Regards, Paul Boven.
>
>
>
> On 01/22/2014 05:20 PM, Paul Robert Marino wrote:
>>
>> are you doing any thin like NATing the VM's on the physical host, or
>> do you have any Iptables forward rules on the physical host.
>> if so you may have a connection tracking issue.
>> there are a couple of ways you can fix that if thats the case the
>> easiest of which is to install conntrackd on the physical hosts and
>> configure it to sync directly into the live connection tracking table;
>> however it does limit your scaling capabilities for your fail over
>> zones.
>> The second is not to do that any more.
>>
>>
>>
>> On Wed, Jan 22, 2014 at 9:38 AM, Paul Boven <boven at jive.nl> wrote:
>>>
>>> Hi Josh, everyone,
>>>
>>> I've just tried the server.allow-insecure option, and it makes no
>>> difference.
>>>
>>> You can find a summary and the logfiles at this URL:
>>>
>>> http://epboven.home.xs4all.nl/gluster-migrate.html
>>>
>>> The migration itself happens at 14:00:00, with the first write access
>>> attempt by the migrated guest at 14:00:25 which results in the
>>> 'permission
>>> denied' errors in the gluster.log. Some highlights from gluster.log:
>>>
>>> [2014-01-22 14:00:00.779741] D
>>> [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-gv0-replicate-0:
>>> /kvmtest.raw: failed to get the gfid from dict
>>>
>>> [2014-01-22 14:00:00.780458] D
>>> [afr-common.c:1380:afr_lookup_select_read_child] 0-gv0-replicate-0:
>>> Source
>>> selected as 1 for /kvmtest.raw
>>>
>>> [2014-01-22 14:00:25.176181] W [client-rpc-fops.c:471:client3_3_open_cbk]
>>> 0-gv0-client-1: remote operation failed: Permission denied. Path:
>>> /kvmtest.raw (f7ed9edd-c6bd-4e86-b448-1d98bb38314b)
>>>
>>> [2014-01-22 14:00:25.176322] W [fuse-bridge.c:2167:fuse_writev_cbk]
>>> 0-glusterfs-fuse: 2494829: WRITE => -1 (Permission denied)
>>>
>>> Regards, Paul Boven.
>>>
>>>
>>> On 01/21/2014 05:35 PM, Josh Boon wrote:
>>>>
>>>>
>>>> Hey Paul,
>>>>
>>>>
>>>> Have you tried server.allow-insecure: on as a volume option? If that
>>>> doesn't work we'll need the logs for both bricks.
>>>>
>>>> Best,
>>>> Josh
>>>>
>>>> ----- Original Message -----
>>>> From: "Paul Boven" <boven at jive.nl>
>>>> To: gluster-users at gluster.org
>>>> Sent: Tuesday, January 21, 2014 11:12:03 AM
>>>> Subject: Re: [Gluster-users] Migrating a VM makes its gluster storage
>>>> inaccessible
>>>>
>>>> Hi Josh, everyone,
>>>>
>>>> Glad you're trying to help, so no need to apologize at all.
>>>>
>>>> mount output:
>>>> /dev/sdb1 on /export/brick0 type xfs (rw)
>>>>
>>>> localhost:/gv0 on /gluster type fuse.glusterfs
>>>> (rw,default_permissions,allow_other,max_read=131072)
>>>>
>>>> gluster volume info all:
>>>> Volume Name: gv0
>>>> Type: Replicate
>>>> Volume ID: ee77a036-50c7-4a41-b10d-cc0703769df9
>>>> Status: Started
>>>> Number of Bricks: 1 x 2 = 2
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: 10.88.4.0:/export/brick0/sdb1
>>>> Brick2: 10.88.4.1:/export/brick0/sdb1
>>>> Options Reconfigured:
>>>> diagnostics.client-log-level: INFO
>>>> diagnostics.brick-log-level: INFO
>>>>
>>>> Regards, Paul Boven.
>>>>
>>>>
>>>>
>>>>
>>>> On 01/21/2014 05:02 PM, Josh Boon wrote:
>>>>>
>>>>>
>>>>> Hey Paul,
>>>>>
>>>>> Definitely looks to be gluster. Sorry about the wrong guess on UID/GID.
>>>>> What's the output of "mount" and "gluster volume info all"?
>>>>>
>>>>> Best,
>>>>> Josh
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Paul Boven" <boven at jive.nl>
>>>>> To: gluster-users at gluster.org
>>>>> Sent: Tuesday, January 21, 2014 10:56:34 AM
>>>>> Subject: Re: [Gluster-users] Migrating a VM makes its gluster storage
>>>>> inaccessible
>>>>>
>>>>> Hi Josh,
>>>>>
>>>>> I've taken great care that /etc/passwd and /etc/group are the same on
>>>>> both machines. When the problem occurs, even root gets 'permission
>>>>> denied' when trying to read /gluster/guest.raw. So my first reaction
>>>>> was
>>>>> that it cannot be a uid problem.
>>>>>
>>>>> In the normal situation, the storage for a running guest is owned by
>>>>> libvirt-qemu:kvm. When I shut a guest down (virsh destroy), the
>>>>> ownership changes to root:root on both cluster servers.
>>>>>
>>>>> During a migration (that fails), the ownership also ends up as
>>>>> root:root
>>>>> on both, which I hadn't noticed before. Filemode is 0644.
>>>>>
>>>>> On the originating server, root can still read /gluster/guest.raw,
>>>>> whereas on the destination, this gives me 'permission denied'.
>>>>>
>>>>> The qemu logfile for the guest doesn't show much interesting
>>>>> information, merely 'shutting down' on the originating server, and the
>>>>> startup on de destination server. Libvirt/qemu does not seem to be
>>>>> aware
>>>>> of the situation that the guest ends up in. I'll post the gluster logs
>>>>> somewhere, too.
>>>>>
>>>>>     From the destination server:
>>>>>
>>>>> LC_ALL=C
>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
>>>>> /usr/bin/kvm -name kvmtest -S -M pc-i440fx-1.4 -m 1024 -smp
>>>>> 1,sockets=1,cores=1,threads=1 -uuid
>>>>> 97db2d3f-c8e4-31de-9f89-848356b20da5
>>>>> -nographic -no-user-config -nodefaults -chardev
>>>>>
>>>>>
>>>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/kvmtest.monitor,server,nowait
>>>>> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
>>>>> -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
>>>>> -drive
>>>>>
>>>>>
>>>>> file=/gluster/kvmtest.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none
>>>>> -device
>>>>>
>>>>>
>>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>>>>> -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device
>>>>>
>>>>>
>>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:01:01:11,bus=pci.0,addr=0x3
>>>>> -chardev pty,id=charserial0 -device
>>>>> isa-serial,chardev=charserial0,id=serial0 -incoming tcp:0.0.0.0:49166
>>>>> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>>>>> W: kvm binary is deprecated, please use qemu-system-x86_64 instead
>>>>> char device redirected to /dev/pts/4 (label charserial0)
>>>>>
>>>>> Regards, Paul Boven.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 01/21/2014 04:22 PM, Josh Boon wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Paul,
>>>>>>
>>>>>> Sounds like a potential uid/gid problem.  Would you be able to update
>>>>>> with the logs from cd /var/log/libvirt/qemu/ for the guest from both
>>>>>> source
>>>>>> and destination? Also the gluster logs for the volume would be
>>>>>> awesome.
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Josh
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: "Paul Boven" <boven at jive.nl>
>>>>>> To: gluster-users at gluster.org
>>>>>> Sent: Tuesday, January 21, 2014 9:36:06 AM
>>>>>> Subject: Re: [Gluster-users] Migrating a VM makes its gluster storage
>>>>>> inaccessible
>>>>>>
>>>>>> Hi James,
>>>>>>
>>>>>> Thanks for the quick reply.
>>>>>>
>>>>>> We are only using the fuse mounted paths at the moment. So
>>>>>> libvirt/qemu
>>>>>> simply know of these files as /gluster/guest.raw, and the guests are
>>>>>> not
>>>>>> aware of libgluster.
>>>>>>
>>>>>> Some version numbers:
>>>>>>
>>>>>> Kernel: Ubuntu 3.8.0-35-generic (13.10, Raring)
>>>>>> Glusterfs: 3.4.1-ubuntu1~raring1
>>>>>> qemu: 1.4.0+dfsg-1expubuntu4
>>>>>> libvirt0: 1.0.2-0ubuntu11.13.04.4
>>>>>> The gluster bricks are on xfs.
>>>>>>
>>>>>> Regards, Paul Boven.
>>>>>>
>>>>>>
>>>>>> On 01/21/2014 03:25 PM, James wrote:
>>>>>>>
>>>>>>>
>>>>>>> Are you using the qemu gluster:// storage or are you using a fuse
>>>>>>> mounted file path?
>>>>>>>
>>>>>>> I would actually expect it to work with either, however I haven't had
>>>>>>> a chance to test this yet.
>>>>>>>
>>>>>>> It's probably also useful if you post your qemu versions...
>>>>>>>
>>>>>>> James
>>>>>>>
>>>>>>> On Tue, Jan 21, 2014 at 9:15 AM, Paul Boven <boven at jive.nl> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi everyone
>>>>>>>>
>>>>>>>> We've been running glusterfs-3.4.0 on Ubuntu 13.04, using semiosis'
>>>>>>>> packages. We're using kvm (libvrt) to host guest installs, and
>>>>>>>> thanks
>>>>>>>> to
>>>>>>>> gluster and libvirt, we can live-migrate guests between the two
>>>>>>>> hosts.
>>>>>>>>
>>>>>>>> Recently I ran an apt-get update/upgrade to stay up-to-date with
>>>>>>>> security
>>>>>>>> patches, and this also upgraded our glusterfs to the 3.4.1 version
>>>>>>>> of
>>>>>>>> the
>>>>>>>> packages.
>>>>>>>>
>>>>>>>> Since this upgrade (which updated the gluster packages, but also the
>>>>>>>> Ubuntu
>>>>>>>> kernel package), kvm live migration fails in a most unusual manner.
>>>>>>>> The live
>>>>>>>> migration itself succeeds, but on the receiving machine, the
>>>>>>>> vm-storage for
>>>>>>>> that machine becomes inaccessible. Which in turn causes the guest OS
>>>>>>>> to no
>>>>>>>> longer be able to read or write its filesystem, with of course
>>>>>>>> fairly
>>>>>>>> disastrous consequences for such a guest.
>>>>>>>>
>>>>>>>> So before a migration, everything is running smoothly. The two
>>>>>>>> cluster
>>>>>>>> nodes
>>>>>>>> are 'cl0' and 'cl1', and we do the migration like this:
>>>>>>>>
>>>>>>>> virsh migrate --live --persistent --undefinesource <guest>
>>>>>>>> qemu+tls://cl1/system
>>>>>>>>
>>>>>>>> The migration itself works, but soon as you do the migration, the
>>>>>>>> /gluster/guest.raw file (which holds the filesystem for the guest)
>>>>>>>> becomes
>>>>>>>> completely inaccessible: trying to read it (e.g. with dd or md5sum)
>>>>>>>> results
>>>>>>>> in a 'permission denied' on the destination cluster node, whereas
>>>>>>>> the
>>>>>>>> file
>>>>>>>> is still perfectly fine on the machine that the migration originated
>>>>>>>> from.
>>>>>>>>
>>>>>>>> As soon as I stop the guest (virsh destroy), the /gluster/guest.raw
>>>>>>>> file
>>>>>>>> becomes readable again and I can start up the guest on either server
>>>>>>>> without
>>>>>>>> further issues. It does not affect any of the other files in
>>>>>>>> /gluster/.
>>>>>>>>
>>>>>>>> The problem seems to be in the gluster or fuse part, because once
>>>>>>>> this
>>>>>>>> error
>>>>>>>> condition is triggered, the /gluster/guest.raw cannot be read by any
>>>>>>>> application on the destination server. This situation is 100%
>>>>>>>> reproducible,
>>>>>>>> every attempted live migration fails in this way.
>>>>>>>>
>>>>>>>> Has anyone else experienced this? Is this a known or new bug?
>>>>>>>>
>>>>>>>> We've done some troubleshooting already in the irc channel (thanks
>>>>>>>> to
>>>>>>>> everyone for their help) but haven't found the smoking gun yet. I
>>>>>>>> would
>>>>>>>> appreciate any help in debugging and resolving this.
>>>>>>>>
>>>>>>>> Regards, Paul Boven.
>>>>>>>> --
>>>>>>>> Paul Boven <boven at jive.nl> +31 (0)521-596547
>>>>>>>> Unix/Linux/Networking specialist
>>>>>>>> Joint Institute for VLBI in Europe - www.jive.nl
>>>>>>>> VLBI - It's a fringe science
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Paul Boven <boven at jive.nl> +31 (0)521-596547
>>> Unix/Linux/Networking specialist
>>> Joint Institute for VLBI in Europe - www.jive.nl
>>> VLBI - It's a fringe science
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
> --
> Paul Boven <boven at jive.nl> +31 (0)521-596547
> Unix/Linux/Networking specialist
> Joint Institute for VLBI in Europe - www.jive.nl
> VLBI - It's a fringe science
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

More information about the Gluster-users mailing list