[Gluster-devel] Glusterfs 3.7.11 with LibGFApi in Qemu on Ubuntu Xenial does not work

[Prasanna Kalever]

On Wed, Jun 15, 2016 at 2:41 PM, André Bauer <abauer at magix.net> wrote:
>
> Hi Lists,
>
> i just updated on of my Ubuntu KVM Servers from 14.04 (Trusty) to 16.06
> (Xenial).
>
> I use the Glusterfs packages from the officail Ubuntu PPA and my own
> Qemu packages (
> https://launchpad.net/~monotek/+archive/ubuntu/qemu-glusterfs-3.7 )
> which have libgfapi enabled.
>
> On Ubuntu 14.04 everything is working fine. I only had to add the
> following lines to the Apparmor config in
> /etc/apparmor.d/abstractions/libvirt-qemu to get it work:
>
> # for glusterfs
> /proc/sys/net/ipv4/ip_local_reserved_ports r,
> /usr/lib/@{multiarch}/glusterfs/**.so mr,
> /tmp/** rw,
>
> In Ubuntu 16.04 i'm not able to start the my VMs via libvirt or to
> create new images via qemu-img using libgfapi.
>
> Mounting the volume via fuse does work without problems.
>
> Examples:
>
> qemu-img create gluster://storage.mydomain/vmimages/kvm2test.img 1G
> Formatting 'gluster://storage.intdmz.h1.mdd/vmimages/kvm2test.img',
> fmt=raw size=1073741824
> [2016-06-15 08:15:26.710665] E [MSGID: 108006]
> [afr-common.c:4046:afr_notify] 0-vmimages-replicate-0: All subvolumes
> are down. Going offline until atleast one of them comes back up.
> [2016-06-15 08:15:26.710736] E [MSGID: 108006]
> [afr-common.c:4046:afr_notify] 0-vmimages-replicate-1: All subvolumes
> are down. Going offline until atleast one of them comes back up.
>
> Libvirtd log:
>
> [2016-06-13 16:53:57.055113] E [MSGID: 104007]
> [glfs-mgmt.c:637:glfs_mgmt_getspec_cbk] 0-glfs-mgmt: failed to fetch
> volume file (key:vmimages) [Invalid argument]
> [2016-06-13 16:53:57.055196] E [MSGID: 104024]
> [glfs-mgmt.c:738:mgmt_rpc_notify] 0-glfs-mgmt: failed to connect with
> remote-host: storage.intdmz.h1.mdd (Permission denied) [Permission denied]
> 2016-06-13T16:53:58.049945Z qemu-system-x86_64: -drive
> file=gluster://storage.intdmz.h1.mdd/vmimages/checkbox.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=writeback:
> Gluster connection failed for server=storage.intdmz.h1.mdd port=0
> volume=vmimages image=checkbox.qcow2 transport=tcp: Permission denied

I think you have missed enabling bind insecure which is needed by
libgfapi access, please try again after following below steps

=> edit /etc/glusterfs/glusterd.vol by add "option
rpc-auth-allow-insecure on" #(on all nodes)
=> gluster vol set $volume server.allow-insecure on
=> systemctl restart glusterd #(on all nodes)

In case this does not work,
provide help us with the below, along with the logfiles
# gluster vol info
# gluster vol status
# gluster peer status

--
Prasanna

>
> I don't see anything in the apparmor logs when setting everything to
> complain or audit.
>
> It also seems GlusterFS servers don't get any request because brick logs
> are not complaining anything.
>
> Any hints?
>
>
> --
> Regards
> André Bauer
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel