[Gluster-devel] "failed to fetch volume file (key:dpkvol)" error , when tried as non-root (vdsm) user

[Deepak C Shetty]

Hi All,
   I am trying to setup a devpt stack as below
ovirt->vdsm->libvirt->qemu(using gluster native integration)

So before i even go to vdsm, i wanted to ensure that things work fine 
via virsh, the stack being the below
virsh->libvirt->qemu(using gluster native integration)

*** Note that qemu and libvirt both are having gluster native support 
enabled and upstream, and I am using the version which has these support.

*** Glsuter is configured as below...

$ gluster volume info dpkvol

Volume Name: dpkvol
Type: Distribute
Volume ID: 846494e5-035a-4658-9611-7f5f3a306b50
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: vm-vdsm-de-1:/home/dpkshetty/mybrick2
Options Reconfigured:
server.allow-insecure: on

*** Other system info

$ hostname
vm-vdsm-de-1

$ ls -al /home/dpkshetty/mybrick2/
total 931928
drwxrwxrwx.  3 root      root           4096 Dec 12 14:18 .
drwx------. 30 dpkshetty dpkshetty      4096 Dec 12 17:48 ..
-rwxrwxrwx.  2 vdsm      kvm       954269696 Dec 12 17:57 
debian_lenny_i386_standard.qcow2
drw-------.  7 vdsm      kvm            4096 Dec 12 14:18 .glusterfs


*** Now when i use the below qemu cmdline, logged in as root, it all 
works fine...

$ qemu-system-x86_64 -drive 
file=gluster+tcp://vm-vdsm-de-1/dpkvol/debian_lenny_i386_standard.qcow2,if=none,id=drive-ide0-0-0 
-device 
ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 
-vnc :1 --enable-kvm -smp 2 -m 1G

$ id
uid=0(root) gid=0(root) groups=0(root) 
context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

*** The below ( exact same qemu cmdline ) fails, when logged in as user 
vdsm...

$ qemu-system-x86_64 -drive 
file=gluster+tcp://vm-vdsm-de-1/dpkvol/debian_lenny_i386_standard.qcow2,if=none,id=drive-ide0-0-0 
-device 
ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 
-vnc :1 --enable-kvm -smp 2 -m 1G
qemu-system-x86_64: -drive 
file=gluster+tcp://vm-vdsm-de-1/dpkvol/debian_lenny_i386_standard.qcow2,if=none,id=drive-ide0-0-0: 
Gluster connection failed for server=vm-vdsm-de-1 port=0 volume=dpkvol 
image=debian_lenny_i386_standard.qcow2 transport=tcp

qemu-system-x86_64: -drive 
file=gluster+tcp://vm-vdsm-de-1/dpkvol/debian_lenny_i386_standard.qcow2,if=none,id=drive-ide0-0-0: 
could not open disk image 
gluster+tcp://vm-vdsm-de-1/dpkvol/debian_lenny_i386_standard.qcow2: No 
data available

$ id
uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) 
context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

I had enabled qemu's gluster block backend logs ( with help from 
bharata, CCing him here ) and this is what i see there ...

[2012-12-12 13:29:53.988722] I [socket.c:3390:socket_init] 0-gfapi: SSL 
support is NOT enabled
[2012-12-12 13:29:53.988768] I [socket.c:3405:socket_init] 0-gfapi: 
using system polling thread
[2012-12-12 13:29:53.995202] W [socket.c:501:__socket_rwv] 0-gfapi: 
readv failed (No data available)
[2012-12-12 13:29:53.995247] W 
[socket.c:1932:__socket_proto_state_machine] 0-gfapi: reading from 
socket failed. Error (No data available), peer (192.168.122.139:24007)
[2012-12-12 13:29:53.995736] E [rpc-clnt.c:368:saved_frames_unwind] 
(-->/lib64/libgfrpc.so.0(rpc_clnt_notify+0xd0) [0x7fd8089c1cb0] 
(-->/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) 
[0x7fd8089c0113] (-->/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) 
[0x7fd8089c002e]))) 0-gfapi: forced unwinding frame type(GlusterFS 
Handshake) op(GETSPEC(2)) called at 2012-12-12 13:29:53.994867 (xid=0x1x)
[2012-12-12 13:29:53.995787] E [glfs-mgmt.c:486:mgmt_getspec_cbk] 
0-glfs-mgmt: failed to fetch volume file (key:dpkvol)
[2012-12-12 13:29:53.995819] E [glfs-mgmt.c:543:mgmt_rpc_notify] 
0-glfs-mgmt: failed to connect with remote-host: No data available
[2012-12-12 13:29:53.995833] I [glfs-mgmt.c:546:mgmt_rpc_notify] 
0-glfs-mgmt: 1 connect attempts left
[2012-12-12 13:29:57.007468] W [socket.c:501:__socket_rwv] 0-gfapi: 
readv failed (No data available)
[2012-12-12 13:29:57.007553] W 
[socket.c:1932:__socket_proto_state_machine] 0-gfapi: reading from 
socket failed. Error (No data available), peer (192.168.122.139:24007)
[2012-12-12 13:29:57.007724] E [rpc-clnt.c:368:saved_frames_unwind] 
(-->/lib64/libgfrpc.so.0(rpc_clnt_notify+0xd0) [0x7fd8089c1cb0] 
(-->/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) 
[0x7fd8089c0113] (-->/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) 
[0x7fd8089c002e]))) 0-gfapi: forced unwinding frame type(GlusterFS 
Handshake) op(GETSPEC(2)) called at 2012-12-12 13:29:57.007102 (xid=0x2x)
[2012-12-12 13:29:57.007755] E [glfs-mgmt.c:486:mgmt_getspec_cbk] 
0-glfs-mgmt: failed to fetch volume file (key:dpkvol)
[2012-12-12 13:29:57.007860] E [glfs-mgmt.c:543:mgmt_rpc_notify] 
0-glfs-mgmt: failed to connect with remote-host: No data available
[2012-12-12 13:29:57.007886] I [glfs-mgmt.c:546:mgmt_rpc_notify] 
0-glfs-mgmt: 0 connect attempts left

*** Few things i tried...

* I tried `chmod -R 777 *` to the /var/lib/glusterd/ - just in case the 
user:group:other perms are inhibiting glusterd to fetch volfile, but 
that also didn't work.

* When i try from virsh, the exact same error I see as above. I even 
tried changing dynamic_ownership and user/group of qemu process started 
by libvirt to root, vdsm, etc , nothing works. Gluster just keeps 
throwing the same error as above for all cases.

*** Summary

So net net, the way i see it.. the reason for the above error is 
something to do with running the qemu cmdline as non-root.
Q is... What needs to be done so that gluster volume works fine when 
accessed from a client that is running as non root ( 'vdsm' here), the 
client here being qemu accessing glsuter volume via libgfapi


Appreciate any pointers to help narrow down & resolve the issue.
Let me know if more info on the setup is needed.


thanx,
deepak
P.S I am not copying qemu-devel , since I feel this is somthing to do 
with gluster setup/issue.