[Gluster-devel] "failed to fetch volume file (key:dpkvol)" error , when tried as non-root (vdsm) user

Deepak C Shetty deepakcs at linux.vnet.ibm.com
Thu Dec 13 04:50:50 UTC 2012


Here is the qemu process spawned by virsh ( via libvirt )...

root      2912     1  1 10:16 ?        00:00:00 
/usr/local/bin/qemu-system-x86_64 -name virsh-vm-backed-by-gluster -S -M 
pc -cpu qemu64,-svm -enable-kvm -m 1024 -smp 
1,sockets=1,cores=1,threads=1 -uuid bdbae806-c272-4b87-ae69-274fd4d57c5f 
-smbios type=1,manufacturer=oVirt,product=oVirt 
Node,version=17-1,serial=762589AD-3D52-42C3-6F65-D682277D5B37_52:54:00:c7:66:ec,uuid=bdbae806-c272-4b87-ae69-274fd4d57c5f 
-no-user-config -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/virsh-vm-backed-by-gluster.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc 
base=2012-12-13T04:46:48,driftfix=slew -no-shutdown -device 
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device 
virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x3 
-drive 
file=gluster+tcp://vm-vdsm-de-1/dpkvol/debian_lenny_i386_standard.qcow2,if=none,id=drive-ide0-0-0 
-device 
ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 
-chardev pty,id=charconsole0 -device 
virtconsole,chardev=charconsole0,id=console0 -vnc 0:0,password -vga cirrus

With this also, i still get the same error which says "unabel to fetch 
volfile"

But gluster volume and status are fine...

$ gluster volume info dpkvol

Volume Name: dpkvol
Type: Distribute
Volume ID: 846494e5-035a-4658-9611-7f5f3a306b50
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: vm-vdsm-de-1:/home/dpkshetty/mybrick2
Options Reconfigured:
server.allow-insecure: on

$ gluster volume status dpkvol
Status of volume: dpkvol
Gluster process                        Port    Online    Pid
------------------------------------------------------------------------------
Brick vm-vdsm-de-1:/home/dpkshetty/mybrick2        49152    Y 1752
NFS Server on localhost                    38467    Y    1763


thanx,
deepak


On 12/12/2012 07:26 PM, Deepak C Shetty wrote:
> Hi All,
>   I am trying to setup a devpt stack as below
> ovirt->vdsm->libvirt->qemu(using gluster native integration)
>
> So before i even go to vdsm, i wanted to ensure that things work fine 
> via virsh, the stack being the below
> virsh->libvirt->qemu(using gluster native integration)
>
> *** Note that qemu and libvirt both are having gluster native support 
> enabled and upstream, and I am using the version which has these support.
>
> *** Glsuter is configured as below...
>
> $ gluster volume info dpkvol
>
> Volume Name: dpkvol
> Type: Distribute
> Volume ID: 846494e5-035a-4658-9611-7f5f3a306b50
> Status: Started
> Number of Bricks: 1
> Transport-type: tcp
> Bricks:
> Brick1: vm-vdsm-de-1:/home/dpkshetty/mybrick2
> Options Reconfigured:
> server.allow-insecure: on
>
> *** Other system info
>
> $ hostname
> vm-vdsm-de-1
>
> $ ls -al /home/dpkshetty/mybrick2/
> total 931928
> drwxrwxrwx.  3 root      root           4096 Dec 12 14:18 .
> drwx------. 30 dpkshetty dpkshetty      4096 Dec 12 17:48 ..
> -rwxrwxrwx.  2 vdsm      kvm       954269696 Dec 12 17:57 
> debian_lenny_i386_standard.qcow2
> drw-------.  7 vdsm      kvm            4096 Dec 12 14:18 .glusterfs
>
>
> *** Now when i use the below qemu cmdline, logged in as root, it all 
> works fine...
>
> $ qemu-system-x86_64 -drive 
> file=gluster+tcp://vm-vdsm-de-1/dpkvol/debian_lenny_i386_standard.qcow2,if=none,id=drive-ide0-0-0 
> -device 
> ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 
> -vnc :1 --enable-kvm -smp 2 -m 1G
>
> $ id
> uid=0(root) gid=0(root) groups=0(root) 
> context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
>
> *** The below ( exact same qemu cmdline ) fails, when logged in as 
> user vdsm...
>
> $ qemu-system-x86_64 -drive 
> file=gluster+tcp://vm-vdsm-de-1/dpkvol/debian_lenny_i386_standard.qcow2,if=none,id=drive-ide0-0-0 
> -device 
> ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 
> -vnc :1 --enable-kvm -smp 2 -m 1G
> qemu-system-x86_64: -drive 
> file=gluster+tcp://vm-vdsm-de-1/dpkvol/debian_lenny_i386_standard.qcow2,if=none,id=drive-ide0-0-0: 
> Gluster connection failed for server=vm-vdsm-de-1 port=0 volume=dpkvol 
> image=debian_lenny_i386_standard.qcow2 transport=tcp
>
> qemu-system-x86_64: -drive 
> file=gluster+tcp://vm-vdsm-de-1/dpkvol/debian_lenny_i386_standard.qcow2,if=none,id=drive-ide0-0-0: 
> could not open disk image 
> gluster+tcp://vm-vdsm-de-1/dpkvol/debian_lenny_i386_standard.qcow2: No 
> data available
>
> $ id
> uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) 
> context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
>
> I had enabled qemu's gluster block backend logs ( with help from 
> bharata, CCing him here ) and this is what i see there ...
>
> [2012-12-12 13:29:53.988722] I [socket.c:3390:socket_init] 0-gfapi: 
> SSL support is NOT enabled
> [2012-12-12 13:29:53.988768] I [socket.c:3405:socket_init] 0-gfapi: 
> using system polling thread
> [2012-12-12 13:29:53.995202] W [socket.c:501:__socket_rwv] 0-gfapi: 
> readv failed (No data available)
> [2012-12-12 13:29:53.995247] W 
> [socket.c:1932:__socket_proto_state_machine] 0-gfapi: reading from 
> socket failed. Error (No data available), peer (192.168.122.139:24007)
> [2012-12-12 13:29:53.995736] E [rpc-clnt.c:368:saved_frames_unwind] 
> (-->/lib64/libgfrpc.so.0(rpc_clnt_notify+0xd0) [0x7fd8089c1cb0] 
> (-->/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) 
> [0x7fd8089c0113] (-->/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) 
> [0x7fd8089c002e]))) 0-gfapi: forced unwinding frame type(GlusterFS 
> Handshake) op(GETSPEC(2)) called at 2012-12-12 13:29:53.994867 (xid=0x1x)
> [2012-12-12 13:29:53.995787] E [glfs-mgmt.c:486:mgmt_getspec_cbk] 
> 0-glfs-mgmt: failed to fetch volume file (key:dpkvol)
> [2012-12-12 13:29:53.995819] E [glfs-mgmt.c:543:mgmt_rpc_notify] 
> 0-glfs-mgmt: failed to connect with remote-host: No data available
> [2012-12-12 13:29:53.995833] I [glfs-mgmt.c:546:mgmt_rpc_notify] 
> 0-glfs-mgmt: 1 connect attempts left
> [2012-12-12 13:29:57.007468] W [socket.c:501:__socket_rwv] 0-gfapi: 
> readv failed (No data available)
> [2012-12-12 13:29:57.007553] W 
> [socket.c:1932:__socket_proto_state_machine] 0-gfapi: reading from 
> socket failed. Error (No data available), peer (192.168.122.139:24007)
> [2012-12-12 13:29:57.007724] E [rpc-clnt.c:368:saved_frames_unwind] 
> (-->/lib64/libgfrpc.so.0(rpc_clnt_notify+0xd0) [0x7fd8089c1cb0] 
> (-->/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) 
> [0x7fd8089c0113] (-->/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) 
> [0x7fd8089c002e]))) 0-gfapi: forced unwinding frame type(GlusterFS 
> Handshake) op(GETSPEC(2)) called at 2012-12-12 13:29:57.007102 (xid=0x2x)
> [2012-12-12 13:29:57.007755] E [glfs-mgmt.c:486:mgmt_getspec_cbk] 
> 0-glfs-mgmt: failed to fetch volume file (key:dpkvol)
> [2012-12-12 13:29:57.007860] E [glfs-mgmt.c:543:mgmt_rpc_notify] 
> 0-glfs-mgmt: failed to connect with remote-host: No data available
> [2012-12-12 13:29:57.007886] I [glfs-mgmt.c:546:mgmt_rpc_notify] 
> 0-glfs-mgmt: 0 connect attempts left
>
> *** Few things i tried...
>
> * I tried `chmod -R 777 *` to the /var/lib/glusterd/ - just in case 
> the user:group:other perms are inhibiting glusterd to fetch volfile, 
> but that also didn't work.
>
> * When i try from virsh, the exact same error I see as above. I even 
> tried changing dynamic_ownership and user/group of qemu process 
> started by libvirt to root, vdsm, etc , nothing works. Gluster just 
> keeps throwing the same error as above for all cases.
>
> *** Summary
>
> So net net, the way i see it.. the reason for the above error is 
> something to do with running the qemu cmdline as non-root.
> Q is... What needs to be done so that gluster volume works fine when 
> accessed from a client that is running as non root ( 'vdsm' here), the 
> client here being qemu accessing glsuter volume via libgfapi
>
>
> Appreciate any pointers to help narrow down & resolve the issue.
> Let me know if more info on the setup is needed.
>
>
> thanx,
> deepak
> P.S I am not copying qemu-devel , since I feel this is somthing to do 
> with gluster setup/issue.
>
>
>





More information about the Gluster-devel mailing list