[Gluster-users] Unable to start/deploy VMs after Qemu/Gluster upgrade to 2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1
Martin Toth
snowmailer at gmail.com
Fri Nov 25 15:11:41 UTC 2016
Hello all,
we are using your qemu packages to deploy qemu VMs on our gluster via gfsapi.
Recent upgrade broken our qemu and we are not able to deploy / start VMs anymore.
Gluster is running OK, mounted with FUSE, everything looks ok, there is probably some problem with qemu while accessing gluster with gfsapi.
These are our current versions (Qemu is from https://launchpad.net/~monotek/+archive/ubuntu/qemu-glusterfs-3.7 ):
ii glusterfs-client 3.7.17-ubuntu1~trusty2 amd64 clustered file-system (client package)
ii glusterfs-common 3.7.17-ubuntu1~trusty2 amd64 GlusterFS common libraries and translator modules
ii glusterfs-server 3.7.17-ubuntu1~trusty2 amd64 clustered file-system (server package)
ii qemu-keymaps 2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 all QEMU keyboard maps
ii qemu-kvm 2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 amd64 QEMU Full virtualization
ii qemu-system-common 2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-x86 2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 amd64 QEMU utilities
We see error attached lower in mail. Do you have any suggestions what can cause this problem ?
Thanks in advance for your help.
Regards,
Martin
Volume Name: vmvol
Type: Replicate
Volume ID: a72b5c9e-b8ff-488e-b10f-5ba4b71e62b8
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: node1.storage.internal:/gluster/vmvol/brick1
Brick2: node2.storage.internal:/gluster/vmvol/brick1
Brick3: node3.storage.internal:/gluster/vmvol/brick1
Options Reconfigured:
cluster.self-heal-daemon: on
nfs.disable: on
cluster.server-quorum-type: server
cluster.quorum-type: auto
network.remote-dio: enable
performance.stat-prefetch: on
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
cluster.eager-lock: enable
storage.owner-gid: 9869
storage.owner-uid: 9869
server.allow-insecure: on
performance.readdir-ahead: on
2016-11-25 12:54:06.121+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name one-67 -S -machine pc-i440fx-trusty,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 3c703b26-3f57-44d0-8d76-bb281fd8902c -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-67.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=gluster://node1:24007/vmvol/67/disk.1,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 -drive file=/var/lib/one//datastores/112/67/disk.0,if=none,id=drive-ide0-0-1,readonly=on,format=raw,cache=none -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1,bootindex=3 -drive file=/var/lib/one//datastores/112/67/disk.2,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=24,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=02:00:0a:c8:64:1f,bus=pci.0,addr=0x3,bootindex=1 -vnc 0.0.0.0:67 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
[2016-11-25 12:54:06.254217] I [MSGID: 104045] [glfs-master.c:96:notify] 0-gfapi: New graph 6e6f6465-322d-3231-3330-392d32303136 (0) coming up
[2016-11-25 12:54:06.254246] I [MSGID: 114020] [client.c:2113:notify] 0-vmvol-client-0: parent translators are ready, attempting connect on transport
[2016-11-25 12:54:06.254473] I [MSGID: 114020] [client.c:2113:notify] 0-vmvol-client-1: parent translators are ready, attempting connect on transport
[2016-11-25 12:54:06.254672] I [MSGID: 114020] [client.c:2113:notify] 0-vmvol-client-2: parent translators are ready, attempting connect on transport
[2016-11-25 12:54:06.254844] I [rpc-clnt.c:1960:rpc_clnt_reconfig] 0-vmvol-client-0: changing port to 49152 (from 0)
[2016-11-25 12:54:06.255303] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmvol-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-11-25 12:54:06.255391] I [rpc-clnt.c:1960:rpc_clnt_reconfig] 0-vmvol-client-2: changing port to 49152 (from 0)
[2016-11-25 12:54:06.255844] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmvol-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-11-25 12:54:06.256259] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmvol-client-2: Connected to vmvol-client-2, attached to remote volume '/gluster/vmvol/brick1'.
[2016-11-25 12:54:06.256268] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmvol-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-11-25 12:54:06.256288] I [MSGID: 108005] [afr-common.c:4299:afr_notify] 0-vmvol-replicate-0: Subvolume 'vmvol-client-2' came back up; going online.
[2016-11-25 12:54:06.256464] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmvol-client-2: Server lk version = 1
[2016-11-25 12:54:06.262716] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmvol-client-0: Connected to vmvol-client-0, attached to remote volume '/gluster/vmvol/brick1'.
[2016-11-25 12:54:06.262729] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmvol-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-11-25 12:54:06.262863] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmvol-client-0: Server lk version = 1
[2016-11-25 12:54:06.267906] I [rpc-clnt.c:1960:rpc_clnt_reconfig] 0-vmvol-client-1: changing port to 49152 (from 0)
[2016-11-25 12:54:06.268148] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmvol-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-11-25 12:54:06.287724] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmvol-client-1: Connected to vmvol-client-1, attached to remote volume '/gluster/vmvol/brick1'.
[2016-11-25 12:54:06.287734] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmvol-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-11-25 12:54:06.313439] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmvol-client-1: Server lk version = 1
[2016-11-25 12:54:06.314113] I [MSGID: 108031] [afr-common.c:2071:afr_local_discovery_cbk] 0-vmvol-replicate-0: selecting local read_child vmvol-client-1
[2016-11-25 12:54:06.314928] I [MSGID: 104041] [glfs-resolve.c:890:__glfs_active_subvol] 0-vmvol: switched to graph 6e6f6465-322d-3231-3330-392d32303136 (0)
[2016-11-25 12:54:16.331479] I [MSGID: 114021] [client.c:2122:notify] 0-vmvol-client-0: current graph is no longer active, destroying rpc_client
[2016-11-25 12:54:16.331507] I [MSGID: 114021] [client.c:2122:notify] 0-vmvol-client-1: current graph is no longer active, destroying rpc_client
[2016-11-25 12:54:16.331517] I [MSGID: 114018] [client.c:2037:client_rpc_notify] 0-vmvol-client-0: disconnected from vmvol-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2016-11-25 12:54:16.331531] I [MSGID: 114018] [client.c:2037:client_rpc_notify] 0-vmvol-client-1: disconnected from vmvol-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2016-11-25 12:54:16.331534] I [MSGID: 114021] [client.c:2122:notify] 0-vmvol-client-2: current graph is no longer active, destroying rpc_client
[2016-11-25 12:54:16.331543] W [MSGID: 108001] [afr-common.c:4379:afr_notify] 0-vmvol-replicate-0: Client-quorum is not met
[2016-11-25 12:54:16.331667] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f4dcee94502] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f4dcec642be] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f4dcec643ce] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f4dcec65aa4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f4dcec66360] ))))) 0-vmvol-client-2: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2016-11-25 12:54:16.331128 (xid=0x2f62)
[2016-11-25 12:54:16.331684] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-vmvol-client-2: remote operation failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331701] I [MSGID: 114018] [client.c:2037:client_rpc_notify] 0-vmvol-client-2: disconnected from vmvol-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2016-11-25 12:54:16.331710] E [MSGID: 108006] [afr-common.c:4321:afr_notify] 0-vmvol-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2016-11-25 12:54:16.331712] E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-0: remote operation failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331727] E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-1: remote operation failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331735] E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-2: remote operation failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331749] E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-0: remote operation failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331756] E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-1: remote operation failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331762] E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-2: remote operation failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331768] W [inode.c:1814:inode_table_destroy] (-->/usr/lib/x86_64-linux-gnu/libgfapi.so.0(glfs_fini+0x3cf) [0x7f4dd57ef49f] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy_all+0x51) [0x7f4dceebe4d1] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy+0xd4) [0x7f4dceebe3d4] ) 0-gfapi: Active inode(0x7f4db83101a4) with refcount(1) found during cleanup
[2016-11-25 12:54:16.331834] E [inode.c:468:__inode_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.17/xlator/cluster/replicate.so(afr_local_cleanup+0x128) [0x7f4dbb5dafd8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x21) [0x7f4dceebc7d1] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x4a0b6) [0x7f4dceebc0b6] ) 0-: Assertion failed: inode->ref
[2016-11-25 12:54:16.331876] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-gfapi: size=84 max=2 total=2
[2016-11-25 12:54:16.331924] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-gfapi: size=156 max=3 total=3
[2016-11-25 12:54:16.332037] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-gfapi: size=108 max=2 total=2
[2016-11-25 12:54:16.332046] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-client-0: size=1300 max=2 total=3038
[2016-11-25 12:54:16.332054] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-client-1: size=1300 max=2 total=3039
[2016-11-25 12:54:16.332060] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-client-2: size=1300 max=3 total=3038
[2016-11-25 12:54:16.332068] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-replicate-0: size=10556 max=4 total=3038
[2016-11-25 12:54:16.332107] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-dht: size=1148 max=0 total=0
[2016-11-25 12:54:16.332141] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-dht: size=2316 max=2 total=6
[2016-11-25 12:54:16.332200] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-readdir-ahead: size=60 max=0 total=0
[2016-11-25 12:54:16.332212] I [io-stats.c:2951:fini] 0-vmvol: io-stats translator unloaded
[2016-11-25 12:54:16.332245] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 1
[2016-11-25 12:54:16.332259] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 2
qemu-system-x86_64: -drive file=gluster://node1:24007/vmvol/67/disk.1,if=none,id=drive-ide0-0-0,format=qcow2,cache=none: could not open disk image gluster://node1:24007/vmvol/67/disk.1: Could not read L1 table: Bad file descriptor
2016-11-25 12:54:17.199+0000: shutting down
More information about the Gluster-users
mailing list