[Gluster-users] Freezing during heal

Kevin Lemonnier lemonnierk at ulrar.net
Mon Apr 18 07:10:16 UTC 2016


Yes, but as I was saying I don't believe KVM is using a mount point, I think it uses
the API (http://www.gluster.org/community/documentation/index.php/Libgfapi_with_qemu_libvirt).
Might be mistaken ofcourse. Proxmox does have a mountpoint for conveniance, I'll attach those
logs, hoping they contain the informations you need. They do seem to contain a lot of errors
for the 15.
For reference, there was a disconnect of the first brick (10.10.0.1) in the morning and then a successfull
heal that caused about 40 minutes downtime of the VMs. Right after that heal finished (if my memory is
correct it was about noon or close) the second brick (10.10.0.2) rebooted, and that's the one I disconnected
to prevent the heal from causing another downtime.
I reconnected it one at the end of the afternoon, hoping the heal would go well but everything went down
like in the morning so I disconnected it again, and waited 11pm (23:00) to reconnect it and let it finish.

Thanks for your help,


On Mon, Apr 18, 2016 at 12:28:28PM +0530, Krutika Dhananjay wrote:
> Sorry, I was referring to the glusterfs client logs.
> 
> Assuming you are using FUSE mount, your log file will be in
> /var/log/glusterfs/<hyphenated-mount-point-path>.log
> 
> -Krutika
> 
> On Sun, Apr 17, 2016 at 9:37 PM, Kevin Lemonnier <lemonnierk at ulrar.net>
> wrote:
> 
> > I believe Proxmox is just an interface to KVM that uses the lib, so if I'm
> > not mistaken there isn't client logs ?
> >
> > It's not the first time I have the issue, it happens on every heal on the
> > 2 clusters I have.
> >
> > I did let the heal finish that night and the VMs are working now, but it
> > is pretty scarry for future crashes or brick replacement.
> > Should I maybe lower the shard size ? Won't solve the fact that 2 bricks
> > on 3 aren't keeping the filesystem usable but might make the healing
> > quicker right ?
> >
> > Thanks
> >
> > Le 17 avril 2016 17:56:37 GMT+02:00, Krutika Dhananjay <
> > kdhananj at redhat.com> a écrit :
> > >Could you share the client logs and information about the approx
> > >time/day
> > >when you saw this issue?
> > >
> > >-Krutika
> > >
> > >On Sat, Apr 16, 2016 at 12:57 AM, Kevin Lemonnier
> > ><lemonnierk at ulrar.net>
> > >wrote:
> > >
> > >> Hi,
> > >>
> > >> We have a small glusterFS 3.7.6 cluster with 3 nodes running with
> > >proxmox
> > >> VM's on it. I did set up the different recommended option like the
> > >virt
> > >> group, but
> > >> by hand since it's on debian. The shards are 256MB, if that matters.
> > >>
> > >> This morning the second node crashed, and as it came back up started
> > >a
> > >> heal, but that basically froze all the VM's running on that volume.
> > >Since
> > >> we really really
> > >> can't have 40 minutes down time in the middle of the day, I just
> > >removed
> > >> the node from the network and that stopped the heal, allowing the
> > >VM's to
> > >> access
> > >> their disks again. The plan was to re-connecte the node in a couple
> > >of
> > >> hours to let it heal at night.
> > >> But a VM crashed now, and it can't boot up again : seems to freez
> > >trying
> > >> to access the disks.
> > >>
> > >> Looking at the heal info for the volume, it has gone way up since
> > >this
> > >> morning, it looks like the VM's aren't writing to both nodes, just
> > >the one
> > >> they are on.
> > >> It seems pretty bad, we have 2 nodes on 3 up, I would expect the
> > >volume to
> > >> work just fine since it has quorum. What am I missing ?
> > >>
> > >> It is still too early to start the heal, is there a way to start the
> > >VM
> > >> anyway right now ? I mean, it was running a moment ago so the data is
> > >> there, it just needs
> > >> to let the VM access it.
> > >>
> > >>
> > >>
> > >> Volume Name: vm-storage
> > >> Type: Replicate
> > >> Volume ID: a5b19324-f032-4136-aaac-5e9a4c88aaef
> > >> Status: Started
> > >> Number of Bricks: 1 x 3 = 3
> > >> Transport-type: tcp
> > >> Bricks:
> > >> Brick1: first_node:/mnt/vg1-storage
> > >> Brick2: second_node:/mnt/vg1-storage
> > >> Brick3: third_node:/mnt/vg1-storage
> > >> Options Reconfigured:
> > >> cluster.quorum-type: auto
> > >> cluster.server-quorum-type: server
> > >> network.remote-dio: enable
> > >> cluster.eager-lock: enable
> > >> performance.readdir-ahead: on
> > >> performance.quick-read: off
> > >> performance.read-ahead: off
> > >> performance.io-cache: off
> > >> performance.stat-prefetch: off
> > >> features.shard: on
> > >> features.shard-block-size: 256MB
> > >> cluster.server-quorum-ratio: 51%
> > >>
> > >>
> > >> Thanks for your help
> > >>
> > >> --
> > >> Kevin Lemonnier
> > >> PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
> > >>
> > >> _______________________________________________
> > >> Gluster-users mailing list
> > >> Gluster-users at gluster.org
> > >> http://www.gluster.org/mailman/listinfo/gluster-users
> > >>
> >
> > --
> > Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser ma brièveté.
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> >

-- 
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
-------------- next part --------------
[2016-04-15 08:27:18.806106] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 (args: /usr/sbin/glusterfs --volfile-server=ips1adm.hobb --volfile-id=vm-storage /mnt/pve/vm-storage)
[2016-04-15 08:27:18.851311] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-04-15 08:27:18.898961] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-04-15 08:27:18.899331] I [MSGID: 114020] [client.c:2118:notify] 0-vm-storage-client-0: parent translators are ready, attempting connect on transport
[2016-04-15 08:27:18.899590] I [MSGID: 114020] [client.c:2118:notify] 0-vm-storage-client-1: parent translators are ready, attempting connect on transport
[2016-04-15 08:27:18.899699] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-vm-storage-client-0: changing port to 49152 (from 0)
[2016-04-15 08:27:18.899807] I [MSGID: 114020] [client.c:2118:notify] 0-vm-storage-client-2: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
  1: volume vm-storage-client-0
  2:     type protocol/client
  3:     option ping-timeout 42
  4:     option remote-host ips1adm.hobb
  5:     option remote-subvolume /mnt/vg1-storage
  6:     option transport-type socket
  7:     option username 65dafda7-3a22-4b44-9cb1-748c05b65a7f
  8:     option password 013a64fb-ae18-43cb-9426-81d1b1828cf0
  9:     option filter-O_DIRECT enable
 10:     option send-gids true
 11: end-volume
 12:  
 13: volume vm-storage-client-1
 14:     type protocol/client
 15:     option ping-timeout 42
 16:     option remote-host ips2adm.hobb
 17:     option remote-subvolume /mnt/vg1-storage
 18:     option transport-type socket
 19:     option username 65dafda7-3a22-4b44-9cb1-748c05b65a7f
 20:     option password 013a64fb-ae18-43cb-9426-81d1b1828cf0
 21:     option filter-O_DIRECT enable
 22:     option send-gids true
 23: end-volume
 24:  
 25: volume vm-storage-client-2
 26:     type protocol/client
 27:     option ping-timeout 42
 28:     option remote-host ips3adm.hobb
 29:     option remote-subvolume /mnt/vg1-storage
 30:     option transport-type socket
 31:     option username 65dafda7-3a22-4b44-9cb1-748c05b65a7f
 32:     option password 013a64fb-ae18-43cb-9426-81d1b1828cf0
[2016-04-15 08:27:18.900023] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vm-storage-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
 33:     option filter-O_DIRECT enable
 34:     option send-gids true
 35: end-volume
 36:  
 37: volume vm-storage-replicate-0
 38:     type cluster/replicate
 39:     option eager-lock enable
 40:     option quorum-type auto
 41:     subvolumes vm-storage-client-0 vm-storage-client-1 vm-storage-client-2
 42: end-volume
 43:  
 44: volume vm-storage-dht
 45:     type cluster/distribute
 46:     subvolumes vm-storage-replicate-0
 47: end-volume
 48:  
 49: volume vm-storage-shard
 50:     type features/shard
 51:     option shard-block-size 256MB
 52:     subvolumes vm-storage-dht
 53: end-volume
 54:  
 55: volume vm-storage-write-behind
 56:     type performance/write-behind
 57:     subvolumes vm-storage-shard
 58: end-volume
 59:  
 60: volume vm-storage-readdir-ahead
 61:     type performance/readdir-ahead
 62:     subvolumes vm-storage-write-behind
 63: end-volume
 64:  
 65: volume vm-storage-open-behind
 66:     type performance/open-behind
 67:     subvolumes vm-storage-readdir-ahead
 68: end-volume
 69:  
 70: volume vm-storage
 71:     type debug/io-stats
 72:     option latency-measurement off
 73:     option count-fop-hits off
 74:     subvolumes vm-storage-open-behind
 75: end-volume
 76:  
 77: volume meta-autoload
 78:     type meta
 79:     subvolumes vm-storage
 80: end-volume
 81:  
+------------------------------------------------------------------------------+
[2016-04-15 08:27:18.900170] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-vm-storage-client-1: changing port to 49152 (from 0)
[2016-04-15 08:27:18.900188] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vm-storage-client-0: Connected to vm-storage-client-0, attached to remote volume '/mnt/vg1-storage'.
[2016-04-15 08:27:18.900202] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vm-storage-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-04-15 08:27:18.900237] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-vm-storage-replicate-0: Subvolume 'vm-storage-client-0' came back up; going online.
[2016-04-15 08:27:18.900275] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vm-storage-client-0: Server lk version = 1
[2016-04-15 08:27:18.900579] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vm-storage-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-04-15 08:27:18.900752] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vm-storage-client-1: Connected to vm-storage-client-1, attached to remote volume '/mnt/vg1-storage'.
[2016-04-15 08:27:18.900761] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vm-storage-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-04-15 08:27:18.900838] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vm-storage-client-1: Server lk version = 1
[2016-04-15 08:27:18.905142] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-vm-storage-client-2: changing port to 49152 (from 0)
[2016-04-15 08:27:18.908828] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vm-storage-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-04-15 08:27:18.910657] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vm-storage-client-2: Connected to vm-storage-client-2, attached to remote volume '/mnt/vg1-storage'.
[2016-04-15 08:27:18.910669] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vm-storage-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-04-15 08:27:18.913358] I [fuse-bridge.c:5137:fuse_graph_setup] 0-fuse: switched to graph 0
[2016-04-15 08:27:18.913403] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vm-storage-client-2: Server lk version = 1
[2016-04-15 08:27:18.913440] I [fuse-bridge.c:4030:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.23
[2016-04-15 08:27:18.915674] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 09:42:03.180355] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-vm-storage-client-1: server 10.10.0.2:49152 has not responded in the last 42 seconds, disconnecting.
[2016-04-15 09:42:03.180590] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 09:41:00.398507 (xid=0xbda)
[2016-04-15 09:42:03.180608] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2016-04-15 09:42:03.180717] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 09:41:01.515128 (xid=0xbdb)
[2016-04-15 09:42:03.180811] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 09:41:18.129979 (xid=0xbdc)
[2016-04-15 09:42:03.180931] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-1: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2016-04-15 09:41:21.177170 (xid=0xbdd)
[2016-04-15 09:42:03.181100] I [socket.c:3308:socket_submit_request] 0-vm-storage-client-1: not connected (priv->connected = 0)
[2016-04-15 09:42:03.181111] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-vm-storage-client-1: failed to submit rpc-request (XID: 0xbde Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vm-storage-client-1)
[2016-04-15 09:42:03.181125] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-1: remote operation failed. Path: /template (af20ae4d-28c3-4434-bc5c-50258b588c77) [Transport endpoint is not connected]
[2016-04-15 09:42:03.181166] W [rpc-clnt-ping.c:208:rpc_clnt_ping_cbk] 0-vm-storage-client-1: socket disconnected
[2016-04-15 09:42:03.181175] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-vm-storage-client-1: disconnected from vm-storage-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2016-04-15 09:42:03.184969] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2016-04-15 09:42:03.434728] E [socket.c:2278:socket_connect_finish] 0-vm-storage-client-1: connection to 10.10.0.2:24007 failed (No route to host)
[2016-04-15 09:42:13.274018] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
The message "W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]" repeated 2 times between [2016-04-15 09:42:03.180608] and [2016-04-15 09:42:03.180822]
[2016-04-15 09:43:26.192916] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-vm-storage-client-1: changing port to 49152 (from 0)
[2016-04-15 09:43:26.193448] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vm-storage-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-04-15 09:43:26.193738] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vm-storage-client-1: Connected to vm-storage-client-1, attached to remote volume '/mnt/vg1-storage'.
[2016-04-15 09:43:26.193747] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vm-storage-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-04-15 09:43:26.193752] I [MSGID: 114042] [client-handshake.c:1056:client_post_handshake] 0-vm-storage-client-1: 2 fds open - Delaying child_up until they are re-opened
[2016-04-15 09:43:26.199263] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-vm-storage-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2016-04-15 09:43:26.199415] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vm-storage-client-1: Server lk version = 1
[2016-04-15 09:43:33.601890] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 09:47:18.208149] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-vm-storage-client-1: server 10.10.0.2:49152 has not responded in the last 42 seconds, disconnecting.
[2016-04-15 09:47:18.208296] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 09:46:13.292844 (xid=0xc59)
[2016-04-15 09:47:18.208313] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2016-04-15 09:47:18.208424] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-1: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2016-04-15 09:46:36.205172 (xid=0xc5a)
[2016-04-15 09:47:18.208434] W [rpc-clnt-ping.c:208:rpc_clnt_ping_cbk] 0-vm-storage-client-1: socket disconnected
[2016-04-15 09:47:18.208489] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 09:47:01.533488 (xid=0xc5b)
[2016-04-15 09:47:18.208724] I [socket.c:3308:socket_submit_request] 0-vm-storage-client-1: not connected (priv->connected = 0)
[2016-04-15 09:47:18.208735] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-vm-storage-client-1: failed to submit rpc-request (XID: 0xc5c Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vm-storage-client-1)
[2016-04-15 09:47:18.208745] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-1: remote operation failed. Path: /template (af20ae4d-28c3-4434-bc5c-50258b588c77) [Transport endpoint is not connected]
[2016-04-15 09:47:18.208749] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-vm-storage-client-1: disconnected from vm-storage-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2016-04-15 09:47:18.212709] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2016-04-15 09:47:18.208500] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2016-04-15 09:47:19.442755] E [socket.c:2278:socket_connect_finish] 0-vm-storage-client-1: connection to 10.10.0.2:24007 failed (No route to host)
[2016-04-15 09:47:28.405510] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 14:06:19.121634] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-vm-storage-replicate-0: performing entry selfheal on be318638-e8a0-4c6d-977d-7a937aa84806
[2016-04-15 14:07:25.236763] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-vm-storage-replicate-0: Completed entry selfheal on be318638-e8a0-4c6d-977d-7a937aa84806. source=-1 sinks=0 2 
[2016-04-15 14:07:44.137699] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-vm-storage-replicate-0: performing entry selfheal on be318638-e8a0-4c6d-977d-7a937aa84806
[2016-04-15 14:08:50.688189] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-vm-storage-replicate-0: Completed entry selfheal on be318638-e8a0-4c6d-977d-7a937aa84806. source=-1 sinks=0 2 
[2016-04-15 14:09:05.510301] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-vm-storage-replicate-0: performing entry selfheal on be318638-e8a0-4c6d-977d-7a937aa84806
[2016-04-15 14:10:11.812496] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-vm-storage-replicate-0: Completed entry selfheal on be318638-e8a0-4c6d-977d-7a937aa84806. source=-1 sinks=0 2 
[2016-04-15 14:28:31.020249] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-vm-storage-client-1: changing port to 49152 (from 0)
[2016-04-15 14:28:31.020745] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vm-storage-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-04-15 14:28:31.021002] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vm-storage-client-1: Connected to vm-storage-client-1, attached to remote volume '/mnt/vg1-storage'.
[2016-04-15 14:28:31.021010] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vm-storage-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-04-15 14:28:31.021015] I [MSGID: 114042] [client-handshake.c:1056:client_post_handshake] 0-vm-storage-client-1: 2 fds open - Delaying child_up until they are re-opened
[2016-04-15 14:28:31.021517] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-vm-storage-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2016-04-15 14:28:31.021617] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vm-storage-client-1: Server lk version = 1
[2016-04-15 14:28:39.461195] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 15:00:24.160533] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-vm-storage-client-1: server 10.10.0.2:49152 has not responded in the last 42 seconds, disconnecting.
[2016-04-15 15:00:24.160759] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 14:59:22.589158 (xid=0x1128)
[2016-04-15 15:00:24.160781] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2016-04-15 15:00:24.160936] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 14:59:25.409734 (xid=0x1129)
[2016-04-15 15:00:24.161077] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-1: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2016-04-15 14:59:42.156826 (xid=0x112a)
[2016-04-15 15:00:24.161089] W [rpc-clnt-ping.c:208:rpc_clnt_ping_cbk] 0-vm-storage-client-1: socket disconnected
[2016-04-15 15:00:24.161159] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 15:00:01.234841 (xid=0x112b)
[2016-04-15 15:00:24.161292] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 15:00:06.653217 (xid=0x112c)
[2016-04-15 15:00:24.161421] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 15:00:15.194406 (xid=0x112d)
[2016-04-15 15:00:24.161468] I [socket.c:3308:socket_submit_request] 0-vm-storage-client-1: not connected (priv->connected = 0)
[2016-04-15 15:00:24.161483] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-vm-storage-client-1: failed to submit rpc-request (XID: 0x112e Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vm-storage-client-1)
[2016-04-15 15:00:24.161496] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-1: remote operation failed. Path: /template (af20ae4d-28c3-4434-bc5c-50258b588c77) [Transport endpoint is not connected]
[2016-04-15 15:00:24.161501] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-vm-storage-client-1: disconnected from vm-storage-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2016-04-15 15:00:24.165773] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2016-04-15 15:00:26.058748] E [socket.c:2278:socket_connect_finish] 0-vm-storage-client-1: connection to 10.10.0.2:24007 failed (No route to host)
[2016-04-15 15:00:34.298042] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
The message "W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]" repeated 4 times between [2016-04-15 15:00:24.160781] and [2016-04-15 15:00:24.161434]
[2016-04-15 15:18:51.942356] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:18:51.948216] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:18:51.956037] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 15:18:54.208264] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 15:31:51.951287] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:31:51.956267] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:31:51.965608] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 15:31:54.812827] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 15:33:15.138565] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:33:15.145630] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:33:15.147358] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:33:15.152674] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 15:33:15.157142] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 15:33:24.326016] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 15:33:29.650585] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:33:29.655647] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:33:29.656908] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 15:33:29.661859] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 15:33:36.190312] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:33:36.212708] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:33:36.214418] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:33:36.223744] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 15:33:36.228262] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 15:33:40.219444] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:33:40.226319] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:33:40.227927] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 15:33:40.233211] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 15:33:40.237811] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 15:33:44.413467] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 16:01:23.020834] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 16:01:23.025839] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 16:01:23.028039] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 16:01:23.034614] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 16:01:36.582024] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 16:01:37.970665] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 16:01:37.972338] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 16:01:37.978373] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 16:01:37.983307] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 16:01:43.981396] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 16:01:43.988224] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 16:01:43.990047] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 16:01:44.002144] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 16:01:44.006330] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 16:01:44.267476] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 16:01:49.212756] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 16:01:49.219513] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 16:01:49.221289] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 16:01:49.227821] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 16:01:49.233684] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 16:01:54.334146] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 19:01:47.233929] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:01:47.241107] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:01:47.242855] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:01:47.251549] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:01:47.254723] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:01:49.585044] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:01:49.591619] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:01:49.593217] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:01:49.605769] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:01:49.606759] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:01:54.126596] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:01:54.131606] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:01:54.132583] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:01:54.140738] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:01:54.410922] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 19:01:57.882337] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:01:57.889207] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:01:57.890874] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:01:57.897900] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:01:57.903602] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:03:26.014836] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:03:26.021430] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:03:26.023078] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:03:26.028662] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:03:26.033493] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:02:01.182563] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 19:03:29.898555] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:03:29.905260] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:03:29.906950] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:03:29.912271] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:03:29.916664] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:03:31.549362] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 19:03:33.882880] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:03:33.888042] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:03:33.891536] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:03:33.895357] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:03:41.576081] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:03:41.582773] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:03:41.584502] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-04-15 19:03:41.589679] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:03:41.594300] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-04-15 19:03:44.046623] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 21:03:35.430243] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-vm-storage-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2016-04-15 21:03:35.430284] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-vm-storage-client-1: disconnected from vm-storage-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2016-04-15 21:03:37.599459] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-vm-storage-client-1: changing port to 49152 (from 0)
[2016-04-15 21:03:37.599924] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vm-storage-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-04-15 21:03:37.601305] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vm-storage-client-1: Connected to vm-storage-client-1, attached to remote volume '/mnt/vg1-storage'.
[2016-04-15 21:03:37.601313] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vm-storage-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-04-15 21:03:37.601323] I [MSGID: 114042] [client-handshake.c:1056:client_post_handshake] 0-vm-storage-client-1: 1 fds open - Delaying child_up until they are re-opened
[2016-04-15 21:03:37.601713] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-vm-storage-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2016-04-15 21:03:37.601846] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vm-storage-client-1: Server lk version = 1
[2016-04-15 21:03:44.061674] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 23:51:59.298179] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-vm-storage-client-2: server 10.10.0.3:49152 has not responded in the last 42 seconds, disconnecting.
[2016-04-15 23:51:59.298611] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-2: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 23:50:56.961052 (xid=0x244df)
[2016-04-15 23:51:59.298629] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2016-04-15 23:51:59.298750] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-2: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 23:51:01.342112 (xid=0x244e0)
[2016-04-15 23:51:59.298853] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-2: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 23:51:14.365308 (xid=0x244e1)
[2016-04-15 23:51:59.298956] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-2: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2016-04-15 23:51:17.295219 (xid=0x244e2)
[2016-04-15 23:51:59.299058] I [socket.c:3308:socket_submit_request] 0-vm-storage-client-2: not connected (priv->connected = 0)
[2016-04-15 23:51:59.299069] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-vm-storage-client-2: failed to submit rpc-request (XID: 0x244e3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vm-storage-client-2)
[2016-04-15 23:51:59.299083] W [rpc-clnt-ping.c:208:rpc_clnt_ping_cbk] 0-vm-storage-client-2: socket disconnected
[2016-04-15 23:51:59.299083] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-2: remote operation failed. Path: /template (af20ae4d-28c3-4434-bc5c-50258b588c77) [Transport endpoint is not connected]
[2016-04-15 23:51:59.299106] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-vm-storage-client-2: disconnected from vm-storage-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2016-04-15 23:51:59.299942] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2016-04-15 23:52:01.339489] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-15 23:52:02.258720] E [socket.c:2278:socket_connect_finish] 0-vm-storage-client-2: connection to 10.10.0.3:24007 failed (No route to host)
[2016-04-15 23:53:27.315460] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-vm-storage-client-2: changing port to 49152 (from 0)
[2016-04-15 23:53:27.319145] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vm-storage-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-04-15 23:53:27.320954] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vm-storage-client-2: Connected to vm-storage-client-2, attached to remote volume '/mnt/vg1-storage'.
[2016-04-15 23:53:27.320965] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vm-storage-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-04-15 23:53:27.322701] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vm-storage-client-2: Server lk version = 1
[2016-04-15 23:53:29.766555] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
The message "W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]" repeated 2 times between [2016-04-15 23:51:59.298629] and [2016-04-15 23:51:59.298867]
[2016-04-15 23:56:26.324270] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-vm-storage-client-2: server 10.10.0.3:49152 has not responded in the last 42 seconds, disconnecting.
[2016-04-15 23:56:26.324416] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-2: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 23:55:41.564023 (xid=0x24547)
[2016-04-15 23:56:26.324434] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2016-04-15 23:56:26.324547] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-2: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2016-04-15 23:55:44.320134 (xid=0x24548)
[2016-04-15 23:56:26.324560] W [rpc-clnt-ping.c:208:rpc_clnt_ping_cbk] 0-vm-storage-client-2: socket disconnected
[2016-04-15 23:56:26.324615] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-2: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 23:56:01.791993 (xid=0x24549)
[2016-04-15 23:56:26.324709] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa6bb73e4a3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fa6bb50855f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa6bb50867e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fa6bb509e3e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fa6bb50a658] ))))) 0-vm-storage-client-2: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-04-15 23:56:19.713780 (xid=0x2454a)
[2016-04-15 23:56:26.324839] I [socket.c:3308:socket_submit_request] 0-vm-storage-client-2: not connected (priv->connected = 0)
[2016-04-15 23:56:26.324850] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-vm-storage-client-2: failed to submit rpc-request (XID: 0x2454b Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vm-storage-client-2)
[2016-04-15 23:56:26.324860] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-2: remote operation failed. Path: /template (af20ae4d-28c3-4434-bc5c-50258b588c77) [Transport endpoint is not connected]
[2016-04-15 23:56:26.324863] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-vm-storage-client-2: disconnected from vm-storage-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2016-04-15 23:56:26.325422] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2016-04-15 23:56:28.926712] E [socket.c:2278:socket_connect_finish] 0-vm-storage-client-2: connection to 10.10.0.3:24007 failed (No route to host)
[2016-04-15 23:56:34.289067] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
The message "W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vm-storage-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]" repeated 2 times between [2016-04-15 23:56:26.324434] and [2016-04-15 23:56:26.324720]
[2016-04-15 23:58:13.197418] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-vm-storage-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2016-04-15 23:58:13.197449] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-vm-storage-client-2: disconnected from vm-storage-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2016-04-15 23:58:15.344543] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-vm-storage-client-2: changing port to 49152 (from 0)
[2016-04-15 23:58:15.348326] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vm-storage-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-04-15 23:58:15.350121] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vm-storage-client-2: Connected to vm-storage-client-2, attached to remote volume '/mnt/vg1-storage'.
[2016-04-15 23:58:15.350130] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vm-storage-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-04-15 23:58:15.351841] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vm-storage-client-2: Server lk version = 1
[2016-04-15 23:58:16.883063] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-vm-storage-replicate-0: selecting local read_child vm-storage-client-0
[2016-04-16 09:07:28.411137] I [MSGID: 109036] [dht-common.c:7869:dht_log_new_layout_for_dir_selfheal] 0-vm-storage-dht: Setting layout of /images/100 with [Subvol_name: vm-storage-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], 
[2016-04-16 09:33:46.317027] I [MSGID: 109066] [dht-rename.c:1411:dht_rename] 0-vm-storage-dht: renaming /images/101/vm-101-disk-1.qcow2 (hash=vm-storage-replicate-0/cache=vm-storage-replicate-0) => /images/100/vm-100-disk-1.qcow2 (hash=vm-storage-replicate-0/cache=vm-storage-replicate-0)
[2016-04-16 09:38:13.922988] I [MSGID: 109066] [dht-rename.c:1411:dht_rename] 0-vm-storage-dht: renaming /images/103/vm-103-disk-1.qcow2 (hash=vm-storage-replicate-0/cache=vm-storage-replicate-0) => /images/101/vm-101-disk-1.qcow2 (hash=vm-storage-replicate-0/cache=vm-storage-replicate-0)
[2016-04-16 09:46:23.718197] I [MSGID: 109066] [dht-rename.c:1411:dht_rename] 0-vm-storage-dht: renaming /images/100/vm-100-disk-1.qcow2 (hash=vm-storage-replicate-0/cache=vm-storage-replicate-0) => /images/103/vm-103-disk-1.qcow2 (hash=vm-storage-replicate-0/cache=vm-storage-replicate-0)
[2016-04-18 06:48:14.304668] I [MSGID: 109066] [dht-rename.c:1411:dht_rename] 0-vm-storage-dht: renaming /images/100/vm-100-disk-1.qcow2 (hash=vm-storage-replicate-0/cache=vm-storage-replicate-0) => /images/103/vm-103-disk-1.qcow2 (hash=vm-storage-replicate-0/cache=vm-storage-replicate-0)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160418/ce6050a4/attachment.sig>


More information about the Gluster-users mailing list