[Bugs] [Bug 1205128] New: Disperse volume: "df -h" on a cifs mount throws IO error and no file systems processed message
bugzilla at redhat.com
bugzilla at redhat.com
Tue Mar 24 10:05:55 UTC 2015
https://bugzilla.redhat.com/show_bug.cgi?id=1205128
Bug ID: 1205128
Summary: Disperse volume: "df -h" on a cifs mount throws IO
error and no file systems processed message
Product: GlusterFS
Version: mainline
Component: disperse
Assignee: bugs at gluster.org
Reporter: byarlaga at redhat.com
CC: bugs at gluster.org, gluster-bugs at redhat.com
Description of problem:
=======================
Configured samba on one of the server, mounted a 2x(8+3) disperse volume on
linux using cifs. Ran IO operations for some time, like files and directory
creates / deletes and did "df -h ." from one of the directory of the mount.
[root at dhcp37-61 truncation]# df -h .
df: `.': Input/output error
df: no file systems processed
[root at dhcp37-61 truncation]#
[root at dhcp37-61 truncation]# mount
/dev/mapper/vg_dhcp3761-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/vda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
ninja:/dispersevol on /mnt/fuse1 type fuse.glusterfs
(rw,default_permissions,allow_other,max_read=131072)
\\\\vertigo\\gluster-testvol on /mnt/cifs type cifs (rw)
Version-Release number of selected component (if applicable):
=============================================================
[root at vertigo ~]# gluster --version
glusterfs 3.7dev built on Mar 17 2015 01:06:35
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General
Public License.
How reproducible:
=================
100%
Steps to Reproduce:
1. Create a distributed disperse volume 2x(8+3)
2. configure samba server and mount on linux client using "mount -t cifs"
3. Run IO, create files / directories and do some deletes.
4. Run df -h . on a directory in the mount.
Actual results:
Input / Output error and no file systems processed errors.
Expected results:
Additional info:
Gluster v status and info :
===========================
[root at vertigo ~]# gluster v status testvol
Status of volume: testvol
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick vertigo:/rhs/brick1/testvol-1 49184 0 Y 28052
Brick ninja:/rhs/brick1/testvol-2 49183 0 Y 28545
Brick vertigo:/rhs/brick2/testvol-3 49185 0 Y 28067
Brick ninja:/rhs/brick2/testvol-4 49184 0 Y 28560
Brick vertigo:/rhs/brick3/testvol-5 49186 0 Y 28082
Brick ninja:/rhs/brick3/testvol-6 49185 0 Y 28575
Brick vertigo:/rhs/brick4/testvol-7 49187 0 Y 28097
Brick ninja:/rhs/brick4/testvol-8 49186 0 Y 3641
Brick vertigo:/rhs/brick1/testvol-9 49188 0 Y 28112
Brick ninja:/rhs/brick1/testvol-10 49187 0 Y 3647
Brick vertigo:/rhs/brick2/testvol-11 49189 0 Y 28127
Brick ninja:/rhs/brick2/testvol-12 49188 0 Y 3673
Brick vertigo:/rhs/brick3/testvol-13 49190 0 Y 3684
Brick ninja:/rhs/brick3/testvol-14 49189 0 Y 3678
Brick vertigo:/rhs/brick4/testvol-15 49191 0 Y 3665
Brick ninja:/rhs/brick4/testvol-16 49190 0 Y 3690
Brick vertigo:/rhs/brick1/testvol-17 49192 0 Y 3710
Brick ninja:/rhs/brick1/testvol-18 49191 0 Y 3689
Brick vertigo:/rhs/brick2/testvol-19 49193 0 Y 3698
Brick ninja:/rhs/brick2/testvol-20 49192 0 Y 3679
Brick vertigo:/rhs/brick3/testvol-21 49194 0 Y 3716
Brick ninja:/rhs/brick3/testvol-22 49193 0 Y 3691
Snapshot Daemon on localhost 49195 0 Y 4041
NFS Server on localhost 2049 0 Y 29601
Quota Daemon on localhost N/A N/A N N/A
Snapshot Daemon on ninja 49194 0 Y 4029
NFS Server on ninja 2049 0 Y 28591
Quota Daemon on ninja N/A N/A Y 28610
Task Status of Volume testvol
------------------------------------------------------------------------------
Task : Rebalance
ID : b41c6131-5c3d-4335-a6f8-ce54cfdb720b
Status : completed
[root at vertigo ~]# gluster v info testvol
Volume Name: testvol
Type: Distributed-Disperse
Volume ID: 8d541198-bd77-4aaa-8b19-a29880cabea5
Status: Started
Number of Bricks: 2 x (8 + 3) = 22
Transport-type: tcp
Bricks:
Brick1: vertigo:/rhs/brick1/testvol-1
Brick2: ninja:/rhs/brick1/testvol-2
Brick3: vertigo:/rhs/brick2/testvol-3
Brick4: ninja:/rhs/brick2/testvol-4
Brick5: vertigo:/rhs/brick3/testvol-5
Brick6: ninja:/rhs/brick3/testvol-6
Brick7: vertigo:/rhs/brick4/testvol-7
Brick8: ninja:/rhs/brick4/testvol-8
Brick9: vertigo:/rhs/brick1/testvol-9
Brick10: ninja:/rhs/brick1/testvol-10
Brick11: vertigo:/rhs/brick2/testvol-11
Brick12: ninja:/rhs/brick2/testvol-12
Brick13: vertigo:/rhs/brick3/testvol-13
Brick14: ninja:/rhs/brick3/testvol-14
Brick15: vertigo:/rhs/brick4/testvol-15
Brick16: ninja:/rhs/brick4/testvol-16
Brick17: vertigo:/rhs/brick1/testvol-17
Brick18: ninja:/rhs/brick1/testvol-18
Brick19: vertigo:/rhs/brick2/testvol-19
Brick20: ninja:/rhs/brick2/testvol-20
Brick21: vertigo:/rhs/brick3/testvol-21
Brick22: ninja:/rhs/brick3/testvol-22
Options Reconfigured:
cluster.disperse-self-heal-daemon: enable
client.event-threads: 4
server.event-threads: 4
features.quota: on
features.uss: on
Log snippet from server :
========================
[2015-03-24 09:44:54.968748] I [event-epoll.c:629:event_dispatch_epoll_worker]
0-epoll: Started thread with index 1
[2015-03-24 09:44:54.987699] I [graph.c:269:gf_add_cmdline_options]
0-testvol-md-cache: adding option 'cache-posix-acl' for volume
'testvol-md-cache' with value 'true'
[2015-03-24 09:44:54.988553] I [event-epoll.c:629:event_dispatch_epoll_worker]
0-epoll: Started thread with index 2
[2015-03-24 09:44:54.988641] I [event-epoll.c:629:event_dispatch_epoll_worker]
0-epoll: Started thread with index 3
[2015-03-24 09:44:54.988695] I [event-epoll.c:629:event_dispatch_epoll_worker]
0-epoll: Started thread with index 4
[2015-03-24 09:44:54.988713] I [client.c:2391:client_check_remote_host]
0-testvol-snapd-client: Remote host is not set. Assuming the volfile server as
remote host.
[2015-03-24 09:44:54.989665] I [dht-shared.c:337:dht_init_regex] 0-testvol-dht:
using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
[2015-03-24 09:44:55.006666] I [glfs-master.c:93:notify] 0-gfapi: New graph
76657274-6967-6f2e-6c61-622e656e672e (0) coming up
[2015-03-24 09:44:55.006709] I [client.c:2350:notify] 0-testvol-client-0:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.013811] I [client.c:2350:notify] 0-testvol-client-1:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.014208] I [rpc-clnt.c:1806:rpc_clnt_reconfig]
0-testvol-client-0: changing port to 49184 (from 0)
[2015-03-24 09:44:55.019587] I [client.c:2350:notify] 0-testvol-client-2:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.020014] I [rpc-clnt.c:1806:rpc_clnt_reconfig]
0-testvol-client-1: changing port to 49183 (from 0)
[2015-03-24 09:44:55.022412] I
[client-handshake.c:1412:select_server_supported_programs] 0-testvol-client-0:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-03-24 09:44:55.022684] I [client-handshake.c:1200:client_setvolume_cbk]
0-testvol-client-0: Connected to testvol-client-0, attached to remote volume
'/rhs/brick1/testvol-1'.
[2015-03-24 09:44:55.022713] I [client-handshake.c:1210:client_setvolume_cbk]
0-testvol-client-0: Server and Client lk-version numbers are not same,
reopening the fds
[2015-03-24 09:44:55.022845] I
[client-handshake.c:187:client_set_lk_version_cbk] 0-testvol-client-0: Server
lk version = 1
[2015-03-24 09:44:55.024963] I [client.c:2350:notify] 0-testvol-client-3:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.025196] I [rpc-clnt.c:1806:rpc_clnt_reconfig]
0-testvol-client-2: changing port to 49185 (from 0)
[2015-03-24 09:44:55.028642] I
[client-handshake.c:1412:select_server_supported_programs] 0-testvol-client-1:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-03-24 09:44:55.029376] I [client-handshake.c:1200:client_setvolume_cbk]
0-testvol-client-1: Connected to testvol-client-1, attached to remote volume
'/rhs/brick1/testvol-2'.
[2015-03-24 09:44:55.029417] I [client-handshake.c:1210:client_setvolume_cbk]
0-testvol-client-1: Server and Client lk-version numbers are not same,
reopening the fds
[2015-03-24 09:44:55.029592] I
[client-handshake.c:187:client_set_lk_version_cbk] 0-testvol-client-1: Server
lk version = 1
[2015-03-24 09:44:55.031706] I [client.c:2350:notify] 0-testvol-client-4:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.032175] I [rpc-clnt.c:1806:rpc_clnt_reconfig]
0-testvol-client-3: changing port to 49184 (from 0)
[2015-03-24 09:44:55.035424] I
[client-handshake.c:1412:select_server_supported_programs] 0-testvol-client-2:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-03-24 09:44:55.035735] I [client-handshake.c:1200:client_setvolume_cbk]
0-testvol-client-2: Connected to testvol-client-2, attached to remote volume
'/rhs/brick2/testvol-3'.
[2015-03-24 09:44:55.035764] I [client-handshake.c:1210:client_setvolume_cbk]
0-testvol-client-2: Server and Client lk-version numbers are not same,
reopening the fds
[2015-03-24 09:44:55.035872] I
[client-handshake.c:187:client_set_lk_version_cbk] 0-testvol-client-2: Server
lk version = 1
[2015-03-24 09:44:55.038987] I [client.c:2350:notify] 0-testvol-client-5:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.039275] I [rpc-clnt.c:1806:rpc_clnt_reconfig]
0-testvol-client-4: changing port to 49186 (from 0)
[2015-03-24 09:44:55.044907] I
[client-handshake.c:1412:select_server_supported_programs] 0-testvol-client-3:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-03-24 09:44:55.045297] I [client-handshake.c:1200:client_setvolume_cbk]
0-testvol-client-3: Connected to testvol-client-3, attached to remote volume
'/rhs/brick2/testvol-4'.
[2015-03-24 09:44:55.045326] I [client-handshake.c:1210:client_setvolume_cbk]
0-testvol-client-3: Server and Client lk-version numbers are not same,
reopening the fds
[2015-03-24 09:44:55.045485] I
[client-handshake.c:187:client_set_lk_version_cbk] 0-testvol-client-3: Server
lk version = 1
[2015-03-24 09:44:55.048855] I [client.c:2350:notify] 0-testvol-client-6:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.049145] W [socket.c:636:__socket_rwv] 0-testvol-client-5:
readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.049849] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-5: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.049014 (xid=0x1)
[2015-03-24 09:44:55.049888] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-5: received
RPC status error
[2015-03-24 09:44:55.049905] I [client.c:2282:client_rpc_notify]
0-testvol-client-5: disconnected from testvol-client-5. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.051424] W [socket.c:636:__socket_rwv] 0-testvol-client-4:
readv on 10.70.34.56:49186 failed (No data available)
[2015-03-24 09:44:55.051640] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-4: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.051347 (xid=0x3)
[2015-03-24 09:44:55.051666] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-4: received
RPC status error
[2015-03-24 09:44:55.051676] I [client.c:2282:client_rpc_notify]
0-testvol-client-4: disconnected from testvol-client-4. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.056473] I [client.c:2350:notify] 0-testvol-client-7:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.056658] W [socket.c:636:__socket_rwv] 0-testvol-client-6:
readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.056914] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-6: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.056576 (xid=0x1)
[2015-03-24 09:44:55.056953] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-6: received
RPC status error
[2015-03-24 09:44:55.056971] I [client.c:2282:client_rpc_notify]
0-testvol-client-6: disconnected from testvol-client-6. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.061408] I [client.c:2350:notify] 0-testvol-client-8:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.061738] W [socket.c:636:__socket_rwv] 0-testvol-client-7:
readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.061882] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-7: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.061523 (xid=0x1)
[2015-03-24 09:44:55.061901] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-7: received
RPC status error
[2015-03-24 09:44:55.061912] I [client.c:2282:client_rpc_notify]
0-testvol-client-7: disconnected from testvol-client-7. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.065247] I [client.c:2350:notify] 0-testvol-client-9:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.065372] W [socket.c:636:__socket_rwv] 0-testvol-client-8:
readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.065504] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-8: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.065278 (xid=0x1)
[2015-03-24 09:44:55.065529] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-8: received
RPC status error
[2015-03-24 09:44:55.065544] I [client.c:2282:client_rpc_notify]
0-testvol-client-8: disconnected from testvol-client-8. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.068921] I [client.c:2350:notify] 0-testvol-client-10:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.069235] W [socket.c:636:__socket_rwv] 0-testvol-client-9:
readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.069459] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-9: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.069091 (xid=0x1)
[2015-03-24 09:44:55.069491] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-9: received
RPC status error
[2015-03-24 09:44:55.069507] I [client.c:2282:client_rpc_notify]
0-testvol-client-9: disconnected from testvol-client-9. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.072761] I [client.c:2350:notify] 0-testvol-client-11:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.072899] W [socket.c:636:__socket_rwv] 0-testvol-client-10:
readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.073153] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-10: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.072831 (xid=0x1)
[2015-03-24 09:44:55.073186] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-10: received
RPC status error
[2015-03-24 09:44:55.073202] I [client.c:2282:client_rpc_notify]
0-testvol-client-10: disconnected from testvol-client-10. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.077104] I [client.c:2350:notify] 0-testvol-client-12:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.077423] W [socket.c:636:__socket_rwv] 0-testvol-client-11:
readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.077645] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-11: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.077253 (xid=0x1)
[2015-03-24 09:44:55.077676] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-11: received
RPC status error
[2015-03-24 09:44:55.077694] I [client.c:2282:client_rpc_notify]
0-testvol-client-11: disconnected from testvol-client-11. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.080986] I [client.c:2350:notify] 0-testvol-client-13:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.081115] W [socket.c:636:__socket_rwv] 0-testvol-client-12:
readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.081354] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-12: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.081041 (xid=0x1)
[2015-03-24 09:44:55.081386] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-12: received
RPC status error
[2015-03-24 09:44:55.081408] I [client.c:2282:client_rpc_notify]
0-testvol-client-12: disconnected from testvol-client-12. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.085114] I [client.c:2350:notify] 0-testvol-client-14:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.085351] W [socket.c:636:__socket_rwv] 0-testvol-client-13:
readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.085596] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-13: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.085220 (xid=0x1)
[2015-03-24 09:44:55.085627] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-13: received
RPC status error
[2015-03-24 09:44:55.085645] I [client.c:2282:client_rpc_notify]
0-testvol-client-13: disconnected from testvol-client-13. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.089322] I [client.c:2350:notify] 0-testvol-client-15:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.089477] W [socket.c:636:__socket_rwv] 0-testvol-client-14:
readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.089700] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-14: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.089415 (xid=0x1)
[2015-03-24 09:44:55.089765] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-14: received
RPC status error
[2015-03-24 09:44:55.089793] I [client.c:2282:client_rpc_notify]
0-testvol-client-14: disconnected from testvol-client-14. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.093698] I [client.c:2350:notify] 0-testvol-client-16:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.094060] W [socket.c:636:__socket_rwv] 0-testvol-client-15:
readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.094314] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-15: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.093817 (xid=0x1)
[2015-03-24 09:44:55.094346] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-15: received
RPC status error
[2015-03-24 09:44:55.094363] I [client.c:2282:client_rpc_notify]
0-testvol-client-15: disconnected from testvol-client-15. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.098442] I [client.c:2350:notify] 0-testvol-client-17:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.098636] W [socket.c:636:__socket_rwv] 0-testvol-client-16:
readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.098799] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-16: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.098517 (xid=0x1)
[2015-03-24 09:44:55.098821] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-16: received
RPC status error
[2015-03-24 09:44:55.098832] I [client.c:2282:client_rpc_notify]
0-testvol-client-16: disconnected from testvol-client-16. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.101610] E [snapview-client.c:309:svc_lookup_cbk]
0-testvol-snapview-client: Lookup failed on normal graph with error Transport
endpoint is not connected
[2015-03-24 09:44:55.101666] I [glfs-resolve.c:795:__glfs_active_subvol]
0-testvol: first lookup on graph 76657274-6967-6f2e-6c61-622e656e672e (0)
failed (Transport endpoint is not connected)
[2015-03-24 09:44:55.103740] I [client.c:2350:notify] 0-testvol-client-18:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.104011] W [socket.c:636:__socket_rwv] 0-testvol-client-17:
readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.104319] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-17: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.103840 (xid=0x1)
[2015-03-24 09:44:55.104364] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-17: received
RPC status error
[2015-03-24 09:44:55.104389] I [client.c:2282:client_rpc_notify]
0-testvol-client-17: disconnected from testvol-client-17. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.108992] I [client.c:2350:notify] 0-testvol-client-19:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.109205] W [socket.c:636:__socket_rwv] 0-testvol-client-18:
readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.109446] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-18: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.109072 (xid=0x1)
[2015-03-24 09:44:55.109477] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-18: received
RPC status error
[2015-03-24 09:44:55.109493] I [client.c:2282:client_rpc_notify]
0-testvol-client-18: disconnected from testvol-client-18. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.114609] I [client.c:2350:notify] 0-testvol-client-20:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.114929] W [socket.c:636:__socket_rwv] 0-testvol-client-19:
readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.115259] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-19: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.114780 (xid=0x1)
[2015-03-24 09:44:55.115312] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-19: received
RPC status error
[2015-03-24 09:44:55.115342] I [client.c:2282:client_rpc_notify]
0-testvol-client-19: disconnected from testvol-client-19. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.119475] I [client.c:2350:notify] 0-testvol-client-21:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.119644] W [socket.c:636:__socket_rwv] 0-testvol-client-20:
readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.119901] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-20: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.119547 (xid=0x1)
[2015-03-24 09:44:55.119963] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-20: received
RPC status error
[2015-03-24 09:44:55.119989] I [client.c:2282:client_rpc_notify]
0-testvol-client-20: disconnected from testvol-client-20. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.124589] I [client.c:2350:notify] 0-testvol-snapd-client:
parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.124880] W [socket.c:636:__socket_rwv] 0-testvol-client-21:
readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.125169] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-client-21: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at
2015-03-24 09:44:55.124728 (xid=0x1)
[2015-03-24 09:44:55.125204] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-21: received
RPC status error
[2015-03-24 09:44:55.125221] I [client.c:2282:client_rpc_notify]
0-testvol-client-21: disconnected from testvol-client-21. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.131850] W [socket.c:636:__socket_rwv]
0-testvol-snapd-client: readv on 127.0.0.1:24007 failed (No data available)
[2015-03-24 09:44:55.132118] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] )))))
0-testvol-snapd-client: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called
at 2015-03-24 09:44:55.131786 (xid=0x1)
[2015-03-24 09:44:55.132149] W
[client-handshake.c:1587:client_dump_version_cbk] 0-testvol-snapd-client:
received RPC status error
[2015-03-24 09:44:55.132166] I [client.c:2282:client_rpc_notify]
0-testvol-snapd-client: disconnected from testvol-snapd-client. Client process
will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.201978] E [snapview-client.c:309:svc_lookup_cbk]
0-testvol-snapview-client: Lookup failed on normal graph with error Transport
endpoint is not connected
[2015-03-24 09:44:55.202040] I [glfs-resolve.c:795:__glfs_active_subvol]
0-testvol: first lookup on graph 76657274-6967-6f2e-6c61-622e656e672e (0)
failed (Transport endpoint is not connected)
[2015-03-24 09:44:55.202279] I [mem-pool.c:557:mem_pool_destroy] 0-gfapi:
size=84 max=0 total=0
[2015-03-24 09:44:55.202863] I [mem-pool.c:557:mem_pool_destroy] 0-gfapi:
size=156 max=1 total=1
[2015-03-24 09:44:55.203221] I [mem-pool.c:557:mem_pool_destroy] 0-gfapi:
size=108 max=0 total=0
[2015-03-24 09:44:55.203280] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-0: size=1300 max=0 total=0
[2015-03-24 09:44:55.203299] I [client.c:2282:client_rpc_notify]
0-testvol-client-0: disconnected from testvol-client-0. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.203322] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-1: size=1300 max=0 total=0
[2015-03-24 09:44:55.203335] I [client.c:2282:client_rpc_notify]
0-testvol-client-1: disconnected from testvol-client-1. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.203383] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-2: size=1300 max=0 total=0
[2015-03-24 09:44:55.203386] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-1: size=588 max=3 total=6
[2015-03-24 09:44:55.203401] I [client.c:2282:client_rpc_notify]
0-testvol-client-2: disconnected from testvol-client-2. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.203422] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-3: size=1300 max=0 total=0
[2015-03-24 09:44:55.203457] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-2: size=588 max=3 total=6
[2015-03-24 09:44:55.203467] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-1: size=124 max=3 total=6
[2015-03-24 09:44:55.203442] I [client.c:2282:client_rpc_notify]
0-testvol-client-3: disconnected from testvol-client-3. Client process will
keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.203477] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-4: size=588 max=2 total=3
[2015-03-24 09:44:55.203516] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-2: size=124 max=3 total=6
[2015-03-24 09:44:55.203352] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-0: size=588 max=3 total=6
[2015-03-24 09:44:55.203543] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-3: size=588 max=3 total=6
[2015-03-24 09:44:55.203588] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-4: size=124 max=2 total=3
[2015-03-24 09:44:55.203637] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-4: size=1300 max=0 total=0
[2015-03-24 09:44:55.203654] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-0: size=124 max=3 total=6
[2015-03-24 09:44:55.203661] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-5: size=588 max=1 total=1
[2015-03-24 09:44:55.203746] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-3: size=124 max=3 total=6
[2015-03-24 09:44:55.203789] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-5: size=124 max=1 total=1
[2015-03-24 09:44:55.203813] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-5: size=1300 max=0 total=0
[2015-03-24 09:44:55.203838] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-6: size=588 max=1 total=1
[2015-03-24 09:44:55.203897] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-6: size=124 max=1 total=1
[2015-03-24 09:44:55.203917] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-6: size=1300 max=0 total=0
[2015-03-24 09:44:55.203937] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-7: size=588 max=1 total=1
[2015-03-24 09:44:55.204001] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-7: size=124 max=1 total=1
[2015-03-24 09:44:55.204025] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-7: size=1300 max=0 total=0
[2015-03-24 09:44:55.204045] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-8: size=588 max=1 total=1
[2015-03-24 09:44:55.204108] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-8: size=124 max=1 total=1
[2015-03-24 09:44:55.204134] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-8: size=1300 max=0 total=0
[2015-03-24 09:44:55.204153] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-9: size=588 max=1 total=1
[2015-03-24 09:44:55.204212] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-9: size=124 max=1 total=1
[2015-03-24 09:44:55.204233] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-9: size=1300 max=0 total=0
[2015-03-24 09:44:55.204251] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-10: size=588 max=1 total=1
[2015-03-24 09:44:55.204305] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-10: size=124 max=1 total=1
[2015-03-24 09:44:55.204325] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-10: size=1300 max=0 total=0
[2015-03-24 09:44:57.204410] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-disperse-0: size=1764 max=1 total=2
[2015-03-24 09:44:57.204705] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-disperse-0: size=1932 max=0 total=0
[2015-03-24 09:44:57.204964] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-disperse-0: size=1260 max=0 total=0
[2015-03-24 09:44:57.205162] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-11: size=588 max=1 total=1
[2015-03-24 09:44:57.205228] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-11: size=124 max=1 total=1
[2015-03-24 09:44:57.205257] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-11: size=1300 max=0 total=0
[2015-03-24 09:44:57.205277] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-12: size=588 max=1 total=1
[2015-03-24 09:44:57.205336] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-12: size=124 max=1 total=1
[2015-03-24 09:44:57.205357] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-12: size=1300 max=0 total=0
[2015-03-24 09:44:57.205375] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-13: size=588 max=1 total=1
[2015-03-24 09:44:57.205432] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-13: size=124 max=1 total=1
[2015-03-24 09:44:57.205452] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-13: size=1300 max=0 total=0
[2015-03-24 09:44:57.205469] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-14: size=588 max=1 total=1
[2015-03-24 09:44:57.205525] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-14: size=124 max=1 total=1
[2015-03-24 09:44:57.205545] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-14: size=1300 max=0 total=0
[2015-03-24 09:44:57.205564] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-15: size=588 max=1 total=1
[2015-03-24 09:44:57.205619] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-15: size=124 max=1 total=1
[2015-03-24 09:44:57.205639] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-15: size=1300 max=0 total=0
[2015-03-24 09:44:57.205658] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-16: size=588 max=1 total=1
[2015-03-24 09:44:57.205717] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-16: size=124 max=1 total=1
[2015-03-24 09:44:57.205736] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-16: size=1300 max=0 total=0
[2015-03-24 09:44:57.205754] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-17: size=588 max=1 total=1
[2015-03-24 09:44:57.205810] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-17: size=124 max=1 total=1
[2015-03-24 09:44:57.205835] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-17: size=1300 max=0 total=0
[2015-03-24 09:44:57.205862] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-18: size=588 max=1 total=1
[2015-03-24 09:44:57.205928] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-18: size=124 max=1 total=1
[2015-03-24 09:44:57.205949] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-18: size=1300 max=0 total=0
[2015-03-24 09:44:57.205967] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-19: size=588 max=1 total=1
[2015-03-24 09:44:57.206026] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-19: size=124 max=1 total=1
[2015-03-24 09:44:57.206062] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-19: size=1300 max=0 total=0
[2015-03-24 09:44:57.206081] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-20: size=588 max=1 total=1
[2015-03-24 09:44:57.206142] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-20: size=124 max=1 total=1
[2015-03-24 09:44:57.206163] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-20: size=1300 max=0 total=0
[2015-03-24 09:44:57.206186] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-21: size=588 max=1 total=1
[2015-03-24 09:44:57.206249] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-21: size=124 max=1 total=1
[2015-03-24 09:44:57.206270] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-client-21: size=1300 max=0 total=0
[2015-03-24 09:44:58.973320] I [timer.c:44:gf_timer_call_after] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_reconnect+0x107)[0x7ff40c08c5f7] (-->
/usr/lib64/libglusterfs.so.0(gf_timer_proc+0xc8)[0x7ff40bc01b98] (-->
/lib64/libpthread.so.0(+0x3d2fc079d1)[0x7ff4223909d1] ))))) 0-timer: ctx
cleanup started
[2015-03-24 09:44:58.973694] I [timer.c:44:gf_timer_call_after] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (-->
/usr/lib64/libgfrpc.so.0(__save_frame+0x6e)[0x7ff40c08c7ae] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x3e2)[0x7ff40c08d102] (-->
/usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_submit_request+0x196)[0x7ff402761e26]
))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.974020] I [timer.c:44:gf_timer_call_after] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (-->
/usr/lib64/libgfrpc.so.0(__save_frame+0x6e)[0x7ff40c08c7ae] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x3e2)[0x7ff40c08d102] (-->
/usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_submit_request+0x196)[0x7ff402761e26]
))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.974078] I [rpc-clnt.c:1806:rpc_clnt_reconfig]
0-testvol-snapd-client: changing port to 49195 (from 0)
[2015-03-24 09:44:58.974425] I [timer.c:44:gf_timer_call_after] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1e7)[0x7ff40c08e157] (-->
/usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7ff40c0897a8] (-->
/usr/lib64/glusterfs/3.7dev/rpc-transport/socket.so(+0xb221)[0x7ff4080de221]
))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.984287] I [timer.c:44:gf_timer_call_after] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_reconnect+0x107)[0x7ff40c08c5f7] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_start+0x17)[0x7ff40c08c617] (-->
/usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_rpc_notify+0x3d2)[0x7ff4027628a2]
))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.984588] I [timer.c:44:gf_timer_call_after] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (-->
/usr/lib64/libgfrpc.so.0(__save_frame+0x6e)[0x7ff40c08c7ae] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x3e2)[0x7ff40c08d102] (-->
/usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_submit_request+0x196)[0x7ff402761e26]
))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.984650] I
[client-handshake.c:1412:select_server_supported_programs]
0-testvol-snapd-client: Using Program GlusterFS 3.3, Num (1298437), Version
(330)
[2015-03-24 09:44:58.984957] I [timer.c:44:gf_timer_call_after] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (-->
/usr/lib64/libgfrpc.so.0(__save_frame+0x6e)[0x7ff40c08c7ae] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x3e2)[0x7ff40c08d102] (-->
/usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_submit_request+0x196)[0x7ff402761e26]
))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.985029] I [client-handshake.c:1200:client_setvolume_cbk]
0-testvol-snapd-client: Connected to testvol-snapd-client, attached to remote
volume 'snapd-testvol'.
[2015-03-24 09:44:58.985056] I [client-handshake.c:1210:client_setvolume_cbk]
0-testvol-snapd-client: Server and Client lk-version numbers are not same,
reopening the fds
[2015-03-24 09:44:58.985313] I [timer.c:44:gf_timer_call_after] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (-->
/usr/lib64/libgfrpc.so.0(__save_frame+0x6e)[0x7ff40c08c7ae] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x3e2)[0x7ff40c08d102] (-->
/usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_submit_request+0x196)[0x7ff402761e26]
))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.985527] I [timer.c:44:gf_timer_call_after] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (-->
/usr/lib64/libgfrpc.so.0(+0x3d318128a8)[0x7ff40c0908a8] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x390)[0x7ff40c08d0b0] (-->
/usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_submit_request+0x196)[0x7ff402761e26]
))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.985556] W [rpc-clnt-ping.c:296:rpc_clnt_start_ping]
0-testvol-snapd-client: unable to setup ping timer
[2015-03-24 09:44:58.985600] I
[client-handshake.c:187:client_set_lk_version_cbk] 0-testvol-snapd-client:
Server lk version = 1
[2015-03-24 09:44:59.206359] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-disperse-1: size=1764 max=1 total=2
[2015-03-24 09:44:59.206661] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-disperse-1: size=1932 max=0 total=0
[2015-03-24 09:44:59.207173] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-disperse-1: size=1260 max=0 total=0
[2015-03-24 09:44:59.207373] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-dht:
size=1148 max=0 total=0
[2015-03-24 09:44:59.207482] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-dht:
size=2148 max=1 total=2
[2015-03-24 09:44:59.207646] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-read-ahead: size=188 max=0 total=0
[2015-03-24 09:44:59.207668] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-io-cache: size=68 max=0 total=0
[2015-03-24 09:44:59.207683] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-io-cache: size=252 max=1 total=2
[2015-03-24 09:44:59.207732] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-snapd-client: size=1300 max=0 total=0
[2015-03-24 09:44:59.207772] I [client.c:2282:client_rpc_notify]
0-testvol-snapd-client: disconnected from testvol-snapd-client. Client process
will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:59.207825] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-snapd-client: size=588 max=2 total=6
[2015-03-24 09:44:59.207954] I [mem-pool.c:557:mem_pool_destroy]
0-testvol-snapd-client: size=124 max=2 total=6
[2015-03-24 09:44:59.208134] E [mem-pool.c:250:__gf_free] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libglusterfs.so.0(__gf_free+0xb4)[0x7ff40bc13be4] (-->
/usr/lib64/glusterfs/3.7dev/xlator/features/snapview-client.so(fini+0x2e)[0x7ff4014408ae]
(--> /usr/lib64/libglusterfs.so.0(+0x3d3141f341)[0x7ff40bbe2341] (-->
/usr/lib64/libglusterfs.so.0(+0x3d3141f2d5)[0x7ff40bbe22d5] ))))) 0-: Assertion
failed: GF_MEM_HEADER_MAGIC == *(uint32_t *)ptr
[2015-03-24 09:45:09.433579] I [event-epoll.c:629:event_dispatch_epoll_worker]
0-epoll: Started thread with index 1
[2015-03-24 09:45:09.441878] W [socket.c:636:__socket_rwv] 0-gfapi: readv on
127.0.0.1:24007 failed (No data available)
[2015-03-24 09:45:09.442464] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-gfapi:
forced unwinding frame type(GlusterFS Handshake) op(GETSPEC(2)) called at
2015-03-24 09:45:09.441697 (xid=0x1)
[2015-03-24 09:45:09.442510] E [glfs-mgmt.c:605:glfs_mgmt_getspec_cbk]
0-glfs-mgmt: failed to fetch volume file (key:testvol)
[2015-03-24 09:45:09.442531] E [glfs-mgmt.c:702:mgmt_rpc_notify] 0-glfs-mgmt:
failed to connect with remote-host: localhost (No data available)
[2015-03-24 09:45:09.442545] I [glfs-mgmt.c:707:mgmt_rpc_notify] 0-glfs-mgmt:
Exhausted all volfile servers
[2015-03-24 09:45:09.442833] I [event-epoll.c:659:event_dispatch_epoll_worker]
0-epoll: Exited thread with index 1
==============================================
sosreport of the server will be attached.
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list