[Gluster-users] glusterfs 3.2.1 nfs hangs on opensuse 11.4
Christopher Anderlik
christopher.anderlik at xidras.com
Tue Jul 12 14:50:41 UTC 2011
hi.
I don't know if this is a similar issue - but try to compile glusterfs with debugging enabled:
#export CFLAGS='-g3 -O0'
http://gluster.org/pipermail/gluster-users/2011-January/006530.html
cheers,
christopher
Am 12.07.2011 16:16, schrieb Uwe Weiss:
> Hi list,
>
> I have some trouble with glusterfs 3.2.1. compiled from tarball in two
> opensue 11.4 boxes.
> Compiling an Installing of glusterfs went fine.
> Afterwards I have configured a distributed replicated volume named
> gluster_vol1 (peers are of course connected to each other)
> As long as I only use the native gluster-client everything works fine an I
> am reallay happy about glusterfs.
> But if I try to use nfs I am running in trouble.
> In details: I have two hosts called vmhost1 (192.168.50.1) and
> vmhost2(192.168.50.2). On both hosts glusterfs is started.
> On both hosts I have mounted a directory named /nfs_mount with:
> mount -t nfs -o mountproto=tcp 192.168.50.1:/gluster_vol1 /nfs_mount
> Afterward I can ls, df, du and so on with this directory on both hosts. Even
> copying of small files works mostly but not every times.
>
> But if I try to create bigfiles with dd for example [], the connections
> hangs after one or two minutes. Sometimes it helps to restart glusterfs on
> the mounted node, sometimes only a reset of the master node solves the
> problem.
> This happens if I run the copying or the creation of bigfile on any node
> that has mounted the gluster nfs share independently if I run this command
> from a single node or concurrently from multiple nodes.
>
> Here is my logfile from my masternode after starting glusterfs on both
> nodes:
>
> cat etc-glusterfs-glusterd.vol.log
> [2011-07-12 15:19:08.455046] I [glusterd.c:564:init] 0-management: Using
> /etc/glusterd as working directory
> [2011-07-12 15:19:08.467129] E [rpc-transport.c:676:rpc_transport_load]
> 0-rpc-transport: /usr/local/lib/glusterfs/3.2.1/rpc-transport/rdma.so:
> cannot open shared object file: No such file or directory
> [2011-07-12 15:19:08.467151] E [rpc-transport.c:680:rpc_transport_load]
> 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not
> valid or not found on this machine
> [2011-07-12 15:19:08.467164] W [rpcsvc.c:1288:rpcsvc_transport_create]
> 0-rpc-service: cannot create listener, initing the transport failed
> [2011-07-12 15:19:08.475337] I [glusterd.c:88:glusterd_uuid_init]
> 0-glusterd: retrieved UUID: 358770a7-d1ee-4fbd-8552-b354a2295d79
> [2011-07-12 15:19:09.852048] E
> [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key:
> brick-0
> [2011-07-12 15:19:09.852093] E
> [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key:
> brick-1
> [2011-07-12 15:19:10.72382] I [glusterd-handler.c:3399:glusterd_friend_add]
> 0-glusterd: connect returned 0
> [2011-07-12 15:19:10.74324] I
> [glusterd-utils.c:1092:glusterd_volume_start_glusterfs] 0-: About to start
> glusterfs for brick 192.168.50.1:/sp1/glusterfs
> [2011-07-12 15:19:10.98987] I
> [glusterd-utils.c:2217:glusterd_nfs_pmap_deregister] 0-: De-registered
> MOUNTV3 successfully
> [2011-07-12 15:19:10.100043] I
> [glusterd-utils.c:2222:glusterd_nfs_pmap_deregister] 0-: De-registered
> MOUNTV1 successfully
> [2011-07-12 15:19:10.101123] I
> [glusterd-utils.c:2227:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3
> successfully
> Given volfile:
> +---------------------------------------------------------------------------
> ---+
> 1: volume management
> 2: type mgmt/glusterd
> 3: option working-directory /etc/glusterd
> 4: option transport-type socket,rdma
> 5: option transport.socket.keepalive-time 10
> 6: option transport.socket.keepalive-interval 2
> 7: end-volume
> 8:
>
> +---------------------------------------------------------------------------
> ---+
> [2011-07-12 15:19:10.115334] E [socket.c:1685:socket_connect_finish]
> 0-management: connection to failed (Connection refused)
> [2011-07-12 15:19:10.193987] I [glusterd-pmap.c:237:pmap_registry_bind]
> 0-pmap: adding brick /sp1/glusterfs on port 24010
> [2011-07-12 15:19:10.224800] W [socket.c:1494:__socket_proto_state_machine]
> 0-socket.management: reading from socket failed. Error (Transport endpoint
> is not connected), peer (192.168.50.1:1023)
> [2011-07-12 15:21:39.983189] W [socket.c:1494:__socket_proto_state_machine]
> 0-socket.management: reading from socket failed. Error (Transport endpoint
> is not connected), peer (192.168.50.2:1021)
> [2011-07-12 15:21:40.318042] I
> [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program
> glusterd clnt mgmt, Num (1238433), Version (1)
>
> Here is my nfs.log before mounting anything:
> cat nfs.log
> [2011-07-12 15:19:10.213621] I [nfs.c:675:init] 0-nfs: NFS service started
> [2011-07-12 15:19:10.213743] W [write-behind.c:3029:init]
> 0-gluster_vol1-write-behind: disabling write-behind for first 0 bytes
> [2011-07-12 15:19:10.216610] I [client.c:1935:notify]
> 0-gluster_vol1-client-0: parent translators are ready, attempting connect on
> transport
> [2011-07-12 15:19:10.220675] I [client.c:1935:notify]
> 0-gluster_vol1-client-1: parent translators are ready, attempting connect on
> transport
> Given volfile:
> +---------------------------------------------------------------------------
> ---+
> 1: volume gluster_vol1-client-0
> 2: type protocol/client
> 3: option remote-host 192.168.50.1
> 4: option remote-subvolume /sp1/glusterfs
> 5: option transport-type tcp
> 6: end-volume
> 7:
> 8: volume gluster_vol1-client-1
> 9: type protocol/client
> 10: option remote-host 192.168.50.2
> 11: option remote-subvolume /sp1/glusterfs
> 12: option transport-type tcp
> 13: end-volume
> 14:
> 15: volume gluster_vol1-replicate-0
> 16: type cluster/replicate
> 17: subvolumes gluster_vol1-client-0 gluster_vol1-client-1
> 18: end-volume
> 19:
> 20: volume gluster_vol1-write-behind
> 21: type performance/write-behind
> 22: subvolumes gluster_vol1-replicate-0
> 23: end-volume
> 24:
> 25: volume gluster_vol1-read-ahead
> 26: type performance/read-ahead
> 27: subvolumes gluster_vol1-write-behind
> 28: end-volume
> 29:
> 30: volume gluster_vol1-io-cache
> 31: type performance/io-cache
> 32: subvolumes gluster_vol1-read-ahead
> 33: end-volume
> 34:
> 35: volume gluster_vol1-quick-read
> 36: type performance/quick-read
> 37: subvolumes gluster_vol1-io-cache
> 38: end-volume
> 39:
> 40: volume gluster_vol1
> 41: type debug/io-stats
> 42: option latency-measurement off
> 43: option count-fop-hits off
> 44: subvolumes gluster_vol1-quick-read
> 45: end-volume
> 46:
> 47: volume nfs-server
> 48: type nfs/server
> 49: option nfs.dynamic-volumes on
> 50: option rpc-auth.addr.gluster_vol1.allow *
> 51: option nfs3.gluster_vol1.volume-id
> 41ec51e6-e67a-45d4-bc80-be06c2453333
> 52: subvolumes gluster_vol1
> 53: end-volume
>
> +---------------------------------------------------------------------------
> ---+
> [2011-07-12 15:19:10.224263] E [socket.c:1685:socket_connect_finish]
> 0-gluster_vol1-client-1: connection to failed (Connection refused)
> [2011-07-12 15:19:10.224670] I [rpc-clnt.c:1531:rpc_clnt_reconfig]
> 0-gluster_vol1-client-0: changing port to 24010 (from 0)
> [2011-07-12 15:19:14.229314] I
> [client-handshake.c:1082:select_server_supported_programs]
> 0-gluster_vol1-client-0: Using Program GlusterFS-3.1.0, Num (1298437),
> Version (310)
> [2011-07-12 15:19:14.231655] I [client-handshake.c:913:client_setvolume_cbk]
> 0-gluster_vol1-client-0: Connected to 192.168.50.1:24010, attached to remote
> volume '/sp1/glusterfs'.
> [2011-07-12 15:19:14.231690] I [afr-common.c:2591:afr_notify]
> 0-gluster_vol1-replicate-0: Subvolume 'gluster_vol1-client-0' came back up;
> going online.
> [2011-07-12 15:19:14.232730] I [afr-common.c:902:afr_fresh_lookup_cbk]
> 0-gluster_vol1-replicate-0: added root inode
> [2011-07-12 15:21:39.850087] E
> [client-handshake.c:1163:client_query_portmap_cbk] 0-gluster_vol1-client-1:
> failed to get the port number for remote subvolume
> [2011-07-12 15:21:39.850177] I [client.c:1883:client_rpc_notify]
> 0-gluster_vol1-client-1: disconnected
> [2011-07-12 15:21:43.473577] I [rpc-clnt.c:1531:rpc_clnt_reconfig]
> 0-gluster_vol1-client-1: changing port to 24010 (from 0)
> [2011-07-12 15:21:47.478306] I
> [client-handshake.c:1082:select_server_supported_programs]
> 0-gluster_vol1-client-1: Using Program GlusterFS-3.1.0, Num (1298437),
> Version (310)
> [2011-07-12 15:21:47.478804] I [client-handshake.c:913:client_setvolume_cbk]
> 0-gluster_vol1-client-1: Connected to 192.168.50.2:24010, attached to remote
> volume '/sp1/glusterfs'.
>
> This is the bricks log after starting glusterfs on both nodes
> cat bricks/sp1-glusterfs.log
> [2011-07-12 15:19:10.109639] W [socket.c:419:__socket_keepalive] 0-socket:
> failed to set keep idle on socket 8
> [2011-07-12 15:19:10.109714] W [socket.c:1846:socket_server_event_handler]
> 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
> [2011-07-12 15:19:10.175912] W [graph.c:274:gf_add_cmdline_options]
> 0-gluster_vol1-server: adding option 'listen-port' for volume
> 'gluster_vol1-server' with value '24010'
> [2011-07-12 15:19:10.176601] W [rpc-transport.c:447:validate_volume_options]
> 0-tcp.gluster_vol1-server: option 'listen-port' is deprecated, preferred is
> 'transport.socket.listen-port', continuing with correction
> Given volfile:
> +---------------------------------------------------------------------------
> ---+
> 1: volume gluster_vol1-posix
> 2: type storage/posix
> 3: option directory /sp1/glusterfs
> 4: end-volume
> 5:
> 6: volume gluster_vol1-access-control
> 7: type features/access-control
> 8: subvolumes gluster_vol1-posix
> 9: end-volume
> 10:
> 11: volume gluster_vol1-locks
> 12: type features/locks
> 13: subvolumes gluster_vol1-access-control
> 14: end-volume
> 15:
> 16: volume gluster_vol1-io-threads
> 17: type performance/io-threads
> 18: subvolumes gluster_vol1-locks
> 19: end-volume
> 20:
> 21: volume gluster_vol1-marker
> 22: type features/marker
> 23: option volume-uuid 41ec51e6-e67a-45d4-bc80-be06c2453333
> 24: option timestamp-file /etc/glusterd/vols/gluster_vol1/marker.tstamp
> 25: option xtime off
> 26: option quota off
> 27: subvolumes gluster_vol1-io-threads
> 28: end-volume
> 29:
> 30: volume /sp1/glusterfs
> 31: type debug/io-stats
> 32: option latency-measurement off
> 33: option count-fop-hits off
> 34: subvolumes gluster_vol1-marker
> 35: end-volume
> 36:
> 37: volume gluster_vol1-server
> 38: type protocol/server
> 39: option transport-type tcp
> 40: option auth.addr./sp1/glusterfs.allow *
> 41: subvolumes /sp1/glusterfs
> 42: end-volume
>
> +---------------------------------------------------------------------------
> ---+
> [2011-07-12 15:19:14.229546] I [server-handshake.c:534:server_setvolume]
> 0-gluster_vol1-server: accepted client from 192.168.50.1:1022
> [2011-07-12 15:21:43.990367] I [server-handshake.c:534:server_setvolume]
> 0-gluster_vol1-server: accepted client from 192.168.50.2:1019
>
>
> After a successful mount of vmhost1 from vmhost2 I see the following from
> the mount command:
>
>
> 192.168.50.1:/gluster_vol1 on /nfs_mount type nfs
> (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=3
> 8467,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.1,mountvers=3,mountpor
> t=38465,mountproto=tcp,local_lock=none,addr=192.168.50.1)
>
> Afterwards I tried to create a bigfile in /nfs_mount with:
>
> dd< /dev/zero> /nfs_mount/bigfile2.img bs=1M count=20480
>
> The command runs a few seconds and than it hangs.
> I found the following entrys in nfs log on the master masterserver
> (vmhost1)
>
> [2011-07-12 15:54:40.840976] C
> [client-handshake.c:121:rpc_client_ping_timer_expired]
> 0-gluster_vol1-client-1: server 192.168.50.2:24010 has not responded in the
> last 42 seconds, disconnecting.
> [2011-07-12 15:54:40.841670] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.841839] I [client3_1-fops.c:683:client3_1_writev_cbk]
> 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
> connected
> [2011-07-12 15:54:40.846839] I [socket.c:2275:socket_submit_request]
> 0-gluster_vol1-client-1: not connected (priv->connected = 0)
> [2011-07-12 15:54:40.847046] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263239x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.847091] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263240x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.847130] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.847159] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.847193] I [client3_1-fops.c:683:client3_1_writev_cbk]
> 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
> connected
> [2011-07-12 15:54:40.847423] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263241x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.847477] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.847495] I [client3_1-fops.c:683:client3_1_writev_cbk]
> 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
> connected
> [2011-07-12 15:54:40.847571] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263242x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.847626] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.847643] I [client3_1-fops.c:683:client3_1_writev_cbk]
> 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
> connected
> [2011-07-12 15:54:40.847716] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263243x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.847771] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.847798] I [client3_1-fops.c:683:client3_1_writev_cbk]
> 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
> connected
> [2011-07-12 15:54:40.847874] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263244x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.847984] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.848006] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.848071] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.848106] I [client3_1-fops.c:683:client3_1_writev_cbk]
> 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
> connected
> [2011-07-12 15:54:40.848199] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263245x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.848265] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.848283] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.848320] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.848338] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.848435] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.848466] I [client3_1-fops.c:683:client3_1_writev_cbk]
> 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
> connected
> [2011-07-12 15:54:40.848578] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263246x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.848630] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.848657] I [client3_1-fops.c:683:client3_1_writev_cbk]
> 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
> connected
> [2011-07-12 15:54:40.848738] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263247x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.848808] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.848820] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263248x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.848848] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.849145] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.849261] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.849304] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.849403] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.849429] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.849513] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.849545] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.849605] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.849613] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263249x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.849678] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.849649] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.849938] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.849974] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.850083] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.850114] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263250x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.850151] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.850116] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.850285] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.850324] W [client-handshake.c:264:client_ping_cbk]
> 0-gluster_vol1-client-1: timer must have expired
> [2011-07-12 15:54:40.850388] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.850416] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.850557] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.850630] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.850694] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263251x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.850745] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.850767] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.850855] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.850984] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.851055] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.851152] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263252x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.851186] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.851163] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.851309] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.851407] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.851436] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.851544] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.851573] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.851648] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.851670] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.851742] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.851822] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.851899] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263253x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.851935] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.851944] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.852080] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.852224] W [rpc-clnt.c:1411:rpc_clnt_submit]
> 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263254x
> Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
> (gluster_vol1-$
> [2011-07-12 15:54:40.852255] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.852227] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.852302] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.852379] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.852402] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.852477] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.852501] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.852573] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.852594] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.852665] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.852693] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.852769] E [rpc-clnt.c:338:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
> [2011-07-12 15:54:40.852792] I
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
> remote operation failed: Transport endpoint is not connected
> [2011-07-12 15:54:40.852839] I [client.c:1883:client_rpc_notify]
> 0-gluster_vol1-client-1: disconnected
> [2011-07-12 15:54:40.852935] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.852972] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.853082] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.853112] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.853228] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.853262] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.853324] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.853347] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.853423] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.853460] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.853509] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.853528] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.853552] I
> [client-handshake.c:1082:select_server_supported_programs]
> 0-gluster_vol1-client-1: Using Program GlusterFS-3.1.0, Num (1298437),
> Version (310)
> [2011-07-12 15:54:40.853637] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx.
> EBADFD[2011-07-12 15:54:40.854121] I
> [client-handshake.c:913:client_setvolume_cbk] 0-gluster_vol1-client-1:
> Connected to 192.168.50.2:24010, attached to remote volume '/sp1/glusterfs'.
> [2011-07-12 15:54:40.854150] I
> [client-handshake.c:779:client_post_handshake] 0-gluster_vol1-client-1: 1
> fds open - Delaying child_up until they are re-opened
> [2011-07-12 15:54:40.854228] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.854255] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.854348] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.854373] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.854473] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.854499] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.854720] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.854750] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.854868] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.854897] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.855076] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.855107] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.855241] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.855277] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.855453] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.855488] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.855665] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.855690] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.855813] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.855842] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.856037] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.856067] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.856181] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.856208] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.856307] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.856335] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.856447] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.856474] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.856579] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.856607] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.857560] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.857590] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.858438] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.858470] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.858570] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.858599] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.858830] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.858861] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.859057] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.853659] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.853702] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.853720] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.853893] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.853912] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.859086] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.859198] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.859226] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.860050] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.860084] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.860196] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.860226] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.861396] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.861428] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.861516] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.861538] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.861732] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.861754] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.862049] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.862077] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.862165] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.862187] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.862334] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.862358] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.862579] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.862602] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:40.862874] W [client3_1-fops.c:4777:client3_1_finodelk]
> 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
> [2011-07-12 15:54:40.862909] W [client3_1-fops.c:4824:client3_1_finodelk]
> 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
> state
> [2011-07-12 15:54:41.21179] I [client-lk.c:617:decrement_reopen_fd_count]
> 0-gluster_vol1-client-1: last fd open'd/lock-self-heal'd - notifying
> CHILD-UP
> [2011-07-12 15:54:41.21567] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.22347] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.22831] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.23295] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.23775] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.24338] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.24781] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.25357] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.26039] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.26628] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.27038] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.27602] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.28089] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.28577] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.29033] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
> [2011-07-12 15:54:41.29601] W [afr-open.c:624:afr_openfd_flush]
> 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
> (/bigfile2.img)
>
> Has anyone an ideas?
> Help is appreciated
>
> Thanks
> ewuewu
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
More information about the Gluster-users
mailing list