[Gluster-users] glusterfs 3.2.1 nfs hangs on opensuse 11.4
Uwe Weiss
u.weiss at netz-objekte.de
Tue Jul 12 14:16:29 UTC 2011
Hi list,
I have some trouble with glusterfs 3.2.1. compiled from tarball in two
opensue 11.4 boxes.
Compiling an Installing of glusterfs went fine.
Afterwards I have configured a distributed replicated volume named
gluster_vol1 (peers are of course connected to each other)
As long as I only use the native gluster-client everything works fine an I
am reallay happy about glusterfs.
But if I try to use nfs I am running in trouble.
In details: I have two hosts called vmhost1 (192.168.50.1) and
vmhost2(192.168.50.2). On both hosts glusterfs is started.
On both hosts I have mounted a directory named /nfs_mount with:
mount -t nfs -o mountproto=tcp 192.168.50.1:/gluster_vol1 /nfs_mount
Afterward I can ls, df, du and so on with this directory on both hosts. Even
copying of small files works mostly but not every times.
But if I try to create bigfiles with dd for example [], the connections
hangs after one or two minutes. Sometimes it helps to restart glusterfs on
the mounted node, sometimes only a reset of the master node solves the
problem.
This happens if I run the copying or the creation of bigfile on any node
that has mounted the gluster nfs share independently if I run this command
from a single node or concurrently from multiple nodes.
Here is my logfile from my masternode after starting glusterfs on both
nodes:
cat etc-glusterfs-glusterd.vol.log
[2011-07-12 15:19:08.455046] I [glusterd.c:564:init] 0-management: Using
/etc/glusterd as working directory
[2011-07-12 15:19:08.467129] E [rpc-transport.c:676:rpc_transport_load]
0-rpc-transport: /usr/local/lib/glusterfs/3.2.1/rpc-transport/rdma.so:
cannot open shared object file: No such file or directory
[2011-07-12 15:19:08.467151] E [rpc-transport.c:680:rpc_transport_load]
0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not
valid or not found on this machine
[2011-07-12 15:19:08.467164] W [rpcsvc.c:1288:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the transport failed
[2011-07-12 15:19:08.475337] I [glusterd.c:88:glusterd_uuid_init]
0-glusterd: retrieved UUID: 358770a7-d1ee-4fbd-8552-b354a2295d79
[2011-07-12 15:19:09.852048] E
[glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-0
[2011-07-12 15:19:09.852093] E
[glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-1
[2011-07-12 15:19:10.72382] I [glusterd-handler.c:3399:glusterd_friend_add]
0-glusterd: connect returned 0
[2011-07-12 15:19:10.74324] I
[glusterd-utils.c:1092:glusterd_volume_start_glusterfs] 0-: About to start
glusterfs for brick 192.168.50.1:/sp1/glusterfs
[2011-07-12 15:19:10.98987] I
[glusterd-utils.c:2217:glusterd_nfs_pmap_deregister] 0-: De-registered
MOUNTV3 successfully
[2011-07-12 15:19:10.100043] I
[glusterd-utils.c:2222:glusterd_nfs_pmap_deregister] 0-: De-registered
MOUNTV1 successfully
[2011-07-12 15:19:10.101123] I
[glusterd-utils.c:2227:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3
successfully
Given volfile:
+---------------------------------------------------------------------------
---+
1: volume management
2: type mgmt/glusterd
3: option working-directory /etc/glusterd
4: option transport-type socket,rdma
5: option transport.socket.keepalive-time 10
6: option transport.socket.keepalive-interval 2
7: end-volume
8:
+---------------------------------------------------------------------------
---+
[2011-07-12 15:19:10.115334] E [socket.c:1685:socket_connect_finish]
0-management: connection to failed (Connection refused)
[2011-07-12 15:19:10.193987] I [glusterd-pmap.c:237:pmap_registry_bind]
0-pmap: adding brick /sp1/glusterfs on port 24010
[2011-07-12 15:19:10.224800] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (192.168.50.1:1023)
[2011-07-12 15:21:39.983189] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (192.168.50.2:1021)
[2011-07-12 15:21:40.318042] I
[glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program
glusterd clnt mgmt, Num (1238433), Version (1)
Here is my nfs.log before mounting anything:
cat nfs.log
[2011-07-12 15:19:10.213621] I [nfs.c:675:init] 0-nfs: NFS service started
[2011-07-12 15:19:10.213743] W [write-behind.c:3029:init]
0-gluster_vol1-write-behind: disabling write-behind for first 0 bytes
[2011-07-12 15:19:10.216610] I [client.c:1935:notify]
0-gluster_vol1-client-0: parent translators are ready, attempting connect on
transport
[2011-07-12 15:19:10.220675] I [client.c:1935:notify]
0-gluster_vol1-client-1: parent translators are ready, attempting connect on
transport
Given volfile:
+---------------------------------------------------------------------------
---+
1: volume gluster_vol1-client-0
2: type protocol/client
3: option remote-host 192.168.50.1
4: option remote-subvolume /sp1/glusterfs
5: option transport-type tcp
6: end-volume
7:
8: volume gluster_vol1-client-1
9: type protocol/client
10: option remote-host 192.168.50.2
11: option remote-subvolume /sp1/glusterfs
12: option transport-type tcp
13: end-volume
14:
15: volume gluster_vol1-replicate-0
16: type cluster/replicate
17: subvolumes gluster_vol1-client-0 gluster_vol1-client-1
18: end-volume
19:
20: volume gluster_vol1-write-behind
21: type performance/write-behind
22: subvolumes gluster_vol1-replicate-0
23: end-volume
24:
25: volume gluster_vol1-read-ahead
26: type performance/read-ahead
27: subvolumes gluster_vol1-write-behind
28: end-volume
29:
30: volume gluster_vol1-io-cache
31: type performance/io-cache
32: subvolumes gluster_vol1-read-ahead
33: end-volume
34:
35: volume gluster_vol1-quick-read
36: type performance/quick-read
37: subvolumes gluster_vol1-io-cache
38: end-volume
39:
40: volume gluster_vol1
41: type debug/io-stats
42: option latency-measurement off
43: option count-fop-hits off
44: subvolumes gluster_vol1-quick-read
45: end-volume
46:
47: volume nfs-server
48: type nfs/server
49: option nfs.dynamic-volumes on
50: option rpc-auth.addr.gluster_vol1.allow *
51: option nfs3.gluster_vol1.volume-id
41ec51e6-e67a-45d4-bc80-be06c2453333
52: subvolumes gluster_vol1
53: end-volume
+---------------------------------------------------------------------------
---+
[2011-07-12 15:19:10.224263] E [socket.c:1685:socket_connect_finish]
0-gluster_vol1-client-1: connection to failed (Connection refused)
[2011-07-12 15:19:10.224670] I [rpc-clnt.c:1531:rpc_clnt_reconfig]
0-gluster_vol1-client-0: changing port to 24010 (from 0)
[2011-07-12 15:19:14.229314] I
[client-handshake.c:1082:select_server_supported_programs]
0-gluster_vol1-client-0: Using Program GlusterFS-3.1.0, Num (1298437),
Version (310)
[2011-07-12 15:19:14.231655] I [client-handshake.c:913:client_setvolume_cbk]
0-gluster_vol1-client-0: Connected to 192.168.50.1:24010, attached to remote
volume '/sp1/glusterfs'.
[2011-07-12 15:19:14.231690] I [afr-common.c:2591:afr_notify]
0-gluster_vol1-replicate-0: Subvolume 'gluster_vol1-client-0' came back up;
going online.
[2011-07-12 15:19:14.232730] I [afr-common.c:902:afr_fresh_lookup_cbk]
0-gluster_vol1-replicate-0: added root inode
[2011-07-12 15:21:39.850087] E
[client-handshake.c:1163:client_query_portmap_cbk] 0-gluster_vol1-client-1:
failed to get the port number for remote subvolume
[2011-07-12 15:21:39.850177] I [client.c:1883:client_rpc_notify]
0-gluster_vol1-client-1: disconnected
[2011-07-12 15:21:43.473577] I [rpc-clnt.c:1531:rpc_clnt_reconfig]
0-gluster_vol1-client-1: changing port to 24010 (from 0)
[2011-07-12 15:21:47.478306] I
[client-handshake.c:1082:select_server_supported_programs]
0-gluster_vol1-client-1: Using Program GlusterFS-3.1.0, Num (1298437),
Version (310)
[2011-07-12 15:21:47.478804] I [client-handshake.c:913:client_setvolume_cbk]
0-gluster_vol1-client-1: Connected to 192.168.50.2:24010, attached to remote
volume '/sp1/glusterfs'.
This is the bricks log after starting glusterfs on both nodes
cat bricks/sp1-glusterfs.log
[2011-07-12 15:19:10.109639] W [socket.c:419:__socket_keepalive] 0-socket:
failed to set keep idle on socket 8
[2011-07-12 15:19:10.109714] W [socket.c:1846:socket_server_event_handler]
0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
[2011-07-12 15:19:10.175912] W [graph.c:274:gf_add_cmdline_options]
0-gluster_vol1-server: adding option 'listen-port' for volume
'gluster_vol1-server' with value '24010'
[2011-07-12 15:19:10.176601] W [rpc-transport.c:447:validate_volume_options]
0-tcp.gluster_vol1-server: option 'listen-port' is deprecated, preferred is
'transport.socket.listen-port', continuing with correction
Given volfile:
+---------------------------------------------------------------------------
---+
1: volume gluster_vol1-posix
2: type storage/posix
3: option directory /sp1/glusterfs
4: end-volume
5:
6: volume gluster_vol1-access-control
7: type features/access-control
8: subvolumes gluster_vol1-posix
9: end-volume
10:
11: volume gluster_vol1-locks
12: type features/locks
13: subvolumes gluster_vol1-access-control
14: end-volume
15:
16: volume gluster_vol1-io-threads
17: type performance/io-threads
18: subvolumes gluster_vol1-locks
19: end-volume
20:
21: volume gluster_vol1-marker
22: type features/marker
23: option volume-uuid 41ec51e6-e67a-45d4-bc80-be06c2453333
24: option timestamp-file /etc/glusterd/vols/gluster_vol1/marker.tstamp
25: option xtime off
26: option quota off
27: subvolumes gluster_vol1-io-threads
28: end-volume
29:
30: volume /sp1/glusterfs
31: type debug/io-stats
32: option latency-measurement off
33: option count-fop-hits off
34: subvolumes gluster_vol1-marker
35: end-volume
36:
37: volume gluster_vol1-server
38: type protocol/server
39: option transport-type tcp
40: option auth.addr./sp1/glusterfs.allow *
41: subvolumes /sp1/glusterfs
42: end-volume
+---------------------------------------------------------------------------
---+
[2011-07-12 15:19:14.229546] I [server-handshake.c:534:server_setvolume]
0-gluster_vol1-server: accepted client from 192.168.50.1:1022
[2011-07-12 15:21:43.990367] I [server-handshake.c:534:server_setvolume]
0-gluster_vol1-server: accepted client from 192.168.50.2:1019
After a successful mount of vmhost1 from vmhost2 I see the following from
the mount command:
192.168.50.1:/gluster_vol1 on /nfs_mount type nfs
(rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=3
8467,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.1,mountvers=3,mountpor
t=38465,mountproto=tcp,local_lock=none,addr=192.168.50.1)
Afterwards I tried to create a bigfile in /nfs_mount with:
dd < /dev/zero > /nfs_mount/bigfile2.img bs=1M count=20480
The command runs a few seconds and than it hangs.
I found the following entrys in nfs log on the master masterserver
(vmhost1)
[2011-07-12 15:54:40.840976] C
[client-handshake.c:121:rpc_client_ping_timer_expired]
0-gluster_vol1-client-1: server 192.168.50.2:24010 has not responded in the
last 42 seconds, disconnecting.
[2011-07-12 15:54:40.841670] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.841839] I [client3_1-fops.c:683:client3_1_writev_cbk]
0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
connected
[2011-07-12 15:54:40.846839] I [socket.c:2275:socket_submit_request]
0-gluster_vol1-client-1: not connected (priv->connected = 0)
[2011-07-12 15:54:40.847046] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263239x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.847091] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263240x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.847130] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.847159] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.847193] I [client3_1-fops.c:683:client3_1_writev_cbk]
0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
connected
[2011-07-12 15:54:40.847423] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263241x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.847477] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.847495] I [client3_1-fops.c:683:client3_1_writev_cbk]
0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
connected
[2011-07-12 15:54:40.847571] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263242x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.847626] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.847643] I [client3_1-fops.c:683:client3_1_writev_cbk]
0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
connected
[2011-07-12 15:54:40.847716] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263243x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.847771] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.847798] I [client3_1-fops.c:683:client3_1_writev_cbk]
0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
connected
[2011-07-12 15:54:40.847874] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263244x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.847984] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.848006] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.848071] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.848106] I [client3_1-fops.c:683:client3_1_writev_cbk]
0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
connected
[2011-07-12 15:54:40.848199] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263245x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.848265] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.848283] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.848320] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.848338] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.848435] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.848466] I [client3_1-fops.c:683:client3_1_writev_cbk]
0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
connected
[2011-07-12 15:54:40.848578] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263246x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.848630] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.848657] I [client3_1-fops.c:683:client3_1_writev_cbk]
0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not
connected
[2011-07-12 15:54:40.848738] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263247x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.848808] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.848820] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263248x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.848848] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.849145] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.849261] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.849304] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.849403] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.849429] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.849513] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.849545] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.849605] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.849613] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263249x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.849678] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.849649] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.849938] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.849974] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.850083] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.850114] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263250x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.850151] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.850116] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.850285] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.850324] W [client-handshake.c:264:client_ping_cbk]
0-gluster_vol1-client-1: timer must have expired
[2011-07-12 15:54:40.850388] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.850416] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.850557] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.850630] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.850694] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263251x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.850745] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.850767] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.850855] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.850984] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.851055] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.851152] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263252x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.851186] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.851163] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.851309] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.851407] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.851436] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.851544] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.851573] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.851648] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.851670] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.851742] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.851822] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.851899] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263253x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.851935] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.851944] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.852080] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.852224] W [rpc-clnt.c:1411:rpc_clnt_submit]
0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263254x
Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport
(gluster_vol1-$
[2011-07-12 15:54:40.852255] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.852227] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.852302] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.852379] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.852402] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.852477] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.852501] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.852573] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.852594] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.852665] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.852693] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.852769] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$
[2011-07-12 15:54:40.852792] I
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1:
remote operation failed: Transport endpoint is not connected
[2011-07-12 15:54:40.852839] I [client.c:1883:client_rpc_notify]
0-gluster_vol1-client-1: disconnected
[2011-07-12 15:54:40.852935] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.852972] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.853082] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.853112] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.853228] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.853262] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.853324] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.853347] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.853423] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.853460] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.853509] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.853528] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.853552] I
[client-handshake.c:1082:select_server_supported_programs]
0-gluster_vol1-client-1: Using Program GlusterFS-3.1.0, Num (1298437),
Version (310)
[2011-07-12 15:54:40.853637] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx.
EBADFD[2011-07-12 15:54:40.854121] I
[client-handshake.c:913:client_setvolume_cbk] 0-gluster_vol1-client-1:
Connected to 192.168.50.2:24010, attached to remote volume '/sp1/glusterfs'.
[2011-07-12 15:54:40.854150] I
[client-handshake.c:779:client_post_handshake] 0-gluster_vol1-client-1: 1
fds open - Delaying child_up until they are re-opened
[2011-07-12 15:54:40.854228] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.854255] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.854348] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.854373] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.854473] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.854499] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.854720] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.854750] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.854868] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.854897] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.855076] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.855107] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.855241] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.855277] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.855453] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.855488] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.855665] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.855690] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.855813] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.855842] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.856037] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.856067] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.856181] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.856208] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.856307] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.856335] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.856447] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.856474] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.856579] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.856607] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.857560] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.857590] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.858438] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.858470] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.858570] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.858599] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.858830] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.858861] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.859057] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.853659] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.853702] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.853720] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.853893] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.853912] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.859086] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.859198] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.859226] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.860050] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.860084] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.860196] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.860226] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.861396] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.861428] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.861516] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.861538] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.861732] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.861754] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.862049] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.862077] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.862165] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.862187] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.862334] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.862358] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.862579] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.862602] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:40.862874] W [client3_1-fops.c:4777:client3_1_finodelk]
0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD
[2011-07-12 15:54:40.862909] W [client3_1-fops.c:4824:client3_1_finodelk]
0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad
state
[2011-07-12 15:54:41.21179] I [client-lk.c:617:decrement_reopen_fd_count]
0-gluster_vol1-client-1: last fd open'd/lock-self-heal'd - notifying
CHILD-UP
[2011-07-12 15:54:41.21567] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.22347] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.22831] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.23295] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.23775] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.24338] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.24781] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.25357] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.26039] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.26628] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.27038] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.27602] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.28089] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.28577] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.29033] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
[2011-07-12 15:54:41.29601] W [afr-open.c:624:afr_openfd_flush]
0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024
(/bigfile2.img)
Has anyone an ideas?
Help is appreciated
Thanks
ewuewu
More information about the Gluster-users
mailing list