[Gluster-users] Problems running benchamrks

Fri Aug 8 14:39:27 UTC 2008

I'm having some problems with my glusterfs file system when I run
benchmarks on it. I'm running iozone in throughput mode and I keep
losing the glusterfs mount point on random machines. I get the error
"Transport endpoint is not connected" when I try and run df or ls on the
node it failed on. The one thing I did notice this morning one of the
systems that fail last night started working again when I got in this
morning. The failures I'm seeing are at the end of the log files.
System:
4 Storage nodes strip
46 compute nodes
Centos 5.2
2.6.18-53.el5
Glusterfs 1.3.10
Fuse fuse-2.7.3glfs10
Qlogic QLE7240 IB cards

Heres some of the messages I'm seeing in the log files from the nodes 
that are mounting the file system.

2008-08-07 19:46:22 W [client-protocol.c:205:call_bail] client-stripe-1: 
activating bail-out. pending frames = 1. last sent = 2008-08-07 
19:45:32. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:46:22 C [client-protocol.c:212:call_bail] client-stripe-1: 
bailing transport
2008-08-07 19:46:22 W [client-protocol.c:205:call_bail] client-stripe-4: 
activating bail-out. pending frames = 1. last sent = 2008-08-07 
19:45:32. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:46:22 C [client-protocol.c:212:call_bail] client-stripe-4: 
bailing transport
2008-08-07 19:46:22 W [client-protocol.c:205:call_bail] client-stripe-2: 
activating bail-out. pending frames = 1. last sent = 2008-08-07 
19:45:32. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:46:22 C [client-protocol.c:212:call_bail] client-stripe-2: 
bailing transport
2008-08-07 19:46:22 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-2: cleaning up state in transport object 0x9625260
2008-08-07 19:46:22 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-2: forced unwinding frame type(1) op(34) reply=@0x962f1f0
2008-08-07 19:46:22 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-2: no proper reply from server, returning ENOTCONN
2008-08-07 19:46:22 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-2 returned errno 107
2008-08-07 19:46:22 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-4: cleaning up state in transport object 0x9629f20
2008-08-07 19:46:22 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-4: forced unwinding frame type(1) op(34) reply=@0x962e190
2008-08-07 19:46:22 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-4: no proper reply from server, returning ENOTCONN
2008-08-07 19:46:22 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-4 returned errno 107
2008-08-07 19:46:22 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-1: cleaning up state in transport object 0x961f350
2008-08-07 19:46:22 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-1: forced unwinding frame type(1) op(34) reply=@0x9627630
2008-08-07 19:46:22 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-1: no proper reply from server, returning ENOTCONN
2008-08-07 19:46:22 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-1 returned errno 107
2008-08-07 19:47:25 W [client-protocol.c:205:call_bail] client-stripe-3: 
activating bail-out. pending frames = 1. last sent = 2008-08-07 
19:45:32. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:47:25 C [client-protocol.c:212:call_bail] client-stripe-3: 
bailing transport
2008-08-07 19:47:25 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-3: cleaning up state in transport object 0x96278c0
2008-08-07 19:47:25 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-3: forced unwinding frame type(1) op(34) reply=@0x962c2f0
2008-08-07 19:47:25 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-3: no proper reply from server, returning ENOTCONN
2008-08-07 19:47:25 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-3 returned errno 107
2008-08-07 19:47:25 E [fuse-bridge.c:468:fuse_entry_cbk] glusterfs-fuse: 
63760: (34) / => -1 (107)
2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] 
trans: attempting to pipeline request type(2) op(4) with handshake
2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-1: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-2: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-3: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-4: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:53:23 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-1: attempting to pipeline request type(1) op(15) with 
handshake
2008-08-07 19:53:23 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-2: attempting to pipeline request type(1) op(15) with 
handshake
2008-08-07 19:53:23 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-3: attempting to pipeline request type(1) op(15) with 
handshake
2008-08-07 19:53:23 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-4: attempting to pipeline request type(1) op(15) with 
handshake
2008-08-07 19:54:06 W [client-protocol.c:205:call_bail] client-stripe-1: 
activating bail-out. pending frames = 2. last sent = 2008-08-07 
19:53:23. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:54:06 C [client-protocol.c:212:call_bail] client-stripe-1: 
bailing transport
2008-08-07 19:54:06 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-1: cleaning up state in transport object 0x961f350
2008-08-07 19:54:06 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-1: forced unwinding frame type(1) op(34) reply=@0x9629c90
2008-08-07 19:54:06 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-1: no proper reply from server, returning ENOTCONN
2008-08-07 19:54:06 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-1 returned errno 107
2008-08-07 19:54:06 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-1: forced unwinding frame type(1) op(15) reply=@0x9629c90
2008-08-07 19:54:06 E [client-protocol.c:3884:client_statfs_cbk] 
client-stripe-1: no proper reply from server, returning ENOTCONN
2008-08-07 19:54:09 W [client-protocol.c:205:call_bail] client-stripe-4: 
activating bail-out. pending frames = 2. last sent = 2008-08-07 
19:53:23. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:54:09 C [client-protocol.c:212:call_bail] client-stripe-4: 
bailing transport
2008-08-07 19:54:09 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-4: cleaning up state in transport object 0x9629f20
2008-08-07 19:54:09 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-4: forced unwinding frame type(1) op(34) reply=@0x962e190
2008-08-07 19:54:09 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-4: no proper reply from server, returning ENOTCONN
2008-08-07 19:54:09 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-4 returned errno 107
2008-08-07 19:54:09 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-4: forced unwinding frame type(1) op(15) reply=@0x962e190
2008-08-07 19:54:09 E [client-protocol.c:3884:client_statfs_cbk] 
client-stripe-4: no proper reply from server, returning ENOTCONN
2008-08-07 19:54:10 W [client-protocol.c:205:call_bail] client-stripe-3: 
activating bail-out. pending frames = 2. last sent = 2008-08-07 
19:53:23. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:54:10 C [client-protocol.c:212:call_bail] client-stripe-3: 
bailing transport
2008-08-07 19:54:10 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-3: cleaning up state in transport object 0x96278c0
2008-08-07 19:54:10 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-3: forced unwinding frame type(1) op(34) reply=@0x9627630
2008-08-07 19:54:10 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-3: no proper reply from server, returning ENOTCONN
2008-08-07 19:54:10 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-3 returned errno 107
2008-08-07 19:54:10 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-3: forced unwinding frame type(1) op(15) reply=@0x9627630
2008-08-07 19:54:10 E [client-protocol.c:3884:client_statfs_cbk] 
client-stripe-3: no proper reply from server, returning ENOTCONN
2008-08-07 19:54:13 W [client-protocol.c:205:call_bail] client-stripe-2: 
activating bail-out. pending frames = 2. last sent = 2008-08-07 
19:53:23. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:54:13 C [client-protocol.c:212:call_bail] client-stripe-2: 
bailing transport
2008-08-07 19:54:13 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-2: cleaning up state in transport object 0x9625260
2008-08-07 19:54:13 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-2: forced unwinding frame type(1) op(34) reply=@0x962c870
2008-08-07 19:54:13 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-2: no proper reply from server, returning ENOTCONN
2008-08-07 19:54:13 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-2 returned errno 107
2008-08-07 19:54:13 E [fuse-bridge.c:468:fuse_entry_cbk] glusterfs-fuse: 
63761: (34) / => -1 (107)
2008-08-07 19:54:13 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-1: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:54:13 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-2: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:54:13 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-3: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:54:13 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-4: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:54:13 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-2: forced unwinding frame type(1) op(15) reply=@0x962c870
2008-08-07 19:54:13 E [client-protocol.c:3884:client_statfs_cbk] 
client-stripe-2: no proper reply from server, returning ENOTCONN
2008-08-07 19:54:13 E [fuse-bridge.c:2064:fuse_statfs_cbk] 
glusterfs-fuse: 63762: ERR => -1 (107)
2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] 
trans: attempting to pipeline request type(2) op(4) with handshake
2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-1: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-2: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-3: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-4: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:55:14 W [client-protocol.c:205:call_bail] client-stripe-3: 
activating bail-out. pending frames = 2. last sent = 2008-08-07 
19:54:31. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:55:14 C [client-protocol.c:212:call_bail] client-stripe-3: 
bailing transport
2008-08-07 19:55:14 W [client-protocol.c:205:call_bail] client-stripe-4: 
activating bail-out. pending frames = 2. last sent = 2008-08-07 
19:54:31. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:55:14 C [client-protocol.c:212:call_bail] client-stripe-4: 
bailing transport
2008-08-07 19:55:14 W [client-protocol.c:205:call_bail] client-stripe-1: 
activating bail-out. pending frames = 2. last sent = 2008-08-07 
19:54:31. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:55:14 C [client-protocol.c:212:call_bail] client-stripe-1: 
bailing transport
2008-08-07 19:55:14 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-1: cleaning up state in transport object 0x961f350
2008-08-07 19:55:14 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-1: forced unwinding frame type(1) op(34) reply=@0x9629c90
2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-1: no proper reply from server, returning ENOTCONN
2008-08-07 19:55:14 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-1 returned errno 107
2008-08-07 19:55:14 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-1: forced unwinding frame type(1) op(34) reply=@0x9629c90
2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-1: no proper reply from server, returning ENOTCONN
2008-08-07 19:55:14 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-1 returned errno 107
2008-08-07 19:55:14 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-4: cleaning up state in transport object 0x9629f20
2008-08-07 19:55:14 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-4: forced unwinding frame type(1) op(34) reply=@0x962c870
2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-4: no proper reply from server, returning ENOTCONN
2008-08-07 19:55:14 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-4 returned errno 107
2008-08-07 19:55:14 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-4: forced unwinding frame type(1) op(34) reply=@0x962c870
2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-4: no proper reply from server, returning ENOTCONN
2008-08-07 19:55:14 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-4 returned errno 107
2008-08-07 19:55:14 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-3: cleaning up state in transport object 0x96278c0
2008-08-07 19:55:14 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-3: forced unwinding frame type(1) op(34) reply=@0x9627630
2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-3: no proper reply from server, returning ENOTCONN
2008-08-07 19:55:14 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-3 returned errno 107
2008-08-07 19:55:14 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-3: forced unwinding frame type(1) op(34) reply=@0x9627630
2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-3: no proper reply from server, returning ENOTCONN
2008-08-07 19:55:14 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-3 returned errno 107
2008-08-07 19:55:21 W [client-protocol.c:205:call_bail] client-stripe-2: 
activating bail-out. pending frames = 2. last sent = 2008-08-07 
19:54:31. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:55:21 C [client-protocol.c:212:call_bail] client-stripe-2: 
bailing transport
2008-08-07 19:55:21 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-2: cleaning up state in transport object 0x9625260
2008-08-07 19:55:21 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-2: forced unwinding frame type(1) op(34) reply=@0x962e3b0
2008-08-07 19:55:21 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-2: no proper reply from server, returning ENOTCONN
2008-08-07 19:55:21 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-2 returned errno 107
2008-08-07 19:55:21 E [fuse-bridge.c:468:fuse_entry_cbk] glusterfs-fuse: 
63761: (34) / => -1 (107)
2008-08-07 19:55:21 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-2: forced unwinding frame type(1) op(34) reply=@0x962e3b0
2008-08-07 19:55:21 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-2: no proper reply from server, returning ENOTCONN
2008-08-07 19:55:21 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-2 returned errno 107
2008-08-07 19:55:21 E [fuse-bridge.c:468:fuse_entry_cbk] glusterfs-fuse: 
63763: (34) / => -1 (107)
2008-08-07 19:55:21 E [fuse-bridge.c:2699:init] glusterfs-fuse: 
fuse_mount failed (Transport endpoint is not connected)

2008-08-07 19:55:21 E [glusterfs.c:547:main] glusterfs: Initializing 
FUSE failed
2008-08-07 19:55:21 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-1: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:55:21 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-2: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:55:21 E [ib-verbs.c:1107:ib_verbs_send_completion_proc] 
transport/ib-verbs: send work request on `ipath0' returned error 
wc.status = 12, wc.vendor_err = 0, post->buf = 0x2aaaac530000, 
wc.byte_len = 0, post->reused = 35
2008-08-07 19:55:21 E [ib-verbs.c:1500:ib_verbs_except] 
transport/ib-verbs: shutdown () - error: Transport endpoint is not connected
2008-08-07 19:55:24 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-3: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:55:24 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-4: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:55:45 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-1: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:55:45 E [ib-verbs-client.c:141:ib_verbs_client_connect] 
client-stripe-2: connection not in progress - trace: Network is unreachable
2008-08-07 19:55:45 W [client-protocol.c:332:client_protocol_xfer] 
client-stripe-2: not connected at the moment to submit frame type(1) op(34)
2008-08-07 19:55:45 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-2: no proper reply from server, returning ENOTCONN
2008-08-07 19:55:45 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-2 returned errno 107
2008-08-07 19:55:45 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-3: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:55:45 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-4: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:56:32 W [client-protocol.c:205:call_bail] client-stripe-1: 
activating bail-out. pending frames = 2. last sent = 2008-08-07 
19:55:45. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:56:32 C [client-protocol.c:212:call_bail] client-stripe-1: 
bailing transport
2008-08-07 19:56:32 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-1: cleaning up state in transport object 0x961f350
2008-08-07 19:56:32 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-1: forced unwinding frame type(1) op(34) reply=@0x962f690
2008-08-07 19:56:32 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-1: no proper reply from server, returning ENOTCONN
2008-08-07 19:56:32 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-1 returned errno 107
2008-08-07 19:56:32 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-1: forced unwinding frame type(1) op(34) reply=@0x962f690
2008-08-07 19:56:32 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-1: no proper reply from server, returning ENOTCONN
2008-08-07 19:56:32 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-1 returned errno 107
2008-08-07 19:56:35 W [client-protocol.c:205:call_bail] client-stripe-4: 
activating bail-out. pending frames = 2. last sent = 2008-08-07 
19:55:45. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:56:35 C [client-protocol.c:212:call_bail] client-stripe-4: 
bailing transport
2008-08-07 19:56:35 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-4: cleaning up state in transport object 0x9629f20
2008-08-07 19:56:35 W [client-protocol.c:205:call_bail] client-stripe-3: 
activating bail-out. pending frames = 2. last sent = 2008-08-07 
19:55:45. last received = 1969-12-31 19:00:00 transport-timeout = 42
2008-08-07 19:56:35 C [client-protocol.c:212:call_bail] client-stripe-3: 
bailing transport
2008-08-07 19:56:35 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-4: forced unwinding frame type(1) op(34) reply=@0x962c870
2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-4: no proper reply from server, returning ENOTCONN
2008-08-07 19:56:35 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-4 returned errno 107
2008-08-07 19:56:35 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-4: forced unwinding frame type(1) op(34) reply=@0x962c870
2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-4: no proper reply from server, returning ENOTCONN
2008-08-07 19:56:35 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-4 returned errno 107
2008-08-07 19:56:35 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-3: cleaning up state in transport object 0x96278c0
2008-08-07 19:56:35 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-3: forced unwinding frame type(1) op(34) reply=@0x9629c90
2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-3: no proper reply from server, returning ENOTCONN
2008-08-07 19:56:35 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-3 returned errno 107
2008-08-07 19:56:35 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-3: forced unwinding frame type(1) op(34) reply=@0x9629c90
2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-3: no proper reply from server, returning ENOTCONN
2008-08-07 19:56:35 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-3 returned errno 107
2008-08-07 19:56:35 E [fuse-bridge.c:468:fuse_entry_cbk] glusterfs-fuse: 
63764: (34) / => -1 (107)
2008-08-07 19:56:35 E [ib-verbs-client.c:141:ib_verbs_client_connect] 
client-stripe-1: connection not in progress - trace: Network is unreachable
2008-08-07 19:56:35 W [client-protocol.c:332:client_protocol_xfer] 
client-stripe-1: not connected at the moment to submit frame type(1) op(34)
2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-1: no proper reply from server, returning ENOTCONN
2008-08-07 19:56:35 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-1 returned errno 107
2008-08-07 19:56:35 E [ib-verbs-client.c:141:ib_verbs_client_connect] 
client-stripe-2: connection not in progress - trace: Network is unreachable
2008-08-07 19:56:35 W [client-protocol.c:332:client_protocol_xfer] 
client-stripe-2: not connected at the moment to submit frame type(1) op(34)
2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-2: no proper reply from server, returning ENOTCONN
2008-08-07 19:56:35 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-2 returned errno 107
2008-08-07 19:56:35 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-3: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 19:56:35 E [ib-verbs-client.c:141:ib_verbs_client_connect] 
client-stripe-4: connection not in progress - trace: Network is unreachable
2008-08-07 19:56:35 W [client-protocol.c:332:client_protocol_xfer] 
client-stripe-4: not connected at the moment to submit frame type(1) op(34)
2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-4: no proper reply from server, returning ENOTCONN
2008-08-07 19:56:35 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-4 returned errno 107
2008-08-07 19:56:45 E [ib-verbs-client.c:141:ib_verbs_client_connect] 
client-stripe-4: connection not in progress - trace: Network is unreachable
2008-08-07 19:57:25 E [ib-verbs-client.c:141:ib_verbs_client_connect] 
client-stripe-3: connection not in progress - trace: Network is unreachable
2008-08-07 19:57:28 E [ib-verbs-client.c:141:ib_verbs_client_connect] 
client-stripe-1: connection not in progress - trace: Network is unreachable
2008-08-07 19:57:28 W [client-protocol.c:332:client_protocol_xfer] 
client-stripe-1: not connected at the moment to submit frame type(1) op(34)
2008-08-07 19:57:28 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-1: no proper reply from server, returning ENOTCONN
2008-08-07 19:57:28 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-1 returned errno 107
2008-08-07 19:57:28 E [ib-verbs-client.c:141:ib_verbs_client_connect] 
client-stripe-2: connection not in progress - trace: Network is unreachable
2008-08-07 19:57:28 W [client-protocol.c:332:client_protocol_xfer] 
client-stripe-2: not connected at the moment to submit frame type(1) op(34)
2008-08-07 19:57:28 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-2: no proper reply from server, returning ENOTCONN
2008-08-07 19:57:28 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-2 returned errno 107
2008-08-07 19:57:28 E [ib-verbs-client.c:141:ib_verbs_client_connect] 
client-stripe-3: connection not in progress - trace: Network is unreachable
2008-08-07 19:57:28 W [client-protocol.c:332:client_protocol_xfer] 
client-stripe-3: not connected at the moment to submit frame type(1) op(34)
2008-08-07 19:57:28 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-3: no proper reply from server, returning ENOTCONN
2008-08-07 19:57:28 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-3 returned errno 107
2008-08-07 19:57:28 E [ib-verbs-client.c:141:ib_verbs_client_connect] 
client-stripe-4: connection not in progress - trace: Network is unreachable
2008-08-07 19:57:28 W [client-protocol.c:332:client_protocol_xfer] 
client-stripe-4: not connected at the moment to submit frame type(1) op(34)
2008-08-07 19:57:28 E [client-protocol.c:4423:client_lookup_cbk] 
client-stripe-4: no proper reply from server, returning ENOTCONN
2008-08-07 19:57:28 W 
[stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: 
client-stripe-4 returned errno 107
2008-08-07 19:57:28 E [fuse-bridge.c:468:fuse_entry_cbk] glusterfs-fuse: 
63765: (34) /iozone.DUMMY.14 => -1 (107)
2008-08-07 19:57:29 E [ib-verbs-client.c:141:ib_verbs_client_connect] 
client-stripe-2: connection not in progress - trace: Network is unreachable
2008-08-07 19:57:30 W [client-protocol.c:1296:client_close] 
client-stripe-2: no valid fd found, returning
2008-08-07 19:57:30 W [client-protocol.c:1296:client_close] 
client-stripe-3: no valid fd found, returning
2008-08-07 19:57:30 W [client-protocol.c:1296:client_close] 
client-stripe-4: no valid fd found, returning
2008-08-07 19:57:30 W [client-protocol.c:1296:client_close] 
client-stripe-1: no valid fd found, returning
2008-08-07 19:57:33 E [ib-verbs-client.c:141:ib_verbs_client_connect] 
client-stripe-1: connection not in progress - trace: Network is unreachable
2008-08-07 20:00:04 W [client-protocol.c:280:client_protocol_xfer] 
trans: attempting to pipeline request type(2) op(4) with handshake
2008-08-07 20:00:04 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-2: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 20:00:04 W [fuse-bridge.c:398:fuse_entry_cbk] glusterfs-fuse: 
2: (34) / => 1 Rehashing 0/0
2008-08-07 23:31:29 W [client-protocol.c:280:client_protocol_xfer] 
trans: attempting to pipeline request type(2) op(4) with handshake
2008-08-07 23:31:29 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-3: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 23:31:29 W [fuse-bridge.c:398:fuse_entry_cbk] glusterfs-fuse: 
2: (34) / => 1 Rehashing 0/0
2008-08-07 23:59:06 W [client-protocol.c:205:call_bail] client-stripe-2: 
activating bail-out. pending frames = 1. last sent = 2008-08-07 
23:58:23. last received = 2008-08-07 23:58:17 transport-timeout = 42
2008-08-07 23:59:06 C [client-protocol.c:212:call_bail] client-stripe-2: 
bailing transport
2008-08-07 23:59:06 W [client-protocol.c:4777:client_protocol_cleanup] 
client-stripe-2: cleaning up state in transport object 0x87eb260
2008-08-07 23:59:06 E [client-protocol.c:4827:client_protocol_cleanup] 
client-stripe-2: forced unwinding frame type(1) op(14) reply=@0x87f3d10
2008-08-07 23:59:06 E [client-protocol.c:3247:client_write_cbk] 
client-stripe-2: no proper reply from server, returning ENOTCONN
2008-08-07 23:59:06 W [stripe.c:3070:stripe_writev_cbk] stripe: 
client-stripe-2 returned errno 107
2008-08-07 23:59:06 E [fuse-bridge.c:1645:fuse_writev_cbk] 
glusterfs-fuse: 128670: WRITE => -1 (107)
2008-08-07 23:59:06 W [client-protocol.c:280:client_protocol_xfer] 
client-stripe-2: attempting to pipeline request type(1) op(34) with 
handshake
2008-08-07 23:59:28 E [client-protocol.c:1238:client_flush] 
client-stripe-2: : returning EBADFD
2008-08-07 23:59:28 W [stripe.c:198:stripe_stack_unwind_cbk] stripe: 
client-stripe-2 returned errno 77
2008-08-07 23:59:28 W [client-protocol.c:1296:client_close] 
client-stripe-2: no valid fd found, returning
2008-08-08 00:51:00 W [client-protocol.c:1296:client_close] 
client-stripe-2: no valid fd found, returning

-Peter

-- 
Peter Pelekis  		5444 Napa Street
Western Scientific 	San Diego CA, 92110
Senior Systems Engineer		www.wsm.com
pete.pelekis at wsm.com		Fax 619-220-6590
Phone 619-220-6580 x212 Toll Free 800-443-6699
GSA# GS-35F-5009H

Visit us at:

"I'm a great believer in luck, and I find the harder I work the more I 
have of it."
-Thomas Jefferson

"If everything seems under control, you're not going fast enough."
-Mario Andretti