[Gluster-users] glusterfs problem

[Matt Lawrence]

Raghavendra G wrote:
> Hi Matt,
> Can you please attach the whole log files? we need both client and 
> server log files.

 I now have a solid failure.  I really need some help figuring out what 
is wrong.

-bash-3.2# cat /etc/glusterfs/glusterfs-server.vol
##############################################
###  GlusterFS Server Volume Specification  ##
##############################################

volume shelf-1-raw
  type storage/posix
  option directory /glusterfs/shelf-1
end-volume

volume shelf-1
  type performance/io-threads
  option thread-count 2
  option cache-size 32MB
  subvolumes shelf-1-raw
end-volume

### Add network serving capability to above brick.
volume server
  type protocol/server
  option transport-type tcp/server     # For TCP/IP transport-bash-3.2# 
cat /etc/glusterfs/glusterfs-client.vol
##############################################
###  GlusterFS Client Volume Specification  ##
##############################################

volume storage
  type protocol/client
  option transport-type tcp/client
  option remote-host 127.0.0.1
  option remote-subvolume shelf-1
end-volume

volume storage-readahead
  type performance/read-ahead
  option page-size      128KB
  option page-count     4
  option force-atime-update off
  subvolumes storage
end-volume

volume storage-writebehind
  type performance/write-behind
  option aggregate-size 512KB
  option window-size 3MB
  option flush-behind on
  subvolumes storage-readahead
end-volume


  subvolumes shelf-1
  option auth.ip.shelf-1.allow 127.0.0.1
end-volume


-bash-3.2# dd if=/dev/zero of=/storage/testing bs=64K count=256K
dd: writing `/storage/testing': Transport endpoint is not connected
dd: closing output file `/storage/testing': Transport endpoint is not 
connected

-bash-3.2# cat /var/log/glusterfs/glusterfsd.log
2008-12-15 12:25:49 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:49 E [tcp.c:124:tcp_except] server: shutdown () - 
error: Bad file descriptor
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed
2008-12-15 12:25:50 E [server-protocol.c:186:generic_reply] server: 
transport_writev failed



-bash-3.2# cat /var/log/glusterfs/glusterfs.log
2008-12-15 12:24:09 W [client-protocol.c:280:client_protocol_xfer] 
storage: attempting to pipeline request type(1) op(34) with handshake
2008-12-15 12:25:49 W [client-protocol.c:205:call_bail] storage: 
activating bail-out. pending frames = 75. last sent = 2008-12-15 
12:25:05. last received = 2008-12-15 12:25:05 transport-timeout = 42
2008-12-15 12:25:49 C [client-protocol.c:212:call_bail] storage: bailing 
transport
2008-12-15 12:25:49 W [client-protocol.c:4777:client_protocol_cleanup] 
storage: cleaning up state in transport object 0x1c6f3c0
2008-12-15 12:25:49 E [client-protocol.c:325:client_protocol_xfer] 
storage: transport_submit failed
2008-12-15 12:25:49 E [client-protocol.c:4827:client_protocol_cleanup] 
storage: forced unwinding frame type(1) op(14) reply=@0x2aaab008a270
2008-12-15 12:25:49 E [client-protocol.c:3247:client_write_cbk] storage: 
no proper reply from server, returning ENOTCONN
2008-12-15 12:25:49 E [client-protocol.c:4827:client_protocol_cleanup] 
storage: forced unwinding frame type(1) op(14) reply=@0x2aaab008a270
2008-12-15 12:25:49 E [client-protocol.c:3247:client_write_cbk] storage: 
no proper reply from server, returning ENOTCONN
2008-12-15 12:25:49 E [client-protocol.c:4827:client_protocol_cleanup] 
storage: forced unwinding frame type(1) op(14) reply=@0x2aaab008a270
2008-12-15 12:25:49 E [write-behind.c:602:wb_writev] 
storage-writebehind: delayed error : 107
2008-12-15 12:25:49 E [fuse-bridge.c:1609:fuse_writev_cbk] 
glusterfs-fuse: 39589: WRITE => -1 (107)
2008-12-15 12:25:49 E [client-protocol.c:3247:client_write_cbk] storage: 
no proper reply from server, returning ENOTCONN
2008-12-15 12:25:49 E [client-protocol.c:4827:client_protocol_cleanup] 
storage: forced unwinding frame type(1) op(14) reply=@0x2aaab008a270
2008-12-15 12:25:49 E [client-protocol.c:3247:client_write_cbk] storage: 
no proper reply from server, returning ENOTCONN
2008-12-15 12:25:49 E [client-protocol.c:4827:client_protocol_cleanup] 
storage: forced unwinding frame type(1) op(14) reply=@0x2aaab008a270
2008-12-15 12:25:49 E [client-protocol.c:3247:client_write_cbk] storage: 
no proper reply from server, returning ENOTCONN
2008-12-15 12:25:49 E [client-protocol.c:4827:client_protocol_cleanup] 
storage: forced unwinding frame type(1) op(14) reply=@0x2aaab008a270
2008-12-15 12:25:49 E [client-protocol.c:3247:client_write_cbk] storage: 
no proper reply from server, returning ENOTCONN
2008-12-15 12:25:49 E [client-protocol.c:4827:client_protocol_cleanup] 
storage: forced unwinding frame type(1) op(14) reply=@0x2aaab008a270
2008-12-15 12:25:49 E [client-protocol.c:3247:client_write_cbk] storage: 
no proper reply from server, returning ENOTCONN
2008-12-15 12:25:49 E [client-protocol.c:4827:client_protocol_cleanup] 
storage: forced unwinding frame type(1) op(14) reply=@0x2aaab008a270
2008-12-15 12:25:49 E [client-protocol.c:3247:client_write_cbk] storage: 
no proper reply from server, returning ENOTCONN
2008-12-15 12:25:49 E [client-protocol.c:4827:client_protocol_cleanup] 
storage: forced unwinding frame type(1) op(14) reply=@0x2aaab008a270
2008-12-15 12:25:49 E [client-protocol.c:3247:client_write_cbk] storage: 
no proper reply from server, returning ENOTCONN
2008-12-15 12:25:49 E [client-protocol.c:4827:client_protocol_cleanup] 
storage: forced unwinding frame type(1) op(14) reply=@0x2aaab008a270
2008-12-15 12:25:49 E [client-protocol.c:3247:client_write_cbk] storage: 
no proper reply from server, returning ENOTCONN
2008-12-15 12:25:49 E [fuse-bridge.c:922:fuse_err_cbk] glusterfs-fuse: 
39590: (16) ERR => -1 (107)
2008-12-15 12:25:49 E [client-protocol.c:1238:client_flush] storage: : 
returning EBADFD
2008-12-15 12:25:49 W [client-protocol.c:1296:client_close] storage: no 
valid fd found, returning
2008-12-15 12:25:49 E [client-protocol.c:4827:client_protocol_cleanup] 
storage: forced unwinding frame type(1) op(14) reply=@0x2aaab008a270
2008-12-15 12:25:49 E [client-protocol.c:3247:client_write_cbk] storage: 
no proper reply from server, returning ENOTCONN


Any ideas what I should try next?

-- Matt