[Gluster-devel] Error: Transport endpoint is not connected

[JV]

Hello.

I see an error, when I try to

find /mnt/glusterfs/ -type f -exec md5sum {} \; >/dev/null

it fails as:
md5sum: /mnt/glusterfs/2009/1/18/13/E4E4EF76/AF3A768D/F777FC5E: 
Transport endpoint is not connected

Problem seems not to be related to any single file - it failed twice 
already on different files, after about 60 minutes. This particular file 
exists on both storage nodes, md5sums are the same. Extended attributes 
are the same:

trusted.afr.g1b2=0sAAAAAAAAAAAAAAAA
trusted.afr.g2b2=0sAAAAAAAAAAAAAAAA

Backend FS is ext4, xfs was failing for some strange reason.


in client log is only this:

pending frames:

patchset: 82394d484803e02e28441bc0b988efaaff60dd94
signal received: 6
configuration details:argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 2.0.0rc8
[0xb8067400]
/lib/i686/cmov/libc.so.6(abort+0x188)[0xb7ede018]
/lib/i686/cmov/libc.so.6(__assert_fail+0xee)[0xb7ed55be]
/lib/i686/cmov/libpthread.so.0(pthread_mutex_lock+0x5d4)[0xb8013f54]
/usr/lib/libfuse.so.2[0xb75d3147]
/usr/lib/libfuse.so.2(fuse_session_process+0x26)[0xb75d4bf6]
/usr/local/lib/glusterfs/2.0.0rc8/xlator/mount/fuse.so[0xb75eef31]
/lib/i686/cmov/libpthread.so.0[0xb80124c0]
/lib/i686/cmov/libc.so.6(clone+0x5e)[0xb7f916de]
---------

There are no errors in storage server logs.

Systems:
Debian Stable,
kernel 2.6.29.1
gcc (Debian 4.3.2-1.1) 4.3.2
glusterfs 2.0.0rc8 built on Apr 20 2009 23:04:47
Repository revision: 82394d484803e02e28441bc0b988efaaff60dd94

fuse from debian 2.7.4

Configuration:
2 storage nodes (Dual Core, 2GB RAM, 3x1TB SATA)
1 client (Dual Core, 6GB RAM)

Storage config:
===========================================
volume gdisk1
   type storage/posix
   option directory /export/gdisk1/storage/
end-volume

volume brick1
   type features/posix-locks
   subvolumes gdisk1
end-volume

volume gdisk2
   type storage/posix
   option directory /export/gdisk2/storage/
end-volume

volume brick2
   type features/posix-locks
   subvolumes gdisk2
end-volume


volume gdisk3
   type storage/posix
   option directory /export/gdisk3/storage/
end-volume

volume brick3
   type features/posix-locks
   subvolumes gdisk3
end-volume


volume server
   type protocol/server
   option transport-type tcp
   option auth.addr.brick1.allow *
   option auth.addr.brick2.allow *
   option auth.addr.brick3.allow *
   subvolumes brick1 brick2 brick3
end-volume


Client config:
===========================

volume g1b1
         type protocol/client
         option transport-type tcp/client
         option remote-host 10.10.70.2
         option remote-subvolume brick1
end-volume
volume g1b2
         type protocol/client
         option transport-type tcp
         option remote-host 10.10.70.2
         option remote-subvolume brick2
end-volume
volume g1b3
         type protocol/client
         option transport-type tcp
         option remote-host 10.10.70.2
         option remote-subvolume brick3
end-volume

volume g2b1
         type protocol/client
         option transport-type tcp/client
         option remote-host 10.10.70.3
         option remote-subvolume brick1
end-volume
volume g2b2
         type protocol/client
         option transport-type tcp
         option remote-host 10.10.70.3
         option remote-subvolume brick2
end-volume
volume g2b3
         type protocol/client
         option transport-type tcp
         option remote-host 10.10.70.3
         option remote-subvolume brick3
end-volume

volume replicate1
         type cluster/replicate
         subvolumes g1b1 g2b1
end-volume

volume replicate2
         type cluster/replicate
         subvolumes g1b2 g2b2
end-volume

volume replicate3
         type cluster/replicate
         subvolumes g1b3 g2b3
end-volume



volume distribute
   type cluster/distribute
   subvolumes replicate1 replicate2 replicate3
end-volume

volume readahead
   type performance/read-ahead
   option page-size 128kB        # 256KB is the default option
   option page-count 4           # 2 is default option
   option force-atime-update off # default is off
   subvolumes distribute
end-volume

volume io-cache
   type performance/io-cache
   option cache-size 64MB             # default is 32MB
   option page-size 1MB               #128KB is default option
   option cache-timeout 2             # default is 1 second
   subvolumes readahead
end-volume


volume writebehind
   type performance/write-behind
   option aggregate-size 128KB # default is 0bytes
   option window-size 4MB    # default is equal to aggregate-size
   option flush-behind on    # default is 'off'
   subvolumes io-cache
end-volume


Is there anything I can do help debugging it?

Also it would seem that there are some significant changes to 
write-behind translator, as I understand it aggregate-size is no longer 
used?

Thanks.
JV