[Gluster-users] Transport endpoint is not connected

[Ivan Dimitrov]

Hi group,
I'm in production with gluster for the last 2 weeks. No problems until 
today.
As of today I've got the "Transport endpoint is not connected" problem 
on the client, maybe once every hour.
df: `/services/users/6': Transport endpoint is not connected

Here is my setup:
I have 1 Client and 2 Servers with 2 Disks each for bricks. Glusterfs 
3.3 compiled from source.

# gluster volume info

Volume Name: freecloud
Type: Distributed-Replicate
Volume ID: 1cf4804f-12aa-4cd1-a892-cec69fc2cf22
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: XX.25.137.252:/mnt/35be42b4-afb3-48a2-8b3c-17a422fd1e15
Brick2: YY.40.3.216:/mnt/7ee4f117-8aee-4cae-b08c-5e441b703886
Brick3: XX.25.137.252:/mnt/9ee7c816-085d-4c5c-9276-fd3dadac6c72
Brick4: YY.40.3.216:/mnt/311399bc-4d55-445d-8480-286c56cf493e
Options Reconfigured:
cluster.self-heal-daemon: on
performance.cache-size: 256MB
performance.io-thread-count: 32
features.quota: on

Quota is ON but not used
---------------------------------------------

# gluster volume status all detail
Status of volume: freecloud
------------------------------------------------------------------------------
Brick                : Brick 
XX.25.137.252:/mnt/35be42b4-afb3-48a2-8b3c-17a422fd1e15
Port                 : 24009
Online               : Y
Pid                  : 29221
File System          : xfs
Device               : /dev/sdd1
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 659.7GB
Total Disk Space     : 698.3GB
Inode Count          : 732571968
Free Inodes          : 730418928
------------------------------------------------------------------------------
Brick                : Brick 
YY.40.3.216:/mnt/7ee4f117-8aee-4cae-b08c-5e441b703886
Port                 : 24009
Online               : Y
Pid                  : 15496
File System          : xfs
Device               : /dev/sdc1
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 659.7GB
Total Disk Space     : 698.3GB
Inode Count          : 732571968
Free Inodes          : 730410396
------------------------------------------------------------------------------
Brick                : Brick 
XX.25.137.252:/mnt/9ee7c816-085d-4c5c-9276-fd3dadac6c72
Port                 : 24010
Online               : Y
Pid                  : 29227
File System          : xfs
Device               : /dev/sdc1
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 659.9GB
Total Disk Space     : 698.3GB
Inode Count          : 732571968
Free Inodes          : 730417864
------------------------------------------------------------------------------
Brick                : Brick 
YY.40.3.216:/mnt/311399bc-4d55-445d-8480-286c56cf493e
Port                 : 24010
Online               : Y
Pid                  : 15502
File System          : xfs
Device               : /dev/sdb1
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 659.9GB
Total Disk Space     : 698.3GB
Inode Count          : 732571968
Free Inodes          : 730409337


On server1 I mount the volume and start copying files to it. Server1 is 
used like storage.

209.25.137.252:freecloud                      1.4T   78G  1.3T   6% 
/home/freecloud

One thing to mention is that I have a large list of subdirectories in 
the main directory and the list keeps getting bigger.
client1# ls | wc -l
42424

---------------------------------------
I have one client server that mounts glusterfs and uses the files 
directly as the files are for low traffic web sites. On the client, 
there is no gluster daemon, just the mount.

client1# mount -t glusterfs rscloud1.domain.net:/freecloud 
/services/users/6/

This all worked fine for the last 2-3 weeks. Here is a log from the 
crash client1:/var/log/glusterfs/services-users-6-.log

pending frames:
frame : type(1) op(RENAME)
frame : type(1) op(RENAME)
frame : type(1) op(RENAME)
frame : type(1) op(RENAME)

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2012-07-12 14:51:01
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.3.0
/lib/x86_64-linux-gnu/libc.so.6(+0x32480)[0x7f1e0e9f0480]
/services/glusterfs//lib/libglusterfs.so.0(uuid_unpack+0x0)[0x7f1e0f79d760]
/services/glusterfs//lib/libglusterfs.so.0(+0x4c526)[0x7f1e0f79d526]
/services/glusterfs//lib/libglusterfs.so.0(uuid_utoa+0x26)[0x7f1e0f77ca66]
/services/glusterfs//lib/glusterfs/3.3.0/xlator/features/quota.so(quota_rename_cbk+0x308)[0x7f1e09b940c8]
/services/glusterfs//lib/glusterfs/3.3.0/xlator/cluster/distribute.so(dht_rename_unlink_cbk+0x454)[0x7f1e09dad264]
/services/glusterfs//lib/glusterfs/3.3.0/xlator/cluster/replicate.so(afr_unlink_unwind+0xf7)[0x7f1e09ff23c7]
/services/glusterfs//lib/glusterfs/3.3.0/xlator/cluster/replicate.so(afr_unlink_wind_cbk+0xb6)[0x7f1e09ff43d6]
/services/glusterfs//lib/glusterfs/3.3.0/xlator/protocol/client.so(client3_1_unlink_cbk+0x526)[0x7f1e0a2792c6]
/services/glusterfs//lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x7f1e0f548e45]
/services/glusterfs//lib/libgfrpc.so.0(rpc_clnt_notify+0xa5)[0x7f1e0f549825]
/services/glusterfs//lib/libgfrpc.so.0(rpc_transport_notify+0x27)[0x7f1e0f5458b7]
/services/glusterfs//lib/glusterfs/3.3.0/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7f1e0b84de14]
/services/glusterfs//lib/glusterfs/3.3.0/rpc-transport/socket.so(socket_event_handler+0xc7)[0x7f1e0b84e167]
/services/glusterfs//lib/libglusterfs.so.0(+0x40047)[0x7f1e0f791047]
/services/glusterfs/sbin/glusterfs(main+0x34d)[0x404c5d]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f1e0e9dcead]
/services/glusterfs/sbin/glusterfs[0x404f21]
---------


Where should I look for more clues?