[Gluster-users] Transport endpoint is not connected with fuse

Tue May 23 13:12:09 UTC 2017

I'm getting Transport endpoint is not connected on two client nodes, but I
don't see any errors on the server, any idea where to look next?

I'm using gluster 3.10 from the ppa for ubuntu-server, any idea if this
version is known to be broken?

I'm mounting the file systems using fuse, is that supposed to work?

I'm using tcp on a 40 Gb/s Infiniband with ipoib and so far it has seemed
to be reliable enough when I tested it, does anybody use IB?

+------------------------------------------------------------------------------+
[2017-05-23 12:54:47.716519] I [rpc-clnt.c:2000:rpc_clnt_reconfig]
0-prism-client-0: changing port to 49158 (from 0)
[2017-05-23 12:54:47.717202] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs]
0-prism-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-23 12:54:47.717703] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 0-prism-client-0: Connected
to prism-client-0, attached to remote volume '/bricks/prism/glenlivet'.
[2017-05-23 12:54:47.717729] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 0-prism-client-0: Server and
Client lk-version numbers are not same, reopening the fds
[2017-05-23 12:54:47.718513] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-prism-client-0: Server
lk version = 1
[2017-05-23 12:54:47.718623] I [fuse-bridge.c:4146:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel
7.23
[2017-05-23 12:54:47.718643] I [fuse-bridge.c:4831:fuse_graph_sync] 0-fuse:
switched to graph 0
[2017-05-23 13:04:26.767636] C
[rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-prism-client-0: server
192.168.42.118:49158 has not responded in the last 42 seconds,
disconnecting.
[2017-05-23 13:04:26.767821] I [MSGID: 114018]
[client.c:2276:client_rpc_notify] 0-prism-client-0: disconnected from
prism-client-0. Client process will keep trying to connect to glusterd
until brick's port is available
[2017-05-23 13:04:26.768308] E [rpc-clnt.c:365:saved_frames_unwind] (-->
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f950c64962b]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1d1)[0x7f950c416eb1]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f950c416fce]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x94)[0x7f950c418654]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f950c419138]
))))) 0-prism-client-0: forced unwinding frame type(GlusterFS 3.3)
op(READDIRP(40)) called at 2017-05-23 13:03:43.999907 (xid=0x63a7)
[2017-05-23 13:04:26.768337] W [MSGID: 114031]
[client-rpc-fops.c:2640:client3_3_readdirp_cbk] 0-prism-client-0: remote
operation failed [Transport endpoint is not connected]
[2017-05-23 13:04:26.768557] E [rpc-clnt.c:365:saved_frames_unwind] (-->
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f950c64962b]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1d1)[0x7f950c416eb1]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f950c416fce]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x94)[0x7f950c418654]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f950c419138]
))))) 0-prism-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2))
called at 2017-05-23 13:03:44.763233 (xid=0x63a8)
[2017-05-23 13:04:26.768581] W [rpc-clnt-ping.c:202:rpc_clnt_ping_cbk]
0-prism-client-0: socket disconnected
[2017-05-23 13:04:26.768625] W [MSGID: 114031]
[client-rpc-fops.c:503:client3_3_stat_cbk] 0-prism-client-0: remote
operation failed [Transport endpoint is not connected]
[2017-05-23 13:04:26.768682] W [fuse-bridge.c:767:fuse_attr_cbk]
0-glusterfs-fuse: 30770: STAT() /gradle-metrics/hulo/36 => -1 (Transport
endpoint is not connected)
[2017-05-23 13:04:26.779347] W [fuse-bridge.c:767:fuse_attr_cbk]
0-glusterfs-fuse: 30813: STAT() /gradle-metrics/hulo => -1 (Transport
endpoint is not connected)
[2017-05-23 13:04:26.788802] W [fuse-bridge.c:767:fuse_attr_cbk]
0-glusterfs-fuse: 30855: STAT() /gradle-metrics/hulo => -1 (Transport
endpoint is not connected)
[2017-05-23 13:04:26.792743] W [fuse-bridge.c:767:fuse_attr_cbk]
0-glusterfs-fuse: 30897: STAT() /gradle-metrics/hulo => -1 (Transport
endpoint is not connected)
[2017-05-23 13:04:26.797661] W [fuse-bridge.c:767:fuse_attr_cbk]
0-glusterfs-fuse: 30940: STAT() /gradle-metrics => -1 (Transport endpoint
is not connected)
[2017-05-23 13:04:26.800631] W [fuse-bridge.c:767:fuse_attr_cbk]
0-glusterfs-fuse: 30982: STAT() /gradle-metrics => -1 (Transport endpoint
is not connected)
[2017-05-23 13:04:26.802215] W [MSGID: 114031]
[client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-prism-client-0: remote
operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport
endpoint is not connected]
[2017-05-23 13:04:26.804198] W [fuse-bridge.c:767:fuse_attr_cbk]
0-glusterfs-fuse: 31025: LOOKUP() / => -1 (Transport endpoint is not
connected)
[2017-05-23 13:04:26.809985] W [fuse-bridge.c:767:fuse_attr_cbk]
0-glusterfs-fuse: 31067: LOOKUP() / => -1 (Transport endpoint is not
connected)
[2017-05-23 13:04:26.813651] W [fuse-bridge.c:767:fuse_attr_cbk]
0-glusterfs-fuse: 31109: LOOKUP() / => -1 (Transport endpoint is not
connected)
[2017-05-23 13:04:36.769691] I [rpc-clnt.c:2000:rpc_clnt_reconfig]
0-prism-client-0: changing port to 49158 (from 0)
[2017-05-23 13:04:37.938120] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs]
0-prism-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-23 13:04:38.628669] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 0-prism-client-0: Connected
to prism-client-0, attached to remote volume '/bricks/prism/glenlivet'.
[2017-05-23 13:04:38.628718] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 0-prism-client-0: Server and
Client lk-version numbers are not same, reopening the fds
[2017-05-23 13:04:38.628738] I [MSGID: 114042]
[client-handshake.c:1047:client_post_handshake] 0-prism-client-0: 4 fds
open - Delaying child_up until they are re-opened
[2017-05-23 13:04:39.326687] I [MSGID: 114060]
[client-handshake.c:817:client3_3_reopendir_cbk] 0-prism-client-0:
reopendir on <gfid:00000000-0000-0000-0000-000000000001> succeeded (fd = 0)
[2017-05-23 13:04:40.723521] I [MSGID: 114060]
[client-handshake.c:817:client3_3_reopendir_cbk] 0-prism-client-0:
reopendir on <gfid:72da7dff-13f2-44af-bdc0-5b9910ff91a3> succeeded (fd = 1)
[2017-05-23 13:04:41.069774] I [MSGID: 114060]
[client-handshake.c:817:client3_3_reopendir_cbk] 0-prism-client-0:
reopendir on <gfid:46f04844-5e8f-4760-8546-40f8ea99aa70> succeeded (fd = 2)
[2017-05-23 13:04:41.721634] I [MSGID: 114060]
[client-handshake.c:817:client3_3_reopendir_cbk] 0-prism-client-0:
reopendir on <gfid:cd1eafe8-f1e1-473e-a53e-9148afb7d7c6> succeeded (fd = 2)
[2017-05-23 13:04:41.721756] I [MSGID: 114041]
[client-handshake.c:678:client_child_up_reopen_done] 0-prism-client-0: last
fd open'd/lock-self-heal'd - notifying CHILD-UP
[2017-05-23 13:04:42.405328] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-prism-client-0: Server
lk version = 1

==> gluster-meta.log <==
[2017-05-23 13:04:43.910069] C
[rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-meta-client-0: server
192.168.42.118:49154 has not responded in the last 42 seconds,
disconnecting.
[2017-05-23 13:04:43.910264] I [MSGID: 114018]
[client.c:2276:client_rpc_notify] 0-meta-client-0: disconnected from
meta-client-0. Client process will keep trying to connect to glusterd until
brick's port is available
[2017-05-23 13:04:43.910596] E [rpc-clnt.c:365:saved_frames_unwind] (-->
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f56a520962b]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1d1)[0x7f56a4fd6eb1]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f56a4fd6fce]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x94)[0x7f56a4fd8654]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f56a4fd9138]
))))) 0-meta-client-0: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-05-23 13:04:01.809391 (xid=0x80)
[2017-05-23 13:04:43.910621] W [MSGID: 114031]
[client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-meta-client-0: remote
operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport
endpoint is not connected]
[2017-05-23 13:04:43.910676] W [fuse-bridge.c:767:fuse_attr_cbk]
0-glusterfs-fuse: 173: LOOKUP() / => -1 (Transport endpoint is not
connected)
[2017-05-23 13:04:43.910839] E [rpc-clnt.c:365:saved_frames_unwind] (-->
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f56a520962b]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1d1)[0x7f56a4fd6eb1]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f56a4fd6fce]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x94)[0x7f56a4fd8654]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f56a4fd9138]
))))) 0-meta-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2))
called at 2017-05-23 13:04:01.809418 (xid=0x81)
[2017-05-23 13:04:43.910858] W [rpc-clnt-ping.c:202:rpc_clnt_ping_cbk]
0-meta-client-0: socket disconnected

==> gluster-prism.log <==
The message "W [MSGID: 114031] [client-rpc-fops.c:503:client3_3_stat_cbk]
0-prism-client-0: remote operation failed [Transport endpoint is not
connected]" repeated 232 times between [2017-05-23 13:04:26.768625] and
[2017-05-23 13:04:26.802082]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-prism-client-0: remote
operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport
endpoint is not connected]" repeated 105 times between [2017-05-23
13:04:26.802215] and [2017-05-23 13:04:26.813835]

-- 
Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170523/9c925768/attachment.html>