<div dir="ltr"><div><div><div>I'm getting Transport endpoint is not connected on two client nodes, but I don't see any errors on the server, any idea where to look next?<br><br></div>I'm using gluster 3.10 from the ppa for ubuntu-server, any idea if this version is known to be broken?<br><br></div>I'm mounting the file systems using fuse, is that supposed to work?<br><br></div>I'm using tcp on a 40 Gb/s Infiniband with ipoib and so far it has seemed to be reliable enough when I tested it, does anybody use IB?<br><br><div><div><div><div><br>+------------------------------------------------------------------------------+<br>[2017-05-23 12:54:47.716519] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-prism-client-0: changing port to 49158 (from 0)<br>[2017-05-23 12:54:47.717202] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-prism-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>[2017-05-23 12:54:47.717703] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-prism-client-0: Connected to prism-client-0, attached to remote volume '/bricks/prism/glenlivet'.<br>[2017-05-23 12:54:47.717729] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-prism-client-0: Server and Client lk-version numbers are not same, reopening the fds<br>[2017-05-23 12:54:47.718513] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-prism-client-0: Server lk version = 1<br>[2017-05-23 12:54:47.718623] I [fuse-bridge.c:4146:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.23<br>[2017-05-23 12:54:47.718643] I [fuse-bridge.c:4831:fuse_graph_sync] 0-fuse: switched to graph 0<br>[2017-05-23 13:04:26.767636] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-prism-client-0: server <a href="http://192.168.42.118:49158">192.168.42.118:49158</a> has not responded in the last 42 seconds, disconnecting.<br>[2017-05-23 13:04:26.767821] I [MSGID: 114018] [client.c:2276:client_rpc_notify] 0-prism-client-0: disconnected from prism-client-0. Client process will keep trying to connect to glusterd until brick's port is available<br>[2017-05-23 13:04:26.768308] E [rpc-clnt.c:365:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f950c64962b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1d1)[0x7f950c416eb1] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f950c416fce] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x94)[0x7f950c418654] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f950c419138] ))))) 0-prism-client-0: forced unwinding frame type(GlusterFS 3.3) op(READDIRP(40)) called at 2017-05-23 13:03:43.999907 (xid=0x63a7)<br>[2017-05-23 13:04:26.768337] W [MSGID: 114031] [client-rpc-fops.c:2640:client3_3_readdirp_cbk] 0-prism-client-0: remote operation failed [Transport endpoint is not connected]<br>[2017-05-23 13:04:26.768557] E [rpc-clnt.c:365:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f950c64962b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1d1)[0x7f950c416eb1] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f950c416fce] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x94)[0x7f950c418654] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f950c419138] ))))) 0-prism-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2017-05-23 13:03:44.763233 (xid=0x63a8)<br>[2017-05-23 13:04:26.768581] W [rpc-clnt-ping.c:202:rpc_clnt_ping_cbk] 0-prism-client-0: socket disconnected<br>[2017-05-23 13:04:26.768625] W [MSGID: 114031] [client-rpc-fops.c:503:client3_3_stat_cbk] 0-prism-client-0: remote operation failed [Transport endpoint is not connected]<br>[2017-05-23 13:04:26.768682] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 30770: STAT() /gradle-metrics/hulo/36 => -1 (Transport endpoint is not connected)<br>[2017-05-23 13:04:26.779347] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 30813: STAT() /gradle-metrics/hulo => -1 (Transport endpoint is not connected)<br>[2017-05-23 13:04:26.788802] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 30855: STAT() /gradle-metrics/hulo => -1 (Transport endpoint is not connected)<br>[2017-05-23 13:04:26.792743] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 30897: STAT() /gradle-metrics/hulo => -1 (Transport endpoint is not connected)<br>[2017-05-23 13:04:26.797661] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 30940: STAT() /gradle-metrics => -1 (Transport endpoint is not connected)<br>[2017-05-23 13:04:26.800631] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 30982: STAT() /gradle-metrics => -1 (Transport endpoint is not connected)<br>[2017-05-23 13:04:26.802215] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-prism-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]<br>[2017-05-23 13:04:26.804198] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 31025: LOOKUP() / => -1 (Transport endpoint is not connected)<br>[2017-05-23 13:04:26.809985] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 31067: LOOKUP() / => -1 (Transport endpoint is not connected)<br>[2017-05-23 13:04:26.813651] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 31109: LOOKUP() / => -1 (Transport endpoint is not connected)<br>[2017-05-23 13:04:36.769691] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-prism-client-0: changing port to 49158 (from 0)<br>[2017-05-23 13:04:37.938120] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-prism-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>[2017-05-23 13:04:38.628669] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-prism-client-0: Connected to prism-client-0, attached to remote volume '/bricks/prism/glenlivet'.<br>[2017-05-23 13:04:38.628718] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-prism-client-0: Server and Client lk-version numbers are not same, reopening the fds<br>[2017-05-23 13:04:38.628738] I [MSGID: 114042] [client-handshake.c:1047:client_post_handshake] 0-prism-client-0: 4 fds open - Delaying child_up until they are re-opened<br>[2017-05-23 13:04:39.326687] I [MSGID: 114060] [client-handshake.c:817:client3_3_reopendir_cbk] 0-prism-client-0: reopendir on <gfid:00000000-0000-0000-0000-000000000001> succeeded (fd = 0)<br>[2017-05-23 13:04:40.723521] I [MSGID: 114060] [client-handshake.c:817:client3_3_reopendir_cbk] 0-prism-client-0: reopendir on <gfid:72da7dff-13f2-44af-bdc0-5b9910ff91a3> succeeded (fd = 1)<br>[2017-05-23 13:04:41.069774] I [MSGID: 114060] [client-handshake.c:817:client3_3_reopendir_cbk] 0-prism-client-0: reopendir on <gfid:46f04844-5e8f-4760-8546-40f8ea99aa70> succeeded (fd = 2)<br>[2017-05-23 13:04:41.721634] I [MSGID: 114060] [client-handshake.c:817:client3_3_reopendir_cbk] 0-prism-client-0: reopendir on <gfid:cd1eafe8-f1e1-473e-a53e-9148afb7d7c6> succeeded (fd = 2)<br>[2017-05-23 13:04:41.721756] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-prism-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP<br>[2017-05-23 13:04:42.405328] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-prism-client-0: Server lk version = 1<br><br>==> gluster-meta.log <==<br>[2017-05-23 13:04:43.910069] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-meta-client-0: server <a href="http://192.168.42.118:49154">192.168.42.118:49154</a> has not responded in the last 42 seconds, disconnecting.<br>[2017-05-23 13:04:43.910264] I [MSGID: 114018] [client.c:2276:client_rpc_notify] 0-meta-client-0: disconnected from meta-client-0. Client process will keep trying to connect to glusterd until brick's port is available<br>[2017-05-23 13:04:43.910596] E [rpc-clnt.c:365:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f56a520962b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1d1)[0x7f56a4fd6eb1] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f56a4fd6fce] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x94)[0x7f56a4fd8654] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f56a4fd9138] ))))) 0-meta-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-05-23 13:04:01.809391 (xid=0x80)<br>[2017-05-23 13:04:43.910621] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-meta-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]<br>[2017-05-23 13:04:43.910676] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 173: LOOKUP() / => -1 (Transport endpoint is not connected)<br>[2017-05-23 13:04:43.910839] E [rpc-clnt.c:365:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f56a520962b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1d1)[0x7f56a4fd6eb1] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f56a4fd6fce] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x94)[0x7f56a4fd8654] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f56a4fd9138] ))))) 0-meta-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2017-05-23 13:04:01.809418 (xid=0x81)<br>[2017-05-23 13:04:43.910858] W [rpc-clnt-ping.c:202:rpc_clnt_ping_cbk] 0-meta-client-0: socket disconnected<br><br>==> gluster-prism.log <==<br>The message "W [MSGID: 114031] [client-rpc-fops.c:503:client3_3_stat_cbk] 0-prism-client-0: remote operation failed [Transport endpoint is not connected]" repeated 232 times between [2017-05-23 13:04:26.768625] and [2017-05-23 13:04:26.802082]<br>The message "W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-prism-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]" repeated 105 times between [2017-05-23 13:04:26.802215] and [2017-05-23 13:04:26.813835]<br><br clear="all"><br>-- <br><div class="gmail_signature">Flemming Frandsen - YAPH - <a href="http://osaa.dk" target="_blank">http://osaa.dk</a> - <a href="http://dren.dk/" target="_blank">http://dren.dk/</a></div>
</div></div></div></div></div>