<div dir="ltr">can you please share the glusterd log file?<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 15, 2017 at 5:18 PM, Guy Cukierman <span dir="ltr"><<a href="mailto:guyc@elminda.com" target="_blank">guyc@elminda.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="#0563C1" vlink="#954F72" lang="EN-US">
<div class="m_292242184082376356WordSection1">
<p class="MsoNormal">Hi,<u></u><u></u></p>
<p class="MsoNormal">I’m having a similar issue, were you able to solve it?<u></u><u></u></p>
<p class="MsoNormal">Thanks.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Hey all,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">I've got a strange problem going on here. I've installed glusterfs-server<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">on ubuntu 16.04:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">glusterfs-client/xenial,now 3.7.6-1ubuntu1 amd64 [installed,automatic]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">glusterfs-common/xenial,now 3.7.6-1ubuntu1 amd64 [installed,automatic]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">glusterfs-server/xenial,now 3.7.6-1ubuntu1 amd64 [installed]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">I can successfully probe another peer at this point. Then, after installing<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">kubernetes via kargo, peer probing begins failing with a timeout. I've<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">tried stopping all kubernetes related services, and flushing all iptables<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">rules, however I don't see any packets leaving any interface when<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">attempting to peer probe.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">from cli.log:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.704900] I [MSGID: 101190]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[event-epoll.c:632:event_<wbr>dispatch_epoll_worker] 0-epoll: Started thread<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">with index 1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.704973] T [cli.c:273:cli_rpc_notify] 0-glusterfs: got<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">RPC_CLNT_CONNECT<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.705001] T [cli-quotad-client.c:94:cli_<wbr>quotad_notify]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-glusterfs: got RPC_CLNT_CONNECT<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.705014] I [socket.c:2355:socket_event_<wbr>handler]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-transport: disconnecting now<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.705204] T [rpc-clnt.c:1404:rpc_clnt_<wbr>record]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.705256] T<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[rpc-clnt.c:1261:rpc_clnt_<wbr>record_build_header] 0-rpc-clnt: Request fraglen<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">156, payload: 92, rpc hdr: 64<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.705662] T [socket.c:2879:socket_connect] (--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libglusterfs.so.0(_gf_log_<wbr>callingfn+0x1a3)[<wbr>0x7f012fd21953]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--> /usr/lib/x86_64-linux-gnu<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/libgfrpc.so.0(rpc_clnt_<wbr>remove_ping_timer_locked+0x84)<wbr>[0x7f012f69add4] (--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x55)[<wbr>0x7f012f697af5]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--> /usr/lib/x8<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">6_64-linux-gnu/libgfrpc.so.0(<wbr>rpc_clnt_notify+0x88)[<wbr>0x7f012f698338] (--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libgfrpc.so.0(rpc_transport_<wbr>notify+0x23)[0x7f012f6945b3]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">))))) 0-glusterfs: connect<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">() called on transport already connected<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.705680] D<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[rpc-clnt-ping.c:98:rpc_clnt_<wbr>remove_ping_timer_locked] (--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libglusterfs.so.0(_gf_log_<wbr>callingfn+0x1a3)[<wbr>0x7f012fd21953]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--> /<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">usr/lib/x86_64-linux-gnu/<wbr>libgfrpc.so.0(rpc_clnt_remove_<wbr>ping_timer_locked+0x84)[<wbr>0x7f012f69add4]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x55)[<wbr>0x7f012f<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">697af5] (--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libgfrpc.so.0(rpc_clnt_notify+<wbr>0x88)[0x7f012f698338]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libgfrpc.so.0(rpc_transport_<wbr>notify+0x23)[0x7f012f6945b3]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">)))<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">)) 0-: /var/run/gluster/quotad.<wbr>socket: ping timer event already removed<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.705710] T [cli-quotad-client.c:100:cli_<wbr>quotad_notify]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-glusterfs: got RPC_CLNT_DISCONNECT<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.705718] T [rpc-clnt.c:1598:rpc_clnt_<wbr>submit]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Proc: 1) to rpc-transport (glusterfs)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.705739] D [rpc-clnt-ping.c:281:rpc_clnt_<wbr>start_ping]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-glusterfs: ping timeout is 0, returning<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.705723] D [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[event-epoll.c:591:event_<wbr>dispatch_epoll_handler] 0-epoll: generation bumped<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">on idx=1 from gen=1 to slot->gen=2, fd=7, slot->fd=7<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:27.614881] T [rpc-clnt.c:418:rpc_clnt_<wbr>reconnect]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-glusterfs: attempting reconnect<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:27.615151] T [socket.c:2879:socket_connect] (--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libglusterfs.so.0(_gf_log_<wbr>callingfn+0x1a3)[<wbr>0x7f012fd21953]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>glusterfs/3.7.6/rpc-transport/<wbr>socket.so(+0x6c1b)[<wbr>0x7f012a697c1b]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libgfrpc.so.0(rpc_clnt_<wbr>reconnect+0xb9)[<wbr>0x7f012f695999]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libglusterfs.so.0(gf_timer_<wbr>proc+0xfc)[0x7f012fd3d70c]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--> /lib/x86_64-linux-gnu/<wbr>libpthread.so.0(+0x76ba)[<wbr>0x7f012f0b86ba] )))))<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-glusterfs: connect () called on transport already connected<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">it then repeats the following:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:27.615177] T [rpc-clnt.c:418:rpc_clnt_<wbr>reconnect]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-glusterfs: attempting reconnect<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:27.615188] T [socket.c:2887:socket_connect] 0-glusterfs:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">connecting 0x25d3550, state=0 gen=0 sock=-1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:27.615200] T<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[name.c:295:af_unix_client_<wbr>get_remote_sockaddr] 0-glusterfs: using<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">connect-path /var/run/gluster/quotad.socket<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:27.615218] T [name.c:111:af_unix_client_<wbr>bind]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-glusterfs: bind-path not specified for unix socket, letting connect to<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">assign default value<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:27.615329] T [cli-quotad-client.c:94:cli_<wbr>quotad_notify]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-glusterfs: got RPC_CLNT_CONNECT<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:27.615355] I [socket.c:2355:socket_event_<wbr>handler]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-transport: disconnecting now<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:27.615567] D<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[rpc-clnt-ping.c:98:rpc_clnt_<wbr>remove_ping_timer_locked] (--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libglusterfs.so.0(_gf_log_<wbr>callingfn+0x1a3)[<wbr>0x7f012fd21953]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libgfrpc.so.0(rpc_clnt_remove_<wbr>ping_timer_locked+0x84)[<wbr>0x7f012f69add4]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x55)[<wbr>0x7f012f697af5]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libgfrpc.so.0(rpc_clnt_notify+<wbr>0x88)[0x7f012f698338]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libgfrpc.so.0(rpc_transport_<wbr>notify+0x23)[0x7f012f6945b3]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">))))) 0-: /var/run/gluster/quotad.<wbr>socket: ping timer event already removed<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:27.615598] T [cli-quotad-client.c:100:cli_<wbr>quotad_notify]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-glusterfs: got RPC_CLNT_DISCONNECT<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:27.615612] D [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[event-epoll.c:591:event_<wbr>dispatch_epoll_handler] 0-epoll: generation bumped<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">on idx=1 from gen=4 to slot->gen=5, fd=7, slot->fd=7<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:30.615864] T [rpc-clnt.c:418:rpc_clnt_<wbr>reconnect]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-glusterfs: attempting reconnect<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:30.616113] T [socket.c:2879:socket_connect] (--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libglusterfs.so.0(_gf_log_<wbr>callingfn+0x1a3)[<wbr>0x7f012fd21953]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>glusterfs/3.7.6/rpc-transport/<wbr>socket.so(+0x6c1b)[<wbr>0x7f012a697c1b]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libgfrpc.so.0(rpc_clnt_<wbr>reconnect+0xb9)[<wbr>0x7f012f695999]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>libglusterfs.so.0(gf_timer_<wbr>proc+0xfc)[0x7f012fd3d70c]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(--> /lib/x86_64-linux-gnu/<wbr>libpthread.so.0(+0x76ba)[<wbr>0x7f012f0b86ba] )))))<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-glusterfs: connect () called on transport already connected<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">and from etc-glusterfs-glusterd.vol.log i see the probe request and then:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.707258] D [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[glusterd-peer-utils.c:133:<wbr>glusterd_peerinfo_find_by_<wbr>hostname]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-management: Unable to find friend: 10.1.0.7<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.707260] D [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[glusterd-peer-utils.c:247:<wbr>glusterd_peerinfo_find] 0-management: Unable to<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">find hostname: 10.1.0.7<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.707293] I [MSGID: 106129]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[glusterd-handler.c:3600:<wbr>glusterd_probe_begin] 0-glusterd: Unable to find<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">peerinfo for host: 10.1.0.7 (24007)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.707314] D [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[glusterd-peer-utils.c:458:<wbr>glusterd_peer_hostname_new] 0-glusterd:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Returning 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.707329] D [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[glusterd-utils.c:5974:<wbr>glusterd_sm_tr_log_init] 0-glusterd: returning 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.707353] D [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[glusterd-store.c:3802:<wbr>glusterd_store_create_peer_<wbr>dir] 0-glusterd:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Returning with 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.728550] D [MSGID: 0] [store.c:421:gf_store_handle_<wbr>new]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-: Returning 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.728679] D [MSGID: 0] [store.c:375:gf_store_save_<wbr>value]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-management: returning: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.728720] D [logging.c:1952:_gf_msg_<wbr>internal]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">to flush least recently used log message to disk<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">The message "D [MSGID: 0] [store.c:375:gf_store_save_<wbr>value] 0-management:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">returning: 0" repeated 2 times between [2017-04-03 22:20:24.728679] and<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.728718]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.728720] D [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[glusterd-store.c:3931:<wbr>glusterd_store_peer_write] 0-glusterd: Returning<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">with 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.786973] D [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[glusterd-store.c:3957:<wbr>glusterd_store_perform_peer_<wbr>store] 0-glusterd:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Returning 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787007] D [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[glusterd-store.c:3978:<wbr>glusterd_store_peerinfo] 0-glusterd: Returning with 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787046] D [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[glusterd-handler.c:3391:<wbr>glusterd_transport_inet_<wbr>options_build] 0-glusterd:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Returning 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787077] I [rpc-clnt.c:984:rpc_clnt_<wbr>connection_init]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-management: setting frame-timeout to 600<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787090] D [rpc-clnt.c:996:rpc_clnt_<wbr>connection_init]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-management: setting ping-timeout to 30<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787106] D [rpc-transport.c:288:rpc_<wbr>transport_load]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-rpc-transport: attempt to load file<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">/usr/lib/x86_64-linux-gnu/<wbr>glusterfs/3.7.6/rpc-transport/<wbr>socket.so<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787147] T [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[options.c:91:xlator_option_<wbr>validate_int] 0-management: no range check<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">required for 'option transport.tcp-user-timeout -1'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787166] T [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[options.c:91:xlator_option_<wbr>validate_int] 0-management: no range check<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">required for 'option transport.socket.keepalive-<wbr>time 10'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787182] T [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[options.c:91:xlator_option_<wbr>validate_int] 0-management: no range check<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">required for 'option transport.socket.keepalive-<wbr>interval 2'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787203] T [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[options.c:91:xlator_option_<wbr>validate_int] 0-management: no range check<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">required for 'option remote-port 24007'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787221] D [socket.c:3845:socket_init] 0-management:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Configued transport.tcp-user-timeout=-1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787233] D [socket.c:3928:socket_init] 0-management:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">SSL support on the I/O path is NOT enabled<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787242] D [socket.c:3931:socket_init] 0-management:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">SSL support for glusterd is NOT enabled<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787251] D [socket.c:3948:socket_init] 0-management:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">using system polling thread<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787263] T [rpc-clnt.c:418:rpc_clnt_<wbr>reconnect]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-management: attempting reconnect<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787273] T [socket.c:2887:socket_connect] 0-management:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">connecting 0x7ff7fc005640, state=0 gen=0 sock=-1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787288] T [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[common-utils.c:290:gf_<wbr>resolve_ip6] 0-resolver: DNS cache not present,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">freshly probing hostname: 10.1.0.7<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787843] D [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[common-utils.c:333:gf_<wbr>resolve_ip6] 0-resolver: returning ip-10.1.0.7<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">(port-24007) for hostname: 10.1.0.7 and port: 24007<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787863] D [socket.c:2833:socket_fix_ssl_<wbr>opts]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-management: disabling SSL for portmapper connection<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787883] T [socket.c:805:__socket_<wbr>nodelay]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-management: NODELAY enabled for socket 8<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787899] W [socket.c:869:__socket_<wbr>keepalive] 0-socket:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">failed to set TCP_USER_TIMEOUT -1000 on socket 8, Invalid argument<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.787910] E [socket.c:2965:socket_connect] 0-management:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Failed to set keep-alive: Invalid argument<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:20:24.788092] D [MSGID: 0]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[common-utils.c:2887:gf_ports_<wbr>reserved] 0-glusterfs: lower: 30000, higher:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">32767<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:21:52.911012] D [logging.c:1828:gf_log_flush_<wbr>timeout_cbk]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-logging-infra: Log timer timed out. About to flush outstanding messages<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">if present<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:21:52.911131] D [logging.c:1790:__gf_log_<wbr>inject_timer_event]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:22:24.707002] D [socket.c:558:__socket_rwv]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-socket.management: EOF on socket<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">[2017-04-03 22:22:24.707032] D [socket.c:2355:socket_event_<wbr>handler]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">0-transport: disconnecting now<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">I've debugged this to the best of my ability but haven't made any progress<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">narrowing it down, nor has google turned up anything useful.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Thanks for the help!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Ken<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>