[Gluster-users] Unable to peer probe between new servers and old servers

[Krishnan Parthasarathi]

[Sorry for replying to multiple mails in the same thread]

----- Original Message -----
> I have two production machines (web1, web2) that are currently using
> Glusterfs. I added two new machines, web3 and web4. Web1 and web2 are peered
> and are running great. Web3 and web4 will peer with each other but web1 and
> web2 will not peer with web3 or web4 and vice versa.

Did you add web3 and web4 using the following procedure?

from web1 (or web2),
web1# gluster peer probe web3
web1# gluster peer probe web4

Only a peer that is (already) part of a cluster can add a new peer to
the (same) cluster. i.e a peer cannot seek membership into the cluster,
only a member can 'invite' a new peer.

HTH,
kp

> 
> All machines are running Gluster 3.6.1:
> ii glusterfs-client 3.6.1-1 amd64 clustered file-system (client package)
> ii glusterfs-common 3.6.1-1 amd64 GlusterFS common libraries and translator
> modules
> ii glusterfs-server 3.6.1-1 amd64 clustered file-system (server package)
> 
> I ran "gluster volume set all cluster.op-version 30600" on web2, now all four
> machines have "operating-version=30600" in /var/lib/glusterd/ glusterd.info
> 
> All machines have the same kernel:
> Linux web1 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08) x86_64
> GNU/Linux
> 
> /etc/hosts is managed with Ansible, all machines resolve by their names
> (web1, web2, web3, web4) to their internal IP.
> 
> "hping3 -8 111,2049,24007,49152-49162 -S [web1-web4]" shows all Gluster ports
> open from all servers.
> 
> root at web1:~# gluster peer probe web2
> peer probe: success. Host web2 port 24007 already in peer list
> root at web1:~# gluster peer probe web3
> peer probe: failed: Probe returned with unknown errno 107
> root at web1:~# gluster peer probe web4
> peer probe: failed: Probe returned with unknown errno 107
> 
> 
> root at web3:~# gluster peer probe web1
> peer probe: failed: Probe returned with unknown errno 107
> root at web3:~# gluster peer probe web2
> peer probe: failed: Probe returned with unknown errno 107
> root at web3:~# gluster peer probe web4
> peer probe: success. Host web4 port 24007 already in peer list
> 
> 
> from /var/log/glusterfs/cli.log:
> 
> [2015-01-07 20:38:33.978304] T [cli.c:264:cli_rpc_notify] 0-glusterfs: got
> RPC_CLNT_CONNECT
> [2015-01-07 20:38:33.978322] T [cli-quotad-client.c:94:cli_quotad_notify]
> 0-glusterfs: got RPC_CLNT_CONNECT
> [2015-01-07 20:38:33.978333] I [socket.c:2344:socket_event_handler]
> 0-transport: disconnecting now
> [2015-01-07 20:38:33.978354] T [cli-quotad-client.c:100:cli_quotad_notify]
> 0-glusterfs: got RPC_CLNT_DISCONNECT
> [2015-01-07 20:38:33.978622] T [rpc-clnt.c:1381:rpc_clnt_record] 0-glusterfs:
> Auth Info: pid: 0, uid: 0, gid: 0, owner:
> [2015-01-07 20:38:33.978681] T [rpc-clnt.c:1238:rpc_clnt_record_build_header]
> 0-rpc-clnt: Request fraglen 144, payload: 80, rpc hdr: 64
> [2015-01-07 20:38:33.979037] T [socket.c:2863:socket_connect] (-->
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1cd)[0x7f9e8ef0ac1d]
> (-->
> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.1/rpc-transport/socket.so(+0x637c)[0x7f9e8bcc637c]
> (-->
> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x3a4)[0x7f9e8e40fad4]
> (--> gluster(cli_submit_request+0x15f)[0x7f9e8f3c52df] (-->
> gluster(cli_cmd_submit+0x8b)[0x7f9e8f3c6eeb] ))))) 0-glusterfs: connect ()
> called on transport already connected
> [2015-01-07 20:38:33.979100] T [rpc-clnt.c:1573:rpc_clnt_submit] 0-rpc-clnt:
> submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 1) to
> rpc-transport (glusterfs)
> [2015-01-07 20:38:33.979111] D [rpc-clnt-ping.c:231:rpc_clnt_start_ping]
> 0-glusterfs: ping timeout is 0, returning
> [2015-01-07 20:38:33.986269] T [rpc-clnt.c:660:rpc_clnt_reply_init]
> 0-glusterfs: received rpc message (RPC XID: 0x1 Program: Gluster CLI,
> ProgVers: 2, Proc: 1) from rpc-transport (glusterfs)
> [2015-01-07 20:38:33.986311] I [cli-rpc-ops.c:131:gf_cli_probe_cbk] 0-cli:
> Received resp to probe
> [2015-01-07 20:38:33.986321] E [cli-rpc-ops.c:136:gf_cli_probe_cbk] 0-cli:
> Probe returned with unknown errno 107
> [2015-01-07 20:38:33.986620] D [cli-cmd.c:384:cli_cmd_submit] 0-cli:
> Returning -1
> [2015-01-07 20:38:33.986638] D [cli-rpc-ops.c:3021:gf_cli_probe] 0-cli:
> Returning -1
> [2015-01-07 20:38:33.986650] D [cli-cmd-peer.c:96:cli_cmd_peer_probe_cbk]
> 0-cli: frame->local is not NULL (0x7f9e7c000980)
> [2015-01-07 20:38:33.986667] I [input.c:36:cli_batch] 0-: Exiting with: -1
> 
> 
> 
> Thanks in advance for any help.
> 
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users