[Gluster-users] Unable to peer probe between new servers and old servers

Wed Jan 7 23:19:12 UTC 2015

Saw the ping command, but I'm not familiar with that and port checking
which is why I asked about telnet.

If you turn off the Ubuntu firewall as a test, does that help?

On Wed, Jan 7, 2015 at 4:41 PM, Michael Bushey <michael at realtymogul.com>
wrote:

> Hi Ryan.  Thanks for responding. :) Yes, as I mentioned fping3 shows all
> ports open to all machines.
>
> root at web3:~# telnet web1 24007
> Trying 10.10.x.x...
> Connected to web1.
> Escape character is '^]'.
>
> root at web3:~# traceroute web1
> traceroute to web1 (10.10.x.x), 30 hops max, 60 byte packets
>  1  10.11.0.1 (10.11.0.1)  0.967 ms  0.968 ms  0.960 ms
>  2  web1 (10.10.x.x)  0.581 ms  0.621 ms  0.665 ms
>
> ufw status
> ....
> glusterfs                  ALLOW       10.10.0.0/16
> glusterfs                  ALLOW       10.11.0.0/16
>
>
> cat /etc/ufw/applications.d/ufw-glusterfs
> [glusterfs]
> title=GlusterFS
> description=GlusterFS is an open source, distributed file system capable
> of scaling to several petabytes
> ports=111,2049,24007,49152:49162/tcp|111/udp
>
>
>
>
>
> On Wed, Jan 7, 2015 at 2:05 PM, Ryan Nix <ryan.nix at gmail.com> wrote:
>
>> Have you been able to telnet to the Gluster ports on the other peers?  If
>> you run a traceroute to the affected peers, what does it return?
>>
>> On Wed, Jan 7, 2015 at 3:44 PM, Michael Bushey <michael at realtymogul.com>
>> wrote:
>>
>>> I have two production machines (web1, web2) that are currently using
>>> Glusterfs. I added two new machines, web3 and web4. Web1 and web2 are
>>> peered and are running great. Web3 and web4 will peer with each other but
>>> web1 and web2 will not peer with web3 or web4 and vice versa.
>>>
>>> All machines are running Gluster 3.6.1:
>>> ii  glusterfs-client               3.6.1-1
>>> amd64        clustered file-system (client package)
>>> ii  glusterfs-common               3.6.1-1
>>> amd64        GlusterFS common libraries and translator modules
>>> ii  glusterfs-server               3.6.1-1
>>> amd64        clustered file-system (server package)
>>>
>>> I ran "gluster volume set all cluster.op-version 30600" on web2, now all
>>> four machines have "operating-version=30600" in /var/lib/glusterd/
>>> glusterd.info
>>>
>>> All machines have the same kernel:
>>> Linux web1 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08)
>>> x86_64 GNU/Linux
>>>
>>> /etc/hosts is managed with Ansible, all machines resolve by their names
>>> (web1, web2, web3, web4) to their internal IP.
>>>
>>> "hping3 -8 111,2049,24007,49152-49162 -S [web1-web4]" shows all Gluster
>>> ports open from all servers.
>>>
>>> root at web1:~# gluster peer probe web2
>>> peer probe: success. Host web2 port 24007 already in peer list
>>> root at web1:~# gluster peer probe web3
>>> peer probe: failed: Probe returned with unknown errno 107
>>> root at web1:~# gluster peer probe web4
>>> peer probe: failed: Probe returned with unknown errno 107
>>>
>>>
>>> root at web3:~# gluster peer probe web1
>>> peer probe: failed: Probe returned with unknown errno 107
>>> root at web3:~# gluster peer probe web2
>>> peer probe: failed: Probe returned with unknown errno 107
>>> root at web3:~# gluster peer probe web4
>>> peer probe: success. Host web4 port 24007 already in peer list
>>>
>>>
>>> from /var/log/glusterfs/cli.log:
>>>
>>> [2015-01-07 20:38:33.978304] T [cli.c:264:cli_rpc_notify] 0-glusterfs:
>>> got RPC_CLNT_CONNECT
>>> [2015-01-07 20:38:33.978322] T
>>> [cli-quotad-client.c:94:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT
>>> [2015-01-07 20:38:33.978333] I [socket.c:2344:socket_event_handler]
>>> 0-transport: disconnecting now
>>> [2015-01-07 20:38:33.978354] T
>>> [cli-quotad-client.c:100:cli_quotad_notify] 0-glusterfs: got
>>> RPC_CLNT_DISCONNECT
>>> [2015-01-07 20:38:33.978622] T [rpc-clnt.c:1381:rpc_clnt_record]
>>> 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner:
>>> [2015-01-07 20:38:33.978681] T
>>> [rpc-clnt.c:1238:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen
>>> 144, payload: 80, rpc hdr: 64
>>> [2015-01-07 20:38:33.979037] T [socket.c:2863:socket_connect] (-->
>>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1cd)[0x7f9e8ef0ac1d]
>>> (-->
>>> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.1/rpc-transport/socket.so(+0x637c)[0x7f9e8bcc637c]
>>> (-->
>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x3a4)[0x7f9e8e40fad4]
>>> (--> gluster(cli_submit_request+0x15f)[0x7f9e8f3c52df] (-->
>>> gluster(cli_cmd_submit+0x8b)[0x7f9e8f3c6eeb] ))))) 0-glusterfs: connect ()
>>> called on transport already connected
>>> [2015-01-07 20:38:33.979100] T [rpc-clnt.c:1573:rpc_clnt_submit]
>>> 0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2,
>>> Proc: 1) to rpc-transport (glusterfs)
>>> [2015-01-07 20:38:33.979111] D [rpc-clnt-ping.c:231:rpc_clnt_start_ping]
>>> 0-glusterfs: ping timeout is 0, returning
>>> [2015-01-07 20:38:33.986269] T [rpc-clnt.c:660:rpc_clnt_reply_init]
>>> 0-glusterfs: received rpc message (RPC XID: 0x1 Program: Gluster CLI,
>>> ProgVers: 2, Proc: 1) from rpc-transport (glusterfs)
>>> [2015-01-07 20:38:33.986311] I [cli-rpc-ops.c:131:gf_cli_probe_cbk]
>>> 0-cli: Received resp to probe
>>> [2015-01-07 20:38:33.986321] E [cli-rpc-ops.c:136:gf_cli_probe_cbk]
>>> 0-cli: Probe returned with unknown errno 107
>>> [2015-01-07 20:38:33.986620] D [cli-cmd.c:384:cli_cmd_submit] 0-cli:
>>> Returning -1
>>> [2015-01-07 20:38:33.986638] D [cli-rpc-ops.c:3021:gf_cli_probe] 0-cli:
>>> Returning -1
>>> [2015-01-07 20:38:33.986650] D
>>> [cli-cmd-peer.c:96:cli_cmd_peer_probe_cbk] 0-cli: frame->local is not NULL
>>> (0x7f9e7c000980)
>>> [2015-01-07 20:38:33.986667] I [input.c:36:cli_batch] 0-: Exiting with:
>>> -1
>>>
>>>
>>>
>>> Thanks in advance for any help.
>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>
>
> --
> *Michael Bushey*
> Director of Technology
> Realty Mogul, Co.
> michael at realtymogul.com
> Cell: (310) 619-0414
> via Front Desk: (310) 907-7129 (ext. 124)
>
>
> *Realty Mogul in 19 words:*  Realty Mogul is a marketplace for accredited
> investors to pool money online and buy shares of pre-vetted investment
> properties.
>
> Please Note: This email, including attachments, is intended only for the
> addresses(s) indicated, and may include non-public, proprietary,
> confidential or legally privileged information.  If you are not an intended
> recipient or an authorized agent of an intended recipient, you are hereby
> notified that any dissemination, distribution or copying of the information
> contained in or transmitted with this e-mail is unauthorized and strictly
> prohibited.  If you have received this email in error, delete it and any
> attachments or copies immediately.
>
> Realty Mogul does not make investment recommendations, and this message
> should not be construed as such.  This message is not an offer to sell or
> the solicitation of an offer to buy any security or interest in any fund,
> which only can be made through official offering documents that contain
> important information about risks, fees and expenses.  Any investment
> information contained herein has been secured from sources Realty Mogul
> believes are reliable, but we make no representations or warranties as to
> the accuracy of such information and accept no liability therefor.
> RealtyMogul offers equity securities through WealthForge, LLC
> <http://www.wealthforgeholdings.com>, member FINRA <http://www.finra.org>/
> SIPC <http://www.sipc.org>.  Realty Mogul operates in California under
> DRE license #01926613.  We recommend that you consult with a financial
> advisor, attorney, accountant, and any other professional that can help you
> to understand and assess the risks associated with any investment
> opportunity.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150107/057e9019/attachment.html>