[Gluster-users] make gluster run on internal interfaces only

[Stefan Becker]

Hi together,

I still did not manage gluster to run on internal interfaces only. With public interface I had it running in like 30 minutes and it worked like a charm. Then I changed the bind-address for glusterd and I just cannot probe the other server. I did a debian dist-upgrade today and removed all gluster packages (I wanted to start from scratch). Then I reinstalled a brand new .deb downloaded today from gluster.org.

Here is my setup:

+++ Node 1 +++

s1-new:~# cat /etc/hostname
s1-new
s1-new:~#
s1-new:~# cat /etc/hosts
127.0.0.1       localhost
10.10.100.31    s1-new
10.10.100.223   s2-new

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
s1-new:~#
s1-new:~# cat /etc/glusterfs/glusterd.vol
volume management
    type mgmt/glusterd
    option working-directory /etc/glusterd
    option transport-type tcp
    option transport.socket.bind-address 10.10.100.31
    option transport.socket.keepalive-time 10
    option transport.socket.keepalive-interval 2
end-volume

s1-new:~#
s1-new:~# netstat -lpn | grep gluster
tcp        0      0 10.10.100.31:24007      0.0.0.0:*               LISTEN      2336/glusterd

+++ Node 2 +++

s2-new:~# cat /etc/hostname
s2-new
s2-new:~#
s2-new:~# cat /etc/hosts
127.0.0.1       localhost
10.10.100.31    s1-new
10.10.100.223   s2-new

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
s2-new:~#
s2-new:~# cat /etc/glusterfs/glusterd.vol
volume management
    type mgmt/glusterd
    option working-directory /etc/glusterd
    option transport-type tcp
    option transport.socket.bind-address 10.10.100.223
    option transport.socket.keepalive-time 10
    option transport.socket.keepalive-interval 2
end-volume

s2-new:~#
s2-new:~# netstat -lpn | grep gluster
tcp        0      0 10.10.100.223:24007     0.0.0.0:*               LISTEN      1850/glusterd

Both nodes/daemons are reachable from each other:

s1-new:~# ping s2-new
PING s2-new (10.10.100.223) 56(84) bytes of data.
64 bytes from s2-new (10.10.100.223): icmp_req=1 ttl=64 time=0.099 ms
64 bytes from s2-new (10.10.100.223): icmp_req=2 ttl=64 time=0.093 ms
^C
--- s2-new ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.093/0.096/0.099/0.003 ms
s1-new:~#
s1-new:~# telnet s2-new 24007
Trying 10.10.100.223...
Connected to s2-new.
Escape character is '^]'.
Connection closed by foreign host.

s2-new:~# ping s1-new
PING s1-new (10.10.100.31) 56(84) bytes of data.
64 bytes from s1-new (10.10.100.31): icmp_req=1 ttl=64 time=0.096 ms
64 bytes from s1-new (10.10.100.31): icmp_req=2 ttl=64 time=0.092 ms
s2-new:~#
s2-new:~# telnet s1-new 24007
Trying 10.10.100.31...
Connected to s1-new.
Escape character is '^]'.
Connection closed by foreign host.

But probing just does not work:

s1-new:~# gluster peer probe s2-new
Connection failed. Please check if gluster daemon is operational.
s1-new:~#

And vice versa:

s2-new:~# gluster peer probe s1-new
Connection failed. Please check if gluster daemon is operational.
s2-new:~#

Next I straced those calls to see what they do:

s1-new:~# cat strace.out | grep 'connect('
connect(4, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
connect(4, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
connect(4, {sa_family=AF_INET, sin_port=htons(24007), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
connect(4, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
connect(4, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
connect(4, {sa_family=AF_INET, sin_port=htons(24007), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
s1-new:~#

So I think its not working because the probe process is first trying to connect to the local glusterd which is not listening under 127.0.0.1?

I have no idea, please help me. My two servers are totally naked if you need more information just ask, thank you!

Bye,
Stefan