[Gluster-users] Gluster using multiple VLANs for traffic separation clients <-> servers

Thu Apr 15 09:40:29 UTC 2021

Hi to all,

i'm sorry for the long post, but just to explain all we tried to reach 
our goal.

We are new to gluster and we are trying to create a new gluster 
infrastructure, we are using gluster v9.1 on ubuntu servers.

Our mail goal is to separate/balance traffic from clients <-> servers 
(by volume or subvolumes if possible or using different instances of 
gluster daemon) using several VLANs (with different IP address).

Just to clear about it, we have:

1) For each gluster node, 5 ip address on 2 interface 10GB in bonding mode:

first IP on VLAN 1 default, "public ip" on subnet 192.168.30.0/24, 
gluster{01..12} (name of the servers), used only for management via SSH

second IP on VLAN 260 server, "private ip on subnet 172.26.0.0/24", 
glustersrv{01..12} (name of the servers), used for server <-> server 
traffic only

third IP on VLAN 261 web, "private ip on subnet 172.26.1.0/24", 
glusterweb{01..12} (name of the servers), used for servers <-> clients 
traffic only for specific volumes/subvolumes

fourth IP on VLAN 262 sys, "private ip on subnet 172.26.2.0/24", 
glustersys{01..12} (name of the servers), used for servers <-> clients 
traffic only for specific volumes/subvolumes

fifth IP on VLAN 263 share, "private ip on subnet 172.26.3.0/24" , 
glustershare{01..12} (name of the servers), used for servers <-> clients 
traffic only for specific volumes/subvolumes

2) VLAN 260,261,262 and 263 are not routed on the switch, so only switch 
ports mapped to those VLAN can talk with other "nodes" on the same VLAN

3) For each client, 2 ip address on 2 interface:

first IP on VLAN 1 default, "public ip", glustercli{01..04} (name of the 
clients), used only for management via SSH

second IP on VLAN 261/262/263 web/sys/share, 
glustercliweb/sys/share{01..04} (name of the clients), used for servers 
<-> clients traffic only for specific volumes/subvolumes

4) We created the cluster adding peer using names glustersrv{01..12} to 
ensure each server is talking to eachother using the IP VLAN server

root at gluster01:~# netstat -ant | grep -i list
tcp        0      0 0.0.0.0:24007 0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:6010 0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:49152 0.0.0.0:*               LISTEN

All the gluster services are listening to 0.0.0.0:49152 (so here i don't 
see any problem connecting to them using several IP/VLANs).

5) We created a volume web01 using names glustersrv{01..12} also because 
if we tried to use glusterweb{01..12} we get

root at gluster01:~# gluster volume create scratch3 disperse-data 8 
redundancy 4 transport tcp glusterweb{01..12}:/mnt/hdd4a/area1/web01

volume create: web01: failed: Host glusterweb03 (this name changes each 
time we try to use this command) is not in 'Peer in Cluster' state

and if we try to add glusterweb03 like a peer we get

root at gluster01:~# gluster peer probe glusterweb03

peer probe: Host glusterweb03 port 24007 already in peer list (i suppose 
because the daemon is listening on each address)

6) Finally we tried to mount that volume from several clients who are 
attached to the VLAN 261 web, using

root at glustercli01:~# mount.glusterfs glusterweb01:/scratch /mnt

Mounting glusterfs on /mnt failed.     (after about 1 minute it failed)

so we tried instead

root at glustercli01:~# mount.glusterfs glustersrv01:/scratch /mnt

Mounting glusterfs on /mnt failed.     (it fails instantly but i 
understand this because glustersrv01 is on another subnet/VLAN 260 that 
the clients can't reach anyway)

7) To understand why it was not working using IP VLAN 261 names 
glusterweb{01..12} we enabled IP routing on the VLAN 260 and magically 
the command

root at glustercli01:~# mount.glusterfs glustersrv01:/scratch /mnt

was successfull instantly

So we suppose that the client reach the gluster nodes, but the gluster 
node that received the request from the client sends to the client the 
servernames/IPs using glustersrv{01..12} using the gluster peer names we 
used in the creation of the cluster Gluster and the client (that without 
routing can't reach the servers) doesn't receive information back.

We tried to search some informations about Gluster with multihomed 
servers, or Gluster with multiple instances on the same servers but we 
found only old documentation and also some proposal for v4.0 of gluster 
multihomed but nothing more, this is a link for multiple instances

https://bugzilla.redhat.com/show_bug.cgi?id=913555

Instead we found some webpages talking about using glusterd.vol file to 
make listening on different binding address but no examples for doing it 
and i don't know if that will work for our use case, these are some 
links for glusterd.vol configuration

https://serverfault.com/questions/864634/how-can-i-bind-glusterfsd-process-to-an-interface-ip

http://netkiller.sourceforge.net/linux/storage/gluster.html

I tried to replicate my scenario but glusterd doesn't start when i add 
the volume server section (i tried only modifying glusterd.vol on a 
server could be that the problem ?).

I know we could use Gluster-NFS to trying to do that but we would like 
to use gluster native client directly, now my questions are:

a) is there a way to add Peer using different IPs for each servers ? in 
this way will we get the features we are especting clients <-> 
communication on different VLANs ?

b) if a is not possible, is there a way to implent what we would like to 
have with gluster, client <-> server communication using different 
interfaces/ip/vlan ? If yes how ?

c) if a and b are not possible, is there a way to share a volume on a 
specific IP address/network ? if yes how ?

d) if a, b and c are not possible, is there a way to have multiple 
instances of gluster running on the same server so we can create 
different clusters using in the peer probe different names for each 
cluster ?

e) if a,b,c and e are not possible, glusterd.vol is a possible solution 
? if yes where we can find some examples or documentation about it ? we 
tried but we can't find any docs

Thanks anyway for each hints or for an answer.

Claudio

-- 

        /        |    /   _____/           /    |    /   _____/    |    /
       /       / |   /   /                /   / |   /   /        / |   /
      /       /  |  /   ___/   _____/    /   /  |  /   ___/     /  |  /
     /       /   | /   /                /   /   | /   /        /   | /
   ______/ _/   __/  _/               _/  _/   __/  _/       _/   __/

Claudio Soprano                phone:  (+39)-06-9403.2349/2355
Computing Service              fax:    (+39)-06-9403.2649
LNF-INFN                       e-mail: Claudio.Soprano at lnf.infn.it
Via Enrico Fermi, 40           www:    http://www.lnf.infn.it/
I-00044 Frascati, Italy