[Gluster-users] Network Topology Question

[Jeff Darcy]

----- Original Message ----- 

> Hi there,

> I'm planning to setup a 3-Node Cluster for oVirt and would like to use 56 GBe
> (RoCe)
> exclusively for GlusterFS. Since 56 GBe switches are far too expensive and
> it's not
> planned to add more nodes and furthermore this would add a SPOF I'd like to
> cross connect the nodes as shown in the diagram below:

> Node 1 Node 2 Node3
> ||_______||________||
> |___________________|

> This way there's a dedicated 56 Gbit connection to/from each member node.

> Is is possible to do this with GlusterFS?
> My first thought was to have different IPs on each node's /etc/host mapped to
> the node
> hostnames but I'm unsure if I can force GlusterFS to hostnames instead of
> IPs.

There are two ways you can do this.  Both involve asymmetric configurations.
Imagine that you have three subnets, one per wire:

  192.168.1.1 and 192.168.1.2 between Node1 and Node2
  192.168.2.1 and 192.168.2.2 between Node2 and Node3
  192.168.3.1 and 192.168.3.2 between Node1 and Node3

So, /etc/hosts on Node1 would look like this:

  192.168.1.2 node2
  192.168.3.2 node3

On Node2 you'd have this:

  192.168.1.1 node1
  192.168.2.2 node3

And so on.  Note that these are all different than the clients, which would
have entries (probably in DNS rather than /etc/hosts) for the servers'
slower external addresses.

The other way to do the same thing is with explicit host routes or iptables
rules.  In that kind of setup, you put each server into its own subnet,
then add routes on the others to go through the interfaces you want.  For
example:

  node1 is 172.30.16.1
  node2 is 172.30.17.1
  node3 is 172.30.18.1

Therefore, on node1 (using the interface addresses above):

  route add -host node2 gw 192.168.1.2
  route add -host node3 gw 192.168.3.2

On node2:

  route add -host node1 gw 192.168.1.1
  route add -host node3 gw 192.168.2.2

And so on, again.  Don't forget to turn on IP forwarding.  Also, this
still requires that the servers have a different /etc/hosts than clients,
but at least it can be the same across all servers.  Alternatively, you
could use the same /etc/hosts (or DNS) everywhere, if you can add routes
on the clients as well.

All that said, the benefit of such a configuration is rather limited.
Using FUSE or GFAPI, replication will still occur over the slow client
network because it's being driven by the clients (this is likely to
change in 4.0).  On the other hand, self-heal and rebalance traffic
will use the faster internal network.  SMB and NFS will use both, so
they might see some benefit in *aggregate* but not per-client
throughput.  Depending on your usage pattern, the extra complexity of
setting up this kind of routing might not be worth the effort.