[Gluster-users] Network Topology Question

[Joe Julian]

This is a concept that needs some work to implement, but could 
potentially see a benefit from that topography: 
https://docs.google.com/document/d/15IiPVIPMzgGwkt1sKuIusRE2l3pQY6NnA4WmWaMsFJE/edit?usp=sharing

On 05/22/2015 09:02 AM, Jeff Darcy wrote:
> ----- Original Message -----
>
>> Hi there,
>> I'm planning to setup a 3-Node Cluster for oVirt and would like to use 56 GBe
>> (RoCe)
>> exclusively for GlusterFS. Since 56 GBe switches are far too expensive and
>> it's not
>> planned to add more nodes and furthermore this would add a SPOF I'd like to
>> cross connect the nodes as shown in the diagram below:
>> Node 1 Node 2 Node3
>> ||_______||________||
>> |___________________|
>> This way there's a dedicated 56 Gbit connection to/from each member node.
>> Is is possible to do this with GlusterFS?
>> My first thought was to have different IPs on each node's /etc/host mapped to
>> the node
>> hostnames but I'm unsure if I can force GlusterFS to hostnames instead of
>> IPs.
> There are two ways you can do this.  Both involve asymmetric configurations.
> Imagine that you have three subnets, one per wire:
>
>    192.168.1.1 and 192.168.1.2 between Node1 and Node2
>    192.168.2.1 and 192.168.2.2 between Node2 and Node3
>    192.168.3.1 and 192.168.3.2 between Node1 and Node3
>
> So, /etc/hosts on Node1 would look like this:
>
>    192.168.1.2 node2
>    192.168.3.2 node3
>
> On Node2 you'd have this:
>
>    192.168.1.1 node1
>    192.168.2.2 node3
>
> And so on.  Note that these are all different than the clients, which would
> have entries (probably in DNS rather than /etc/hosts) for the servers'
> slower external addresses.
>
> The other way to do the same thing is with explicit host routes or iptables
> rules.  In that kind of setup, you put each server into its own subnet,
> then add routes on the others to go through the interfaces you want.  For
> example:
>
>    node1 is 172.30.16.1
>    node2 is 172.30.17.1
>    node3 is 172.30.18.1
>
> Therefore, on node1 (using the interface addresses above):
>
>    route add -host node2 gw 192.168.1.2
>    route add -host node3 gw 192.168.3.2
>
> On node2:
>
>    route add -host node1 gw 192.168.1.1
>    route add -host node3 gw 192.168.2.2
>
> And so on, again.  Don't forget to turn on IP forwarding.  Also, this
> still requires that the servers have a different /etc/hosts than clients,
> but at least it can be the same across all servers.  Alternatively, you
> could use the same /etc/hosts (or DNS) everywhere, if you can add routes
> on the clients as well.
>
> All that said, the benefit of such a configuration is rather limited.
> Using FUSE or GFAPI, replication will still occur over the slow client
> network because it's being driven by the clients (this is likely to
> change in 4.0).  On the other hand, self-heal and rebalance traffic
> will use the faster internal network.  SMB and NFS will use both, so
> they might see some benefit in *aggregate* but not per-client
> throughput.  Depending on your usage pattern, the extra complexity of
> setting up this kind of routing might not be worth the effort.
>
>
>     
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users