[Gluster-devel] Multi-network support proposal

[Jeff Darcy]

> 1) Do we have to specify each host's address rewriting in your example - why
> not something like this?
> 
> # gluster network add client-net 1.2.3.0/24
> 
> glusterd could then use a discovery process as I described earlier to
> determine for each server what its IP address is on that subnet and rewrite
> volfiles accordingly.
> 
> The advantage of this subnet-based specification IMHO is that it scales - as
> you add and remove nodes, you do not have to change "client-net" entity, you
> just make sure that Gluster servers provide the appropriate network
> interface with appropriate IP address and subnet mask.

Good idea.  For most cases this would be sufficient, and it's certainly
simpler.  As long as we allow multiple address ranges to be associated
with a network (and a single address is just a very small range) we'll
still be able to handle the more complex cases as well.

> 2) Could we keep the number of roles and the sysadmin interface in general
> from getting too complicated?  Here's an oversimplified model of Gluster
> networking - there are at most 2 kinds of subnets on each server in use by
> Gluster or apps:
> 
> - replication subnet - this is the subnet used at volume creation time.  It
> is used by any background activity involving communication between servers,
> including NSR for replication traffic, self-heal, DHT rebalance.
> - "access" subnet(s?) - used by all access protocols, including glusterfs,
> NFS, SMB, Swift, client-side libgfapi, geo-replication, etc.

One way to do this would be to support macros or wildcards, so that e.g.
"all-server" or "server-*" would match all of the roles you've identified
as the replication subnet (assuming that they all have a "server-" prefix).
We could support volume wildcards as well, so this would probably be a
pretty common setup:

    gluster network route * server-* back-end-net
    gluster network route * client-* front-end-net

> In some sites, we can use a single subnet that does both "roles", but
> separation of these two types of traffic can reduce network contention and
> latency and may be desirable for security reasons, etc.
> 
> If you want to get more throughput for either of these subnets, you can use
> Linux bonding to do so.

Bonding is about *combining* traffic; this is really more about *separating*
it.  There's no reason someone couldn't use both.