[Gluster-devel] Multi-network support proposal

[Jeff Darcy]

This is a proposal for a "lightweight" version of multi-network support,
somewhat limited in functionality but implementable quickly because it
seems unlikely that anyone will be able to spend much time on a full
version.  Here's the corresponding feature page:

http://www.gluster.org/community/documentation/index.php/Features/SplitNetwork

There are three things a user must be able to do:

 * Create new named networks (a "default" network always exists)

 * Associate specific host interface names/addresses with networks

 * Associate volume roles (e.g. I/O, rebalance) with networks

The crux of this proposal is to do what we can by rewriting volfiles
between when they're fetched and when they're used to construct a
translator graph.  Each protocol/client translator within a volfile
originally uses a server's *canonical* name - on the default network,
used for "peer probe" and other glusterd-to-glusterd operations.  For
each such translator, the volfile consumer must do the following:

 (1) Determine its own role (e.g. client, self-heal daemon)

 (2) Use the {volume,role} tuple to find the right *network* to use

 (3) Use the {network,canonical_name} tuple to find the right
     *interface* to use

 (4) Replace the server's canonical name ("remote-host" option) with the
     interface name

For example, consider the following (don't worry about the syntax yet).

  gluster volume create fubar server1:/brick server2:/brick

  gluster network create client-net

  gluster network associate client-net server1 server1-public

  gluster network associate client-net server2 server2-public

  gluster volume set-role fubar mounts client-net

According to step (2) above, any native mount of fubar will know to use
client-net.  Therefore, according to step (3) it should make the
following substitutions:

  server1 => server1-public

  server2 => server2-public

By contrast, glusterd would still use the plain old "server1" and
"server2" on the "default" network.

Three kinds of enhancements are being left to the future:

 (a) Determine the network to use based on *client* identity as well as
     {volume,role}

 (b) Allow a volume to be used only through certain networks

 (c) Modify other parameters (e.g. SSL usage/credentials) based on
     network or interface

Any other thoughts?