[Gluster-devel] Multi-network support proposal

[Ben England]

It's really important for glusterfs not to require that the clients mount volumes using same subnet that is used by servers, and clearly your very general-purpose proposal could address that.  For example, in a site where non-glusterfs protocols are used, there are already good reasons for using multiple subnets, and we want glusterfs to be able to coexist with non-glusterfs protocols at a site.

However, is there a simpler way to allow glusterfs clients to connect to servers through more than one subnet.  For example, suppose your Gluster volume subnet is 172.17.50.0/24 and your "public" network used by glusterfs clients is 1.2.3.0/22, but one of the servers also has an interface on subnet 4.5.6.0/24 .  So at the time that the volume is either created or bricks are added/removed:

- determine what servers are actually in the volume
- ask each server to return the subnet for each of its active network interfaces 
- determine set of subnets that are directly accessible to ALL the volume's servers
- write a glusterfs volfile for each of these subnets and save it

This process is O(N) where N is number of servers, but it only happens for volume creation or addition/removal of bricks, these events do not happen very often (do they?).  In example, 1.2.3.0/22 and 172.17.50.0/24 would have glusterfs volfiles, but 4.5.6.0/22 would not.

So now when a client connects, the server knows which subnet the request came through (getsockaddr), so it can just return the volfile for that subnet.  If there is no volfile for that subnet, the client mount request is rejected..  But what about existing Gluster volumes?  When software is upgraded, we should provide a mechanism for triggering this volfile generation process to open up additional subnets for glusterfs clients.

This proposal requires additional work to be done where volfiles are generated and where glusterfs mount processing is done, but does not require any additional configuration commands or extra user knowledge of Gluster.  glusterfs clients can then use *any* subnet that is accessible to all the servers.  

----- Original Message -----
> From: "Jeff Darcy" <jdarcy at redhat.com>
> To: "Gluster Devel" <gluster-devel at gluster.org>
> Sent: Friday, February 13, 2015 10:57:54 PM
> Subject: [Gluster-devel] Multi-network support proposal
> 
> This is a proposal for a "lightweight" version of multi-network support,
> somewhat limited in functionality but implementable quickly because it
> seems unlikely that anyone will be able to spend much time on a full
> version.  Here's the corresponding feature page:
> 
> http://www.gluster.org/community/documentation/index.php/Features/SplitNetwork
> 
> There are three things a user must be able to do:
> 
>  * Create new named networks (a "default" network always exists)
> 
>  * Associate specific host interface names/addresses with networks
> 
>  * Associate volume roles (e.g. I/O, rebalance) with networks
> 
> The crux of this proposal is to do what we can by rewriting volfiles
> between when they're fetched and when they're used to construct a
> translator graph.  Each protocol/client translator within a volfile
> originally uses a server's *canonical* name - on the default network,
> used for "peer probe" and other glusterd-to-glusterd operations.  For
> each such translator, the volfile consumer must do the following:
> 
>  (1) Determine its own role (e.g. client, self-heal daemon)
> 
>  (2) Use the {volume,role} tuple to find the right *network* to use
> 
>  (3) Use the {network,canonical_name} tuple to find the right
>      *interface* to use
> 
>  (4) Replace the server's canonical name ("remote-host" option) with the
>      interface name
> 
> For example, consider the following (don't worry about the syntax yet).
> 
>   gluster volume create fubar server1:/brick server2:/brick
> 
>   gluster network create client-net
> 
>   gluster network associate client-net server1 server1-public
> 
>   gluster network associate client-net server2 server2-public
> 
>   gluster volume set-role fubar mounts client-net
> 
> According to step (2) above, any native mount of fubar will know to use
> client-net.  Therefore, according to step (3) it should make the
> following substitutions:
> 
>   server1 => server1-public
> 
>   server2 => server2-public
> 
> By contrast, glusterd would still use the plain old "server1" and
> "server2" on the "default" network.
> 
> Three kinds of enhancements are being left to the future:
> 
>  (a) Determine the network to use based on *client* identity as well as
>      {volume,role}
> 
>  (b) Allow a volume to be used only through certain networks
> 
>  (c) Modify other parameters (e.g. SSL usage/credentials) based on
>      network or interface
> 
> Any other thoughts?
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>