[Gluster-users] Stupid question re multiple networks

[Anders Blomdell]

On 2014-11-13 20:15, Jeff Darcy wrote:
>> AFAIK multiple network scenario only works if you are not using Gluster via
>> FUSE mounts or gfapi from remote hosts. It will work for NFS access or when
>> you set up something like Samba with CTDB. Just not with native Gluster as
>> the server always tells the clients which addresses to connect to: ie your
>> storage hosts will always supply the connection details of the hosts that
>> are configured in gluster to your storage clients.
>>
>> I wonder if this could be gaffer-taped with some bridging/vlan/arp spoofing
>> trickery but I'm not sure I'd trust such a hack.
>>
>> It would be *really* nice if there was a way to set up gluster so you could
>> specify different IPs for backend and frontend operations.
> 
> As you suggest, there are various kinds of "trickery" that can be used
> to fake multi-network support even for native mounts.  I've seen it done
> via split-horizon DNS, explicit host routes, and iptables.  *Proper*
> support for multiple networks is part of the proposed 4.0 feature set.
> 
> http://www.gluster.org/community/documentation/index.php/Planning40
> 
> In fact, I would greatly appreciate your help defining what "proper"
> means in this context.  Clearly, we need to add the concept of a network
> to our (informal) object model, and sort out the host/address/network
> relationships.  Then we need a way to direct certain traffic flows to
> certain networks.  The question is: how do we present this to the user?
> Let's take a whack at how to define networks etc. using the CLI's
> current object-verb syntax (even though it's a bit clunky).
> 
>    gluster> network add user-net 1.2.3.0/24
>    gluster> network add back-end 5.6.0.0/16
>    gluster> peer probe 1.2.3.4
>    gluster> peer probe 5.6.7.8
> 
> So far, so good.  Note that on the second probe we should be able to
> recognize that this is just a new address (on another network) for the
> host we already added with the first probe.  Heartbeats, quorums, etc.
> should also be aware of multi-homed hosts.  Maybe there's a better
> syntax, but this will do for now.  Let's add a volume.
> 
>    gluster> volume create silly-vol 1.2.3.4:/brick
> 
> So, which network address should the daemon for 1.2.3.4:/brick expose
> for clients?  Which address should it use for internal traffic such as
> rebalance or self-heal?  This is where it gets tricky.  Let's start by
> saying that *by default* all traffic is on the interface specified on
> the "volume create" line.  If we want to do something different...
> 
>    # ONLY redirect rebalance traffic.
>    gluster> volume route silly-vol rebalance back-end
> 
> Now rebalance traffic goes through 5.6.7.8 instead.  Is that intuitive?
> What about these?
> 
>    # Export a volume on multiple networks.
>    gluster> volume route silly-vol client user-net some-other-net
> 
>    # Redirect rebalance, self-heal, anything else we think of.
>    gluster> volume route silly-vol all-mgmt back-end
> 
>    # Redirect GLOBALLY instead of per volume.
>    gluster> cluster route rebalance back-end
> 
> Does this seem like it's heading in the right direction?  It doesn't
> look too bad to me, but my perspective is hardly typical.  Is there
> something *users* would like to be able to do with multiple networks
> that can't be expressed this way, or is there some better way to define
> how these multiple networks should be used?  Please let us know.
What if the gluster servers are also clients? I locally plan to use
a number of servers acting as gluster and VM servers, so that gluster serves
both the VM's and other clients.

/Anders


-- 
Anders Blomdell                  Email: anders.blomdell at control.lth.se
Department of Automatic Control
Lund University                  Phone:    +46 46 222 4625
P.O. Box 118                     Fax:      +46 46 138118
SE-221 00 Lund, Sweden