[Gluster-users] Stupid question re multiple networks

[Jeff Darcy]

> AFAIK multiple network scenario only works if you are not using Gluster via
> FUSE mounts or gfapi from remote hosts. It will work for NFS access or when
> you set up something like Samba with CTDB. Just not with native Gluster as
> the server always tells the clients which addresses to connect to: ie your
> storage hosts will always supply the connection details of the hosts that
> are configured in gluster to your storage clients.
>
> I wonder if this could be gaffer-taped with some bridging/vlan/arp spoofing
> trickery but I'm not sure I'd trust such a hack.
>
> It would be *really* nice if there was a way to set up gluster so you could
> specify different IPs for backend and frontend operations.

As you suggest, there are various kinds of "trickery" that can be used
to fake multi-network support even for native mounts.  I've seen it done
via split-horizon DNS, explicit host routes, and iptables.  *Proper*
support for multiple networks is part of the proposed 4.0 feature set.

http://www.gluster.org/community/documentation/index.php/Planning40

In fact, I would greatly appreciate your help defining what "proper"
means in this context.  Clearly, we need to add the concept of a network
to our (informal) object model, and sort out the host/address/network
relationships.  Then we need a way to direct certain traffic flows to
certain networks.  The question is: how do we present this to the user?
Let's take a whack at how to define networks etc. using the CLI's
current object-verb syntax (even though it's a bit clunky).

   gluster> network add user-net 1.2.3.0/24
   gluster> network add back-end 5.6.0.0/16
   gluster> peer probe 1.2.3.4
   gluster> peer probe 5.6.7.8

So far, so good.  Note that on the second probe we should be able to
recognize that this is just a new address (on another network) for the
host we already added with the first probe.  Heartbeats, quorums, etc.
should also be aware of multi-homed hosts.  Maybe there's a better
syntax, but this will do for now.  Let's add a volume.

   gluster> volume create silly-vol 1.2.3.4:/brick

So, which network address should the daemon for 1.2.3.4:/brick expose
for clients?  Which address should it use for internal traffic such as
rebalance or self-heal?  This is where it gets tricky.  Let's start by
saying that *by default* all traffic is on the interface specified on
the "volume create" line.  If we want to do something different...

   # ONLY redirect rebalance traffic.
   gluster> volume route silly-vol rebalance back-end

Now rebalance traffic goes through 5.6.7.8 instead.  Is that intuitive?
What about these?

   # Export a volume on multiple networks.
   gluster> volume route silly-vol client user-net some-other-net

   # Redirect rebalance, self-heal, anything else we think of.
   gluster> volume route silly-vol all-mgmt back-end

   # Redirect GLOBALLY instead of per volume.
   gluster> cluster route rebalance back-end

Does this seem like it's heading in the right direction?  It doesn't
look too bad to me, but my perspective is hardly typical.  Is there
something *users* would like to be able to do with multiple networks
that can't be expressed this way, or is there some better way to define
how these multiple networks should be used?  Please let us know.