[Gluster-users] Configure separate network for inter-node communication

[Jeff Darcy]

> I have two gluster nodes in a replicated setup and have connected the two
> nodes together directly through a 10 Gbit/s crossover cable. Now I would
> like to tell gluster to use this seperate private network for any
> communications between the two nodes. Does that make sense? Will this bring
> me any performance gain? and if yes how do I configure that?

It is possible, but it's not likely to improve performance much (yet).

The easiest way to do this is to use a custom /etc/hosts on the servers,
so that *on a server* every other server's name resolves to its private
back-end address.  Meanwhile, clients resolve that same name to the
server's front-end address.  You can get a similar effect with explicit
host routes or iptables rules on the servers.

The reason this won't have much effect on performance is that the
servers do not (currently) replicate to one another.  Instead, clients
send data directly to every replica themselves.  The only time time a
private network would see much traffic would be when the "clients" are
actually the servers performing administrative operations - self heal,
rebalance, and so on.

In 4.0, both parts of this answer would be different.  First, we expect
to have better handling of multiple networks and multi-homed hosts,
including user specification of which networks to use for which
traffic[1].  Second, 4.0 will have a new form of replication which
*does* replicate directly between servers[2].  Parts of this second
feature are in fact likely to appear well before the rest of 4.0, using
the server-to-server data flow but retaining our current methods of
tracking changes and re-syncing servers after a failure.  In fact I'm
writing a blog post right now about this, including some performance
measurements.  I'll respond again here when it's done.

[1] http://www.gluster.org/community/documentation/index.php/Features/SplitNetwork
[2] http://www.gluster.org/community/documentation/index.php/Features/new-style-replication