[Gluster-devel] Re: AFR load-balancing

[Kevan Benson]

Krishna Srinivas wrote:
> On Nov 21, 2007 1:05 AM, Anand Avati <avati at zresearch.com> wrote:
>>>
>>> Have you tried chaining AFR volumes?  There's quite a few ways I can
>>> imagine reducing line saturation if that's the problem.  Here's one:
>>>
>>> server1 has an afr of a local volume and a volume from server2
>>>
>>> server3 has an afr of a local volume and a volume from server4
>>>
>>> client afr of server1's afr and server2's afr
>>>
>>> This should allow a single write from a client at 1000/8/2 = 62.5
>>> MByte/s.  Client writes only twice (to server1 and server3) halving the
>>> bandwidth.  Server1 and server3 write only once each to server2 and
>>> server4 respectively and receive the writes from the client, halving
>>> their bandwidth.
>>>
>>> Note: I have no idea how well this will perform in reality.  There may
>>> be enough lag in glusterfs chaining writes that the gains aren't worth
>>> it, but I suspect since it is effectively pipelining the writes along
>>> there won't be too much lag.
>>
>> Good thought kevan, it sounds like a good in-between solution.
>> Jerker, this might work even better for you if the server-server
>> interconnects are on a seperate physical media not cascaded with the
>> server-client interconnect.
> 
> True. Actually if afr is on the server side, write performance will
> increase only
> if the server-server interconnect is on a different network (or VLAN)
> than the server-client network. If they are on same LAN, regardless
> of where the AFR is loaded (either client or server) there should be
> very little difference in performance (atleast theoretically) because
> even if AFR is loaded on the server it will still have to do the writes
> to all the other servers.

Yeah, that's why I was careful to state that the client should AFR the 
servers it connects to directly.  That way, each server has at most two 
network operations per file operation.

You could, theoretically, get gigabit performance if every system in the 
glusterfs cluster has two nics, and you have switch ports, vlans and 
private netblocks to spare.

E.g.
Client[ETH0]->Server(N)[ETH0],Server(N)[ETH1]->Server(N+1)[ETH0],..,Server((N/2)-1)[ETH1]->Server(N/2)[ETH0]
Client[ETH1]->Server((N/2)+1)[ETH0],Server((N/2)+1)[ETH1]->Server((N/2)+2)[ETH0],..,Server(N-1)[ETH1]->Server(N)[ETH0]

I suspect adding additional servers to the chain beyond 1 or 2 would 
introduce some serious lag, and thus make it not very useful.  You would 
also be taxing the backplane of most switches fairly quickly if you used 
vlans and few switches.

-- 

-Kevan Benson
-A-1 Networks