[Gluster-devel] Replicate/AFR Using Broadcast/Multicast?

[Beat Rubischon]

Hello!

Quoting <gordan at bobich.net> (13.10.10 09:11):

>> I would also expect to see network issues as a cluster grows.
> Performance reducing as the node count increases isn't seen as a bigger
> issue?

I have pretty bad experience with Multicast. Running serveral clusters in
the range 500-1000 nodes in a single broadcast domain over the last year
showed that broadcast or multicast is able to kill your fabrics easily.

Even the most expensive GigE switch chassis could be killed by 125+ MBytes
of traffic which is almost nothing :-) The inbound traffic must be routed to
several outbount ports which results in congestion. Throttling and packet
loss is the result, even for the normal unicast traffic. Multicast or
broadcast is a nice way for a denial of service in a LAN.

In Infiniband multicast typically realized by looping through the
destinations directly on the source HBA. One of the main target in the
current development as multicast is a similar network pattern compared to
the MPI collectives. So there is no win in using multicast over such a
fabric.

Using a parallel storage means you have communications from a large amount
of nodes to a large amount of servers. Using unicast looks bad in the first
point of view but I'm confident it's the better solution over all.

Beat

-- 
     \|/                           Beat Rubischon <beat at 0x1b.ch>
   ( 0^0 )                             http://www.0x1b.ch/~beat/
oOO--(_)--OOo---------------------------------------------------
Meine Erlebnisse, Gedanken und Traeume: http://www.0x1b.ch/blog/