[Gluster-devel] Spurious disconnections / connectivity loss

[Gordan Bobic]

Stephan von Krawczynski wrote:
> On Sun, 31 Jan 2010 13:37:49 +0000
> Gordan Bobic <gordan at bobich.net> wrote:
> 
>> Stephan von Krawczynski wrote:
>>> On Sun, 31 Jan 2010 00:29:55 +0000
>>> Gordan Bobic <gordan at bobich.net> wrote:
> 
>>> Slightly offtopic I would like to ask if you, too, experienced glusterfs using
>>> a lot more bandwith than a comparable nfs connection on the server network
>>> side. It really looks a bit like a waste of resources to me...
>> I haven't noticed bandwidth going "missing", if that's what you mean. I 
>> do my replication server-side, so the server replicates the writes n-1 
>> times for n servers, and my cacti graphs are broadly in line with the 
>> bandwidth usage expected. If I disconnect all the mirrors except the 
>> server I'm connecting to,the bandwidth usage between the client and the 
>> server is similar to NFS.
>>
>> What bandwidth "leakage" are you observing?
> 
> My replication is done on client-side, because this is the only way to have it
> redundantly access the data if one server goes down (in theory).
> If I compare the bandwith used by glusterfs and the bandwith used by nfs for
> the same client it is obvious that nfs uses far less bandwith than glusterfs
> (comparing use of only one server of course). Interestingly incoming and
> outgoing server traffic is merely the same, whereas nfs has far less incoming
> traffic (server side), obviously because the client writes a lot less than it
> reads.

That's hardly unexpected. If you are using client-side replicate, I'd 
expect to see the bandwidth requirements multiply with the number of 
replicas. For all clustered configurations (not limited to glfs) I use a 
separate LAN for cluster communication to ensure best possible 
throughput/latencies, and specifically in case of glfs, I do server side 
replicate so that the replicate traffic gets offloaded to that private 
cluster LAN, so the bandwidth requirements to the clients can be kept 
down to sane levels.

Gordan