[Gluster-users] connection requirements

[Sean Davis]

2009/4/23 Harald Stürzebecher <haralds at cs.tu-berlin.de>

> Hello!
>
> 2009/4/21 randall <randall at ciparo.nl>:
> > dear all,
> >
> > second post ;)
> >
> > another question here, also in most examples i noticed the infiniband or
> 10
> > GigE recommendation, does this really do any good for the individual
> server
> > connection?
>
> IIRC, another recommendation is a RAID-6 array with 8-12 disks. I'd
> expect 400-600MB/s on linear read with hardware like that. In that
> case, 1GigE would be a bottleneck so 10 GigE or Infiniband might be a
> reasonable recommendation.
>
> IIRC, some posts on the mailing list point out that the performance of
> GlusterFS (and most or all other distributed filesystems) is limited
> by connection latency (e.g. 'replicate' has to ask each of the servers
> if it has a newer version of a file and wait for the answer). On
> 100Mbit/s it takes longer to push a packet (of equal size) through the
> wire than it takes at 1GBit/s or even 10GBit/s.
>
> Maybe someone with access to a test setup with GigE, 10GigE and/or
> Infiniband could benchmark this so that others might have a base line
> of what to expect?
>
> > another assumption on my side was that 1 individual server can never
> > saturate a full gigabit link due to the disk throughput limitation,
> (maybe a
> > few 100 MBps at most)
>
> AFAIK, a modern disk (>=1TB SATA) can deliver more than 100MB/s
> locally on linear read:
>
> # hdparm -t /dev/sdc
>
> /dev/sdc:
>  Timing buffered disk reads:  306 MB in  3.00 seconds = 101.84 MB/sec
> # fdisk -l /dev/sdc
>
> Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
>
>
> Even a server with just two of these disks in a 'dht' or 'unify'
> configuration might be able to saturate a 1GigE network link under
> specific conditions.
> Some Solid State Disks are even rated at 250MB/s for read.
>
> > so if every server has a 1 Gigabit connection to a switch which in turn
> has
> > a 10 GigE uplink it would not be a bottleneck (as long as there are not
> too
> > many servers sharing the 10 GigE uplink)
> >
> > correct?
>
> I'm not sure about that, but after looking at prices I'd do a lot of
> testing that the 1GigE network adapter really is a bottleneck before
> buying 10GigE or Infiniband hardware. On a cost/value comparison GigE
> might win on small systems with only one or two disks.
>

And, while it may be a pain, I think it is possible to aggregate multiple
GigE connections....

Sean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090423/7afb61c6/attachment.html>