[Gluster-users] connection requirements

[Harald Stürzebecher]

Hello!

2009/4/21 randall <randall at ciparo.nl>:
> dear all,
>
> second post ;)
>
> another question here, also in most examples i noticed the infiniband or 10
> GigE recommendation, does this really do any good for the individual server
> connection?

IIRC, another recommendation is a RAID-6 array with 8-12 disks. I'd
expect 400-600MB/s on linear read with hardware like that. In that
case, 1GigE would be a bottleneck so 10 GigE or Infiniband might be a
reasonable recommendation.

IIRC, some posts on the mailing list point out that the performance of
GlusterFS (and most or all other distributed filesystems) is limited
by connection latency (e.g. 'replicate' has to ask each of the servers
if it has a newer version of a file and wait for the answer). On
100Mbit/s it takes longer to push a packet (of equal size) through the
wire than it takes at 1GBit/s or even 10GBit/s.

Maybe someone with access to a test setup with GigE, 10GigE and/or
Infiniband could benchmark this so that others might have a base line
of what to expect?

> another assumption on my side was that 1 individual server can never
> saturate a full gigabit link due to the disk throughput limitation, (maybe a
> few 100 MBps at most)

AFAIK, a modern disk (>=1TB SATA) can deliver more than 100MB/s
locally on linear read:

# hdparm -t /dev/sdc

/dev/sdc:
 Timing buffered disk reads:  306 MB in  3.00 seconds = 101.84 MB/sec
# fdisk -l /dev/sdc

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes


Even a server with just two of these disks in a 'dht' or 'unify'
configuration might be able to saturate a 1GigE network link under
specific conditions.
Some Solid State Disks are even rated at 250MB/s for read.

> so if every server has a 1 Gigabit connection to a switch which in turn has
> a 10 GigE uplink it would not be a bottleneck (as long as there are not too
> many servers sharing the 10 GigE uplink)
>
> correct?

I'm not sure about that, but after looking at prices I'd do a lot of
testing that the 1GigE network adapter really is a bottleneck before
buying 10GigE or Infiniband hardware. On a cost/value comparison GigE
might win on small systems with only one or two disks.


Harald Stürzebecher