[Gluster-users] Interesting post about glusterfs setup

[Daniel Mons]

On 5 April 2013 14:23,  <tegner at renget.se> wrote:
> Nice read indeed! Question regarding raid. In an hpc-environment we use
> gluster on raid10 as a primary file system, and on raid6 on the back up one.
> Didn't test, but went for raid10 cause I thoght it would be advantageous
> from a performance perspective - can one say something general ragarding
> this?
> Regards,
> /jon

I answer the same question from a forum user in the original link:
http://forums.overclockers.com.au/showpost.php?p=15234327&postcount=14

GlusterFS's main bottleneck is rarely local storage.  I'm running 16
spindles per node in a RAID6+1S setup, with intelligent SSD
caching/reordering for small random IOPS (SSDs are bypassed for large
sequential reads and writes).  The performance of the local disk is
not the problem.  Gluster's single biggest bottleneck (and this is
common to many clustered file systems) is file lookup over the network
for uncached content, and especially negative lookup.  These are
several orders of magnitude slower than the storage, and increasing
the storage IOPS won't help things much at all.

I'm currently losing 3 disks per 16 (~19%) to per-node redundancy.
Moving to RAID10 would bump that to 50% space loss for no real world
net gain, meaning I'd get less storage for the same dollar outlay, and
no measurable speed increase.

What I'd be interested in seeing is someone testing Infiniband/RDMA
compared to Fibre 10GbE on the same storage bricks to see if the
latency reduction has a positive effect on the file lookup speed of
GlusterFS.  If anyone has a link to that sort of test, please post it
in-thread.

-Dan