[Gluster-users] Interesting post about glusterfs setup

[Jon Tegner]

On 04/06/2013 11:50 AM, Daniel Mons wrote:
> On 5 April 2013 14:23,  <tegner at renget.se> wrote:
>> Nice read indeed! Question regarding raid. In an hpc-environment we use
>> gluster on raid10 as a primary file system, and on raid6 on the back up one.
>> Didn't test, but went for raid10 cause I thoght it would be advantageous
>> from a performance perspective - can one say something general ragarding
>> this?
>> Regards,
>> /jon
> I answer the same question from a forum user in the original link:
> http://forums.overclockers.com.au/showpost.php?p=15234327&postcount=14
>
> GlusterFS's main bottleneck is rarely local storage.  I'm running 16
> spindles per node in a RAID6+1S setup, with intelligent SSD
> caching/reordering for small random IOPS (SSDs are bypassed for large
> sequential reads and writes).  The performance of the local disk is
> not the problem.  Gluster's single biggest bottleneck (and this is
> common to many clustered file systems) is file lookup over the network
> for uncached content, and especially negative lookup.  These are
> several orders of magnitude slower than the storage, and increasing
> the storage IOPS won't help things much at all.
>
> I'm currently losing 3 disks per 16 (~19%) to per-node redundancy.
> Moving to RAID10 would bump that to 50% space loss for no real world
> net gain, meaning I'd get less storage for the same dollar outlay, and
> no measurable speed increase.
>
> What I'd be interested in seeing is someone testing Infiniband/RDMA
> compared to Fibre 10GbE on the same storage bricks to see if the
> latency reduction has a positive effect on the file lookup speed of
> GlusterFS.  If anyone has a link to that sort of test, please post it
> in-thread.
>
> -Dan
>
Thanks!

Our primary system is using infiniband, and gluster 3.2.6-1, with ext4. 
Our backup filsystem uses gluster 3.3.1-1, with gigabit and xfs. I'm 
planning to extend this (the backup one) to use infiniband. Then we 
could test it out (bearing in mind that one system is using ext4 and the 
other is using xfs). However, seen some information that rdma is not 
supported on 3.3.x. Would it be waste of effort to upgrade to infiniband 
in this scenario?

Regards,

/jon