[Gluster-users] Gluster 3.1.1 issues over RDMA and HPC environment

Anand Avati anand.avati at gmail.com
Mon Feb 7 19:43:36 UTC 2011

Some commits into RDMA to improve stability has gone in post 3.1.1. Can you
check if 3.1.2 has those issues as well?


On Sun, Feb 6, 2011 at 10:35 AM, Claudio Baeza Retamal <
claudio at dim.uchile.cl> wrote:

> Dear friends,
> I have several problems of stability, reliability in a small-middle sized
> cluster, my configuration is the following:
> 66 compute nodes (IBM idataplex, X5550, 24 GB RAM)
> 1 access node (front end)
> 1 master node (queue manager and monotoring)
> 2 server for I/O with GlusterFS configured in distributed mode (4 TB in
> total)
> All computer have a Mellanox ConnectX QDR (40 Gbps) dual port
> 1 Switch Qlogic 12800-180, 7 leaf of 24 ports each one and two double
> Spines
> QSFP plug
> Centos 5.5 and Xcat as cluster manager
> Ofed 1.5.1
> Gluster 3.1.1 over inbiniband
> When the cluster is full loaded for applications which use heavily  MPI in
> combination with other application which uses a lot of I/O to file system,
>  GlusterFS do not work anymore.
> Also, when gendb uses interproscan bioinformatic applications with 128 o
> more jobs, GlusterFS death  or disconnects clients randomly, so, some
> applicatios become shutdown due they do not see the file system.
> This do not happen with Gluster over tcp (ethernet 1 Gbps)  and neither
> happen with Lustre 1.8.5 over infiniband, under same conditions Lustre work
> fine.
> My question is, exist any documenation where there are information more
> especific for GlusterFS tuning?
> Only I found basic information for configuring Gluster, but I do no have
> information more deep (i.e. for experts), I think must exist  some option
> for manipulate this siuation on GlusterFS, moreover, other people should
> have the same problems, since we replicate
>  the configuration in other site with the same results.
> Perhaps, the question is about  the gluster scalability, how many clients
> is recommended for each gluster server when I use RDMA and infiniband fabric
> at 40 Gbps?
> I would appreciate any help,  I want to use Gluster, but stability and
> reliability  is very important for us. Perhaps
> claudio
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

More information about the Gluster-users mailing list