[Gluster-users] Gluster 3.1.1 issues over RDMA and HPC environment

[Anand Avati]

Some commits into RDMA to improve stability has gone in post 3.1.1. Can you
check if 3.1.2 has those issues as well?

Avati

On Sun, Feb 6, 2011 at 10:35 AM, Claudio Baeza Retamal <
claudio at dim.uchile.cl> wrote:

> Dear friends,
>
> I have several problems of stability, reliability in a small-middle sized
> cluster, my configuration is the following:
>
> 66 compute nodes (IBM idataplex, X5550, 24 GB RAM)
> 1 access node (front end)
> 1 master node (queue manager and monotoring)
> 2 server for I/O with GlusterFS configured in distributed mode (4 TB in
> total)
>
> All computer have a Mellanox ConnectX QDR (40 Gbps) dual port
> 1 Switch Qlogic 12800-180, 7 leaf of 24 ports each one and two double
> Spines
> QSFP plug
>
> Centos 5.5 and Xcat as cluster manager
> Ofed 1.5.1
> Gluster 3.1.1 over inbiniband
>
> When the cluster is full loaded for applications which use heavily  MPI in
> combination with other application which uses a lot of I/O to file system,
>  GlusterFS do not work anymore.
> Also, when gendb uses interproscan bioinformatic applications with 128 o
> more jobs, GlusterFS death  or disconnects clients randomly, so, some
> applicatios become shutdown due they do not see the file system.
>
> This do not happen with Gluster over tcp (ethernet 1 Gbps)  and neither
> happen with Lustre 1.8.5 over infiniband, under same conditions Lustre work
> fine.
>
> My question is, exist any documenation where there are information more
> especific for GlusterFS tuning?
>
> Only I found basic information for configuring Gluster, but I do no have
> information more deep (i.e. for experts), I think must exist  some option
> for manipulate this siuation on GlusterFS, moreover, other people should
> have the same problems, since we replicate
>  the configuration in other site with the same results.
> Perhaps, the question is about  the gluster scalability, how many clients
> is recommended for each gluster server when I use RDMA and infiniband fabric
> at 40 Gbps?
>
> I would appreciate any help,  I want to use Gluster, but stability and
> reliability  is very important for us. Perhaps
>
>
> claudio
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>