[Gluster-users] GlusterFS vs. NFS

[Brian J Mingus]

We recently purchased a new cluster and I have been advocating the use of
glusterfs. I have a lot of experience on the user side with nfs and my
feeling is that it has poor performance. I suggested gluster which got
relayed to our vendor and this is what they relayed back to me:

   - Suggests we stay away from glusterfs and gfs
   - glusterfs is not mature
   - We need a stable parallel file system. nfs is stable, glusterfs is not.
   - Using gluster would be adding complexity and removing reliability.

It's important to note that I am advocating glusterfs in the presence of
issues with our nfs file system practically every time I submit a job. The
cluster admin characterizes the problem thusly: NFS cannot handle getting
hit with many simultaneous requests. Most processes wait for io and this
drives up the load. The result is that we get locked out of the cluster when
the queue builds up.

Here are the stats of our cluster:

   - 26 nodes + head node
   - Infiniband
   - Dual quad core xeons @ 2.53 Ghz
   - 24 GB memory per node
   - 20 TB hardware RAID 6 available over NFS
   - 26 nodes, each of which has a 207GB hard drive. These are presently all
   at 13GB utilization, meaning there is 5 terabytes of free space across the
   nodes that is essentially unavailable for use right now.

My question to you is, how would you respond to these points made by our
vendor, and given this setup how would you configure GlusterFS to

   1. Replace NFS
   2. Maximize write speed and throughput
   3. Maximize read speed and throughput
   4. Maximize reliability

I'm looking for the stable middle of the ground solution that gets rid of
our performance problems and takes advantage of what appears to be kickass
hardware for a file system, all while making the case for gluster. I have a
positive opinion of this project, I don't know why others do not..

Many thanks,

Brian Mingus
Professional Research Assistant
Computational Cognitive Neuroscience Lab
University of Colorado at Boulder