[Gluster-users] volume sizes

[Liam Slusser]

We have a very similar setup.  We have a 6 x 24 bay gluster cluster
with 36TB per node.  We use 3ware raid cards with raid6 over all the
24 drives making a ~32TB usable per node.  We have our gluster cluster
setup like raid 10, so 3 nodes stripped together and then mirrored to
the other 3 nodes.  Performance is very good as so is the reliability
which was more important to us then performance.  I thought about
breaking it into smaller pieces but it gets complicated very quick so
i went with the simpler is better setup.  We also grow about 1tb a
week of data so i have to add 1-2 nodes a year which is a huge pain in
the butt since gluster doesnt make it very easy to do. (ie building
the directory structure on each new node)  Doing a ls -agl on the root
of our cluster takes well over a week - we have around 50+ million
files in there.

The only downside is the rebuild time whenever we loose a drive.  The
3ware controller with such a large array takes about a week to rebuild
from any one drive failure.  Of course, with raid6, we can loose two
drives without any data loss.  Luckily we've never lost two or more
drives within the same week.  However if we DID for whatever reason
loose the whole array we can always pull the data of the other mirror
node.  I do very closely watch the SMART output of each drive and
proactively replace any drive which starts to show any signs of
failing or read/write errors.

I have a smaller cluster of 4 x 24 bay 36TB per node.  This array
pushes well over 500mbit of traffic almost 24/7 with almost zero
issues.  I've been very happy with how well it performs.  I do notice
that during an array rebuild after a failed drive the IOwait time on
the server is a bit higher but over all it does very well.

If you would like more information on my setup or what
hardware/software i run please feel free to contact me privately.

thanks,
liam


On Tue, Dec 29, 2009 at 1:54 PM, Anthony Goddard <agoddard at mbl.edu> wrote:
> First post!
> We're looking at setting up 6x 24 bay storage servers (36TB of JBOD storage per node) and running glusterFS over this cluster.
> We have RAID cards on these boxes and are trying to decide what the best size of each volume should be, for example if we present the OS's (and gluster) with six 36TB volumes, I imagine rebuilding one node would take a long time, and there may be other performance implications of this. On the other hand, if we present gluster / the OS's with 6x 6TB volumes on each node, we might have more trouble in managing a larger number of volumes.
>
> My gut tells me a lot of small (if you can call 6TB small) volumes will be lower risk and offer faster rebuilds from a failure, though I don't know what the pros and cons of these two approaches might be.
>
> Any advice would be much appreciated!
>
>
> Cheers,
> Anthony
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>