[Gluster-users] small files and cluster/stripe

[Craig Carl]

Jeff - 
Thanks for your email, I think I've got a grasp of your environment now and I understand the problem. If we create a "/gluster/small_files" and a "/gluster/large_files" your users are unlikely to respect distinction, plus it is a management nightmare, right? 
If you have time I'd like your help writing a feature request that would implement what you need. Something like - 

Gluster should provide the option of distributing files based on size to different volumes. 
This distribution should be transparent to users. 
This distribution only needs to happen the first time a file is written. 
The Gluster administrator should have the ability to provide a file size range for each volume. 
The different volumes could be different types; mirror, stripe, mirror & distribute, etc. 

What have I missed? 

Craig 





-- 
Craig Carl 



Gluster, Inc. 
Cell - (408) 829-9953 (California, USA) 
Gtalk - craig.carl at gmail.com 

----- Original Message ----- 
From: "Jeff Anderson-Lee" <jonah at eecs.berkeley.edu> 
To: "Craig Carl" <craig at gluster.com> 
Cc: gluster-users at gluster.org 
Sent: Thursday, May 13, 2010 5:27:45 PM GMT -08:00 US/Canada Pacific 
Subject: Re: [Gluster-users] small files and cluster/stripe 

On 5/13/2010 5:05 PM, Craig Carl wrote: 


Jeff - 
Two comments/ideas. 

1. If you are limited to four pieces of hardware, the minimum for stripe, and you want to stripe some of the data and just distribute other files there is a way to do that. Ideally you would use your hardware RAID controllers to create two LUNs on each host, one for distribute, the other for stripe. If you don't have hardware RAID you could use LVM2 or ZFS to achieve the same thing. (or you could use folders) 
1a. Once you have two file systems created use glusterfs-volgen to create the vol files for the distribute export just like you normally would. 
1b. Move the files you just created to the storage servers and clients. 
1c. Re-run glusterfs-volgen this time for the stripe, adding the -p option and specifying a port. (something above 1024, not 6996). 
1d. Move the files you just created to the storage servers and clients. 
1e . Start Gluster twice on all the servers, specifying the different vol files. 
1f. You now have two GlusterFS exports, one distribute, the other mirror. 


1g. You can mount one inside the other on the client if that makes management easier. 
There are advantages to this model, having two separate Gluster instances significantly improves parallelism on the storage servers. You can manage the two instances as if they are on different iron. 




2. The use case for stripe is vanishingly small. If you have very large files (at least 2X the amount of memory in your storage servers and a minimum of 50GB) with very limited writes and simultaneous access from hundreds of clients then maybe stripe might be appropriate. Stripe was designed for a specific type of HPC problem solving, not general file serving. Our video streaming users don't use stripe, even though that is an obvious use, there are better ways to configure Gluster for that. If you could share the type of content/access methods/iops per sec we could make some specific suggestions. 

We *are* a quasi-HPC environment. We have 100+ batch compute servers with 500+ cores, all with GbT interfaces, pounding on an old NAS storage server. We are trying to replace the old shared staging area with new hardware. We've been looking at an Isilon solution, which performs well for the task but costs 4x to 5x what a Gluster solution would price out at for similar-sized hardware/space. 

Some our users have millions of small files, some have thousands of large files, some have one or two humongous files. If all the data was just one size or another all would be well. All files are currently stored in the same shared staging area. Our users are not HPC programmers and tend to program in HLL such as matlab, so we try to be as accommodating as possible, rather than force them to manage the data distribution. 

We'd love a solution that would (a) spread small files over multiple volumes as well as (b) spread large files over multiple volumes. Cluster/distribute would work for the former and cluster/stripe for the latter. A marriage of the two would be great. 

Right now I'm trying to patch together a temporary testbed using a bunch of old machines with two 143GB drives each. The problem is that many files are multi-GB and unless they are striped they could easily fill up a volume with poor hash distributions. Likewise many small files could swamp the low-end disk in a stripe volume. 

I suppose we could create two pools and tell the predominantly small file users to use one and the predominantly large file users to use the other, but somehow I would not hold my breath on it working out. 

Jeff