[Gluster-users] small files and cluster/stripe

[Jeff Darcy]

On 05/13/2010 03:36 PM, Jeff Anderson-Lee wrote:
> cluster/stripe will split large files across multiple volumes, but it
> seems to
> always put the first part of the file on the first volume; if you have a
> bunch of small files they all end up there, and one volume gets heavily
> used by small files while the others are empty.
>
> cluster/distribute spreads files across multiple volumes, but it puts
> the whole file on a single volume.
>
> Some marriage of the two would be helpful for workloads which contain
> both large and small files, like adding an "option block-size ..." to
> cluster/distribute or "option distribute" to cluster/stripe; it would
> use the filename hash modulo nSubvolumes to determine which volume to
> start in for the first block, then rotate around the stripe for the rest.
>
> I suppose I can work-around by creating multiple volumes as
> sub-directories of the same partition, then striping across those in
> rotations, and distributing across the stripes.

I have written such a distribute+stripe hybrid translator, but it's not 
ready to go beyond my desk and I'm distracted by another project right 
now.  Without that, distribute over stripe (as you suggest) would seem 
to be the natural choice, but I've generally had bad luck combining 
stripe with distribute or replicate.  In general, both distribute and 
stripe have a negative effect on reliability and per-node performance, 
though they make up for the latter by scaling horizontally, so you'd 
want to seriously consider adding replicate as well - but then you'll 
have even more complex interactions between complex translators and 
that's where bugs tend to creep in.  In fact, even though I'm from an 
HPC background myself, I've found little enough value in stripe that 
I've considered changing my translator to do distribute+replicate instead.