[Gluster-users] small files and cluster/stripe

[Jeff Anderson-Lee]

On 5/13/2010 6:24 PM, Craig Carl wrote:
> Jeff -
>     Thanks for your email, I think I've got a grasp of your 
> environment now and I understand the problem. If we create a 
> "/gluster/small_files" and a "/gluster/large_files" your users are 
> unlikely to respect distinction, plus it is a management nightmare, 
> right?
> If you have time I'd like your help writing a feature request that 
> would implement what you need.  Something like -
>
> Gluster should provide the option of distributing files based on size 
> to different volumes.
> This distribution should be transparent to users.
> This distribution only needs to happen the first time a file is written.
> The Gluster administrator should have the ability to provide a file 
> size range for each volume.
> The different volumes could be different types; mirror, stripe, mirror 
> & distribute, etc.
>
> What have I missed?
>
> Craig

That would be one solution.  I would target another that I suspecr is 
probably simpler:

Gluster should provide the option of pseudo-randomizing the distribution 
of file stripes across volumes, so that all small files do not end up on 
the same subvolume of a cluster/stripe.
This distribution should be transparent to users.
This distribution only needs to happen the first time a file is written 
and may be based on the file name hash (a la cluster/distribute).

The net behavior could be such that small files (less that the 
block-size) would have the same data distribution pattern as they would 
have with cluster/distribute, while larger files (greater than the 
stripe block-size) would have their upper blocks ditributed in a 
round-robin from that starting place.

Given that the code already exists for distributing files based on 
namehash in cluster/distribute I think this could be an easier feature 
to add.

Jeff