[Gluster-devel] GlusterFS Roadmap: Erasure codes.

[gordan at bobich.net]

On Thu, 24 Apr 2008, Rodney McDuff wrote:

> I was just looking at the GlusterFS Roadmap and thought that a nifty
> feature for the future would be an AFR-like translator that uses
> Reed-Solomon erasures codes instead for file replication. That would add
> many 9s to the reliability without adding much storage overhead.

You mean to have one file split across several servers like in RAID[3-6]? 
That's a lot more complicated, and CPU intensive. It would also be quite 
expensive on the writes across the servers, because each written block 
would require all segments of it to be read first for the R-S checksums to 
be calculated.

Fundamentally, I think it's just not what GlusterFS is intended to do. The 
nice thing about GlusterFS is that it is very similar to Coda, only:

1) without the limitations that reduce Coda's usefulness (limit on 1MB of 
metadata per directory - and metadata includes file names, which makes 
Coda fundamentally unsuitable for things like Maildirs or any application 
that is likely to see more than 1000-4000 files per directory)

2) without the features that get in the way of clean integration into 
existing server clusters - e.g. Coda's permission system is ACL based 
rather than POSIX, which is great for a truly global file system that Coda 
is designed to be, but annoying for more tightly coupled clusters.

A reed-solomon n+m RAID type solution, on the other hand, is a lot less 
flexible, although it does provide more effective storage per physical 
storage. Total failure recovery is also a lot more difficult. With 
GlusterFS, all the files are still there with their original content and 
their original names. In Coda they have hash numbered names, and the real 
names are in the metadata storage, but the content is still as per the 
original file. All this makes recovery from extensive failure a lot more 
sane (you'll know what I mean if you ever had to recover data from a RAID5 
stripe in which 2 disks went bad).

But if storage space effectiveness is important to you, there are at least 
two products/projects that provide this functionality already:

DDRAID
Network RAID 3.5, node number limited to powers of 2. Project seems to be 
unmaintained.
http://sourceware.org/cluster/ddraid/

Cleversafe Dispersed Storage
Network RAIDn (n+m). Awesome idea, but sadly it's Java based (i.e. 
bloatware with the performance of a snail on sedatives)
http://www.cleversafe.org/dispersed-storage

I hope this helps.

Gordan