[Gluster-devel] Full bore.

[Kevan Benson]

Chris Johnson wrote:
> On Thu, 15 Nov 2007, Kevan Benson wrote:
>> Chris Johnson wrote:
>>>      Oh, this is ext3 with xattr turned on.  But we could use Reiserfs
>>> if that would help.  I may run tests on that as well.
>>
>> I don't know about others, but I wouldn't use reiserfs if you care 
>> about your data and aren't supplying redundancy above teh file system 
>> level (with glusterfs, for example).  I've seen multiple reports of 
>> how Reiserfs has a tendency to have unrecoverable errors in the file 
>> system when it gets sufficiently screwed up, basically necessitating a 
>> reformat of the partition.
> 
>      Huh.  Now that's interesting.  We've been running our mail server
> off Reiserfs for a few years now and never an issue.  It's servived
> power outs and dropped drives.  We're using software RAID on it.

I didn't have any problems when I ran it either, but there are lots of 
horror stories out there.  The basic gist was always "fsck did nothing, 
fs was unrecoverable.  WTF?"

At first I dismissed them as people not taking the correct steps to fix 
problems, but after about of year of hearing of these, I decided I'd 
rather be safe than sorry.

I read an interesting overview of file system recoverability once from 
an ext3 developer.  Basically, Ext3 view priority #1 as data integrity, 
and all else flows from that.  he had some interesting things to say 
about ReiserFS and even XFS (or was if JFS?).  Basically that ReiserFS 
suffers from some major integrity problems in the case of crashes, and 
even XFS (JFS?) has an issue to do with crashes in the middle of file 
writes, such that a file being written to might be removed entirely (not 
accessible at all) and unrecoverable if a crash happened at the right 
point during the write process.  It was from an Ext3 developer though, 
so you can read into that what you want.


>>>      Can we have a discussion on whether I'm heading in the right
>>> direction and what order things go in for the config files?
>>
>> That depends on your goals.  What's important here, speed, redundancy, 
>> or a mix of both?
>>
> 
>       Both of course.  Are the mutually exclusive?  Please, I know
> they can be to some extent.  I'm talking about the real world,
> whatever that is.  I need to get the performance up,
> redundancy/failover would be be real good too.  NFS has a few problems
> with that.

Nope, not mutually exclusive, just requires more hardware to be thrown 
at it.  The nice thing with glusterfs is that that it just needs to be 
"more", not necessarily "better".

What I finally settled on for my two server setup after all was said and 
done with my testing, was a simple AFR config with AFR handled on the 
clients.  It writes to both servers and reads from one.  After the AFR 
translator is extended to load balance reads of files across AFR 
members, I'll get a speed boost from that, and if they extend it to 
stripe read blocks between AFR members, I'll get even more speed.

I chose this because:
1) It's extremely simple to administer and check consistency.  I can 
just rsync between servers if I don't want to wait for file accesses to 
triger self heal on everything (I need to sync extended attributes as 
well, but that's not too hard with a little scripting)
2) I found specifying translators on the clients provided cleaner 
failover than when specified on the servers.
3) Unify could be used to add performance, but it complicates the shares 
on the servers.  Most the performance gains will be made up when load 
balancing is integrated into AFR.

Then again, right now my goal isn't performance but redundancy.  I think 
of this as a redundant NFS replacement.

-- 

-Kevan Benson
-A-1 Networks