[Gluster-devel] GlusterFS vs. NFS performance

[Jake Maul]

Greetings,

I've read a few threads like this in the archives, and had a few
additional questions I was wondering if anyone could answer...

Right off the bat, it seems that GlusterFS cannot make use of standard
filesystem caching on the client-side. Instead, one needs to use the
iocache performance translator. That works, but according to IOzone
GlusterFS is just no match for NFS as far as cache performance is
concerned... ~550MB/s vs. 3GB/s+ in some cases (with the GlusterFS
FUSE patch). Is this known, planned-for-fix, or am I doing it wrong?
:)

The GlusterFS iocache in general seems to be more picky about what's
cached, or perhaps I just don't know how to work it. Could someone
explain the options to me? For instance...
It doesn't take a genius to figure out how cache-size works (although
as a side note it would be very nice if it could actually use the
standard FS cache- all these questions and performance differences
might disappear). For the record, I understand it is not per-thread.

What about page-size? For example, if I set page-size to 1MB, and I
have a 512KB file to cache, obviously it fits in 1 page. What happens
to the other 512KB of space? Wasted presumably? How about if I have a
2MB file... can it consume 2 pages and be fine, or is it not cachable?
Sorry, I don't have a really good grasp of paging and VM subsystems...

What effect does force-revalidate-timeout have? I'm guessing that
anything cached more recently than this value is automatically trusted
to be correct/current, and anything over generates an mtime lookup?


As you might have guessed, my main concern is client-side performance.
>From my testing I'm easily able to saturate the server's gigabit link,
so I'm trying to work on what can be done to let the client(s) to hit
that link less often. We'll undoubtedly be using AFR and/or Unify at
some point, but I don't want to have to be adding hardware just to get
back up to NFS-level performance.

One last quick question... what underlying filesystem do you guys like
to use on GlusterFS exports? Any indication that, say, XFS scales
out/up better than ext3?

Many thanks in advance,
Jake