[Gluster-devel] gluster, gfs, ocfs2, and lustre (lustre.org)

[Brandon Lamb]

On Fri, May 2, 2008 at 10:16 AM, Brandon Lamb <brandonlamb at gmail.com> wrote:
>
> On Fri, May 2, 2008 at 10:08 AM,  <gordan at bobich.net> wrote:
> > On Fri, 2 May 2008, Brandon Lamb wrote:
> >
> > > On Fri, May 2, 2008 at 9:30 AM, Brandon Lamb <brandonlamb at gmail.com>
> > wrote:
> > >
> > > > On Fri, May 2, 2008 at 9:21 AM, Shaofeng Yang <syang at incando.com> wrote:
> > > >
> > > > > Can anybody share some thoughts about those cluster file systems? We
> > are
> > > > > trying to compare the pros and cons for each solution.
> > > > >
> > > > > Thanks,
> > > > > Shaofeng
> > > > >
> > > >
> > > > Tought question as it depends on what you are needing. Myself I have
> > > > messed around with 3 of those for the last 2 years, so far I am still
> > > > just using an 2 NFS servers, one for mail and one for web for my 14 or
> > > > so client machines until I figure out how to use glusterfs.
> > > >
> > > > I tried gfs (redhat) and I dont remember if I even ever got it to
> > > > actually run, I was trying it out on fedora distros. It seemed very
> > > > over complicated and not very user friendly (just my experience).
> > > >
> > >
> >
> > The key to your problems is in Fedora. It _REALLY_ isn't fit for anything
> > more than a hobbyist home setup. It is the alphaware versin of RHEL. For
> > example, FC{7,8} ships only with GFS2 which is not yet stable, and nobody
> > claims it to be. RHEL5 comes with GFS1 and GFS2, GSF2 being there just as a
> > tech preview but not for use in production systems.
> >
> > RHCS has a somewhat steep learning curve, but it's not one that can't be
> > overcome in half a day with assistance from the mailing list. Once you
> > figure out what you're doing it's pretty straightforward, and I've deployed
> > quite a few clusters based on it for various clients.
> >
> >
> > >
> > > > OCFS2 seemed very clean and I was able to use with with ISCSI but man
> > > > the load on my server was running at 7 and it was on the slow side.
> > > > What I was trying to do with it was create a single drive to put my
> > > > maildir data onto (millions of small mail files). The way it worked
> > > > was you actually mounted the file system like it was a local file
> > > > system on all machines that needed it and the cluster part would
> > > > handle the locking or whatnot. Cool concept but overkill for what I
> > > > needed.
> > > >
> > >
> >
> > ANY shared storage FS will suffer major performance penalties for this. File
> > write requires a directory lock. If you start getting contention (e.g.
> > shared imap folders), the performance will go through the floor because
> > you're dealing with distributed lock management overhead and network
> > latencies on top of normal disk latencies. Having said that, most POSIX
> > locking supporting cluster FS-es will suffer from this issue, some more than
> > others.
> >
> >
> > >
> > > > Also I believe both GFS and OCFS2 are these "specialized" file
> > > > systems. What happens if it breaks or goes down? How do you access
> > > > your data? Well if gfs or ocfs2 is broken you cant.
> > > >
> > >
> >
> > That's a bit like saying that if ext3 breaks, you can't access your data.
> > The only specilist thing about them is that they are designed for shared
> > storage, i.e. SAN or DRBD replicated volume.
> >
> >
> > >
> > > > With glusterfs,
> > > > you have direct access to your underlying data. So you can have your
> > > > big raid mounted on a server and use XFS file system, glusterfs just
> > > > sits on top of this so if for some reason you break your glusterfs
> > > > setup you *could* revert back to some other form of serving files
> > > > (such as NFS). Obviously this totally depends on your situation and
> > > > how you are using it.
> > > >
> > >
> >
> > Indeed, but it is fundamentally a distributed rather than centralized
> > storage approach. This isn't a bad thing, but it is an important
> > distinction. GlusterFS is a essentially for a cluster oriented NAS. GFS and
> > OCFS2 are SAN oriented. That is a major difference.
> >
> >
> > >
> > > > Hence the reason that *so far* I am still using NFS. It comes on every
> > > > linux installation, its fairly easy to setup by editing what, 4 lines
> > > > or so. GlusterFS takes the same simple approach and if you do break
> > > > it, you still have access to your data.
> > > >
> > > > The learning curve for glusterfs is much better than the others from
> > > > my experience so far. The biggest thing is just learning all of the
> > > > different ways you can configure spec files.
> > > >
> > >
> >
> > IME, RHCS/GFS didn't take me any more head scratching than GlusterFS did
> > when I first got into it. They are designed for different purposes, and
> > chosing one over the other should be based on project requirements, not on
> > simplicity.
> >
> >
> > > I just wanted to add the stressing of simplicity.
> > >
> > > When the *#($ hits the fan, I would much rather be fixing something
> > > that is on the simple side from the start, rather wondering what the
> > > ### is going on with a specialized filesystem and all the extra pieces
> > > it adds and not having access to my data.
> > >
> >
> > Calling GFS or OCFS2 specialized for this reason is bogus, as I explained
> > earlier. You might as well call ext3 specialized then, along with every
> > other FS.
> >
> >
> > > That is what my company
> > > finally decided on. I was looking into buying iscsi hbas and seeing
> > > about upgrading our network, using DRBD and OCFS2 to sync our two RAID
> > > servers and after two weeks we just looked at each other and said, you
> > > know what. NFS may not be the most kickass thing or lightning fast, or
> > > have builtin replication, but it WORKS. And if a server failed well it
> > > would suck but we could copy from a backup onto the other nfs server
> > > and be running again.
> > >
> >
> > *shrug*
> > I got it all working and without any nightmarish effort. Sure, it's more
> > than 4 lines of config than NFS requires, but the benefits are worth it. You
> > only have to put in the setup time once, and any O(1) effort is preferable
> > to dealing with downtimes in the future.
> >
> >
> > > This is the reason I am down to only investing time into glusterfs.
> > > Its simple but powerful! It does all kinds of cool stuff, and if the
> > > worst happens, Im not really all THAT worried because I know I can
> > > still get my files and have a SIMPLE backup plan.
> > >
> >
> > If GlusterFS works for you - great. I use both GlusterFS and GFS, and use
> > them for fundamentally different tasks. They simply aren't interchangeable
> > in a reasonably thought out systems architecture.
> >
> > Gordan
>
> You know I really dont disagree with most of this to be honest, I
> guess it is also a little unfair to compare the code from 2-3 (time
> flies, maybe it was longer than that) years ago. I really dont know
> how easy it is to set up gfs or ocfs2 today, but when I was doing it,
> it was not that friendly.
>
> Now as far as what distros are suitable for what, ehhhh. Ive
> successfully run fedora on all of our systems for the past 5+ years
> with no problems. But that is beyond the scope of this list I suppose
>
> =P
>
> Ok so maybe it was premature of me to reply back, HOWEVER, those
> *were* my experiences in using the other file systems in needing a NAS
> solution. Which is where my it depends on what you need statement came
> from.
>

Oh yea and in reply to the specialized file system comments, yes that
was probably a bad way to put it. Maybe a better way is that to me, it
is an added bonus that if the "clustering file system software" fails,
I can still access my data if using glusterfs because it has a file
system underneath that is mounted on the server with full access to
it.

With the other solutions this is not the case (still correct?).

The point I was trying to make about this was, our team was
discussing, well what happens if ocfs2 crashes or starts fencing nodes
and we dont have a client that can mount our data because the cluster
either failed or wont start?

I gives me a warm fuzzy feeling with glusterfs to know that it has
this second level underneath it with full access. Hopefully this makes
more sense of what I was trying to say

=)