[Gluster-users] Using local writes with gluster for temporary storage
jdarcy at redhat.com
Thu Nov 15 11:41:21 UTC 2012
On Nov 14, 2012, at 12:45 PM, Pat Haley wrote:
> We have a cluster with 130 compute nodes with an NAS-type
> central storage under gluster (3 bricks, ~50TB). When we
> run large number of ocean models we can run into bottlenecks
> with many jobs trying to write to our central storage.
> It was suggested to us that we could also used gluster to
> unite the disks on the compute nodes into a single "disk"
> in which files would be written locally. Then we could
> move the files after the runs were complete in a more
> sequential manner (thus avoiding overloading the network).
> What was originally suggested (the NUFA policy) has since
> been deprecated. What would be the recommended method
> of accomplishing our goal in the latest version of Gluster?
> And where can we find documentation on it?
> We have seen the following links, but would be interested
> in any more pointers you may have. Thanks.
s the author of the last two, I guess I should weigh in here. There's a
danger with these kinds of solutions, which is the creation of
if two clients write to their local copies using bypass? The subsequent
self-heal will fail due to split brain. Sometimes this isn't an issue
of the way applications work, or can be avoided by locking the file, but
otherwise it's a real danger.
Moving from replication to distribution, NUFA is deprecated in truth
regardless of the official status. There's no CLI support, and it
bug fixes and enhancements - e.g. dynamic reconfiguration, rebalance,
replace-brick - that have been applied to DHT. It builds and appears
but in my brief tests just now didn't seem to distribute files any
than DHT would.
All is not lost, however. You can get a limited kind of NUFA-like
using user-specified layouts. Here's some background:
A while back, I added a feature to mark the extended attributes that
we use as
having been set by the user, so that we'll leave them alone when we
What you can do is set these attributes on a directory so that the
"claims" practically all of the consistent-hashing space and new files
placed there but still findable by all of the usual methods and fully
integrated with all the rest of DHT. When your app is done, you can
the layout type back to normal and let DHT recalaculate it (there's a
xattr to do this for a single directory so you don't need to do it
whole volume). If you actually want to migrate the files around
you could do it the same way we do - check whether the file should be
somewhere else, if so create a copy and rename over the original.
I know that all sounds a bit hackish and cumbersome, but it would work
risk incompatibility with the rest of GlusterFS. I even have some
do some similar things, so if you'd like to collaborate with me on
some of this I'd be glad to.
More information about the Gluster-users