[Gluster-users] Using local writes with gluster for temporary storage

[Jeff Darcy]

On Nov 14, 2012, at 12:45 PM, Pat Haley wrote:

> We have a cluster with 130 compute nodes with an NAS-type
> central storage under gluster (3 bricks, ~50TB).  When we
> run large number of ocean models we can run into bottlenecks
> with many jobs trying to write to our central storage.
> It was suggested to us that we could also used gluster to
> unite the disks on the compute nodes into a single "disk"
> in which files would be written locally.  Then we could
> move the files after the runs were complete in a more
> sequential manner (thus avoiding overloading the network).
>
> What was originally suggested (the NUFA policy) has since
> been deprecated.  What would be the recommended method
> of accomplishing our goal in the latest version of Gluster?
> And where can we find documentation on it?
>
> We have seen the following links, but would be interested
> in any more pointers you may have.  Thanks.
>
> http://thr3ads.net/gluster-users/2012/06/1941337-how-to-enable-nufa-in-3.2.6
>
> http://blog.aeste.my/2012/05/15/glusterfs-3-2-updates/
>
> http://www.gluster.org/2012/05/back-door-async-replication/
>
> https://github.com/jdarcy/bypass


s the author of the last two, I guess I should weigh in here. There's a
danger with these kinds of solutions, which is the creation of  
conflicts. What
if two clients write to their local copies using bypass? The subsequent
self-heal will fail due to split brain. Sometimes this isn't an issue  
because
of the way applications work, or can be avoided by locking the file, but
otherwise it's a real danger.

Moving from replication to distribution, NUFA is deprecated in truth
regardless of the official status. There's no CLI support, and it  
lacks many
bug fixes and enhancements - e.g. dynamic reconfiguration, rebalance,
replace-brick - that have been applied to DHT. It builds and appears  
to work,
but in my brief tests just now didn't seem to distribute files any  
differently
than DHT would.

All is not lost, however. You can get a limited kind of NUFA-like  
behavior by
using user-specified layouts. Here's some background:

         http://hekafs.org/index.php/2011/04/glusterfs-extended-attributes/
         http://hekafs.org/index.php/2012/03/glusterfs-algorithms-distribution/

A while back, I added a feature to mark the extended attributes that  
we use as
having been set by the user, so that we'll leave them alone when we  
rebalance.
What you can do is set these attributes on a directory so that the  
local node
"claims" practically all of the consistent-hashing space and new files  
will be
placed there but still findable by all of the usual methods and fully
integrated with all the rest of DHT. When your app is done, you can  
just set
the layout type back to normal and let DHT recalaculate it (there's a  
magic
xattr to do this for a single directory so you don't need to do it  
across the
whole volume).  If you actually want to migrate the files around  
physically,
you could do it the same way we do - check whether the file should be
somewhere else, if so create a copy and rename over the original.

I know that all sounds a bit hackish and cumbersome, but it would work  
and not
risk incompatibility with the rest of GlusterFS. I even have some  
scripts to
do some similar things, so if you'd like to collaborate with me on  
automating
some of this I'd be glad to.