[Gluster-users] Gluster, Samba and MacOSX - possible solution!

[Paul Robert Marino]

Interesting I haven't worked with Netatalk in a decade but it always
had some short comings essentially AFP is a mostly forgotten protocol
I don't even think Apple is doing much development around it any more
so nothing you are saying about it surprises me.

I do have a WARNING for users with Windows clients. Using this will
help your Samba performance significantly but Windows applications
especially ones written by Microsoft do very strange things with vfs
alternate data streams. One example is thats how it snapshots the
files in a volume when it creates a "restore point" among many other
things. Ive seen many applications that break after restore of data
from a backup that didn't include the vfs alternate stream data
streams.
backing up the xattribs requires special handling as a result any one
who uses them to store vfs alernate data streams should look very
closely at the xfsdump and xfsrestore commands because its literally
why they exist.

Additionally if you intend to use the incremental backup feature of
xfsdump, I'm not sure about other Linux Distros but I've noticed Red
Hat does not create the /var/lib/xfsdump/inventory/ directory int the
xfsdump rpm. the easiest way to check if the required directories are
on the host is to run xfsinvutil -i




On Sun, Jan 19, 2014 at 10:23 PM, Dan Mons <dmons at cuttingedge.com.au> wrote:
> I've mentioned a few times previously my frustrations with MacOSX, and
> in particular Mac's Finder application which seems to do odd things
> with network file systems.
>
> The short background is that NFS on MacOSX since 10.5 is broken when
> using GUI (Cocoa/Carbon) applications.  Files and directories
> mysteriously vanish from view at random intervals, renaming files from
> other protocols (FUSE+GlusterFS or SMB) can sometimes trigger this.
> and changing the case of filenames causes havoc. It's all very random
> and frustrating as there's zero consistency to try and map a root
> cause.
>
> SMB works, however is painfully slow with clustered file systems due
> to the requirement for MacOSX to have resource forks stored in
> separate files (a "._filename" for each "filename", along with
> .DS_Store files) causing negative lookups for every single file (which
> can take a very long time on directories with thousands of images
> inside them, which is common for us - up to 3 minutes to open a single
> directory in Finder!).
>
> There is a negative lookup cache for Gluster, but it hasn't been
> touched for years as far as I can tell, and is covered in "don't use
> this in production, it's in testing" warnings that scare me.  It's
> also merely removing the symptoms of this particular problem, and not
> addressing the root cause.
>
> AFP works (via Netatalk3) by storing resource fork information in it's
> own private CNID database, but comes with it's own problems (top level
> directories can't be marked read-only as it causes the whole share to
> be read-only, which is a security problem for us).  AFP is also up to
> 25% slower than SMB for our workloads.
>
> However I think I've finally stumbled upon the solution! (There's a
> lot of noise out there when it comes to technical solutions for OSX,
> which makes finding this stuff incredibly frustrating).
>
> Samba offers a "streams_xattr" VFS object which allows simulation of
> NTFS file streams, which allows the storing of "alternate data
> streams" (i.e.: things like resource forks and other "non-file"
> information) into xattr, which Gluster supports on top of any file
> system that also offers xattr (we use XFS with 512 byte inodes, as per
> the RHEL best practice guides, and that supports xattr natively).
>
> What tipped me off was this post:
> http://www.libertypages.com/clarktech/?p=6284
>
> Which lead me to the appropriate man page:
> http://www.samba.org/samba/docs/man/manpages/vfs_streams_xattr.8.html
>
> While folks are looking for solutions to SMB under Mavericks, the "no
> ugly .DS_Store" comment was what captured my attention (as .DS_Store
> is another resource fork filetype Macs enforce over SMB).
>
> We've tested a handful of machines today on our setup (6 GlusterFS
> nodes running CentOS6 and Samba 3.6 straight from the default repos).
> I added the "vfs objects = streams_xattr" line to our config, and
> instantly speed returned to the Macs mounting SMB from Gluster (even
> without libgfapi, which I'm yet to get working due to time
> constraints).
>
> Clients are directed to the Gluster nodes via a DNS round robin, and
> distribute pretty well over the course of the day.  Throughput is
> higher and latency lower from a GUI file browsing point of view than
> AFP, and the breakages of NFS aren't there.
>
> The glaring downside of course is that we're not in NFS land, so
> system-wide network mounts aren't happening.  That sucks for us due to
> the way we work, and a working NFS stack in Mac is far more desirable.
>  However this is a better place than having to fall back to AFP for us
> with respect to performance, ease of distributing load, and fewer
> protocols/services to manage.
>
> I'll test a bit more, and contribute our complete smb.conf files a
> little later on if it all works nicely.  For now, anyone wanting to
> use Samba with Gluster and connect MacOSX machines, I strongly
> recommend you also test "streams_xattr" as part of your Samba config.
>
> I've tested Samba 3.6 as well as Samba 4.1, and both work fine with
> this VFS object enabled.  Samba 3.6 only offers SMB1, but that's fine
> for OSX 10.6 through 10.8 inclusive.  Samba 4.1 offers SMB2 and SMB3,
> however you need Mavericks to use the later revisions of the
> protocols.  A quick test suggests that Samba 4.1 is slightly faster
> for OSX 10.9 clients, and makes no difference to OSX 10.8 and below.
> I've not seen any change in Windows client behaviour with this VFS
> object enabled
>
> All of this should speed up *any* Samba storage for Mac clients
> theoretically, but is especially useful to clustered filesystems like
> GlusterFS.
>
> Hope that helps anyone also suffering through Mac OSX deployments!
>
> -Dan
>
>
> ----------------
> Dan Mons
> R&D SysAdmin
> Unbreaker of broken things
> Cutting Edge
> http://cuttingedge.com.au
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users