[Gluster-users] Gluster, Samba and MacOSX - possible solution!

[Dan Mons]

I've mentioned a few times previously my frustrations with MacOSX, and
in particular Mac's Finder application which seems to do odd things
with network file systems.

The short background is that NFS on MacOSX since 10.5 is broken when
using GUI (Cocoa/Carbon) applications.  Files and directories
mysteriously vanish from view at random intervals, renaming files from
other protocols (FUSE+GlusterFS or SMB) can sometimes trigger this.
and changing the case of filenames causes havoc. It's all very random
and frustrating as there's zero consistency to try and map a root
cause.

SMB works, however is painfully slow with clustered file systems due
to the requirement for MacOSX to have resource forks stored in
separate files (a "._filename" for each "filename", along with
.DS_Store files) causing negative lookups for every single file (which
can take a very long time on directories with thousands of images
inside them, which is common for us - up to 3 minutes to open a single
directory in Finder!).

There is a negative lookup cache for Gluster, but it hasn't been
touched for years as far as I can tell, and is covered in "don't use
this in production, it's in testing" warnings that scare me.  It's
also merely removing the symptoms of this particular problem, and not
addressing the root cause.

AFP works (via Netatalk3) by storing resource fork information in it's
own private CNID database, but comes with it's own problems (top level
directories can't be marked read-only as it causes the whole share to
be read-only, which is a security problem for us).  AFP is also up to
25% slower than SMB for our workloads.

However I think I've finally stumbled upon the solution! (There's a
lot of noise out there when it comes to technical solutions for OSX,
which makes finding this stuff incredibly frustrating).

Samba offers a "streams_xattr" VFS object which allows simulation of
NTFS file streams, which allows the storing of "alternate data
streams" (i.e.: things like resource forks and other "non-file"
information) into xattr, which Gluster supports on top of any file
system that also offers xattr (we use XFS with 512 byte inodes, as per
the RHEL best practice guides, and that supports xattr natively).

What tipped me off was this post:
http://www.libertypages.com/clarktech/?p=6284

Which lead me to the appropriate man page:
http://www.samba.org/samba/docs/man/manpages/vfs_streams_xattr.8.html

While folks are looking for solutions to SMB under Mavericks, the "no
ugly .DS_Store" comment was what captured my attention (as .DS_Store
is another resource fork filetype Macs enforce over SMB).

We've tested a handful of machines today on our setup (6 GlusterFS
nodes running CentOS6 and Samba 3.6 straight from the default repos).
I added the "vfs objects = streams_xattr" line to our config, and
instantly speed returned to the Macs mounting SMB from Gluster (even
without libgfapi, which I'm yet to get working due to time
constraints).

Clients are directed to the Gluster nodes via a DNS round robin, and
distribute pretty well over the course of the day.  Throughput is
higher and latency lower from a GUI file browsing point of view than
AFP, and the breakages of NFS aren't there.

The glaring downside of course is that we're not in NFS land, so
system-wide network mounts aren't happening.  That sucks for us due to
the way we work, and a working NFS stack in Mac is far more desirable.
 However this is a better place than having to fall back to AFP for us
with respect to performance, ease of distributing load, and fewer
protocols/services to manage.

I'll test a bit more, and contribute our complete smb.conf files a
little later on if it all works nicely.  For now, anyone wanting to
use Samba with Gluster and connect MacOSX machines, I strongly
recommend you also test "streams_xattr" as part of your Samba config.

I've tested Samba 3.6 as well as Samba 4.1, and both work fine with
this VFS object enabled.  Samba 3.6 only offers SMB1, but that's fine
for OSX 10.6 through 10.8 inclusive.  Samba 4.1 offers SMB2 and SMB3,
however you need Mavericks to use the later revisions of the
protocols.  A quick test suggests that Samba 4.1 is slightly faster
for OSX 10.9 clients, and makes no difference to OSX 10.8 and below.
I've not seen any change in Windows client behaviour with this VFS
object enabled

All of this should speed up *any* Samba storage for Mac clients
theoretically, but is especially useful to clustered filesystems like
GlusterFS.

Hope that helps anyone also suffering through Mac OSX deployments!

-Dan


----------------
Dan Mons
R&D SysAdmin
Unbreaker of broken things
Cutting Edge
http://cuttingedge.com.au