[Gluster-users] Gluster, Samba and MacOSX - possible solution!

[Ira Cooper]

Apple is now primarily using SMB2, instead of AFP in 10.9 I believe.

Thanks,

-Ira

----- Original Message -----
> Interesting I haven't worked with Netatalk in a decade but it always
> had some short comings essentially AFP is a mostly forgotten protocol
> I don't even think Apple is doing much development around it any more
> so nothing you are saying about it surprises me.
> 
> I do have a WARNING for users with Windows clients. Using this will
> help your Samba performance significantly but Windows applications
> especially ones written by Microsoft do very strange things with vfs
> alternate data streams. One example is thats how it snapshots the
> files in a volume when it creates a "restore point" among many other
> things. Ive seen many applications that break after restore of data
> from a backup that didn't include the vfs alternate stream data
> streams.
> backing up the xattribs requires special handling as a result any one
> who uses them to store vfs alernate data streams should look very
> closely at the xfsdump and xfsrestore commands because its literally
> why they exist.
> 
> Additionally if you intend to use the incremental backup feature of
> xfsdump, I'm not sure about other Linux Distros but I've noticed Red
> Hat does not create the /var/lib/xfsdump/inventory/ directory int the
> xfsdump rpm. the easiest way to check if the required directories are
> on the host is to run xfsinvutil -i
> 
> 
> 
> 
> On Sun, Jan 19, 2014 at 10:23 PM, Dan Mons <dmons at cuttingedge.com.au> wrote:
> > I've mentioned a few times previously my frustrations with MacOSX, and
> > in particular Mac's Finder application which seems to do odd things
> > with network file systems.
> >
> > The short background is that NFS on MacOSX since 10.5 is broken when
> > using GUI (Cocoa/Carbon) applications.  Files and directories
> > mysteriously vanish from view at random intervals, renaming files from
> > other protocols (FUSE+GlusterFS or SMB) can sometimes trigger this.
> > and changing the case of filenames causes havoc. It's all very random
> > and frustrating as there's zero consistency to try and map a root
> > cause.
> >
> > SMB works, however is painfully slow with clustered file systems due
> > to the requirement for MacOSX to have resource forks stored in
> > separate files (a "._filename" for each "filename", along with
> > .DS_Store files) causing negative lookups for every single file (which
> > can take a very long time on directories with thousands of images
> > inside them, which is common for us - up to 3 minutes to open a single
> > directory in Finder!).
> >
> > There is a negative lookup cache for Gluster, but it hasn't been
> > touched for years as far as I can tell, and is covered in "don't use
> > this in production, it's in testing" warnings that scare me.  It's
> > also merely removing the symptoms of this particular problem, and not
> > addressing the root cause.
> >
> > AFP works (via Netatalk3) by storing resource fork information in it's
> > own private CNID database, but comes with it's own problems (top level
> > directories can't be marked read-only as it causes the whole share to
> > be read-only, which is a security problem for us).  AFP is also up to
> > 25% slower than SMB for our workloads.
> >
> > However I think I've finally stumbled upon the solution! (There's a
> > lot of noise out there when it comes to technical solutions for OSX,
> > which makes finding this stuff incredibly frustrating).
> >
> > Samba offers a "streams_xattr" VFS object which allows simulation of
> > NTFS file streams, which allows the storing of "alternate data
> > streams" (i.e.: things like resource forks and other "non-file"
> > information) into xattr, which Gluster supports on top of any file
> > system that also offers xattr (we use XFS with 512 byte inodes, as per
> > the RHEL best practice guides, and that supports xattr natively).
> >
> > What tipped me off was this post:
> > http://www.libertypages.com/clarktech/?p=6284
> >
> > Which lead me to the appropriate man page:
> > http://www.samba.org/samba/docs/man/manpages/vfs_streams_xattr.8.html
> >
> > While folks are looking for solutions to SMB under Mavericks, the "no
> > ugly .DS_Store" comment was what captured my attention (as .DS_Store
> > is another resource fork filetype Macs enforce over SMB).
> >
> > We've tested a handful of machines today on our setup (6 GlusterFS
> > nodes running CentOS6 and Samba 3.6 straight from the default repos).
> > I added the "vfs objects = streams_xattr" line to our config, and
> > instantly speed returned to the Macs mounting SMB from Gluster (even
> > without libgfapi, which I'm yet to get working due to time
> > constraints).
> >
> > Clients are directed to the Gluster nodes via a DNS round robin, and
> > distribute pretty well over the course of the day.  Throughput is
> > higher and latency lower from a GUI file browsing point of view than
> > AFP, and the breakages of NFS aren't there.
> >
> > The glaring downside of course is that we're not in NFS land, so
> > system-wide network mounts aren't happening.  That sucks for us due to
> > the way we work, and a working NFS stack in Mac is far more desirable.
> >  However this is a better place than having to fall back to AFP for us
> > with respect to performance, ease of distributing load, and fewer
> > protocols/services to manage.
> >
> > I'll test a bit more, and contribute our complete smb.conf files a
> > little later on if it all works nicely.  For now, anyone wanting to
> > use Samba with Gluster and connect MacOSX machines, I strongly
> > recommend you also test "streams_xattr" as part of your Samba config.
> >
> > I've tested Samba 3.6 as well as Samba 4.1, and both work fine with
> > this VFS object enabled.  Samba 3.6 only offers SMB1, but that's fine
> > for OSX 10.6 through 10.8 inclusive.  Samba 4.1 offers SMB2 and SMB3,
> > however you need Mavericks to use the later revisions of the
> > protocols.  A quick test suggests that Samba 4.1 is slightly faster
> > for OSX 10.9 clients, and makes no difference to OSX 10.8 and below.
> > I've not seen any change in Windows client behaviour with this VFS
> > object enabled
> >
> > All of this should speed up *any* Samba storage for Mac clients
> > theoretically, but is especially useful to clustered filesystems like
> > GlusterFS.
> >
> > Hope that helps anyone also suffering through Mac OSX deployments!
> >
> > -Dan
> >
> >
> > ----------------
> > Dan Mons
> > R&D SysAdmin
> > Unbreaker of broken things
> > Cutting Edge
> > http://cuttingedge.com.au
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>