[Gluster-users] [Gluster-devel] OS X porting merged

[Dan Mons]

I should add to all of this, currently we are doing:

* On MacOSX workstations, we have a small shell script that initiates
SMB mounts of our Gluster setup, so that artists can mount up into the
correct absolute path (no, not /Volumes with a capital V, thank you
Apple).  Artists need to run this once per logon on workstations.
Painful, but less painful than Finder NFS dramas.

* On MacOSX render nodes, we're using NFS3 currently, mounted via
autofs. It appears that most apps (the good ones at least - i.e.:
anything not Adobe, which is a whole other rant) are OK with CLI batch
processing and read/write operations to GlusterFS that way.  The
difference appears to be whatever framework the developers use - most
of our developers use the BSD subsystem of OSX which works to NFS.
Ones that use the Carbon/Cocoa/Finder framework break.

* All Linux workstations and render nodes (all Kubuntu 12.04LTS) use
FUSE+GlusterFS mounted via autofs, because it works better.
Exceptions are legacy "black box" vendor machines like Autodesk Flame
(RHEL5.3 base with a custom kernel and no supported way to add new
stuff) that don't offer FUSE natively, so they're stuck on NFS.  Add
Auodesk to the vendor rant list directly behind Adobe and Apple.
Protip: don't start a software company that begins with the letter A.
It's bad mojo.

* Windows - we dumped it at the end of last year.  Useless for us for
so many reasons.  (Microsoft are the exception that proves the "A*"
rule).

The goal is to give this recent MacOSX porting effort a good solid
bash, and get it into production if it appears relatively stable.
Because testing code in production is how we roll. :)

-Dan


----------------
Dan Mons
Unbreaker of broken things
Cutting Edge
http://cuttingedge.com.au


On 28 April 2014 19:44, Dan Mons <dmons at cuttingedge.com.au> wrote:
> On 28 April 2014 16:11, Andrew Hatfield
> <andrew.hatfield at cynosureservices.com> wrote:
>> Hey Dan,
>>
>> Did you ever test SMB with the streams xattr vfs object?
>
> Yes. :)
>
>> Did it work?  Was performance ok?
>
> It functions, but forces us to compromise.  It was the only way to get
> GlusterFS working via SMB on MacOSX.
>
> For anyone who hasn't discovered this: MacOSX's Finder looks for a
> "._filename" resource fork file for every "filename" data file it
> finds.  GlusterFS (and any clustered file system, to be fair) is quite
> poor at negative lookups (i.e.: when it can't find a file on a
> particular brick, it has to run around asking every brick in the
> cluster if they have the file), and so a folder with ~1000 files in it
> (quite common for a VFX studio, across dozens of shots inside dozens
> of projects) can take several minutes to populate when trying to view
> it in OSX's Finder.
>
> streams xattr means that the resource fork information can be held in
> the xattr component of the underlying file system (for us that's XFS
> on our CentOS6 bricks), and completely removes the need for the
> negative lookup, as that resource fork data lives with the file.  (I
> for one find the whole resource fork concept totally outdated and
> silly, but Apple seem keen to keep it around).
>
> Downsides:
>
> 1) SMB is slower than either NFS or FUSE+GlusterFS.  vfs_glusterfs
> helps, but still doesn't compete in real world usage.
>
> 2) MacOSX does not allow system-wide mounting of SMB shares, ala NFS3.
>  Our studio relies heavily on this as machines need network file
> systems mounted up so that multiple users can hit them at once.  We
> run a large number of machines in what's called a "render farm" which
> are batch-processing multiple jobs each in parallel. Our standard spec
> render node currently is a dual Xeon (8 cores per proc, 2 procs, per
> node, with HT = 32 threads / logical CPUs) and 128GB RAM.  These
> sometimes do one very large job, and sometimes many smaller jobs in
> parallel.  Jobs run as the UID of the person who submitted them, so
> NFS3 style network mounting is mandatory for this to work.  There is
> no such thing as "one user on one machine" in our world.
>
> MacOSX can't do this via SMB.  In 10.8 and 10.9, whichever user
> initiates the SMB connection "owns" the mount (regardless of the
> credentials they use at mount time), and any other UID is locked out
> of that share.
>
> Upsides:
>
> 1) At least it works at all, unlike NFS which Apple broke in Finder
> back in 10.5 and have refused to even acknowledge let alone fix.
> Honestly, how does an OS based on UNIX not do NFS?  10.9 now
> represents 5 complete production versions of their OS in a row with
> broken NFS.  Ludicrous.
>
> 2) AFP has the same UID-clobbering mount issues as SMB, but SMB is
> faster than AFP (thanks to streams xattr) and we can cluster SMB more
> easily.
> Long story short, SMB is a stop-gap.  Ideally either Apple fix NFS
> (I'm not holding my breath, as it's clear Apple's priority is selling
> iPhones and media, and not actually offering a usable operating
> system), or the Gluster community add the ever moving target of MacOSX
> support to their FUSE+GlusterFS client, which is now moving along
> nicely as per this thread.
>
> So my eternal thanks to everyone contributing to this OSX porting
> effort (and of course everyone who's ever contributed a line of code
> to GlusterFS in general).  You are all wonderful human beings.  :)
>
> -Dan