[Gluster-users] MacOSX Finder performance woes

Tue Feb 5 01:28:46 UTC 2013

Hi Adam,

Sadly the disabling of writing out .DS_Store and other resource-fork type
files does nothing to assist.  The problem is exactly as you describe it -
the negative lookup via Finder on directory reads, which happens regardless
whether or not the client has been told to disabled writing of these files.

As it stands, performance over Netatalk is fine, with .AppleDouble database
files dealing with the resource forks.  The downside is that I can't run
Netatalk on every node in the cluster like I can with Samba or NFS, so it
forces all Mac traffic through one node, killing off any elegant attempt to
distribute bandwidth.  Still, it's a workaround for now until negative
lookup caching can be implemented, or Finder's default behavior is fixed.
(And asking Apple to fix Finder seems to be a dead end, given the grief I
see people having with Finder and NFS shares).

For now, in a 6 node cluster, I have MacOSX clients using AFP to Netatalk
on the first node only, and round-robin DNS distributing Windows/Samba
clients and Legacy/NFS clients across the other five nodes.  Modern Linux
clients happily use FUSE+GlusterFS, and that configuration is getting me
through production for now.

-Dan

On 20 December 2012 14:00, Adam Tygart <mozes at k-state.edu> wrote:

> Dan,
>
> Does something like this help for SAMBA?
> http://www.e-rave.nl/disable-creation-of-ds_store-files-on-samba-shares
>
> Negative lookups are a particularly expensive operation on distributed
> filesystems, GlusterFS in particular. It has to check through each
> brick to see that the file *really* doesn't exist, increasing the
> lookup overhead immensely. If samba doesn't pass these queries to
> GlusterFS, it might help a lot.
>
> --
> Adam Tygart
> Beocat Sysadmin
>
> On Wed, Dec 19, 2012 at 8:29 PM, Daniel Mons <daemons at kanuka.com.au>
> wrote:
> > I'm rolling out 4 lots of 6-node GlusterFS setups for my employer.  Each
> > node is ~33TB of RAID6 backed storage (16x 3TB SATA disks in RAID6 with a
> > hot spare hanging off an LSI controller, with 2x SSDs configured for
> > caching), and Gluster is configured in distribute-replicate.  Each
> cluster
> > is 200TB of raw space, 100TB usable after replication.  When complete,
> there
> > will be 4 of these clusters.
> >
> > Nodes are configured as XFS with 512byte inodes, running a fully patched
> > CentOS6 and Gluster 3.3.1.  Each node has a 6 core Xeon processor (with
> HT
> > for 12 threads) with 32GB of RAM.  Each node runs 2x 10Gbps Ethernet over
> > fiber in a bonded configuration (single IP address per node) for a full
> > 20Gbits per node.
> >
> > GlusterFS FUSE performance under Linux is great (clients run a mix of
> Ubuntu
> > 12.04 LTS for workstations and CentOS6 for servers).  Samba performance
> back
> > to Windows 7 clients is great.  NFS performance via both Gluster's
> userspace
> > setup as well as CentOS6's native NFS4 kernel server are great to most
> other
> > systems where we can't get the Gluster FUSE client loaded (large
> > industry-specific Linux boxes that are provided by vendors as a "black
> box"
> > solution, and only allow limited access via NFS or SMB/CIFS).  All
> testing
> > so far under those conditions proves orders of magnitude faster
> throughput
> > than our existing single NAS solutions.
> >
> > MacOSX Finder performance is a problem, however.  There's a huge bug in
> > MacOSX itself that prevents using NFS at all (discussions on other
> mailing
> > lists suggest it occurred somewhere around 10.6, and continues through
> into
> > 10.7 and 10.8).
> >
> > Mounting via SMB under OSX is more stable than NFS, however in folders
> with
> > a large amount of files, Finder goes looking for a corresponding Apple
> > Resource Fork file (for every "filename.ext", it looks for a
> > "._filename.ext").  Running tcpdump and wireshark on the Gluster nodes
> shows
> > that the resulting "FILE_NOT_FOUND" error back to the client takes a very
> > long time.  Configuring a single node as a pure NAS with the same
> software
> > (but no Gluster implementation) is lightening fast.   As soon as
> GlusterFS
> > comes in to play, reporting of each "FILE_NOT_FOUND" slows down the
> process
> > dramatically, causing a directory with ~1000 images in it to take well
> over
> > 5 minutes to display the contents in MacOSX finder.
> >
> > This problem is resolved somewhat by switching to AFP (via Netatalk
> loaded
> > on the GlusterFS nodes), but it has it's own problems unique to that
> > protocol, and I'd rather stick to GlusterFS-FUSE, NFS or SMB in that
> order
> > of preference.
> >
> > It's worth noting that through the terminal, these problems don't exist.
> > Mounting via SMB, browsing to the volume in terminal and running "ls" or
> > "find" style commands retrieve file listings at a similar speed to Linux
> and
> > Windows.  The problem is limited to clients using Finder to browse
> > directories, and again particularly ones with a large number of files
> that
> > don't have matching Apple Resource Fork files.  (Of note, creating empty
> > files of the matching "._filename.ext" format solves the performance
> > problem, but litters our filestores with millions of empty files, which
> we
> > don't want).
> >
> > I understand the problem is not strictly Gluster's issue.  Finder is
> looking
> > for a heck of a lot of files that don't exist (which is a pretty silly
> > design), and it tends to occur only with Samba re-exporting GlusterFS
> > volumes that we can see.  And likewise Apple's NFS bug that has now been
> in
> > existence across three releases of their OS is pretty horrible.   But
> > hopefully I can at least describe the problem and prompt some testing by
> > others.
> >
> > I haven't had a chance to test a MacOSX FUSE client due to time
> constraints,
> > but that would at least answer the question if the problem is Gluster's
> lag
> > in reporting of files not found, or Samba's.
> >
> > -Dan
> >
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130205/d62446a1/attachment.html>