[Gluster-devel] opendir/readdir helper

[Jeff Darcy]

As we all know, directory-listing performance (or lack thereof) is a bit 
of a sore spot for many GlusterFS users, because it's one of the few 
places where FUSE really does make a difference.  It will probably 
always be a sore spot even with the readdirp changes that are already 
under way.  The next step would be to add a FUSE enhancement to "inject" 
directory entries before they're requested, but that's a lot of work for 
an uncertain outcome.  The FUSE haters among the kernel leadership would 
probably reject such changes without serious consideration, and even if 
I'm wrong about that it's likely to be a long time before they make it 
into the various distributions (not to mention non-Linux platform issues).

So, let's think outside the box for a bit.  What about an LD_PRELOAD 
helper?  Believe me, I know all about the problems with LD_PRELOAD, but 
I still can't think of any reasonable use case that requires readdir to 
work across a fork (for example).  The basic idea is that the LD_PRELOAD 
would catch calls to opendir/readdir and match them against paths 
matching GlusterFS volumes.  If a match is found, then it would use 
libgfapi to serve the results, without any FUSE involvement and with 
massive prefetching goodness etc.  Without a match, the helper would 
naturally fall back to the system functions.

I suspect that this approach would make listings on very large single 
directories many times faster than would ever be possible with FUSE. 
For deeply nested directories we'd need to add some more complexity so 
that we're not going through the whole connection-establishment path 
(including authentication etc.) for each directory separately, but 
that's all pretty well understood pain for pretty obvious gain.

Any other thoughts?