[Gluster-devel] Report ESTALE as ENOENT

[J. Bruce Fields]

On Thu, Feb 22, 2018 at 01:17:58PM +0530, Raghavendra G wrote:
> On Wed, Oct 11, 2017 at 7:32 PM, J. Bruce Fields <bfields at fieldses.org>
> wrote:
> 
> > On Wed, Oct 11, 2017 at 04:11:51PM +0530, Raghavendra G wrote:
> > > On Thu, Mar 31, 2016 at 1:22 AM, J. Bruce Fields <bfields at fieldses.org>
> > > wrote:
> > >
> > > > On Mon, Mar 28, 2016 at 04:21:00PM -0400, Vijay Bellur wrote:
> > > > > I would prefer to:
> > > > >
> > > > > 1. Return ENOENT for all system calls that operate on a path.
> > > > >
> > > > > 2. ESTALE might be ok for file descriptor based operations.
> > > >
> > > > Note that operations which operate on paths can fail with ESTALE when
> > > > they attempt to look up a component within a directory that no longer
> > > > exists.
> > > >
> > >
> > > But, "man 2 rmdir"  or "man 2 unlink" doesn't list ESTALE as a valid
> > error.
> >
> > In fact, almost no man pages list ESTALE as a valid error:
> >
> >         [bfields at patate man-pages]$ git grep ESTALE
> >         Changes.old:        Change description for ESTALE
> >         man2/open_by_handle_at.2:.B ESTALE
> >         man2/open_by_handle_at.2:.B ESTALE
> >         man3/errno.3:.B ESTALE
> >
> > Cc'ing Michael Kerrisk for advice.  Is there some reason for that, or
> > can we fix those man pages?
> >
> > > Also rm doesn't seem to handle ESTALE too [3]
> > >
> > > [4] https://github.com/coreutils/coreutils/blob/master/src/remove.c#L305
> >
> > I *think* that code is just deciding whether a given error should be
> > silently ignored in the rm -f case.  I don't think -ESTALE (indicating
> > the directory is bad) is such an error, so I think this code is correct.
> > But my understanding may be wrong.
> >
> 
> For a local filesystem, we may not end up in ESTALE errors. But, when rmdir
> is executed from multiple clients of a network fs (like NFS, Glusterfs),
> unlink or rmdir can easily fail with ESTALE as the other rm invocation
> could've deleted it. I think this is what has happened in bugs like:
> https://bugzilla.redhat.com/show_bug.cgi?id=1546717
> https://bugzilla.redhat.com/show_bug.cgi?id=1245065
> 
> This in fact was the earlier motivation to convert ESTALE into ENOENT, so
> that rm would ignore it. Now that I reverted the fix, looks like the bug
> has promptly resurfaced :)
> 
> There is one glitch though. Bug 1245065 mentions that some parts of
> directory structure remain undeleted. From my understanding, atleast one
> instance of rm (which is racing ahead of all others causing others to
> fail), should've delted the directory structure completely. Though, I need
> to understand the directory traversal done by rm to find whether there are
> cyclic dependency between two rms causing both of them to fail.

I don't see how you could avoid that.  The clients are each caching
multiple subdirectories of the tree, and there's no guarantee that 1
client has fresher caches of every subdirectory.  There's also no
guarantee that the client that's ahead stays ahead--another client that
sees which objects the first client has already deleted can leapfrog
ahead.

I think the solution is just not to do that--NFS clients aren't really
equipped to handle directory operations on directories that are deleted
out from under them, and there probably aren't any hacks on the server
side that will fix that.  If there's a real need for this kind of case,
we may need to work on the protocol itself.  For now all we may be able
to do is educate users about what NFS can and can't do.

--b.