<div dir="ltr"><br><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 22, 2018 at 1:17 PM, Raghavendra G <span dir="ltr"><<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-h5">On Wed, Oct 11, 2017 at 7:32 PM, J. Bruce Fields <span dir="ltr"><<a href="mailto:bfields@fieldses.org" target="_blank">bfields@fieldses.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-m_-4482461411044056547gmail-">On Wed, Oct 11, 2017 at 04:11:51PM +0530, Raghavendra G wrote:<br>
> On Thu, Mar 31, 2016 at 1:22 AM, J. Bruce Fields <<a href="mailto:bfields@fieldses.org" target="_blank">bfields@fieldses.org</a>><br>
> wrote:<br>
><br>
> > On Mon, Mar 28, 2016 at 04:21:00PM -0400, Vijay Bellur wrote:<br>
</span><span class="gmail-m_-4482461411044056547gmail-">> > > I would prefer to:<br>
> > ><br>
> > > 1. Return ENOENT for all system calls that operate on a path.<br>
> > ><br>
> > > 2. ESTALE might be ok for file descriptor based operations.<br>
> ><br>
> > Note that operations which operate on paths can fail with ESTALE when<br>
> > they attempt to look up a component within a directory that no longer<br>
> > exists.<br>
> ><br>
><br>
> But, "man 2 rmdir" or "man 2 unlink" doesn't list ESTALE as a valid error.<br>
<br>
</span>In fact, almost no man pages list ESTALE as a valid error:<br>
<br>
[bfields@patate man-pages]$ git grep ESTALE<br>
Changes.old: Change description for ESTALE<br>
man2/open_by_handle_at.2:.B ESTALE<br>
man2/open_by_handle_at.2:.B ESTALE<br>
man3/errno.3:.B ESTALE<br>
<br>
Cc'ing Michael Kerrisk for advice. Is there some reason for that, or<br>
can we fix those man pages?<br>
<span class="gmail-m_-4482461411044056547gmail-"><br>
> Also rm doesn't seem to handle ESTALE too [3]<br>
><br>
> [4] <a href="https://github.com/coreutils/coreutils/blob/master/src/remove.c#L305" rel="noreferrer" target="_blank">https://github.com/coreutils/c<wbr>oreutils/blob/master/src/remov<wbr>e.c#L305</a><br>
<br>
</span>I *think* that code is just deciding whether a given error should be<br>
silently ignored in the rm -f case. I don't think -ESTALE (indicating<br>
the directory is bad) is such an error, so I think this code is correct.<br>
But my understanding may be wrong.<br></blockquote><div><br></div></div></div><div>For a local filesystem, we may not end up in ESTALE errors. But, when rmdir is executed from multiple clients of a network fs (like NFS, Glusterfs), unlink or rmdir can easily fail with ESTALE as the other rm invocation could've deleted it. I think this is what has happened in bugs like:</div><div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1546717" target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=1546717</a></div><div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1245065" target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=1245065</a></div><div><br></div><div>This in fact was the earlier motivation to convert ESTALE into ENOENT, so that rm would ignore it. Now that I reverted the fix, looks like the bug has promptly resurfaced :)<br></div><div><br></div><div>There is one glitch though. Bug 1245065 mentions that some parts of directory structure remain undeleted. From my understanding, atleast one instance of rm (which is racing ahead of all others causing others to fail), should've delted the directory structure completely. Though, I need to understand the directory traversal done by rm to find whether there are cyclic dependency between two rms causing both of them to fail.<br></div></div></div></div></blockquote><div><br></div><div>Also note that VFS retries unlink and rmdir if there is an estale:</div><div><br></div><div><a href="https://github.com/torvalds/linux/blob/master/fs/namei.c#L4056">https://github.com/torvalds/linux/blob/master/fs/namei.c#L4056</a></div><div><a href="https://github.com/torvalds/linux/blob/master/fs/namei.c#L3927">https://github.com/torvalds/linux/blob/master/fs/namei.c#L3927</a><br></div><div><br> </div><div>So, underlying fs like Glusterfs cannot mask ESTALE as it'll break some functionality.</div><div><br></div><div>In this scenario what are the ways to fix a failing rm when run from multiple mount points (Bugs I pointed out above)? I really can't think of a way other than fixing rm to ignore ESTALE error.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><span class="gmail-"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="gmail-m_-4482461411044056547gmail-"><br>
> > Maybe non-creating open("./foo") returning ENOENT would be reasonable in<br>
> > this case since that's what you'd get in the local filesystem case, but<br>
> > creat("./foo") returning ENOENT, for example, isn't something<br>
> > applications will be written to handle.<br>
> ><br>
> > The Linux VFS will retry ESTALE on path-based systemcalls *one* time, to<br>
> > reduce the chance of ESTALE in those cases.<br>
><br>
><br>
> I should've anticipated bug [2] due to this comment. My mistake. Bug [2] is<br>
> indeed due to kernel not retrying open on receiving an ENOENT error.<br>
> Glusterfs sent ENOENT because file's inode-number/nodeid changed but same<br>
> path exists. The correct error would've been ESTALE, but due to our<br>
> conversion of ESTALE to ENOENT, the latter was sent back to kernel.<br>
><br>
> Looking through kernel VFS code, only open *seems* to retry<br>
> (do_filep_open). I couldn't find similar logic to other path based syscalls<br>
> like rmdir, unlink, stat, chmod etc<br>
<br>
</span>I believe there is a retry in those cases, but I'm not sure exactly<br>
where it is. Looking around.... See the retry_estale() checks sprinkled<br>
around namei.c, which were added by Jeff Layton a few years ago.<br>
<span class="gmail-m_-4482461411044056547gmail-"><br>
--b.<br>
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
</span><a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br>
</blockquote></span></div><span class="gmail-HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="gmail-m_-4482461411044056547gmail_signature">Raghavendra G<br></div>
</font></span></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">Raghavendra G<br></div>
</div></div>