[Gluster-devel] [Nfs-ganesha-devel] nfsv3 client writing file gets Invalid argument on glusterfs with quota on

[Kinglong Mee]

On Thu, Mar 8, 2018 at 10:17 PM Daniel Gryniewicz <dang at redhat.com> wrote:

> On 03/07/2018 10:21 PM, Kinglong Mee wrote:
> > On 2018/3/7 21:10, Daniel Gryniewicz wrote:
> >> On 03/06/2018 10:10 PM, Kinglong Mee wrote:
> >>> On 2018/3/7 10:59, Kinglong Mee wrote:
> >>>> When using nfsv3 on glusterfs-3.13.1-1.el7.x86_64 and
> nfs-ganesha-2.6.0-0.2rc3.el7.centos.x86_64,
> >>>> I gets strange "Invalid argument" when writing file.
> >>>>
> >>>> 1. With quota disabled;
> >>>> nfs client mount nfs-ganesha share, and do 'll' in the testing
> directory.
> >>>>
> >>>> 2. Enable quota;
> >>>> # getfattr -d -m . -e hex /root/rpmbuild/gvtest/nfs-ganesha/testfile92
> >>>> getfattr: Removing leading '/' from absolute path names
> >>>> # file: root/rpmbuild/gvtest/nfs-ganesha/testfile92
> >>>> trusted.gfid=0xe2edaac0eca8420ebbbcba7e56bbd240
> >>>>
> trusted.gfid2path.b3250af8fa558e66=0x39663134343566662d653530332d343831352d396635312d3236633565366332633137642f7465737466696c653932
> >>>>
> trusted.glusterfs.quota.9f1445ff-e503-4815-9f51-26c5e6c2c17d.contri.3=0x00000000000002000000000000000001
> >>>>
> >>>> Notice: testfile92 without trusted.pgfid xattr.
> >>>
> >>> The trusted.pgfid will be created by the next name lookup; nameless
> lookup don't create it.
> >>>
> >>>> 3. restart glusterfs volume by "gluster volume stop/start gvtest"
> >>>
> >>> Restarting glusterfsd here cleanup all inode cache from memory;
> >>> after starting, inode of testfile92's parent is NULL.
> >>>
> >>>> 4. echo somedata > testfile92
> >>>
> >>> Because, nfs-ganesha and nfs client has cache for testfile92,
> >>> before write fops, no name lookup happens that trusted.pgfid is not
> created for testfile92.
> >>>
> >>> Quota_writev call quota_build_ancestry building the ancestry in
> quota_check_limit,
> >>> but testfile92 doesn't contain trusted.pgfid, so that write fops
> failed with Invalid argument.
> >>>
> >>> I have no idea of fixing this problem, any comments are welcome.
> >>>
> >>
> >> I think, ideally, Gluster would send an invalidate upcall under the
> circumstances, causing Ganesha do drop it's cached entry.
> >
> > It doesn't work.
> > I try to restarting nfs-ganesha, and echo data to testfile92, the
> problem also exists.
> >
> > After nfs-ganesha restart,
> > 1. A GETATTR is send from nfs-client for the testfile92, ganesha
> translates it to nameless lookup.
> > 2. A ACCESS gets attributes from nfs-ganesha's cache (cached by #1).
> > 3. A SETATTR sets the testfile92's size to 0, ganesha translates it to
> setattr fop.
> > 4. A WRITE also get Invalid argument error.
> >
> > If ganesha drops its cache, nfs client may write file by filehandle;
> > ganesha lookup it by nameless lookup from glusterfs,
> > so that, trusted.pgfid isn't created too.
> >
> > I think, a name lookup is needed for testfile92 after quota enable.
> >
> > thanks,
> > Kinglong Mee
> >
>
> If a name lookup is needed, then gluster cannot provide NFS semantics in
> these circumstances.  NFS *requires* that access by handle continue to
> work once the client has a handle.  In fact, there's no way to get a
> name for this, since the name doesn't exist anywhere in the handle (by
> design, since a name is an attribute of a dirent, not of a handle/inode).
>
> So, this likely is a bug in Gluster, and needs to be fixed there.  Would
> it be possible to enable quota globally at the start as a workaround?
>

Quota is a function that can be enabled at any time.
If enable quota immediately after creating filesystem (contains only
one directory "/"),
the trusted.pgfid extend-attribute is created automatically when creating file,
this problem is not exit.

thanks,
Kinglong Mee



>
> Daniel
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20180311/a2af2f5f/attachment.html>