[Gluster-users] [EXT] Permission denied closing file when accessing GlusterFS via NFS

Fri Aug 13 00:06:39 UTC 2021

Hello,

Strahil, it looks to me like gluster issue #876 is not the same problem,
but I confess that I don't understand all of that issue.

Stefan, thanks very much for the suggestion. Using nfsvers=4.1 does solve
the problem on an NFS client running CentOS 7, while nfsversion=4.0 has the
problem, so that's good. Using nfsvers=4.1 doesn't solve the problem on a
client using CentOS 6 though. I have verified that version 4.1 is reported
by "mount":
gfs1:/mnt/glusterfs on /var/lib/gfs type nfs
(rw,noatime,hard,bg,intr,nfsvers=4,minorversion=1,addr=1.2.3.4,clientaddr=5.6.7.8)

So it's strange that NFS version 4.1 doesn't solve it on the older CentOS
release. Unfortunately our customer does have a few of these old boxes
still around.

Thanks again.

On Fri, 13 Aug 2021 at 02:33, Stefan Solbrig <stefan.solbrig at ur.de> wrote:

> A while ago, I had a similar problem with an nfs re-export of GlusterFS.
> In my case, enforcing NFS version 4.2 solved the issuse:
>
> mount options:  vers=4,rw,hard,nosuid,minorversion=2
>
> best wishes,
> Stefan
>
> --
> Dr. Stefan Solbrig
> Universität Regensburg, Fakultät für Physik,
> 93040 Regensburg, Germany
> Tel +49-941-943-2097
>
> > Am 12.08.2021 um 07:40 schrieb Strahil Nikolov <hunter86_bg at yahoo.com>:
> >
> > The FUSE client mount options "do not ring any bells" to me - they look
> fine.
> >
> > For the ACL stuff, I couldn't find the commit too.Maybe it's for a newer
> version than yours.
> >
> > You can check if  gluster issue #876 matches your case.
> >
> > Which version of gluster are you using ?
> >
> > Best Regards,
> > Strahil Nikolov
> >
> >
> > On Thu, Aug 12, 2021 at 6:34, David Cunningham
> > <dcunningham at voisonics.com> wrote:
> > I've noticed something interesting: when the "cp" over NFS is done the
> first time it gives the "cp: closing" permission denied error as described
> before and the destination file is created with zero size.
> >
> > If we run the same "cp" over NFS a second time the files are written
> correctly, with the full file data, and there's no error. So it appears
> that the problem does not occur if the destination file exists.
> >
> > Does that give anyone a clue as to what's happening? Thanks.
> >
> >
> > On Thu, 12 Aug 2021 at 13:50, David Cunningham <
> dcunningham at voisonics.com> wrote:
> > Hi,
> >
> > Gilberto, sorry I didn't realise those were server options. After adding
> them to the /etc/exports entry and restarting nfsd the same problem still
> occurs.
> >
> > Strahil, I ran "gluster volume set help | egrep -i 'acl|xlator'" on the
> GlusterFS node/FUSE client and got the following results. Is one of them
> the option you mention?
> > Description: inode-read fops happen only on one of the bricks in
> replicate. Afr will prefer the one specified using this option if it is not
> stale. Option value must be one of the xlator names of the children. Ex:
> <volname>-client-0 till <volname>-client-<number-of-bricks - 1>
> > Description: Cache samba metadata (user.DOSATTRIB, security.NTACL xattr)
> > Description: maximum size of cache consumed by readdir-ahead xlator.
> This value is global and total memory consumption by readdir-ahead is
> capped by this value, irrespective of the number/size of directories cached
> >
> > Could the issue be anything to do with the FUSE client mount options?
> The output of "mount | grep glusterfs" gives:
> > gfs1:/gvol0 on /mnt/glusterfs type fuse.glusterfs
> (rw,noatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
> >
> > Thanks again.
> >
> >
> > On Thu, 12 Aug 2021 at 01:25, Gilberto Ferreira <
> gilberto.nunes32 at gmail.com> wrote:
> > Those options you need put it in the NFS server options, generally in
> /etc/exports
> > ---
> > Gilberto Nunes Ferreira
> > (47) 99676-7530 - Whatsapp / Telegram
> >
> >
> >
> >
> >
> > Em ter., 10 de ago. de 2021 às 18:24, David Cunningham <
> dcunningham at voisonics.com> escreveu:
> > Hi Strahil and Gilberto,
> >
> > Thanks very much for your replies. SELinux is disabled on the NFS server
> (and the client too), and both have the same UID and GID for the user who
> owns the files.
> >
> > On the NFS mount we had options "rw,noatime,hard,bg,intr,vers=4". I
> added "async" which did not solve the problem, and the NFS client mount
> gave an error when trying to use "no_root_squash" or "no_subtree_check".
> Gilberto, is there a specific reason why you suggested those options?
> >
> > Thanks again.
> >
> >
> > On Wed, 11 Aug 2021 at 03:55, Gilberto Ferreira <
> gilberto.nunes32 at gmail.com> wrote:
> > HOw about the NFS options?
> > (rw,async,no_root_squash,no_subtree_check)
> > ---
> > Gilberto Nunes Ferreira
> > (47) 99676-7530 - Whatsapp / Telegram
> >
> >
> >
> >
> >
> > Em ter., 10 de ago. de 2021 às 12:46, Strahil Nikolov <
> hunter86_bg at yahoo.com> escreveu:
> > Hey David,
> >
> > can you give the volume info ?
> >
> > Also, I assume SELINUX is in permissive/disabled state.
> >
> > What about the uod of the user on the nfs client and the nfs server ? Is
> it the same ?
> >
> > Best Regards,
> > Strahil Nikolov
> >
> > On Tue, Aug 10, 2021 at 5:52, David Cunningham
> > <dcunningham at voisonics.com> wrote:
> > Hello,
> >
> > We have a GlusterFS node which also uses the FUSE client to mount the
> filesystem. The same GlusterFS node server also runs an NFS server which
> exports the FUSE client mount, and another machine NFS mounts it.
> >
> > When the NFS client writes data to the mounted filesystem we are seeing
> "Permission denied" errors like this:
> >
> > cp: closing
> `/var/lib/gfs/company/david/1075/Copyrec/1628448189883606-203-17184805327-out-08-08-21-14~43~10-203.mp3':
> Permission denied
> >
> > The file mentioned in the error is actually created on the GlusterFS
> filesystem, but has zero size, so the problem is not a normal Linux
> filesystem permission one.
> >
> > In the brick log nodirectwritedata-gluster-gvol0.log on the GlusterFS
> node we see an error as follows. Would anyone have a suggestion on what the
> problem might be? Thank you in advance!
> >
> > [2021-08-10 02:30:20.359159] I [MSGID: 139001]
> [posix-acl.c:262:posix_acl_log_permit_denied] 0-gvol0-access-control:
> client:
> CTX_ID:8f69363a-f0f4-44e1-84e9-69dfa77a8164-GRAPH_ID:0-PID:2657-HOST:gfs1.company.com-PC_NAME:gvol0-client-0-RECON_NO:-0,
> gfid: f70b1cd6-745a-4ea6-b0a5-1fcfef960f15,
> req(uid:106,gid:111,perm:2,ngrps:0),
> ctx(uid:106,gid:111,in-groups:1,perm:000,updated-fop:SETATTR, acl:-)
> [Permission denied]
> > [2021-08-10 02:30:20.359187] E [MSGID: 115070]
> [server-rpc-fops_v2.c:1502:server4_open_cbk] 0-gvol0-server: 5554927: OPEN
> /company/david/1075/Copyrec/1628448189883606-203-17184805327-out-08-08-21-14~43~10-203.mp3
> (f70b1cd6-745a-4ea6-b0a5-1fcfef960f15), client:
> CTX_ID:8f69363a-f0f4-44e1-84e9-69dfa77a8164-GRAPH_ID:0-PID:2657-HOST:gfs1.company.com-PC_NAME:gvol0-client-0-RECON_NO:-0,
> error-xlator: gvol0-access-control [Permission denied]
> >
> > --
> > David Cunningham, Voisonics Limited
> > http://voisonics.com/
> > USA: +1 213 221 1092
> > New Zealand: +64 (0)28 2558 3782
> > ________
> >
> >
> >
> > Community Meeting Calendar:
> >
> > Schedule -
> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> > Bridge: https://meet.google.com/cpu-eiue-hvk
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
> > ________
> >
> >
> >
> > Community Meeting Calendar:
> >
> > Schedule -
> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> > Bridge: https://meet.google.com/cpu-eiue-hvk
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
> >
> >
> > --
> > David Cunningham, Voisonics Limited
> > http://voisonics.com/
> > USA: +1 213 221 1092
> > New Zealand: +64 (0)28 2558 3782
> >
> >
> > --
> > David Cunningham, Voisonics Limited
> > http://voisonics.com/
> > USA: +1 213 221 1092
> > New Zealand: +64 (0)28 2558 3782
> >
> >
> > --
> > David Cunningham, Voisonics Limited
> > http://voisonics.com/
> > USA: +1 213 221 1092
> > New Zealand: +64 (0)28 2558 3782
> > ________
> >
> >
> >
> > Community Meeting Calendar:
> >
> > Schedule -
> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> > Bridge: https://meet.google.com/cpu-eiue-hvk
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>

-- 
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210813/34b4c7ae/attachment.html>