[Bugs] [Bug 1397252] New: [setxattr_cbk] "Permission denied" warning messages are seen in logs while running pjd-fstest suite
bugzilla at redhat.com
bugzilla at redhat.com
Tue Nov 22 03:33:35 UTC 2016
https://bugzilla.redhat.com/show_bug.cgi?id=1397252
Bug ID: 1397252
Summary: [setxattr_cbk] "Permission denied" warning messages
are seen in logs while running pjd-fstest suite
Product: GlusterFS
Version: 3.9
Component: distribute
Severity: medium
Assignee: bugs at gluster.org
Reporter: nbalacha at redhat.com
CC: bugs at gluster.org, rhs-bugs at redhat.com,
srangana at redhat.com, storage-qa-internal at redhat.com,
tdesala at redhat.com
Depends On: 1392772
Blocks: 1391808
+++ This bug was initially created as a clone of Bug #1392772 +++
+++ This bug was initially created as a clone of Bug #1391808 +++
Description of problem:
=======================
Using pjdfstest filesystem test suite when created a directory with its access
and group permissions below warning messages are seen in FUSE and brick logs,
FUSE logs:
=========
[2016-11-03 10:27:10.948703] W [MSGID: 114031]
[client-rpc-fops.c:1031:client3_3_setxattr_cbk] 2-distrep-client-7: remote
operation failed [Permission denied]
Brick logs:
===========
[2016-11-03 10:26:56.494850] I [MSGID: 115060]
[server-rpc-fops.c:865:_gf_server_log_setxattr_failure] 0-distrep-server: 2505:
SETXATTR /dir/dir (ef02bc07-ba92-4e40-bd81-1661b0d41582) ==>
trusted.glusterfs.dht
[2016-11-03 10:26:56.495084] I [MSGID: 115060]
[server-rpc-fops.c:893:server_setxattr_cbk] 0-distrep-server: Permission denied
Version-Release number of selected component (if applicable):
3.8.4-3.el7rhgs.x86_64
How reproducible:
=================
Always
Steps to Reproduce:
===================
1) Create a distributed replica volume and start it.
2) FUSE mount the volume on a client.
3) As a root user run the below command from mount point to clone pjd-fstest
git://git.code.sf.net/p/ntfs-3g/pjd-fstest
4) Run the below commands
a) ./pjd-fstest/fstest mkdir dir 0777
b) ./pjd-fstest/fstest chown dir 65534 65534
c) ./pjd-fstest/fstest -u 65534 -g 65534 create dir/file 0464
d) ./pjd-fstest/fstest -u 65534 -g 65534 mkdir dir/dir 0464
When executed 4d the permission denied messages are seen in FUSE and brick
logs.
Also, after issuing 'ls' on the parent directory created in 4a, we are seeing
the below info message in FUSE logs,
[2016-11-03 10:30:54.284974] I [MSGID: 109063]
[dht-layout.c:713:dht_layout_normalize] 2-distrep-dht: Found anomalies in
/dir/dir (gfid = ef02bc07-ba92-4e40-bd81-1661b0d41582). Holes=1 overlaps=0
But, I did not see any holes on the directory created in 4d,
[root at server7 ~]# getfattr -d -e hex -m trusted.glusterfs.dht
/bricks/brick*/*/dir/dir
getfattr: Removing leading '/' from absolute path names
# file: bricks/brick0/b0/dir/dir
trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff
# file: bricks/brick1/b1/dir/dir
trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553
# file: bricks/brick2/b2/dir/dir
trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7
[root at server141 ~]# getfattr -d -e hex -m trusted.glusterfs.dht
/bricks/brick*/*/dir/dir
getfattr: Removing leading '/' from absolute path names
# file: bricks/brick0/b0/dir/dir
trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9
# file: bricks/brick1/b1/dir/dir
trusted.glusterfs.dht=0x0000000100000000555555547ffffffd
# file: bricks/brick2/b2/dir/dir
trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551
Actual results:
===============
[setxattr_cbk] "Permission denied" warning messages are seen in logs while
running pjd-fstest suite
Expected results:
=================
Permission denied messages should not be seen.
--- Additional comment from Shyamsundar on 2016-11-07 13:44:52 EST ---
(In reply to Prasad Desala from comment #0)
> Also, after issuing 'ls' on the parent directory created in 4a, we are
> seeing the below info message in FUSE logs,
>
> [2016-11-03 10:30:54.284974] I [MSGID: 109063]
> [dht-layout.c:713:dht_layout_normalize] 2-distrep-dht: Found anomalies in
> /dir/dir (gfid = ef02bc07-ba92-4e40-bd81-1661b0d41582). Holes=1 overlaps=0
>
> But, I did not see any holes on the directory created in 4d,
>
> [root at server7 ~]# getfattr -d -e hex -m trusted.glusterfs.dht
> /bricks/brick*/*/dir/dir
This is because the directory was created without the DHT xattrs and as the
client detected an anomaly in the directory, it healed it as root, and so the
layout *finally* on disk was not empty.
Before doing the ls on ./dir/ if you would take a look at the xattrs on the
brick, you would find them missing.
--- Additional comment from Shyamsundar on 2016-11-07 14:07:13 EST ---
I was able to replicate the problem based on the steps given (thanks).
What occurs here is as follows,
- DHT sends the mkdir call to the bricks
- Post the mkdir, it sends the setxattr call to set the layout xattrs for these
directories
The failure is noted in the bricks and on the FUSE client logs, as the setxattr
fails.
The setxattr fails, due to the permissions on the directory being 0464 (set
during the mkdir as requested by the call), where user has only READ
permissions, and posix_acl xlator checks that the UID of the setxattr caller
(internal DHT request above) is the same as the UID of the directory inode, and
hence evaluates permissions based on this, which evaluates to READ ONLY and
denies the setxattr from proceeding.
Code reference:
-
https://github.com/gluster/glusterfs/blob/master/xlators/system/posix-acl/src/posix-acl.c#L2088
posix_acl_setxattr -> calls setxattr_scrutiny
-
https://github.com/gluster/glusterfs/blob/master/xlators/system/posix-acl/src/posix-acl.c#L1920
setxattr_scrutiny -> acl_permits calls
- acl_permits: Has the POSIX_ACL_USER_OBJ in its list first, and checks the
UID equality of the caller to the inode (which matches) and hence uses
permissions of the user to proceed doing a perm_check
The downside of all this, is that the DHT layout on this directory is not set,
only on a subsequent layout anomaly detection by DHT is this corrected, as
anomaly correction is an internal operation that uses UID/GID as root/root and
the ACL xlator passes the request without the checks.
General discussion:
The manner in which gluster ACL xlator checks the permissions (user first and
then group etc.) is correct as per standards, the reference would be,
https://www.gnu.org/software/libc/manual/html_node/Access-Permission.html#Access-Permission
Further on checking the kernel code, the same rules apply there as well, and a
directory created with READ for user, subsequently fails with EPERM for any
setfattr calls. Kernel code reference:
http://lxr.free-electrons.com/source/fs/posix_acl.c#L346
So basically, the implementation in out ACL xlator seems to be right. I.e just
because group bits provide WRITE permissions, it is not allowed as the UID of
the directory to perform any WRITE operations.
What this means:
So currently other than the logs, there is no real bug to start with. But, the
finding is interesting, and we could make the setxattr of DHT xattrs post mkdir
an internal operation to be done as root/root (just like healing an anomaly).
That would fix the logging problem as well. Is it a security violation is
something that is needed to be considered.
For example xattrop that is driven by the client (and is actually a setxattr
call in the end) has no ACL xlator checks against that FOP (and as a result
does not suffer this problem).
Regression notes:
This should not be a regression, and should be present in the prior releases as
well, just a heads up!
--- Additional comment from Worker Ant on 2016-11-08 03:58:55 EST ---
REVIEW: http://review.gluster.org/15794 (cluster/dht Set layout after mkdir as
root) posted (#1) for review on master by N Balachandran (nbalacha at redhat.com)
--- Additional comment from Worker Ant on 2016-11-21 09:40:19 EST ---
COMMIT: http://review.gluster.org/15794 committed in master by Jeff Darcy
(jdarcy at redhat.com)
------
commit 3e405b546e8b9fe15ae477613474e9cd2d2df4e7
Author: N Balachandran <nbalacha at redhat.com>
Date: Tue Nov 8 14:10:49 2016 +0530
cluster/dht Set layout after mkdir as root
DHT does not set the layout for newly created
directories as root. This causes EPERM failures
when a non-root user with insufficient permissions
creates directories.
credit: srangana at redhat.com for RCA
Change-Id: Ia646e41665ce172c43c5f01d2707455e8eb374ed
BUG: 1392772
Signed-off-by: N Balachandran <nbalacha at redhat.com>
Reviewed-on: http://review.gluster.org/15794
Reviewed-by: Susant Palai <spalai at redhat.com>
NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
Smoke: Gluster Build System <jenkins at build.gluster.org>
Reviewed-by: Raghavendra G <rgowdapp at redhat.com>
Reviewed-by: Jeff Darcy <jdarcy at redhat.com>
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1391808
[Bug 1391808] [setxattr_cbk] "Permission denied" warning messages are seen
in logs while running pjd-fstest suite
https://bugzilla.redhat.com/show_bug.cgi?id=1392772
[Bug 1392772] [setxattr_cbk] "Permission denied" warning messages are seen
in logs while running pjd-fstest suite
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list