[Bugs] [Bug 1171847] Some newly created folders have root ownership although created by unprivileged user

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1171847

Pranith Kumar K <pkarampu at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |achauras at redhat.com



--- Comment #26 from Pranith Kumar K <pkarampu at redhat.com> ---
Guys,
    These are my observations, based on which I made some fixes:
1) If we look at the logs from the case, last disconnect happened on 18th
October.

2) The case does suggest that there had been cases where the 'permission
denied' errors were happening intermittently but they were fixed all by itself.
Most of these error-logs related to permissions are coming only in server
xlators and not at all in posix xlator.

Grepped for both EACCES and EPERM (i.e. permission denied and operation not
permitted)
pk1 at pranithk-laptop - ~/sos/1171847
17:49:16 ⚡ grep-bricks -i permi | wc -l
224

pk1 at pranithk-laptop - ~/sos/1171847
17:50:05 ⚡ grep-bricks -i permi | grep posix | grep -vi "lookup issued on
.glusterfs" | wc -l
0

3) The only other xlator which gives out these kinds of errors is posix-acl.
But the directories are created by a process with uid/gid 1000:502.
    There are two possibilities:
        a) If spurious heals are triggered in dht/afr the permissions should
never be set to root:root with out a bug in setattr code some-where. I looked
through the code in afr and dht, except in dht_rmdir fop which triggers dir
self-heal, I didn't find any code where this is happening at the moment (Shyam,
It is better if more experienced eyes with dht knowledge does one more sweep of
the code, I am asking you to go through the code in a round about manner :-D)
http://review.gluster.org/9435 - this only prevents it for dht with one
subvolume, we need to fix it in more general cases. I will leave it to zteam to
do that. Already told Du and Nitya

       b) Other possibility is code-bugs in posix xlator which do not set the
permissions properly. I found one such issue in posix_symlink for setfsgid()
    http://review.gluster.org/9431
        c) when I tested the patch above I found that our code paths are
hitting 'ifndef HAVE_SET_FSID' macro, so there is this very small window where
the new entry is with uid/gid root:root and it can be read through lookup and
readdirp. Both dht and afr can use this stat buffer to perform their respective
heals and the entry will end up with root:root forever. I tried very hard to
hit this race on my laptop but I was not sure if I hit it because of the kind
of test case which was re-creating the issue for me. But it definitely was
producing similar permission denied messages for setattr and setxattr. I sent
out the patch which performs gfid setting at the end of creating the new entry
to prevent these errors in the logs. Lookup waits for gfid to be assigned
before returning the call where as readdirp's output is not valid when gfid is
not set in the stbuf, so both of these problems go away.
http://review.gluster.org/9434

'3c)' explains why there is an intermittent problem with some entry and it is
fixed all by itself as mentioned in '2)'. As soon as chown happens after the
entry creation things will go back to normal and the errors will stop.

test case:

Create a plain replicate volume 'r2'
disable stat-prefetch
Create two fuse-mounts with --attribute-timeout=0 and --entry-timeout=0
chown <normal-user>:<normal-user> <mnt> #On my laptop the command was 'chown
pk1:pk1 /mnt/r2'
On both the mounts execute the following command:
while true; do mkdir d1; touch d1/a; rm d1/a; rmdir d1; done

This gives out the permission denied messages on the bricks for setattr and
setxattr.

My future directions for this bug:
Try to change the test in such a way that we can verify that the file is
forever in root:root state vs intermittently. Any help here is much
appreciated.

Shyam, feel free to let me know if you find something interesting in dht or any
other xlator.

Amit and Bhaskar,
       Basically we need a way to hit the mkdir, lookup race in either afr/dht
for this issue to happen. It is probabilistic-ally difficult in afr because afr
does a network fop (inodelk) before it does a lookup so it is very difficult
for it to get root:root at the time of that lookup in self-heal. But dht seems
to be using the same stbuf it gets at the time of lookup throughout self-heal,
so it probably is easier to hit than the afr race.

The only patch which I feel would be a candidate for the fix is: '3c)' As
setfsuid/gid is not happening in our environment. Not a single rmdir message in
the logs so may be they are not doing a lot of rmdirs.

Pranith

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=vogUophU2a&a=cc_unsubscribe