[Bugs] [Bug 1401376] New: inconsistent file permissions b/ w write permission and sticky bits(---------T ) displayed when IOs are going on with md-cache enabled ( and within the invalidation cycle)
bugzilla at redhat.com
bugzilla at redhat.com
Mon Dec 5 04:55:20 UTC 2016
https://bugzilla.redhat.com/show_bug.cgi?id=1401376
Bug ID: 1401376
Summary: inconsistent file permissions b/w write permission and
sticky bits(---------T ) displayed when IOs are going
on with md-cache enabled (and within the invalidation
cycle)
Product: GlusterFS
Version: 3.9
Component: md-cache
Keywords: Triaged
Assignee: bugs at gluster.org
Reporter: pgurusid at redhat.com
CC: amukherj at redhat.com, bugs at gluster.org,
nbalacha at redhat.com, nchilaka at redhat.com,
rhinduja at redhat.com, rhs-bugs at redhat.com,
rjoseph at redhat.com, storage-qa-internal at redhat.com
Depends On: 1384070, 1392713
+++ This bug was initially created as a clone of Bug #1392713 +++
+++ This bug was initially created as a clone of Bug #1384070 +++
Description of problem:
=======================
I have enabled md-cache with private build available at
http://etherpad.corp.redhat.com/md-cache-3-2
I created a 2x2 volume and mounted it on two different clients:
>From one client i created a 3GB txt file and triggered about 10Million hardlink
creation to it as below
(file mnt-distrep.log is the original txt file)
for i in {1..10000000};do ln mnt-distrep.log hardlink.$i;done
I had the volume mounted on another client, say client2:
>From the client2 , I was accessing files using ll or ls -l, everything was
fine.
I then chnaged file permissions of one of the hardlink files say hardlink.90 to
chmod 0777 hardlink.90(all this while the hardlink creations were still going
on on client1)
Now if I did an ls -l, I saw that some of the hardlink files(which probably got
created recently) were having the sticky bit ---------T
I saw that in the 2x2 volume, all files were created in say one replica pair
while the ---------T files for each file was created on the other replica
pair(this is due to the cached and hashed concept of dht ie data and link to
files)
I then chose one file which was displaying ----T on the client2 and kept
doing ll or ls -l
It was inconsistently toggling b/w data file and link to file as below
[root at dhcp35-180 ]# ll hardlink.43363
---------T. 21860 root root 0 Oct 12 17:33 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
---------T. 21866 root root 0 Oct 12 17:33 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
---------T. 21869 root root 0 Oct 12 17:33 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43729 root root 3156080712 Oct 12 17:29 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
---------T. 21880 root root 0 Oct 12 17:33 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43748 root root 3156080712 Oct 12 17:29 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
---------T. 21888 root root 0 Oct 12 17:33 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43758 root root 3156080712 Oct 12 17:29 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43764 root root 3156080712 Oct 12 17:29 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43768 root root 3156080712 Oct 12 17:29 hardlink.43363
My understanding is that , in this case instead of caching or looking up from
cache it is trying to fetch the file info from the bricks and it is
inconsistently fetching from the hashed brick which is not right
Also, If I stopped the hardlink creation on the first client(by ctrl+c of the
first client command line)
I see now consistently displaying only the right file permission as below
[root at dhcp35-180 ]# ll hardlink.43363
---------T. 21860 root root 0 Oct 12 17:33 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
---------T. 21866 root root 0 Oct 12 17:33 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
---------T. 21869 root root 0 Oct 12 17:33 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43729 root root 3156080712 Oct 12 17:29 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
---------T. 21880 root root 0 Oct 12 17:33 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43748 root root 3156080712 Oct 12 17:29 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
---------T. 21888 root root 0 Oct 12 17:33 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43758 root root 3156080712 Oct 12 17:29 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43764 root root 3156080712 Oct 12 17:29 hardlink.43363
[root at dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43768 root root 3156080712 Oct 12 17:29 hardlink.43363
Version-Release number of selected component (if applicable):
====
as in etherpad
How reproducible:
Steps to Reproduce:
1.create md-cache setup on a 2x2 volume
2. mount volume on two clients
3. from cleint 1: create a file and start create of hardlinks to the file in a
loop of say some 1lakh hardlinks
4. From another client , say client2 change file permissions of a hardlink
already created say hardlink.10
5. as long as the 1lakh hardlink creation is in progress, keep issuing ls -l of
hardlink.20 (some hardlink which has been created)
You will see that the file permissions is sometimes shown as what is expected
and sometimes with just the sticky bit
Actual results:
Expected results:
--- Additional comment from Worker Ant on 2016-11-08 00:07:26 EST ---
REVIEW: http://review.gluster.org/15789 (upcall: Do not invalidate if the file
is made a linkto file) posted (#1) for review on master by Poornima G
(pgurusid at redhat.com)
--- Additional comment from Worker Ant on 2016-11-18 03:59:10 EST ---
REVIEW: http://review.gluster.org/15789 (upcall: Do not invalidate if the file
is made a linkto file) posted (#2) for review on master by Poornima G
(pgurusid at redhat.com)
--- Additional comment from Worker Ant on 2016-11-24 04:12:55 EST ---
REVIEW: http://review.gluster.org/15789 (upcall: Do not invalidate if the file
is made a linkto file) posted (#3) for review on master by Poornima G
(pgurusid at redhat.com)
--- Additional comment from Worker Ant on 2016-11-24 04:15:40 EST ---
REVIEW: http://review.gluster.org/15789 (upcall: Do not invalidate if the file
is made a linkto file) posted (#4) for review on master by Poornima G
(pgurusid at redhat.com)
--- Additional comment from Worker Ant on 2016-11-30 04:29:40 EST ---
REVIEW: http://review.gluster.org/15789 (dht/md-cache: Filter invalidate if the
file is made a linkto file) posted (#5) for review on master by Poornima G
(pgurusid at redhat.com)
--- Additional comment from Worker Ant on 2016-11-30 04:32:58 EST ---
REVIEW: http://review.gluster.org/15789 (dht/md-cache: Filter invalidate if the
file is made a linkto file) posted (#6) for review on master by Poornima G
(pgurusid at redhat.com)
--- Additional comment from Worker Ant on 2016-12-02 05:47:02 EST ---
COMMIT: http://review.gluster.org/15789 committed in master by Rajesh Joseph
(rjoseph at redhat.com)
------
commit 4536f7bdf16f8286d67598eda9a46c029f0c0bf4
Author: Poornima G <pgurusid at redhat.com>
Date: Tue Nov 8 10:32:29 2016 +0530
dht/md-cache: Filter invalidate if the file is made a linkto file
Upcall as a part of setattr, sends an invalidation and the
invalidation carries the resulting stat value. When a file
is converted to linkto files, even then an invalidation
is set and as a result the mountpoint shows the sticky
bit in the stat of the file.
eg: ---------T. 945 root root 0 Nov 8 10:14 hardlink.999
Fix:
When dht recieves a notification of sticky bit change, it updates
the flag, to indicate md-cache to send the subsequent lookup.
Change-Id: Ic2fd7a5b196db0754f9b97072e644e6bf69da606
BUG: 1392713
Signed-off-by: Poornima G <pgurusid at redhat.com>
Reviewed-on: http://review.gluster.org/15789
Smoke: Gluster Build System <jenkins at build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
Reviewed-by: Niels de Vos <ndevos at redhat.com>
CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
Reviewed-by: Susant Palai <spalai at redhat.com>
Reviewed-by: Rajesh Joseph <rjoseph at redhat.com>
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1384070
[Bug 1384070] inconsistent file permissions b/w write permission and sticky
bits(---------T ) displayed when IOs are going on with md-cache enabled
(and within the invalidation cycle)
https://bugzilla.redhat.com/show_bug.cgi?id=1392713
[Bug 1392713] inconsistent file permissions b/w write permission and sticky
bits(---------T ) displayed when IOs are going on with md-cache enabled
(and within the invalidation cycle)
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list