[Bugs] [Bug 1337394] New: DHT : few Files are not accessible and not listed on mount + more than one Directory have same gfid + (sometimes) attributes has ?? in ls output after renaming Directories from multiple client at same time
bugzilla at redhat.com
bugzilla at redhat.com
Thu May 19 06:11:06 UTC 2016
https://bugzilla.redhat.com/show_bug.cgi?id=1337394
Bug ID: 1337394
Summary: DHT : few Files are not accessible and not listed on
mount + more than one Directory have same gfid +
(sometimes) attributes has ?? in ls output after
renaming Directories from multiple client at same time
Product: GlusterFS
Version: 3.8.0
Component: distribute
Severity: high
Assignee: bugs at gluster.org
Reporter: sabansal at redhat.com
CC: bugs at gluster.org, kramdoss at redhat.com,
mzywusko at redhat.com, nbalacha at redhat.com,
racpatel at redhat.com, rcyriac at redhat.com,
rgowdapp at redhat.com, rhinduja at redhat.com,
sabansal at redhat.com, sankarshan at redhat.com
Depends On: 1118762, 1118770, 1336698
Blocks: 1299184, 1337022
+++ This bug was initially created as a clone of Bug #1336698 +++
+++ This bug was initially created as a clone of Bug #1118762 +++
Description of problem:
=======================
try to rename Directories from multiple client at same time. e.g. from one
mount give mv <src> <dest> and from another mv <src> <dest1> (where dest and
dest1 is not hashing to same) or simply from one mount point mv <src> <dest>
and from another mount point mv <src> <dest> .
If they are not done one after another, then due to race after rename is
finished :-
- same gfid for different Directories (at same level and/or diffrent level)
- files inside those directories are not listed on mount and sometime even
Directory is not listed (so unable to access data)
- sometimes ls output shows question mark in attribute filed.
Version-Release number :
=========================
3.6.0.24-1.el6rhs.x86_64
How reproducible:
=================
Intermittent
Steps to Reproduce:
====================
1. create and mount distributed volume. (mount on multiple client)
2. create few files and Directories on mount point.
[root at OVM5 race]# mkdir src dest
[root at OVM5 race]# touch src/f{1..5}
[to reproduce race, we are putting breakpoint at dht_rename_hashed_dir_cbk for
both rename operation]
3. from one mount point execute 'mv src dest' and from other mpount point
execute 'mv dest src'
Now continue execution from break point from both mount point.
4. verify data from another mount and bricks also
mount:-
[root at OVM5 race]# ls -l
total 0
drwxr-xr-x 3 root root 69 Jul 10 12:32 src <---only src present
[root at OVM5 race]# mkdir dest
mkdir: cannot create directory `dest': File exists <--------
[root at OVM5 race]# ls -l <------- dest is not shown
total 0
drwxr-xr-x 3 root root 69 Jul 10 12:32 src
[root at OVM5 race]# touch src/f1
touch: cannot touch `src/f1': Input/output error
[root at OVM5 race]#
[root at OVM5 race]# ls -l src <-------f1, f2 and f5 are not listed
total 0
drwxr-xr-x 3 root root 38 Jul 10 12:32 dest <--- directory having same name
drwxr-xr-x 3 root root 38 Jul 10 12:32 dest
-rw-r--r-- 1 root root 0 Jul 10 12:32 f3
-rw-r--r-- 1 root root 0 Jul 10 12:32 f4
another mount point:-
[root at OVM5 race]# ls -lR
.:
total 0
drwxr-xr-x 3 root root 69 Jul 10 12:32 src
./src:
total 0
drwxr-xr-x 3 root root 70 Jul 10 12:32 dest
drwxr-xr-x 3 root root 70 Jul 10 12:32 dest
-rw-r--r-- 1 root root 0 Jul 10 12:32 f3
-rw-r--r-- 1 root root 0 Jul 10 12:32 f4
./src/dest:
ls: cannot access ./src/dest/src: No such file or directory
ls: cannot access ./src/dest/src: No such file or directory
total 0
?????????? ? ? ? ? ? src
?????????? ? ? ? ? ? src
./src/dest:
ls: cannot access ./src/dest/src: No such file or directory
ls: cannot access ./src/dest/src: No such file or directory
total 0
?????????? ? ? ? ? ? src <------------------ ?? in attribute
?????????? ? ? ? ? ? src
bricks :-
[root at OVM5 race]# tree /brick*/race/
/brick1/race/
├── dest
│ └── src
│ ├── f1
│ ├── f2
│ └── f5
└── src
└── dest
/brick2/race/
└── src
├── dest
│ └── src
└── f4
/brick3/race/
├── dest
└── src
├── dest
│ └── src
└── f3
11 directories, 5 files
Actual results:
===============
- same gfid for different Directories (at same level and/or different level)
- files inside those directories are not listed on mount and sometime even
Directory is not listed (so unable to access data)
- sometimes ls output shows question mark in attribute filed.
Expected results:
=================
- no two directory should have same gfid
- if both renames are fail/successful or one succeed and other fail; regardless
of all these - all files inside those Directories should be accessible from
mount point
--- Additional comment from Rachana Patel on 2014-07-11 09:27:34 EDT ---
[root at OVM3 ~]# gluster v info race
Volume Name: race
Type: Distribute
Volume ID: 30f7ff59-a90b-44e3-a991-223c81c15d67
Status: Started
Snap Volume: no
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: 10.70.35.172:/brick1/race
Brick2: 10.70.35.172:/brick2/race
Brick3: 10.70.35.172:/brick3/race
Options Reconfigured:
performance.readdir-ahead: on
snap-max-hard-limit: 256
snap-max-soft-limit: 90
auto-delete: disable
--- Additional comment from Vijay Bellur on 2016-05-17 04:52:04 EDT ---
REVIEW: http://review.gluster.org/14371 (dht: rename takes lock on parent
directory if destination exists) posted (#2) for review on master by Sakshi
Bansal
--- Additional comment from Vijay Bellur on 2016-05-17 05:32:25 EDT ---
REVIEW: http://review.gluster.org/14371 (dht: rename takes lock on parent
directory if destination exists) posted (#3) for review on master by Sakshi
Bansal
--- Additional comment from Vijay Bellur on 2016-05-18 05:59:28 EDT ---
COMMIT: http://review.gluster.org/14371 committed in master by Raghavendra G
(rgowdapp at redhat.com)
------
commit 6a6b953d4f999b6db11cc6b62880e810aee9a4c7
Author: Sakshi Bansal <sabansal at redhat.com>
Date: Tue May 17 13:23:28 2016 +0530
dht: rename takes lock on parent directory if destination exists
For directory rename if destination exists the source directory
is created as a child of the given destination directory. Since
the new child directory does not exist take lock on parent of the
child directory.
Change-Id: I24a34605a2cd65984910643ff5462f35e8fc7e71
BUG: 1336698
Signed-off-by: Sakshi Bansal <sabansal at redhat.com>
Reviewed-on: http://review.gluster.org/14371
Smoke: Gluster Build System <jenkins at build.gluster.com>
Reviewed-by: Raghavendra G <rgowdapp at redhat.com>
NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1118762
[Bug 1118762] DHT : few Files are not accessible and not listed on mount +
more than one Directory have same gfid + (sometimes) attributes has ?? in
ls output after renaming Directories from multiple client at same time
https://bugzilla.redhat.com/show_bug.cgi?id=1118770
[Bug 1118770] DHT : If Directory creation is in progress and rename of that
Directory comes from another mount point then after both operation few
files are not accessible and not listed on mount and more than one
Directory have same gfid
https://bugzilla.redhat.com/show_bug.cgi?id=1336698
[Bug 1336698] DHT : few Files are not accessible and not listed on mount +
more than one Directory have same gfid + (sometimes) attributes has ?? in
ls output after renaming Directories from multiple client at same time
https://bugzilla.redhat.com/show_bug.cgi?id=1337022
[Bug 1337022] DHT : few Files are not accessible and not listed on mount +
more than one Directory have same gfid + (sometimes) attributes has ?? in
ls output after renaming Directories from multiple client at same time
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list