[Bugs] [Bug 1257894] New: "rm -rf *" from multiple mount points fails to remove directories on all the subvolumes

bugzilla at redhat.com bugzilla at redhat.com
Fri Aug 28 10:49:09 UTC 2015


https://bugzilla.redhat.com/show_bug.cgi?id=1257894

            Bug ID: 1257894
           Summary: "rm -rf *" from multiple mount points fails to remove
                    directories on all the subvolumes
           Product: GlusterFS
           Version: 3.7.3
         Component: distribute
          Keywords: Triaged
          Severity: high
          Assignee: bugs at gluster.org
          Reporter: sabansal at redhat.com
                CC: bugs at gluster.org, gluster-bugs at redhat.com,
                    nsathyan at redhat.com, rgowdapp at redhat.com,
                    sabansal at redhat.com, spalai at redhat.com,
                    spandura at redhat.com, vagarwal at redhat.com
        Depends On: 1115367, 1245065
            Blocks: 960910, 960938, 1121186, 1128737, 1235633



+++ This bug was initially created as a clone of Bug #1245065 +++

+++ This bug was initially created as a clone of Bug #1115367 +++

Description of problem:
=======================
In a distribute-replicate volume when 'rm -rf *' is performed from multiple
mounts, the directories are removed from some sub-volumes but are not removed
from other sub-volumes. 

Because of this , "rm -rf <directory>" fails with "directory not empty". When
we do "ls -l <directory>" the directory is empty. 

Version-Release number of selected component (if applicable):
==============================================================
glusterfs 3.6.0.22 built on Jun 23 2014 10:33:07

How reproducible:
====================
Often

Steps to Reproduce:
====================
1. Create distribute-replicate volume. Start the volume.

2. Create 2 fuse mounts and 2 nfs mount or all 4 fuse mounts. 

3. Create directories ( mkdir -p A{1..1000}/B{1..20}/C{1..20} )

4. From all the mount points execute "rm -rf *"

Actual results:
====================
root at dj [Jul-02-2014- 1:14:38] >rm -rf *
rm: cannot remove `A11': Directory not empty
rm: cannot remove `A111': Directory not empty
rm: cannot remove `A137': Directory not empty
rm: cannot remove `A151/B18': Directory not empty
rm: cannot remove `A153': Directory not empty
rm: cannot remove `A163': Directory not empty
rm: cannot remove `A204': Directory not empty
rm: cannot remove `A480/B16': Directory not empty

On sub-volume1:
===================
brick1:
~~~~~~~~~
root at rhs-client11 [Jul-02-2014-14:40:48] >ls -l /rhs/device0/rep_brick1/A11
total 0
drwxr-xr-x 3 root root 15 Jul  2 12:32 B19
root at rhs-client11 [Jul-02-2014-14:40:50] >

brick2:
~~~~~~~~~
root at rhs-client12 [Jul-02-2014-14:40:48] >ls -l /rhs/device0/rep_brick2/A11
total 0
drwxr-xr-x 3 root root 15 Jul  2 12:32 B19
root at rhs-client12 [Jul-02-2014-14:40:50] >

On sub-volume2:
====================
brick3:
~~~~~~~
root at rhs-client13 [Jul-02-2014-14:40:48] >ls -l /rhs/device0/rep_brick3/A11
total 0
root at rhs-client13 [Jul-02-2014-14:40:50] >

brick4:
~~~~~~~~
root at rhs-client14 [Jul-02-2014-14:40:48] >ls -l /rhs/device0/rep_brick4/A11
total 0
root at rhs-client14 [Jul-02-2014-14:40:50] >
root at rhs-client14 [Jul-02-2014-14:40:51] >


Expected results:
==================
The directories should be removed from all the subvolumes. 

Additional info:
==================
root at mia [Jul-02-2014-14:42:57] >gluster v info rep

Volume Name: rep
Type: Distributed-Replicate
Volume ID: d8d69cec-8bdd-4c9d-b5f5-972b36716b0b
Status: Started
Snap Volume: no
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: rhs-client11:/rhs/device0/rep_brick1
Brick2: rhs-client12:/rhs/device0/rep_brick2
Brick3: rhs-client13:/rhs/device0/rep_brick3
Brick4: rhs-client14:/rhs/device0/rep_brick4
Options Reconfigured:
features.uss: disable
server.statedump-path: /var/run/gluster/statedumps
features.barrier: disable
performance.readdir-ahead: on
snap-max-hard-limit: 256
snap-max-soft-limit: 90
auto-delete: disable
root at mia [Jul-02-2014-14:43:01] >
root at mia [Jul-02-2014-14:43:02] >gluster v status rep
Status of volume: rep
Gluster process                        Port    Online    Pid
------------------------------------------------------------------------------
Brick rhs-client11:/rhs/device0/rep_brick1        49154    Y    2890
Brick rhs-client12:/rhs/device0/rep_brick2        49154    Y    5472
Brick rhs-client13:/rhs/device0/rep_brick3        49153    Y    2869
Brick rhs-client14:/rhs/device0/rep_brick4        49153    Y    5433
NFS Server on localhost                    2049    Y    32441
Self-heal Daemon on localhost                N/A    Y    27961
NFS Server on rhs-client13                2049    Y    20245
Self-heal Daemon on rhs-client13            N/A    Y    2858
NFS Server on 10.70.36.35                2049    Y    20399
Self-heal Daemon on 10.70.36.35                N/A    Y    2885
NFS Server on rhs-client12                2049    Y    11226
Self-heal Daemon on rhs-client12            N/A    Y    5494
NFS Server on rhs-client14                2049    Y    11211
Self-heal Daemon on rhs-client14            N/A    Y    5455

Task Status of Volume rep
------------------------------------------------------------------------------
There are no active volume tasks

root at mia [Jul-02-2014-14:43:05] >

--- Additional comment from RHEL Product and Program Management on 2014-07-02
05:25:23 EDT ---

Since this issue was entered in bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

--- Additional comment from  on 2014-07-02 06:26:32 EDT ---

SOS Reports:
http://rhsqe-repo.lab.eng.blr.redhat.com/bugs_necessary_info/1115367/

--- Additional comment from Vivek Agarwal on 2014-07-03 02:21:05 EDT ---

Per discussion, not a blocker.

--- Additional comment from Anand Avati on 2015-07-21 03:16:14 EDT ---

REVIEW: http://review.gluster.org/11725 (dht : lock on hashed subvol to prevent
lookup vs rmdir race) posted (#1) for review on master by Sakshi Bansal
(sabansal at redhat.com)

--- Additional comment from Niels de Vos on 2015-07-21 08:14:24 EDT ---

Please provide a public facing description of the issue.

--- Additional comment from Anand Avati on 2015-07-23 00:41:02 EDT ---

REVIEW: http://review.gluster.org/11725 (dht:lock on hashed subvol to prevent
lookup vs rmdir race) posted (#2) for review on master by Sakshi Bansal
(sabansal at redhat.com)

--- Additional comment from Anand Avati on 2015-08-07 00:36:01 EDT ---

REVIEW: http://review.gluster.org/11725 (dht: lock on subvols to prevent lookup
vs rmdir race) posted (#5) for review on master by Sakshi Bansal
(sabansal at redhat.com)

--- Additional comment from Anand Avati on 2015-08-07 04:44:35 EDT ---

REVIEW: http://review.gluster.org/11725 (dht: lock on subvols to prevent lookup
vs rmdir race) posted (#6) for review on master by Sakshi Bansal
(sabansal at redhat.com)

--- Additional comment from Anand Avati on 2015-08-10 06:03:26 EDT ---

REVIEW: http://review.gluster.org/11725 (dht : lock on subvols to prevent
lookup vs rmdir race) posted (#7) for review on master by Sakshi Bansal
(sabansal at redhat.com)

--- Additional comment from Anand Avati on 2015-08-14 02:37:08 EDT ---

REVIEW: http://review.gluster.org/11916 (lock : check if inode exists before
granting blocked locks) posted (#1) for review on master by Sakshi Bansal
(sabansal at redhat.com)

--- Additional comment from Anand Avati on 2015-08-17 06:04:47 EDT ---

REVIEW: http://review.gluster.org/11725 (dht: lock on subvols to prevent lookup
vs rmdir race) posted (#8) for review on master by Sakshi Bansal
(sabansal at redhat.com)

--- Additional comment from Anand Avati on 2015-08-19 00:34:11 EDT ---

REVIEW: http://review.gluster.org/11725 (dht: lock on subvols to prevent lookup
vs rmdir race) posted (#9) for review on master by Raghavendra G
(rgowdapp at redhat.com)

--- Additional comment from Anand Avati on 2015-08-25 05:30:02 EDT ---

REVIEW: http://review.gluster.org/11725 (dht : lock on subvols to prevent
lookup vs rmdir race) posted (#10) for review on master by Sakshi Bansal
(sabansal at redhat.com)

--- Additional comment from Anand Avati on 2015-08-25 23:33:41 EDT ---

REVIEW: http://review.gluster.org/11916 (lock : check if inode exists before
granting blocked locks) posted (#3) for review on master by Sakshi Bansal
(sabansal at redhat.com)

--- Additional comment from Anand Avati on 2015-08-25 23:39:40 EDT ---

REVIEW: http://review.gluster.org/11725 (dht: lock on subvols to prevent lookup
vs rmdir race) posted (#11) for review on master by Sakshi Bansal
(sabansal at redhat.com)

--- Additional comment from Anand Avati on 2015-08-26 06:02:27 EDT ---

REVIEW: http://review.gluster.org/11725 (dht : lock on subvols to prevent
lookup vs rmdir race) posted (#12) for review on master by Sakshi Bansal
(sabansal at redhat.com)

--- Additional comment from Anand Avati on 2015-08-27 15:28:37 EDT ---

REVIEW: http://review.gluster.org/12035 (dht: lookup after selfheal acquires
lock in the mkdir phase) posted (#1) for review on master by Sakshi Bansal
(sabansal at redhat.com)

--- Additional comment from Anand Avati on 2015-08-28 05:16:11 EDT ---

REVIEW: http://review.gluster.org/12035 (dht: lookup after selfheal acquires
lock in the mkdir phase) posted (#2) for review on master by Sakshi Bansal
(sabansal at redhat.com)


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=960910
[Bug 960910] "rm -rf" failed to remove directory complained "directory not
empty" from fuse mount
https://bugzilla.redhat.com/show_bug.cgi?id=1115367
[Bug 1115367] "rm -rf *" from multiple mount points fails to remove
directories on all the subvolumes
https://bugzilla.redhat.com/show_bug.cgi?id=1121186
[Bug 1121186] DHT : If Directory deletion is in progress and lookup from
another mount heals that Directory on sub-volumes. then rmdir/rm -rf on
parents fails with error 'Directory not empty'
https://bugzilla.redhat.com/show_bug.cgi?id=1128737
[Bug 1128737] rebalance :- rm -rf failed with with error 'No such file or
directory' for few files and directory while rebalance is in progress
https://bugzilla.redhat.com/show_bug.cgi?id=1235633
[Bug 1235633] [rm -rf]: Recursive remove fails with "[Errno 39] Directory
not empty" on slave when performed rm -rf on Master volume
https://bugzilla.redhat.com/show_bug.cgi?id=1245065
[Bug 1245065] "rm -rf *" from multiple mount points fails to remove
directories on all the subvolumes
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.


More information about the Bugs mailing list