[Bugs] [Bug 1676400] New: rm -rf fails with "Directory not empty"

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1676400

            Bug ID: 1676400
           Summary: rm -rf fails with  "Directory not empty"
           Product: GlusterFS
           Version: mainline
            Status: NEW
         Component: distribute
          Assignee: bugs at gluster.org
          Reporter: nbalacha at redhat.com
                CC: bugs at gluster.org
  Target Milestone: ---
    Classification: Community



Description of problem:

When 2 clients run rm -rf <dirname> concurrently, the operation fails with "
Directory not empty"


ls on the directory from the gluster mount point does not show any entries
however there are directories on the bricks.



Version-Release number of selected component (if applicable):


How reproducible:
Rare.This is a race condition.

Steps to Reproduce:
Steps:
1. Create 3x (2+1) arbiter volume and fuse mount it. Make sure lookup-optimize
is enabled.
2. mkdir -p dir0/dir1/dir2.
3. Unmount and remount the volume to ensure a fresh lookup is sent. GDB into
the fuse process and set a breakpoint at dht_lookup.
4. from the client mount: 
rm -rf mra_sources
5. When gdb breaks at dht_lookup for dir0/dir1/dir2, set a breakpoint at
dht_lookup_cbk. Allow the process to continue until it hits dht_lookup_cbk.
dht_lookup_cbk will return with op_ret = 0 .
6. Delete dir0/dir1/dir2 from every brick on the non-hashed subvols.
7. Set a breakpoint in dht_selfheal_dir_mkdir and allow gdb to continue. 
8. When the process breaks at dht_selfheal_dir_mkdir, delete the directory from
the hashed subvolume bricks.
9. In dht_selfheal_dir_mkdir_lookup_cbk, set a breakpoint at line :
if (local->selfheal.hole_cnt == layout->cnt) {

When gdb breaks at this point, set local->selfheal.hole_cnt to a value
different from that of layout->cnt. Allow gdb to proceed.

DHT will create the directories only on the non-hashed subvolumes as the layout
has not been updated to indicate that the dir no longer exists on the hashed
subvolume.  This directory will no longer be visible on the mount point causing
the rm -rf to fail.


Actual results:
root at server fuse1]# rm -rf mra_sources
rm: cannot remove ‘dir0/dir1’: Directory not empty


Expected results:
rm -rf should succeed.

Additional info:
As lookup-optimize is enabled, subsequent lookups cannot heal the directory.
The same steps with lookup-optimize disabled will work as a subsequent lookup
will lookup everywhere even if the entry does not exist on the hashed subvol.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.