[Bugs] [Bug 1695403] New: rm -rf fails with "Directory not empty"

bugzilla at redhat.com bugzilla at redhat.com
Wed Apr 3 04:10:06 UTC 2019


            Bug ID: 1695403
           Summary: rm -rf fails with  "Directory not empty"
           Product: GlusterFS
           Version: 5
            Status: NEW
         Component: distribute
          Assignee: bugs at gluster.org
          Reporter: nbalacha at redhat.com
                CC: bugs at gluster.org
        Depends On: 1676400
            Blocks: 1458215, 1661258, 1677260, 1686272
  Target Milestone: ---
    Classification: Community

+++ This bug was initially created as a clone of Bug #1676400 +++

Description of problem:

When 2 clients run rm -rf <dirname> concurrently, the operation sometimes fails
with " Directory not empty"

ls on the directory from the gluster mount point does not show any entries
however there are directories on some of the bricks.

Version-Release number of selected component (if applicable):

How reproducible:
Rare.This is a race condition.

Steps to Reproduce:
1. Create 3x (2+1) arbiter volume and fuse mount it. Make sure lookup-optimize
is enabled.
2. mkdir -p dir0/dir1/dir2.
3. Unmount and remount the volume to ensure a fresh lookup is sent. GDB into
the fuse process and set a breakpoint at dht_lookup.
4. from the client mount: 
rm -rf mra_sources
5. When gdb breaks at dht_lookup for dir0/dir1/dir2, set a breakpoint at
dht_lookup_cbk. Allow the process to continue until it hits dht_lookup_cbk.
dht_lookup_cbk will return with op_ret = 0 .
6. Delete dir0/dir1/dir2 from every brick on the non-hashed subvols.
7. Set a breakpoint in dht_selfheal_dir_mkdir and allow gdb to continue. 
8. When the process breaks at dht_selfheal_dir_mkdir, delete the directory from
the hashed subvolume bricks.
9. In dht_selfheal_dir_mkdir_lookup_cbk, set a breakpoint at line :
if (local->selfheal.hole_cnt == layout->cnt) {

When gdb breaks at this point, set local->selfheal.hole_cnt to a value
different from that of layout->cnt. Allow gdb to proceed.

DHT will create the directories only on the non-hashed subvolumes as the layout
has not been updated to indicate that the dir no longer exists on the hashed
subvolume.  This directory will no longer be visible on the mount point causing
the rm -rf to fail.

Actual results:
root at server fuse1]# rm -rf mra_sources
rm: cannot remove ‘dir0/dir1’: Directory not empty

Expected results:
rm -rf should succeed.

Additional info:
As lookup-optimize is enabled, subsequent lookups cannot heal the directory.
The same steps with lookup-optimize disabled will work as a subsequent lookup
will lookup everywhere even if the entry does not exist on the hashed subvol.

--- Additional comment from Nithya Balachandran on 2019-02-12 08:08:31 UTC ---

RCA for the invisible directory left behind with concurrent rm -rf :

dht_selfheal_dir_mkdir_lookup_cbk (...) {

1381         this_call_cnt = dht_frame_return (frame);                          
1383         LOCK (&frame->lock);                                               
1384         {                                                                  
1385                 if ((op_ret < 0) &&                                        
1386                     (op_errno == ENOENT || op_errno == ESTALE)) {          
1387                         local->selfheal.hole_cnt =
!local->selfheal.hole_cnt ? 1
1388                                                 : local->selfheal.hole_cnt
+ 1; 
1389                 }                                                          
1391                 if (!op_ret) {                                             
1392                         dht_iatt_merge (this, &local->stbuf, stbuf, prev); 
1393                 }                                                          
1394                 check_mds = dht_dict_get_array (xattr,
1395                                                 mds_xattr_val, 1, &errst); 
1396                 if (dict_get (xattr, conf->mds_xattr_key) && check_mds &&
!errst) {
1397                         dict_unref (local->xattr);                         
1398                         local->xattr = dict_ref (xattr);                   
1399                 }                                                          
1401         }                                                                  
1402         UNLOCK (&frame->lock);                                             
1404         if (is_last_call (this_call_cnt)) {                                
1405                 if (local->selfheal.hole_cnt == layout->cnt) {             
1406                         gf_msg_debug (this->name, op_errno,                
1407                                       "Lookup failed, an rmdir could have
1408                                       "deleted this entry %s", loc->name); 
1409                         local->op_errno = op_errno;                        
1410                         goto err;                                          
1411                 } else {                                                   
1412                         for (i = 0; i < layout->cnt; i++) {                
1413                                 if (layout->list[i].err == ENOENT ||       
1414                                     layout->list[i].err == ESTALE ||       
1415                                     local->selfheal.force_mkdir)           
1416                                         missing_dirs++;                    
1417                         }      

There are 2 problems here:

1. The layout is not updated with the new subvol status on error. 

In this case, the initial lookup found a directory on the hashed subvol so only
2 entries in the layout indicate missing directories. However, by the time the
selfheal code is executed, the racing rmdir has deleted the directory from all
the subvols.  At this point, the directory does not exist on any subvol and
dht_selfheal_dir_mkdir_lookup_cbk gets an error from all 3 subvols, 
but this new status is not updated in the layout which still has only 2 missing
dirs marked.

2. this_call_cnt = dht_frame_return (frame); is called before processing the
frame. So with a call cnt of 3, it is possible that the second response has
reached 1404 before the third one has started processing the return values. At
this point, 

local->selfheal.hole_cnt != layout->cnt so control goes to line 1412.

At line 1412, since we are still using the old layout, only the directories on
the non-hashed subvols are considered when incrementing missing_dirs and for
the healing.

The combination of these two causes the selfheal to start healing the
directories on the non-hashed subvols. It succeeds in creating the dirs on the
non-hashed subvols. However, to set the layout, dht takes an inodelk on the
hashed subvol which fails because the directory does on exist there. We
therefore end up with directories on the non-hashed subvols with no layouts

--- Additional comment from Worker Ant on 2019-02-12 08:34:01 UTC ---

REVIEW: https://review.gluster.org/22195 (cluster/dht: Fix lookup selfheal and
rmdir race) posted (#1) for review on master by N Balachandran

--- Additional comment from Worker Ant on 2019-02-13 18:20:26 UTC ---

REVIEW: https://review.gluster.org/22195 (cluster/dht: Fix lookup selfheal and
rmdir race) merged (#3) on master by Raghavendra G

Referenced Bugs:

[Bug 1458215] Slave reports ENOTEMPTY when rmdir is executed on master
[Bug 1676400] rm -rf fails with  "Directory not empty"
[Bug 1677260] rm -rf fails with  "Directory not empty"
[Bug 1686272] fuse mount logs inundated with [dict.c:471:dict_get]
[0x7f9029aa12f7] -->/lib64/libglusterfs.so.0(
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.

More information about the Bugs mailing list