[Gluster-devel] rm -r problem on FreeBSD port

[Rick Macklem]

Hi,

I have a simple gluster volume made up of 2 bricks using
distribute (running on the FreeBSD port of 3.7.6).
When I do a "rm -rf <dir>" for a fairly large tree, I
sometimes get "Directory not empty" errors.
When I look in the directory (even after an unmount, shutdown,
restart, remount) the "ls -l" output for the fuse mounted
volume and the 2 underlying bricks looks like...

# ls -l of fuse mounted gv0: directory

total 1
----------  1 root  wheel  345891 Jan 18 14:49 Makefile
----------  1 root  wheel  345891 Jan 18 14:49 Makefile
----------  1 root  wheel    1195 Jan 18 14:49 ata_if.c
----------  1 root  wheel    1195 Jan 18 14:49 ata_if.c
----------  1 root  wheel     576 Jan 18 14:49 fb_if.c
----------  1 root  wheel     576 Jan 18 14:49 fb_if.c
----------  1 root  wheel    1787 Jan 18 14:49 hdac_if.c
----------  1 root  wheel    1787 Jan 18 14:49 hdac_if.c
----------  1 root  wheel     753 Jan 18 14:49 mmcbus_if.c
----------  1 root  wheel     753 Jan 18 14:49 mmcbus_if.c
drwxr-xr-x  3 root  wheel     512 Jan 14 12:34 modules
----------  1 root  wheel     602 Jan 18 14:49 power_if.c
----------  1 root  wheel     602 Jan 18 14:49 power_if.c
----------  1 root  wheel    1819 Jan 18 14:49 uart_if.c
----------  1 root  wheel    1819 Jan 18 14:49 uart_if.c

# ls -l of the directory in one of the 2 bricks

total 36
----------  2 root  wheel    0 Jan 18 14:49 Makefile
----------  2 root  wheel    0 Jan 18 14:49 ata_if.c
----------  2 root  wheel    0 Jan 18 14:49 fb_if.c
----------  2 root  wheel    0 Jan 18 14:49 hdac_if.c
----------  2 root  wheel    0 Jan 18 14:49 mmcbus_if.c
drwxr-xr-x  3 root  wheel  512 Jan 14 17:34 modules
----------  2 root  wheel    0 Jan 18 14:49 power_if.c
----------  2 root  wheel    0 Jan 18 14:49 uart_if.c

# and ls -l of the underlying directory in the other brick

total 400
-rw-r--r--  2 root  wheel  345891 Jan 14 11:50 Makefile
-rw-r--r--  2 root  wheel    1195 Jan 14 13:19 ata_if.c
-rw-r--r--  2 root  wheel     576 Jan 14 12:13 fb_if.c
-rw-r--r--  2 root  wheel    1787 Jan 14 12:13 hdac_if.c
-rw-r--r--  2 root  wheel     753 Jan 14 12:13 mmcbus_if.c
drwxr-xr-x  3 root  wheel     512 Jan 14 12:34 modules
-rw-r--r--  2 root  wheel     602 Jan 14 12:13 power_if.c
-rw-r--r--  2 root  wheel    1819 Jan 14 12:13 uart_if.c

Anyone have an idea of what is causing this?
(I am thinking something like the FreeBSD fuse interface
 is holding opens on the files when I am doing "rm -rf <dir>",
 even though they shouldn't be open and the GlusterFS is doing
 some trickery similar to NFS's silly rename to avoid the
 files being removed before being closed?
 But I have no idea if this theory holds any water.;-)

Thanks in advance for any hints, rick