[Gluster-devel] rm -r problem on FreeBSD port

[Vijay Bellur]

On 01/18/2016 06:18 PM, Rick Macklem wrote:
> Hi,
>
> I have a simple gluster volume made up of 2 bricks using
> distribute (running on the FreeBSD port of 3.7.6).
> When I do a "rm -rf <dir>" for a fairly large tree, I
> sometimes get "Directory not empty" errors.
> When I look in the directory (even after an unmount, shutdown,
> restart, remount) the "ls -l" output for the fuse mounted
> volume and the 2 underlying bricks looks like...
>
> # ls -l of fuse mounted gv0: directory
>
> total 1
> ----------  1 root  wheel  345891 Jan 18 14:49 Makefile
> ----------  1 root  wheel  345891 Jan 18 14:49 Makefile
> ----------  1 root  wheel    1195 Jan 18 14:49 ata_if.c
> ----------  1 root  wheel    1195 Jan 18 14:49 ata_if.c
> ----------  1 root  wheel     576 Jan 18 14:49 fb_if.c
> ----------  1 root  wheel     576 Jan 18 14:49 fb_if.c
> ----------  1 root  wheel    1787 Jan 18 14:49 hdac_if.c
> ----------  1 root  wheel    1787 Jan 18 14:49 hdac_if.c
> ----------  1 root  wheel     753 Jan 18 14:49 mmcbus_if.c
> ----------  1 root  wheel     753 Jan 18 14:49 mmcbus_if.c
> drwxr-xr-x  3 root  wheel     512 Jan 14 12:34 modules
> ----------  1 root  wheel     602 Jan 18 14:49 power_if.c
> ----------  1 root  wheel     602 Jan 18 14:49 power_if.c
> ----------  1 root  wheel    1819 Jan 18 14:49 uart_if.c
> ----------  1 root  wheel    1819 Jan 18 14:49 uart_if.c
>
> # ls -l of the directory in one of the 2 bricks
>
> total 36
> ----------  2 root  wheel    0 Jan 18 14:49 Makefile
> ----------  2 root  wheel    0 Jan 18 14:49 ata_if.c
> ----------  2 root  wheel    0 Jan 18 14:49 fb_if.c
> ----------  2 root  wheel    0 Jan 18 14:49 hdac_if.c
> ----------  2 root  wheel    0 Jan 18 14:49 mmcbus_if.c
> drwxr-xr-x  3 root  wheel  512 Jan 14 17:34 modules
> ----------  2 root  wheel    0 Jan 18 14:49 power_if.c
> ----------  2 root  wheel    0 Jan 18 14:49 uart_if.c
>
> # and ls -l of the underlying directory in the other brick
>
> total 400
> -rw-r--r--  2 root  wheel  345891 Jan 14 11:50 Makefile
> -rw-r--r--  2 root  wheel    1195 Jan 14 13:19 ata_if.c
> -rw-r--r--  2 root  wheel     576 Jan 14 12:13 fb_if.c
> -rw-r--r--  2 root  wheel    1787 Jan 14 12:13 hdac_if.c
> -rw-r--r--  2 root  wheel     753 Jan 14 12:13 mmcbus_if.c
> drwxr-xr-x  3 root  wheel     512 Jan 14 12:34 modules
> -rw-r--r--  2 root  wheel     602 Jan 14 12:13 power_if.c
> -rw-r--r--  2 root  wheel    1819 Jan 14 12:13 uart_if.c
>
> Anyone have an idea of what is causing this?
> (I am thinking something like the FreeBSD fuse interface
>   is holding opens on the files when I am doing "rm -rf <dir>",
>   even though they shouldn't be open and the GlusterFS is doing
>   some trickery similar to NFS's silly rename to avoid the
>   files being removed before being closed?
>   But I have no idea if this theory holds any water.;-)
>
> Thanks in advance for any hints, rick


Debugging by isolation could be a good technique to figure out the layer 
that is causing the problem here. You probably could start with a 
minimal volume file [1] and mounting by pointing to a volume file 
instead of fetching from (glusterfs -s <volfile> -f <logfile> 
/mnt/point) glusterd to determine if the problem is with the fuse 
implementation in FreeBSD. If that doesn't turn out to be a problem, I 
would create a brick with a single brick and see if the problem happens 
there. If not, we could further test with a volume having multiple 
bricks and so on.

HTH,
Vijay

[1] http://paste.fedoraproject.org/312178/45316267/