[Gluster-devel] [Nfs-ganesha-devel] Device or resource busy when runltp cleanup test-files

[Kinglong Mee]

Sorry for the late reply.

On 4/16/2017 22:31, Vijay Bellur wrote:
> 
> 
> On Wed, Apr 12, 2017 at 10:37 PM, Kinglong Mee <mijinlong at open-fs.com <mailto:mijinlong at open-fs.com>> wrote:
> 
>     Yes, this one is silly rename,
> 
>     >>> rm: cannot remove ‘/mnt/nfs/ltp-JEYAuky2dz/.nfsaa46457a6a72f8ea000014f5’: Device or resource busy
> 
>     But, the other one isn't client's silly rename,
> 
>     >>>>     rm: cannot remove ‘/mnt/nfs/ltp-JEYAuky2dz/rmderQsjV’: Directory not empty
> 
>     Also, the second one often appears when testing ganesha nfs by ltp.
> 
>     I try to get an tcpdump, I found, when ls under the nfs client directory,
> 
>     the glusterfs client doesn't send readdir to glusterfs server, so that,
>     the nfs client gets an empty directory information, but it's empty underlying.
> 
>     ls at some other directory,
>     the glusterfs client send readdir to glusterfs server, nfs client gets right dirctory
>     information as the underlying.
> 
>     So, I guess maybe there are some problems in MDCHANE or glusterfs client?
> 
>     ]# cat /var/lib/glusterd/vols/gvtest/trusted-gvtest.tcp-fuse.vol
>     volume gvtest-client-0
>         type protocol/client
>         option send-gids true
>         option password d80765c8-95ae-46ed-912a-0c98fdd1e7cd
>         option username 7bc6276f-245c-477a-99d0-7ead4fcb2968
>         option transport.address-family inet
>         option transport-type tcp
>         option remote-subvolume /gluster-test/gvtest
>         option remote-host 192.168.9.111
>         option ping-timeout 42
>     end-volume
> 
>     volume gvtest-client-1
>         type protocol/client
>         option send-gids true
>         option password d80765c8-95ae-46ed-912a-0c98fdd1e7cd
>         option username 7bc6276f-245c-477a-99d0-7ead4fcb2968
>         option transport.address-family inet
>         option transport-type tcp
>         option remote-subvolume /gluster-test/gvtest
>         option remote-host 192.168.9.111
>         option ping-timeout 42
>     end-volume
> 
>     volume gvtest-dht
>         type cluster/distribute
>         option lock-migration off
>         subvolumes gvtest-client-0 gvtest-client-1
>     end-volume
> 
>     volume gvtest
>         type debug/io-stats
>         option count-fop-hits off
>         option latency-measurement off
>         option log-level INFO
>         subvolumes gvtest-dht
>     end-volume
> 
> 
> 
> 
> Have you checked the gluster export directories after receiving ENOTEMPTY? If you observe any files there, can you please check if they are 0-byte files with access mode 600 and sticky bit set? 
> 

No.
I have check those files at the latest test.

rm: cannot remove ‘/mnt/nfs/ltp-fpeUthdR0W/rmdVGrwdj’: Directory not empty

The back-end directory,
# ll /gvtest/ltp-fpeUthdR0W/
total 264
drwxrwxrwx 2 root root 253952 Apr 20 15:53 fann7LHNa
drwxrwxrwx 2 root root   4096 Apr 20 15:54 inop4q8Wi
drwxrwxrwx 3 root root   4096 Apr 20 15:58 linu6YVST
drwxrwxrwx 3 root root   4096 Apr 20 15:59 rmdVGrwdj
# ll /gvtest/ltp-fpeUthdR0W/rmdVGrwdj/testdir/
total 0

The nfs client directory mounted through ganesha.nfsd, 
# ll /mnt/nfs/ltp-fpeUthdR0W/
total 12
drwxrwxrwx 2 root root 4096 Apr 20 15:53 fann7LHNa
drwxrwxrwx 2 root root 4096 Apr 20 15:54 inop4q8Wi
drwxrwxrwx 3 root root 4096 Apr 20 15:58 linu6YVST
# ll /mnt/nfs/ltp-fpeUthdR0W/rmdVGrwdj/
total 4
drwxrwxrwx 2 root root 4096 Apr 20 15:59 testdir
[root at C0N1 ~]# ll /mnt/nfs/ltp-fpeUthdR0W/rmdVGrwdj/testdir/
total 0
-rwxrwxrwx 1 root root 0 Apr 20 15:59 testfile

1. nfs client can't get "rmdVGrwdj" when ll the upper directory,
2. but nfs client can access it by name directly,
3. A "testfile" under nfs client's "rmdVGrwdj/testdir",
   but it's empty in the back-end gluster filesystem.
4. after ltp's rm finish, a duplicate rm can remove "rmdVGrwdj".

So, I guess there are some problem of the ganesha.nfsd cache when deleting files.

thanks,
Kinglong Mee

> When a file is being renamed, if the hash value of its new name happens to fall in a range that is different than that of the old name, gluster creates one such file. In the past we have seen instances of these files not being cleaned up properly in some cases and that causes a ENOTEMPTY when rmdir lands on the parent directory. Trying to see if we are hitting the same problem here.
> 
> Regards,
> Vijay