[Gluster-devel] race condition in 3.5.0qa3?

[Emmanuel Dreyfus]

I traced glusterfsd on the brick, and here is the error:

   494     92 glusterfsd CALL  link(0xab7ff7a0,0xab7ff540)
   494     92 glusterfsd NAMI  "/export/wd2a/manu/tmp/xxx"
   494     92 glusterfsd NAMI  "/export/wd2a/.glusterfs/06/96/0696ea4e-e193-4f8a-9b07-270e1d997c90"
   494     92 glusterfsd RET   link -1 errno 1 Operation not permitted

I can reproduce this behavior at the comand line:
# ls -ld /export/wd2a/manu/tmp/xxx
lrwxrwxrwx  1 manu  manu  6 Dec 19 08:53 /export/wd2a/manu/tmp/xxx -> target
# ls -ld /export/wd2a/manu/tmp/target
drwxr-xr-x  2 manu  manu  512 Dec 17 16:32 /export/wd2a/manu/tmp/target
# ls -l /export/wd2a/manu/tmp/target  
total 0
lrwxrwxrwx  1 manu  manu  6 Dec 17 16:32 xxx -> target
# ln /export/wd2a/manu/tmp/xxx /export/wd2a/.glusterfs/06/96/0696ea4e-e193>
ln: /export/wd2a/.glusterfs/06/96/0696ea4e-e193-4f8a-9b07-270e1d997c90: Operation not permitted

This is an expected result: the backend filesystem is FFS, which does not
support hard linking to a directory (POSIX says that operation *may* be
supported). I have to look at the sources, but I guess a link(2) has
to be turned into a linkat(2), so that the hard link is done on the 
symlink and not on the symlink target (BSD and Linux link(2) behaviors
differ here, and POSIX does not explicit what is right).

But there is more breakage. I rm -rf target on the client, and redo 
my test:
$ ls -l
$ ln -s target xxx  ; mv xxx  symlink  
$ ls -l                                
total 0
lrwxrwxrwx  0 manu  manu  6 Dec 19 09:04 xxx -> target

The rename operation failed but was not noticed to client proceses. Client
log says:
[2013-12-19 08:05:33.506568] W [client-rpc-fops.c:2619:client3_3_link_cbk] 0-gfs351-client-0: remote operation failed: No such file or directory (/manu/tmp/xxx -> /manu/tmp/symlink)
[2013-12-19 08:05:33.507016] W [client-rpc-fops.c:2619:client3_3_link_cbk] 0-gfs351-client-1: remote operation failed: No such file or directory (/manu/tmp/xxx -> /manu/tmp/symlink)
[2013-12-19 08:05:33.556367] W [dht-rename.c:381:dht_rename_unlink_cbk] 0-gfs351-dht: /manu/tmp/xxx: unlink on gfs351-replicate-0 failed (No such file or directory)
[2013-12-19 08:05:33.566559] W [fuse-bridge.c:1694:fuse_rename_cbk] 0-glusterfs-fuse: 6217918: /manu/tmp/xxx -> /manu/tmp/symlink => -1 (Undefined error: 0)




-- 
Emmanuel Dreyfus
manu at netbsd.org