[Gluster-devel] dht and rename: how is it supposed to work?

[Emmanuel Dreyfus]

Hi everybody

Here is my latest bug: when the client does a rename(2) on a directory, 
dht turns that into some weird dance involving link(2) on the server. Since 
link(2) cannot operate on a directory, the rename(2) fails with EPERM. 

How is rename supposed to work with dht? Is that bug NetBSD specific? 
I just tried linking a directory on Linux, it fails as well. The 
failure is SUSv2-compliant:
http://pubs.opengroup.org/onlinepubs/009695399/functions/link.html

Here are the details. I can reproduce the problem with this script:
client# cat test.sh                                                         
#!/bin/sh

rm -rf i386 machine inst.* *.core
mkdir i386
install -l s -r i386 machine 

client# ./test.sh
install: machine: rename: Operation not permitted

Oddly, this fails only once. If I run the script twice, I get no errors.
I have to umount/mount the volume to reproduce the bug. Of course, the
script only fails with glusterfs. It runs fine on a native filesystem.

Here is what a kernel trace tells me about install(1) actions:
 CALL  __lstat30(0xbfbfc91c,0xbfbfc868)
 NAMI  "inst.04684a"
 RET   __lstat30 -1 errno 2 No such file or directory
 CALL  symlink(0xbfbff990,0xbfbfc91c)
 MISC  link-target: "i386"
 NAMI  "inst.04684a"
 RET   symlink 0
 CALL  __posix_rename(0xbfbfc91c,0xbfbff995)
 NAMI  "inst.04684a"
 NAMI  "machine"
 RET   __posix_rename -1 errno 1 Operation not permitted

Here is glusterfs client log:
[2011-07-29 06:45:18.283832] I [client3_1-fops.c:2056:client3_1_link_cbk] 
  0-gfs-client-0: remote operation failed: Operation not permitted
[2011-07-29 06:45:18.284401] I [client3_1-fops.c:2056:client3_1_link_cbk] 
  0-gfs-client-1: remote operation failed: Operation not permitted
[2011-07-29 06:45:18.290144] I [client3_1-fops.c:502:client3_1_unlink_cbk] 
  0-gfs-client-0: remote operation failed: No such file or directory
[2011-07-29 06:45:18.290699] I [client3_1-fops.c:502:client3_1_unlink_cbk] 
  0-gfs-client-1: remote operation failed: No such file or directory
[2011-07-29 06:45:18.290821] W [dht-rename.c:348:dht_rename_unlink_cbk] 
  0-gfs-dht: /rename/inst.04684a: unlink on gfs-replicate-0 failed 
  (No such file or director)
[2011-07-29 06:45:18.304395] W [fuse-bridge.c:1352:fuse_rename_cbk] 
  0-glusterfs-fuse: 50: /rename/inst.04684a -> /rename/machine => -1 
  (Operation not permitted)

Here is glusterfs server log:
[2011-07-29 06:45:18.283770] E [posix.c:2131:posix_link] 0-gfs-posix: 
  link /rename/inst.04684a to /rename/machine failed: Operation not permitted
[2011-07-29 06:45:18.283896] I [server3_1-fops.c:1050:server_link_cbk] 
  0-gfs-server: 60: LINK /rename/inst.04684a (49876739) ==> -1 
  (Operation not permitted)
[2011-07-29 06:45:18.290028] I [server3_1-fops.c:964:server_unlink_cbk] 
  0-gfs-server: 64: UNLINK /rename/machine (0) ==> -1 
  (No such file or directory)

And here is server kernel trace, which just confirms the logs above do
spot the right error:
 CALL  symlink(0xbb95d078,0xb65ff5c0)
 MISC  symlink-target: "i386"
 NAMI  "/export/wd3a/rename/inst.04684a"
 RET   symlink 0
(...)
 CALL  __lstat30(0xb6
 CALL  link(0xb65ff5c0,0xb65ff550)
 NAMI  "/export/wd3a/rename/inst.04684a"
 NAMI  "/export/wd3a/rename/machine"
 RET   link -1 errno 1 Operation not permitted
(...)
 CALL  unlink(0xb65ff550)
 NAMI  "/export/wd3a/rename/machine"
 RET   unlink -1 errno 2 No such file or directory

-- 
Emmanuel Dreyfus
manu at netbsd.org