[Gluster-devel] race condition in 3.5.0qa3?

[Emmanuel Dreyfus]

On Tue, Dec 17, 2013 at 08:08:45AM +0100, Emmanuel Dreyfus wrote:
> My test fails on that:
> cd /gfs351/manu/usr/src/obj/tooldir.NetBSD-6.1.2-i386/include/nbinclude && rm -f machine &&  /gfs351/manu/usr/src/obj/tooldir.NetBSD-6.1.2-i386/bin/i486--netbsdelf-nstall -l s -r i386 machine
> i486--netbsdelf-install: machine: rename: Operation not permitted

I have been able to reproduce it in another directory:

$ ls -ld target
drwxr-xr-x  2 manu  manu  1024 Dec 17 14:50 target
$ ln -s target xxx  ; mv xxx  symlink 
mv: rename xxx to symlink: Operation not permitted

But the really odd thing is that the filesystem keeps state. If the
temporary link has another name, il works:

$ ln -s target xx  ; mv xx  symlin
$ ls -ld target symlink
lrwxrwxrwx  0 manu  manu     6 Dec 17 16:31 symlink -> target
drwxr-xr-x  2 manu  manu  1024 Dec 17 14:50 target
$ rm symlink
$ ln -s target xxx  ; mv xxx  symlink  
mv: rename xxx to symlink: Operation not permitted

I can strip down the case to just two system calls:
#include <stdio.h>
#include <unistd.h>
#include <sysexits.h>
#include <err.h>

#define TMPLINK "xxx"
#define SYMLINK "symlink"
#define TARGET "target"
int
main(void)
{
        if (symlink(TARGET, TMPLINK) != 0)
                err(EX_OSERR, "symlink %s failed", TMPLINK);

        if (rename(TMPLINK, SYMLINK) != 0)
                err(EX_OSERR, "rename failed");

        return EX_OK;
}

I have a 2x2 replica cluster, the bricsk do not contain file xxx
Here is one brick log:

[2013-12-17 15:34:53.890604] W [posix-handle.c:717:posix_handle_hard] 0-gfs351-posix: link /export/wd2a/manu/tmp/xxx -> /export/wd2a/.glusterfs/91/03/91038022-94f3-4934-9103-e7335f5d3604 failed (Operation not permitted)
[2013-12-17 15:34:53.890668] E [posix.c:1615:posix_symlink] 0-gfs351-posix: setting gfid on /export/wd2a/manu/tmp/xxx failed
[2013-12-17 15:34:53.901788] E [posix.c:1949:posix_link] 0-gfs351-posix: link /export/wd2a/manu/tmp/xxx to /export/wd2a/manu/tmp/symlink failed: Operation not permitted
[2013-12-17 15:34:53.901941] I [server-rpc-fops.c:1166:server_link_cbk] 0-gfs351-server: 7541429: LINK /manu/tmp/xxx (91038022-94f3-4934-9103-e7335f5d3604) -> fa765f5c-aef7-4372-9a84-a30374dedc15/symlink ==> (Operation not permitted)

-- 
Emmanuel Dreyfus
manu at netbsd.org