[Gluster-users] permission denied and no such file for real files

[Lana Deere]

I'm running glusterfs 3.1.1, 4 volumes distributed, tcp transport,
fuse clients.  The servers are running CentOS 5.5, the clients CentOS 5.4.

The symptom I'm having which reproduces the most frequently goes
like this.  There are a bunch of jobs (several hundred) running on
various client nodes (~50), but for no obvious reason one or several of
them will get a failure to open a file (errno 13, permission denied).
Subsequently it will not be possible to delete that directory because it
is not empty.  Attempts to delete the files which make it non-empty fail,
"no such file or directory".

I attached the program and driver script which I use to reproduce this,
but not some log files (I ran them with trace and even compressed they
are 150M).  It is not perfect, it only reproduces maybe one-third of the
runs.

Here's a typical output:


$ ATTACK-QRSH
Quantity of jobs: 1000
Mon Dec 20 14:59:52 EST 2010: rm .....
Mon Dec 20 14:59:52 EST 2010: create data .....
Mon Dec 20 14:59:54 EST 2010: wait for creates.....
Couldn't create
data/051/307-this!"":is:!""the""filename""and""it""is""very""strange.tmp:
errno 13 (Permission denied)
Couldn't create
data/051/563-this!"":is:!""the""filename""and""it""is""very""strange.tmp:
errno 13 (Permission denied)
Mon Dec 20 15:04:45 EST 2010: check data .....
Mon Dec 20 15:04:47 EST 2010: wait for checks.....
Unable to reopen for read
data/051/563-this!"":is:!""the""filename""and""it""is""very""strange,
errno 2 (No such file or directory)
Unable to reopen for read
data/051/307-this!"":is:!""the""filename""and""it""is""very""strange,
errno 2 (No such file or directory)
Mon Dec 20 15:08:53 EST 2010: rm again .....
rm: cannot remove directory `data/051': Directory not empty
Mon Dec 20 15:09:16 EST 2010: done
$ cd data/051
$ ls
179-this!"":is:!""the""filename""and""it""is""very""strange
435-this!"":is:!""the""filename""and""it""is""very""strange
$ rm *
rm: cannot remove
`179-this!"":is:!""the""filename""and""it""is""very""strange': No such
file or directory
rm: cannot remove
`435-this!"":is:!""the""filename""and""it""is""very""strange': No such
file or directory
$ cd ..
$ rm -r 051
rm: cannot remove
`051/179-this!"":is:!""the""filename""and""it""is""very""strange': No
such file or directory
rm: cannot remove
`051/435-this!"":is:!""the""filename""and""it""is""very""strange': No
such file or directory

Note that the files which won't delete are in the same directory as
had the permission error earlier.

.. Lana (lana.deere at gmail.com)