[Gluster-users] permission denied and no such file for real files

Lana Deere lana.deere at gmail.com
Mon Dec 20 23:27:08 UTC 2010

I'm running glusterfs 3.1.1, 4 volumes distributed, tcp transport,
fuse clients.  The servers are running CentOS 5.5, the clients CentOS 5.4.

The symptom I'm having which reproduces the most frequently goes
like this.  There are a bunch of jobs (several hundred) running on
various client nodes (~50), but for no obvious reason one or several of
them will get a failure to open a file (errno 13, permission denied).
Subsequently it will not be possible to delete that directory because it
is not empty.  Attempts to delete the files which make it non-empty fail,
"no such file or directory".

I attached the program and driver script which I use to reproduce this,
but not some log files (I ran them with trace and even compressed they
are 150M).  It is not perfect, it only reproduces maybe one-third of the

Here's a typical output:

Quantity of jobs: 1000
Mon Dec 20 14:59:52 EST 2010: rm .....
Mon Dec 20 14:59:52 EST 2010: create data .....
Mon Dec 20 14:59:54 EST 2010: wait for creates.....
Couldn't create
errno 13 (Permission denied)
Couldn't create
errno 13 (Permission denied)
Mon Dec 20 15:04:45 EST 2010: check data .....
Mon Dec 20 15:04:47 EST 2010: wait for checks.....
Unable to reopen for read
errno 2 (No such file or directory)
Unable to reopen for read
errno 2 (No such file or directory)
Mon Dec 20 15:08:53 EST 2010: rm again .....
rm: cannot remove directory `data/051': Directory not empty
Mon Dec 20 15:09:16 EST 2010: done
$ cd data/051
$ ls
$ rm *
rm: cannot remove
`179-this!"":is:!""the""filename""and""it""is""very""strange': No such
file or directory
rm: cannot remove
`435-this!"":is:!""the""filename""and""it""is""very""strange': No such
file or directory
$ cd ..
$ rm -r 051
rm: cannot remove
`051/179-this!"":is:!""the""filename""and""it""is""very""strange': No
such file or directory
rm: cannot remove
`051/435-this!"":is:!""the""filename""and""it""is""very""strange': No
such file or directory

Note that the files which won't delete are in the same directory as
had the permission error earlier.

.. Lana (lana.deere at gmail.com)

More information about the Gluster-users mailing list