[Gluster-users] permission denied and no such file for real files

[Thai. Ngo Bao]

Dear All,

I am having a problem, and I think it is similar to Lana's case. The problem is glusterfs prevents clients to create/delete files and folders. This problem appears so regularly. I attached the log file of our client.

Please also keep informed that I'm running glusterfs 3.1.1 with 4 bricks, volumes distributed, tcp transport, fuse clients. The servers and clients are running CentOS 5.4.

Would you have any suggestions, please let me know. Thanks.

~Thai

-----Original Message-----
From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Lana Deere
Sent: Tuesday, December 21, 2010 6:27 AM
To: gluster-users at gluster.org
Subject: [Gluster-users] permission denied and no such file for real files

I'm running glusterfs 3.1.1, 4 volumes distributed, tcp transport,
fuse clients.  The servers are running CentOS 5.5, the clients CentOS 5.4.

The symptom I'm having which reproduces the most frequently goes
like this.  There are a bunch of jobs (several hundred) running on
various client nodes (~50), but for no obvious reason one or several of
them will get a failure to open a file (errno 13, permission denied).
Subsequently it will not be possible to delete that directory because it
is not empty.  Attempts to delete the files which make it non-empty fail,
"no such file or directory".

I attached the program and driver script which I use to reproduce this,
but not some log files (I ran them with trace and even compressed they
are 150M).  It is not perfect, it only reproduces maybe one-third of the
runs.

Here's a typical output:


$ ATTACK-QRSH
Quantity of jobs: 1000
Mon Dec 20 14:59:52 EST 2010: rm .....
Mon Dec 20 14:59:52 EST 2010: create data .....
Mon Dec 20 14:59:54 EST 2010: wait for creates.....
Couldn't create
data/051/307-this!"":is:!""the""filename""and""it""is""very""strange.tmp:
errno 13 (Permission denied)
Couldn't create
data/051/563-this!"":is:!""the""filename""and""it""is""very""strange.tmp:
errno 13 (Permission denied)
Mon Dec 20 15:04:45 EST 2010: check data .....
Mon Dec 20 15:04:47 EST 2010: wait for checks.....
Unable to reopen for read
data/051/563-this!"":is:!""the""filename""and""it""is""very""strange,
errno 2 (No such file or directory)
Unable to reopen for read
data/051/307-this!"":is:!""the""filename""and""it""is""very""strange,
errno 2 (No such file or directory)
Mon Dec 20 15:08:53 EST 2010: rm again .....
rm: cannot remove directory `data/051': Directory not empty
Mon Dec 20 15:09:16 EST 2010: done
$ cd data/051
$ ls
179-this!"":is:!""the""filename""and""it""is""very""strange
435-this!"":is:!""the""filename""and""it""is""very""strange
$ rm *
rm: cannot remove
`179-this!"":is:!""the""filename""and""it""is""very""strange': No such
file or directory
rm: cannot remove
`435-this!"":is:!""the""filename""and""it""is""very""strange': No such
file or directory
$ cd ..
$ rm -r 051
rm: cannot remove
`051/179-this!"":is:!""the""filename""and""it""is""very""strange': No
such file or directory
rm: cannot remove
`051/435-this!"":is:!""the""filename""and""it""is""very""strange': No
such file or directory

Note that the files which won't delete are in the same directory as
had the permission error earlier.

.. Lana (lana.deere at gmail.com)
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users