[Gluster-users] GlusterFS 3.1.1: "Permission denied" and "No such file or directory"

Tue Feb 1 09:50:33 UTC 2011

Dear all,

what I tried:
On the client side client as normal user:
mkdir /storage/cluster/zander/mailtest

drwxr-xr-x  2 zander belle    66 Feb  1 10:28 mailtest

Then I checked, where the directory is found:
Brick1: 192.168.101.249:/storage/4/cluster (xfs)  yes!
Brick2: 192.168.101.248:/storage/5/cluster (xfs)  yes!
Brick3: 192.168.101.250:/storage/6/cluster (xfs)  yes!
Brick4: 192.168.101.247:/storage/7/cluster (xfs)  no!
Brick5: 192.168.101.246:/storage/8/cluster (ext4) no!

This confused me, as I was under the impression that this directory 
should be found on all servers. I then created some files in that 
directory and checked, where they appeared. Again nothing found in 
bricks 4 and 5.

I tried to check the logs.They directory /usr/local/var/log/glusterfs 
exists, but is empty. /var/log/glusterfs/ contains some logfiles, 
however. I zipped the directories from Bricks 3,4 and 5.

You can find them here:
http://www-ekp.physik.uni-karlsruhe.de/~zander/2011-02-01-fs6-logs.tar.gz
http://www-ekp.physik.uni-karlsruhe.de/~zander/2011-02-01-fs7-logs.tar.gz
http://www-ekp.physik.uni-karlsruhe.de/~zander/2011-02-01-fs8-logs.tar.gz

I performed the test at 10:45 on February 1st (which was a few minutes ago).

The error always happens, i.e. whenever someone wants to delete a 
directory. I am now sure it has to do with the fact that the directory 
is not written on Bricks 4 and 5. Data is writte onto these new bricks 
however, but it seems to me only in directories that already exist. The 
problem occured since we switched from glusterFS 3.0 to 3.1.1.

Thanks for any any help,
Daniel

On 01/29/2011 04:20 AM, Pranith Kumar. Karampuri wrote:
> If you have never edited vol files in 3.1.1 that means it is generated by glusterd and it generates access-control xlator. So it is definitely BUG 2296. Chown workaround will take care of it for now.
> I need some data about "Delete problem", the circumstances it is happening etc. I am not able to reproduce anything like what you are stating. You said that the bricks you are using have both ext4 and xfs. Do this test. Create one directory, do ls -l on that directory so that we know the permissions/ownership etc. Check on which brick it is created and find out whether that brick is xfs or ext4. Now delete the directory. Do this until you get the error. Glusterfs stores the logs under the directory /usr/local/var/log/glusterfs. Zip that directory along with the "ls -l" outputs you collected and send them across. We can take a look. It is helpful if you can confirm that the error is happening only on the bricks with xfs or ext4 or both.
>
> Pranith.
> ----- Original Message -----
> From: "Daniel Zander"<zander at ekp.uni-karlsruhe.de>
> To: "Pranith Kumar. Karampuri"<pranithk at gluster.com>
> Cc: gluster-users at gluster.org
> Sent: Friday, January 28, 2011 7:16:05 PM
> Subject: Re: [Gluster-users] GlusterFS 3.1.1: "Permission denied" and "No such file or directory"
>
> Hi!
>
>> To confirm the "Permission Denied" problem you are facing is indeed BUG 2296 following conditions should be met:
>> 1) mount is fuse mount.
> This I can confirm.
>
>> 2) access-control translator should be present on top of the posix translator in the brick volfile.
> Unfortunately, neither me nor any colleague has any idea what this
> means. We have no real sysadmins at our institute, sorry. A wild guess:
> As we are using glusterFS 3.1.1, I have never edited or even seen any
> volfiles. Or is that something different?
>
> As the workaround really seems to work, I think we'll stick with that
> for the moment. Any more ideas about the "delete problem"?
>
> Best Regards,
> Daniel
>
>
>> you can see the reports of the following bugs for "Permission Denied on Fuse mount":
>> http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2304
>> http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2312
>> http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2296
>>
>> chown -R user:primary-group is a workaround that should fix the problem for now.
>>
>> Pranith.
>> ----- Original Message -----
>> From: "Daniel Zander"<zander at ekp.uni-karlsruhe.de>
>> To: "Pranith Kumar. Karampuri"<pranithk at gluster.com>
>> Cc: gluster-users at gluster.org
>> Sent: Friday, January 28, 2011 3:44:46 PM
>> Subject: Re: [Gluster-users] GlusterFS 3.1.1: "Permission denied" and "No such file or directory"
>>
>> Hi Pranith and all,
>>
>> I am not entirely sure what you mean by "how to trigger" the second problem.
>>
>> This problem always occurs. I simply create a folder called test and
>> perform an rm -rf on this folder. I get the error message
>>
>> rm: cannot remove directory `test/': No such file or directory
>>
>> but the folder vanishes anyway.
>>
>> About the first problem again: When you say, non-primary group
>> membership is treated as "others", this can't explain, as far as I
>> understand, why sometimes writing into the very same directoy works and
>> sometimes it fails. By the way: where can I see the bug reports?
>>
>> Issuing a chown -R<user>:<primary_group>   should resolve these issues,
>> shouldn't it?
>>
>> Thanks for your help,
>> Daniel
>>
>>
>> On 01/28/2011 09:32 AM, Pranith Kumar. Karampuri wrote:
>>> hi Daniel,
>>>       The first problem seems like bug 2296. Non-primary group membership was treated as "other". It is already resolved and will be available in 3.1.3. Do you know how to trigger the second problem?.
>>>
>>> Pranith.
>>>
>>> ----- Original Message -----
>>> From: "Daniel Zander"<zander at ekp.uni-karlsruhe.de>
>>> To: gluster-users at gluster.org
>>> Sent: Friday, January 28, 2011 1:53:14 PM
>>> Subject: [Gluster-users] GlusterFS 3.1.1: "Permission denied" and "No such	file or directory"
>>>
>>> Dear all,
>>>
>>> since the upgrade from glusterFS 3.0.0 to 3.1.1, some users have
>>> experienced strange problems. When they want to create a folder or a
>>> file, this usually works, but sometimes, it doesn't. The error message
>>> is something like "Permission Denied".
>>>
>>> The permissions and ownership of the affected folders seem to be set
>>> correctly. I tried to chown -R user:group user/ anyway and it helped
>>> once. It even occurs that when jobs try to write output back from our
>>> batch system, most jobs can write files that are accessible, but a small
>>> number cannot write their output (to the same folder at the same time).
>>>
>>> Another problem, that might be related to the ones above is that
>>> whenever a users tries to delete a directory, the following message occurs:
>>>
>>> rm: cannot remove directory `test/': No such file or directory
>>>
>>> but the directory is deleted anyway.
>>>
>>> I've also seen files, that seem to have no permissions at all, e.g.
>>>
>>> ----------T  2 root root 4.1K Jan 17 10:26 mc_jpsi
>>>
>>> This can be fixed manually, but is still very strange.
>>>
>>> About our setup:
>>>
>>> Volume Name: lemmy
>>> Type: Distribute
>>> Status: Started
>>> Number of Bricks: 5
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: 192.168.101.249:/storage/4/cluster
>>> Brick2: 192.168.101.248:/storage/5/cluster
>>> Brick3: 192.168.101.250:/storage/6/cluster
>>> Brick4: 192.168.101.247:/storage/7/cluster
>>> Brick5: 192.168.101.246:/storage/8/cluster
>>>
>>>
>>> There is sufficient space left on all bricks. The operating systems of
>>> the servers are debian5 (once) and Ubuntu 10.04 server (4 times). The
>>> bricks have different filesystems (ext4 and xfs).
>>>
>>> Thanks and Best Regards,
>>> Daniel
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users