[Gluster-users] I/O error repaired only by owner or root access

Mon Feb 25 13:16:47 UTC 2013

Hello Rajesh- Here are the permissions.  The path in question is a 
directory.

[sms05dab at jupiter ~]$ ls -ld /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl
drwxr-xr-x 60 vq901510 nemo 110592 Feb 23 04:37 
/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl
[sms05dab at jupiter ~]$ ls -ld /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN
lrwxrwxrwx 1 gcs nemo 49 Feb  1  2012 
/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN -> 
/data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN
[sms05dab at jupiter ~]$ ls -ld 
/data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN
drwxr-xr-x 27 gcs nemo 99210 Feb 23 03:14 
/data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN

As you can see the parent directory in this case was a symlink but 
that's not significant.  I ran the "ls -l" commands using my account - 
sms05dab, but the problem was originally reported by user vq901510.  
until I did "ls -l" as root neither of us could access the directory, 
because the parent directory was owned by user gcs.  Usually the problem 
is related to ownership of the file or directory itself.  This is the 
first time I have seen the I/O error caused by parent directory permissions.

This problem seems to have started following an add-brick operation a 
few weeks ago, after which I started "gluster volume rebalance <VOLNAME> 
fix-layout" (which is still running). It occurred to me that the problem 
could be related to link files, many of which need to be rewritten 
following add-brick operations.  This could explain why the ownership of 
the parent directory is significant, because users sms05dab and vq901510 
don't have permission to write in the parent directory owned by user 
gcs.  Normally this wouldn't be a problem because only read access to 
other users' data is required, but it appears as though read access was 
being denied because the new link file couldn't be written by 
unprivileged users.  Is this a plausible explanation of the I/O error do 
you think?

-Dan.

--
Dan Bretherton
ESSC Computer System Manager
Department of Meteorology
Harry Pitt Building, 3 Earley Gate
University of Reading
Reading, RG6 7BE (or RG6 6AL for postal service deliveries)
UK
Tel. +44 118 378 5205, Fax: +44 118 378 6413
-- 
## Please sponsor me to run in VSO's 30km Race to the Eye ##
##        http://www.justgiving.com/DanBretherton         ##

On 02/22/2013 11:56 AM, Rajesh Amaravathi wrote:
> Hi Dan,
> Could you please provide the following info>
> (1) the exact permissions of the file you are accessing and
> its parent directory,
> (2) the user from which 'ls -l' is issued, and
> (3) the owner of the file, and the parent directory.
>
> You could open a bug for it if it is seen several times.
>
> Regards,
> Rajesh Amaravathi,
> Software Engineer, GlusterFS
> RedHat Inc.
>
> ----- Original Message -----
>> From: "Dan Bretherton" <d.a.bretherton at reading.ac.uk>
>> To: "gluster-users" <gluster-users at gluster.org>
>> Sent: Thursday, February 21, 2013 8:16:40 PM
>> Subject: [Gluster-users] I/O error repaired only by owner or root access
>>
>> Dear All-
>> Several users are having a lot of trouble reading files belonging to
>> other users.  Here is an example.
>>
>> [sms05dab at jupiter ~]$ ls -l /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl
>> ls: /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl: Input/output error
>>
>> The corresponding nfs.log messages are shown below.
>>
>> [2013-02-21 12:11:39.204659] W [nfs3.c:727:nfs3svc_getattr_stat_cbk]
>> 0-nfs: fe2ba5b8: /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN => -1
>> (Invalid argument)
>> [2013-02-21 12:11:39.204778] W
>> [nfs3-helpers.c:3389:nfs3_log_common_res]
>> 0-nfs-nfsv3: XID: fe2ba5b8, GETATTR: NFS: 22(Invalid argument for
>> operation), POSIX: 22(Invalid argument)
>> [2013-02-21 12:11:39.215345] I
>> [dht-common.c:954:dht_lookup_everywhere_cbk] 0-nemo2-dht: deleting
>> stale
>> linkfile /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN on
>> nemo2-replicate-0
>> [2013-02-21 12:11:39.225674] W
>> [client3_1-fops.c:592:client3_1_unlink_cbk] 0-nemo2-client-1: remote
>> operation failed: Permission denied
>> [2013-02-21 12:11:39.225786] W
>> [client3_1-fops.c:592:client3_1_unlink_cbk] 0-nemo2-client-0: remote
>> operation failed: Permission denied
>> [2013-02-21 12:11:39.681029] W
>> [client3_1-fops.c:258:client3_1_mknod_cbk] 0-nemo2-client-18: remote
>> operation failed: Permission denied. Path:
>> /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN
>> (1662aa0a-d43b-4c2e-9be9-407eb7a89e85)
>> [2013-02-21 12:11:39.681400] W
>> [client3_1-fops.c:258:client3_1_mknod_cbk] 0-nemo2-client-19: remote
>> operation failed: Permission denied. Path:
>> /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN
>> (1662aa0a-d43b-4c2e-9be9-407eb7a89e85)
>> [2013-02-21 12:11:39.682268] W [nfs3.c:1627:nfs3svc_readlink_cbk]
>> 0-nfs:
>> 2ca5b8: /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN => -1 (Invalid
>> argument)
>> [2013-02-21 12:11:39.682338] W
>> [nfs3-helpers.c:3403:nfs3_log_readlink_res] 0-nfs-nfsv3: XID: 2ca5b8,
>> READLINK: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid
>> argument), target: (null)
>>
>> I managed to access the same directory as the owner (or any user with
>> write access including root) without any trouble, and after that
>> access
>> from my normal user account was fine as well.  The permissions on the
>> directory allowed read access by everyone, but the "Permission
>> denied"
>> messages in nfs.log indicate that some sort of operation is not being
>> allowed when the directory is accessed by other users.   I have seen
>> this happen with files and directories, and with the GlusterFS native
>> client and NFS.
>>
>> I presume this is a bug; I would be grateful if someone could confirm
>> this.  I would file a bug report, but the trouble is that I don't
>> know
>> how to reproduce the problem that causes the I/O error in the first
>> place.  It only happens with some files and directories, not all.
>>   Would
>> a bug report without any way to reproduce the error be any use, and
>> can
>> anyone suggest a way to dig deeper (eg looking at xattrs) next time I
>> come across an example?
>>
>> -Dan.
>>
>> --
>> Dan Bretherton
>> ESSC Computer System Manager
>> Department of Meteorology
>> Harry Pitt Building, 3 Earley Gate
>> University of Reading
>> Reading, RG6 7BE (or RG6 6AL for postal service deliveries)
>> UK
>> Tel. +44 118 378 5205, Fax: +44 118 378 6413
>> --
>> ## Please sponsor me to run in VSO's 30km Race to the Eye ##
>> ##        http://www.justgiving.com/DanBretherton         ##
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>