[Gluster-users] I/O error repaired only by owner or root access

[Dan Bretherton]

--
Dan Bretherton
ESSC Computer System Manager
Department of Meteorology
Harry Pitt Building, 3 Earley Gate
University of Reading
Reading, RG6 7BE (or RG6 6AL for postal service deliveries)
UK
Tel. +44 118 378 5205, Fax: +44 118 378 6413
-- 
## Please sponsor me to run in VSO's 30km Race to the Eye ##
##        http://www.justgiving.com/DanBretherton         ##

On 02/25/2013 04:44 PM, Dan Bretherton wrote:
> On 02/25/2013 03:49 PM, Shawn Nock wrote:
>> Dan Bretherton <d.a.bretherton at reading.ac.uk> writes:
>>
>>> Hello Rajesh- Here are the permissions. The path in question is a
>>> directory.
>>>
>>> [sms05dab at jupiter ~]$ ls -ld /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl
>>> drwxr-xr-x 60 vq901510 nemo 110592 Feb 23 04:37
>>> /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl [sms05dab at jupiter ~]$ ls -ld
>>> /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN lrwxrwxrwx 1 gcs nemo 49 Feb 1
>>> 2012 /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN ->
>>> /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN [sms05dab at jupiter
>>> ~]$ ls -ld /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN
>>> drwxr-xr-x 27 gcs nemo 99210 Feb 23 03:14
>>> /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN
>>>
>>> As you can see the parent directory in this case was a symlink but
>>> that's not significant.  I ran the "ls -l" commands using my account -
>>> sms05dab, but the problem was originally reported by user vq901510.
>>> until I did "ls -l" as root neither of us could access the directory,
>>> because the parent directory was owned by user gcs.  Usually the
>>> problem is related to ownership of the file or directory itself.  This
>>> is the first time I have seen the I/O error caused by parent directory
>>> permissions.
>>>
>>> This problem seems to have started following an add-brick operation a
>>> few weeks ago, after which I started "gluster volume rebalance
>>> <VOLNAME> fix-layout" (which is still running). It occurred to me that
>>> the problem could be related to link files, many of which need to be
>>> rewritten following add-brick operations.  This could explain why the
>>> ownership of the parent directory is significant, because users
>>> sms05dab and vq901510 don't have permission to write in the parent
>>> directory owned by user gcs.  Normally this wouldn't be a problem
>>> because only read access to other users' data is required, but it
>>> appears as though read access was being denied because the new link
>>> file couldn't be written by unprivileged users.  Is this a plausible
>>> explanation of the I/O error do you think?
>> This sounds like my recent bug:
>> https://bugzilla.redhat.com/show_bug.cgi?id=913699
>>
>> In the bug, I said that writing on the fuse mount of one of the brick
>> servers fixed the problem.... but those were the only hosts I was
>> attempting access as root.
>>
> Thanks Shawn. To confirm that we are seeing the same bug I will try 
> accessing affected files from a FUSE mount on a server the next time 
> it happens.
> -Dan.
>
I have updated the bug report with another recent example.  In this most 
recent case, attempting to open a file for reading as an unprivileged 
user resulted in the error "Invalid argument", although "ls -l" worked 
without error.  This happened on a compute server and via a GlusterFS 
client mount point on a GlusterFS storage server. Changing the ownership 
of the parent directory to give the unprivileged user write access to 
the directory (making it writeable by group) allowed the user to open 
the file for reading.
-Dan.