[Gluster-users] Cannot rename files with root squashing and r-x folder group permissions
Raghavendra Bhat
rabhat at redhat.com
Fri Jul 18 05:02:24 UTC 2014
On Thursday 17 July 2014 03:46 PM, David Raffelt wrote:
> Hi Raghavendra,
> I'm don't quite understand the issue. Yes, a rebalance was was
> performed in December last year when I added a brick. However, we have
> only just come across this (reproducible) problem upon upgrading to 3.5.
>
> Is there anything I can do to try and correct the issue? Perhaps turn
> off root squashing while running "gluster volume rebalance
> /VOLNAME/ fix-layout start"?
>
> Cheers,
> Dave
>
Hi Dave,
For now, you can turn off root-squashing. I am still trying to root
cause the issue. Will update ASAP with my finding.
Regards,
Raghavendra Bhat
>
>
>
> On 17 July 2014 17:31, Raghavendra Bhat <rabhat at redhat.com
> <mailto:rabhat at redhat.com>> wrote:
>
> On Wednesday 16 July 2014 10:18 AM, David Raffelt wrote:
>> Hi Raghavendra,
>> No
>> Thanks
>> Dave
>>
>>
>
> As per the cmd_log_history file (a hidden file present in the log
> directory which stores the CLI commands executed on that peer),
> rebalance seems to be running (or was run).
>
> [2013-12-17 03:08:59.081232] : volume rebalance data start : SUCCESS
> [2013-12-17 03:09:14.631826] : volume rebalance data status : SUCCESS
> [2013-12-17 03:09:22.761097] : volume rebalance data status : SUCCESS
> [2013-12-17 03:09:27.748014] : volume rebalance data status : SUCCESS
> [2013-12-17 03:09:28.839242] : volume rebalance data status : SUCCESS
> [2013-12-17 03:10:39.982747] : volume rebalance data status : SUCCESS
> [2013-12-17 03:14:30.919676] : volume rebalance data status : SUCCESS
> [2013-12-17 03:14:33.772300] : volume rebalance data status : SUCCESS
> [2013-12-17 03:29:14.467954] : volume rebalance data status : SUCCESS
> [2013-12-17 03:29:43.303852] : volume rebalance data status : SUCCESS
> [2013-12-17 03:30:04.309054] : volume rebalance data status : SUCCESS
> [2013-12-17 04:35:45.631119] : volume rebalance data status : SUCCESS
>
>
> I think this is what has happened. As part of rebalance layout
> might have changed for some directories and distribute tries to
> repair it by doing a self-heal when a lookup is performed on the
> directory. Distribute performs self-heal as root. But when the
> requests from that client comes to brick process, the requests
> from root are changed by default to nfsnobody (uid: 65534) and
> that uid does not have permissions to do some modifications (in
> this case self-heal) on the directory which brick thinks is owned
> by root. So self-heal does not happen properly and because of that
> some operations performed (in this case rename of a file within
> that directory)
> fails.
>
> Dave,
> Please let me know if I have missed anything. This is my
> observation based on the log files.
>
> CCing Raghavendra G who might be able to clarify whether this is
> what happened.
>
> Regards,
> Raghavendra Bhat
>
>
>> On 16 July 2014 14:47, Raghavendra Bhat <rabhat at redhat.com
>> <mailto:rabhat at redhat.com>> wrote:
>>
>> On Tuesday 15 July 2014 01:57 PM, David Raffelt wrote:
>>> Hi Raghavendra,
>>> Thanks for looking into this. Attached are the log files
>>> from the 3 peers. The glusterfs server is running on
>>> "Beauty". All 3 peers mount the native gluster client on
>>> /home. Each peer has a direct connection to each other,
>>> addressable via the /etc/hosts file.
>>>
>>> Note that I do not see any new output in the log when this
>>> error occurs. Also note that I tried to replicate this
>>> issue on Ubuntu 14.04 with a single brick and could not
>>> replicate it.
>>>
>>> Below is some more output that might help.
>>> Thanks!
>>> Dave
>>>
>>>
>>>
>>> *dave at beauty:~$ glusterfs --version*
>>> glusterfs 3.5git built on Jun 30 2014 15:58:19
>>> Repository revision: git://git.gluster.com/glusterfs.git
>>> <http://git.gluster.com/glusterfs.git>
>>> Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
>>> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>>> It is licensed to you under your choice of the GNU Lesser
>>> General Public License, version 3 or any later version (LGPLv3
>>> or later), or the GNU General Public License, version 2 (GPLv2),
>>> in all cases as published by the Free Software Foundation.
>>>
>>>
>>> *dave at beauty:~$ uname -r*
>>> 3.15.4-1-ARCH
>>>
>>>
>>> *dave at beauty:~$ sudo gluster volume info *
>>> Volume Name: data
>>> Type: Distribute
>>> Volume ID: 1d5948c7-9b7a-40ca-8aa7-85c74bcef3bc
>>> Status: Started
>>> Number of Bricks: 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: beauty:/export/beauty
>>> Brick2: beast:/export/beast
>>> Brick3: benji:/export/benji
>>> Options Reconfigured:
>>> performance.cache-size: 32MB
>>> performance.write-behind-window-size: 1MB
>>> auth.allow:
>>> 172.30.25.173,172.30.25.158,172.30.25.234,172.30.26.76,172.30.26.77,192.168.0.1,192.168.1.1,192.168.1.2,192.168.2.2,192.168.3.2,192.168.4.1,192.168.4.2,192.168.5.1,192.168.5.2
>>> nfs.disable: off
>>> diagnostics.brick-log-level: ERROR
>>> diagnostics.client-log-level: ERROR
>>> server.root-squash: enable
>>>
>>>
>>>
>>
>> Hi Dave,
>>
>> Was rebalance running when you did above operations?
>>
>>
>> Regards,
>> Raghavendra Bhat
>>
>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 15 July 2014 15:29, Raghavendra Bhat <rabhat at redhat.com
>>> <mailto:rabhat at redhat.com>> wrote:
>>>
>>> On Monday 14 July 2014 09:10 PM, Pranith Kumar Karampuri
>>> wrote:
>>>> CCed Raghavendra Bhat who may know about the issue
>>>>
>>>> Pranith
>>>> On 07/14/2014 08:01 PM, Joe Julian wrote:
>>>>> https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
>>>>>
>>>>> Please file a bug report.
>>>>>
>>>>> On July 14, 2014 12:38:11 AM PDT, David Raffelt
>>>>> <d.raffelt at brain.org.au>
>>>>> <mailto:d.raffelt at brain.org.au> wrote:
>>>>>
>>>>> Hi All,
>>>>> After a recent update to gluster 3.5 we are having
>>>>> some issues renaming files when root squashing is
>>>>> enabled and the folder group permissions are not
>>>>> set to write.
>>>>>
>>>>> For example if I create a folder with the
>>>>> following permissions
>>>>> $ mkdir test
>>>>> $ chmod g-w test
>>>>> $ ls -l
>>>>> drwxr-xr-x 2 dave dave 22 Jul 14 17:16 test
>>>>>
>>>>> When I create a file /within/ this folder, and try
>>>>> to rename it I get a file permissions error.
>>>>>
>>>>> $ cd test
>>>>> $ touch asdf
>>>>> $ mv asdf asdf2
>>>>> mv: cannot move 'asdf' to 'asdf2': Permission denied
>>>>>
>>>>> A strace on the mv command reveals the rename
>>>>> system call fails with:
>>>>> rename("asdf", "asdf2") = -1 EACCES (Permission
>>>>> denied)
>>>>>
>>>>> However I can copy the file and delete the old one
>>>>> fine.
>>>>>
>>>>> If I either disable gluster root squashing, or
>>>>> change the test_dir folder group permission to
>>>>> write then I can rename the file without any problems.
>>>>>
>>>>> System details are:
>>>>> Arch linux
>>>>> System umask is set to 002
>>>>> Distributed volume, 3 peers, 1 brick per peer.
>>>>>
>>>>> Any help is much appreciated!
>>>>> Dave
>>>>>
>>>>>
>>>
>>> Hi Dave,
>>>
>>> Can you please provide the brick and client log files?
>>> Which client you were using? fuse or nfs?
>>>
>>> Regards,
>>> Raghavendra Bhat
>>>
>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>>
>>>>> --
>>>>> Sent from my Android device with K-9 Mail. Please
>>>>> excuse my brevity.
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>
>>>
>>> --
>>> *David Raffelt (PhD)*
>>> Postdoctoral Fellow
>>>
>>> The Florey Institute of Neuroscience and Mental Health
>>> Melbourne Brain Centre - Austin Campus
>>> 245 Burgundy Street
>>> Heidelberg Vic 3084
>>> Ph: +61 3 9035 7024
>>> www.florey.edu.au
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>>
>> --
>> *David Raffelt (PhD)*
>> Postdoctoral Fellow
>>
>> The Florey Institute of Neuroscience and Mental Health
>> Melbourne Brain Centre - Austin Campus
>> 245 Burgundy Street
>> Heidelberg Vic 3084
>> Ph: +61 3 9035 7024
>> www.florey.edu.au
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> --
> *David Raffelt (PhD)*
> Postdoctoral Fellow
>
> The Florey Institute of Neuroscience and Mental Health
> Melbourne Brain Centre - Austin Campus
> 245 Burgundy Street
> Heidelberg Vic 3084
> Ph: +61 3 9035 7024
> www.florey.edu.au
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140718/a16a1c50/attachment.html>
More information about the Gluster-users
mailing list