[Gluster-users] Cannot rename files with root squashing and r-x folder group permissions
Raghavendra Bhat
rabhat at redhat.com
Thu Jul 17 07:31:44 UTC 2014
On Wednesday 16 July 2014 10:18 AM, David Raffelt wrote:
> Hi Raghavendra,
> No
> Thanks
> Dave
>
>
As per the cmd_log_history file (a hidden file present in the log
directory which stores the CLI commands executed on that peer),
rebalance seems to be running (or was run).
[2013-12-17 03:08:59.081232] : volume rebalance data start : SUCCESS
[2013-12-17 03:09:14.631826] : volume rebalance data status : SUCCESS
[2013-12-17 03:09:22.761097] : volume rebalance data status : SUCCESS
[2013-12-17 03:09:27.748014] : volume rebalance data status : SUCCESS
[2013-12-17 03:09:28.839242] : volume rebalance data status : SUCCESS
[2013-12-17 03:10:39.982747] : volume rebalance data status : SUCCESS
[2013-12-17 03:14:30.919676] : volume rebalance data status : SUCCESS
[2013-12-17 03:14:33.772300] : volume rebalance data status : SUCCESS
[2013-12-17 03:29:14.467954] : volume rebalance data status : SUCCESS
[2013-12-17 03:29:43.303852] : volume rebalance data status : SUCCESS
[2013-12-17 03:30:04.309054] : volume rebalance data status : SUCCESS
[2013-12-17 04:35:45.631119] : volume rebalance data status : SUCCESS
I think this is what has happened. As part of rebalance layout might
have changed for some directories and distribute tries to repair it by
doing a self-heal when a lookup is performed on the directory.
Distribute performs self-heal as root. But when the requests from that
client comes to brick process, the requests from root are changed by
default to nfsnobody (uid: 65534) and that uid does not have permissions
to do some modifications (in this case self-heal) on the directory which
brick thinks is owned by root. So self-heal does not happen properly and
because of that some operations performed (in this case rename of a file
within that directory)
fails.
Dave,
Please let me know if I have missed anything. This is my observation
based on the log files.
CCing Raghavendra G who might be able to clarify whether this is what
happened.
Regards,
Raghavendra Bhat
> On 16 July 2014 14:47, Raghavendra Bhat <rabhat at redhat.com
> <mailto:rabhat at redhat.com>> wrote:
>
> On Tuesday 15 July 2014 01:57 PM, David Raffelt wrote:
>> Hi Raghavendra,
>> Thanks for looking into this. Attached are the log files from the
>> 3 peers. The glusterfs server is running on "Beauty". All 3
>> peers mount the native gluster client on /home. Each peer has a
>> direct connection to each other, addressable via the /etc/hosts
>> file.
>>
>> Note that I do not see any new output in the log when this error
>> occurs. Also note that I tried to replicate this issue on Ubuntu
>> 14.04 with a single brick and could not replicate it.
>>
>> Below is some more output that might help.
>> Thanks!
>> Dave
>>
>>
>>
>> *dave at beauty:~$ glusterfs --version*
>> glusterfs 3.5git built on Jun 30 2014 15:58:19
>> Repository revision: git://git.gluster.com/glusterfs.git
>> <http://git.gluster.com/glusterfs.git>
>> Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
>> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>> It is licensed to you under your choice of the GNU Lesser
>> General Public License, version 3 or any later version (LGPLv3
>> or later), or the GNU General Public License, version 2 (GPLv2),
>> in all cases as published by the Free Software Foundation.
>>
>>
>> *dave at beauty:~$ uname -r*
>> 3.15.4-1-ARCH
>>
>>
>> *dave at beauty:~$ sudo gluster volume info *
>> Volume Name: data
>> Type: Distribute
>> Volume ID: 1d5948c7-9b7a-40ca-8aa7-85c74bcef3bc
>> Status: Started
>> Number of Bricks: 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: beauty:/export/beauty
>> Brick2: beast:/export/beast
>> Brick3: benji:/export/benji
>> Options Reconfigured:
>> performance.cache-size: 32MB
>> performance.write-behind-window-size: 1MB
>> auth.allow:
>> 172.30.25.173,172.30.25.158,172.30.25.234,172.30.26.76,172.30.26.77,192.168.0.1,192.168.1.1,192.168.1.2,192.168.2.2,192.168.3.2,192.168.4.1,192.168.4.2,192.168.5.1,192.168.5.2
>> nfs.disable: off
>> diagnostics.brick-log-level: ERROR
>> diagnostics.client-log-level: ERROR
>> server.root-squash: enable
>>
>>
>>
>
> Hi Dave,
>
> Was rebalance running when you did above operations?
>
>
> Regards,
> Raghavendra Bhat
>
>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On 15 July 2014 15:29, Raghavendra Bhat <rabhat at redhat.com
>> <mailto:rabhat at redhat.com>> wrote:
>>
>> On Monday 14 July 2014 09:10 PM, Pranith Kumar Karampuri wrote:
>>> CCed Raghavendra Bhat who may know about the issue
>>>
>>> Pranith
>>> On 07/14/2014 08:01 PM, Joe Julian wrote:
>>>> https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
>>>>
>>>> Please file a bug report.
>>>>
>>>> On July 14, 2014 12:38:11 AM PDT, David Raffelt
>>>> <d.raffelt at brain.org.au> <mailto:d.raffelt at brain.org.au>
>>>> wrote:
>>>>
>>>> Hi All,
>>>> After a recent update to gluster 3.5 we are having some
>>>> issues renaming files when root squashing is enabled
>>>> and the folder group permissions are not set to write.
>>>>
>>>> For example if I create a folder with the following
>>>> permissions
>>>> $ mkdir test
>>>> $ chmod g-w test
>>>> $ ls -l
>>>> drwxr-xr-x 2 dave dave 22 Jul 14 17:16 test
>>>>
>>>> When I create a file /within/ this folder, and try to
>>>> rename it I get a file permissions error.
>>>>
>>>> $ cd test
>>>> $ touch asdf
>>>> $ mv asdf asdf2
>>>> mv: cannot move ‘asdf’ to ‘asdf2’: Permission denied
>>>>
>>>> A strace on the mv command reveals the rename system
>>>> call fails with:
>>>> rename("asdf", "asdf2") = -1 EACCES (Permission denied)
>>>>
>>>> However I can copy the file and delete the old one fine.
>>>>
>>>> If I either disable gluster root squashing, or change
>>>> the test_dir folder group permission to write then I
>>>> can rename the file without any problems.
>>>>
>>>> System details are:
>>>> Arch linux
>>>> System umask is set to 002
>>>> Distributed volume, 3 peers, 1 brick per peer.
>>>>
>>>> Any help is much appreciated!
>>>> Dave
>>>>
>>>>
>>
>> Hi Dave,
>>
>> Can you please provide the brick and client log files? Which
>> client you were using? fuse or nfs?
>>
>> Regards,
>> Raghavendra Bhat
>>
>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>>
>>>> --
>>>> Sent from my Android device with K-9 Mail. Please excuse my
>>>> brevity.
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>>
>> --
>> *David Raffelt (PhD)*
>> Postdoctoral Fellow
>>
>> The Florey Institute of Neuroscience and Mental Health
>> Melbourne Brain Centre - Austin Campus
>> 245 Burgundy Street
>> Heidelberg Vic 3084
>> Ph: +61 3 9035 7024
>> www.florey.edu.au
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> --
> *David Raffelt (PhD)*
> Postdoctoral Fellow
>
> The Florey Institute of Neuroscience and Mental Health
> Melbourne Brain Centre - Austin Campus
> 245 Burgundy Street
> Heidelberg Vic 3084
> Ph: +61 3 9035 7024
> www.florey.edu.au
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140717/57d88d16/attachment.html>
More information about the Gluster-users
mailing list