[Gluster-users] Cannot rename files with root squashing and r-x folder group permissions

Raghavendra Bhat rabhat at redhat.com
Fri Jul 18 05:02:24 UTC 2014


On Thursday 17 July 2014 03:46 PM, David Raffelt wrote:
> Hi Raghavendra,
> I'm don't quite understand the issue. Yes, a rebalance was was 
> performed in December last year when I added a brick. However, we have 
> only just come across this (reproducible) problem upon upgrading to 3.5.
>
> Is there anything I can do to try and correct the issue? Perhaps turn 
> off root squashing while running "gluster volume rebalance 
> /VOLNAME/ fix-layout start"?
>
> Cheers,
> Dave
>

Hi Dave,

For now, you can turn off root-squashing. I am still trying to root 
cause the issue. Will update ASAP with my finding.

Regards,
Raghavendra Bhat

>
>
>
> On 17 July 2014 17:31, Raghavendra Bhat <rabhat at redhat.com 
> <mailto:rabhat at redhat.com>> wrote:
>
>     On Wednesday 16 July 2014 10:18 AM, David Raffelt wrote:
>>     Hi Raghavendra,
>>     No
>>     Thanks
>>     Dave
>>
>>
>
>     As per the cmd_log_history file (a hidden file present in the log
>     directory which stores the CLI commands executed on that peer),
>     rebalance seems to be running (or was run).
>
>     [2013-12-17 03:08:59.081232]  : volume rebalance data start : SUCCESS
>     [2013-12-17 03:09:14.631826]  : volume rebalance data status : SUCCESS
>     [2013-12-17 03:09:22.761097]  : volume rebalance data status : SUCCESS
>     [2013-12-17 03:09:27.748014]  : volume rebalance data status : SUCCESS
>     [2013-12-17 03:09:28.839242]  : volume rebalance data status : SUCCESS
>     [2013-12-17 03:10:39.982747]  : volume rebalance data status : SUCCESS
>     [2013-12-17 03:14:30.919676]  : volume rebalance data status : SUCCESS
>     [2013-12-17 03:14:33.772300]  : volume rebalance data status : SUCCESS
>     [2013-12-17 03:29:14.467954]  : volume rebalance data status : SUCCESS
>     [2013-12-17 03:29:43.303852]  : volume rebalance data status : SUCCESS
>     [2013-12-17 03:30:04.309054]  : volume rebalance data status : SUCCESS
>     [2013-12-17 04:35:45.631119]  : volume rebalance data status : SUCCESS
>
>
>      I think this is what has happened. As part of rebalance layout
>     might have changed for some directories and distribute tries to
>     repair it by doing a self-heal when a lookup is performed on the
>     directory. Distribute performs self-heal as root. But when the
>     requests from that client comes to brick process, the requests
>     from root are changed by default to nfsnobody (uid: 65534) and
>     that uid does not have permissions to do some modifications (in
>     this case self-heal) on the directory which brick thinks is owned
>     by root. So self-heal does not happen properly and because of that
>     some operations performed (in this case rename of a file within
>     that directory)
>     fails.
>
>     Dave,
>     Please let me know if I have missed anything. This is my
>     observation based on the log files.
>
>     CCing Raghavendra G who might be able to clarify whether this is
>     what happened.
>
>     Regards,
>     Raghavendra Bhat
>
>
>>     On 16 July 2014 14:47, Raghavendra Bhat <rabhat at redhat.com
>>     <mailto:rabhat at redhat.com>> wrote:
>>
>>         On Tuesday 15 July 2014 01:57 PM, David Raffelt wrote:
>>>         Hi Raghavendra,
>>>         Thanks for looking into this. Attached are the log files
>>>         from the 3 peers. The glusterfs server is running on
>>>         "Beauty".  All 3 peers mount the native gluster client on
>>>         /home. Each peer has a direct connection to each other,
>>>         addressable via the /etc/hosts file.
>>>
>>>         Note that I do not see any new output in the log when this
>>>         error occurs.  Also note that I tried to replicate this
>>>         issue on Ubuntu 14.04 with a single brick and could not
>>>         replicate it.
>>>
>>>         Below is some more output that might help.
>>>         Thanks!
>>>         Dave
>>>
>>>
>>>
>>>         *dave at beauty:~$ glusterfs --version*
>>>         glusterfs 3.5git built on Jun 30 2014 15:58:19
>>>         Repository revision: git://git.gluster.com/glusterfs.git
>>>         <http://git.gluster.com/glusterfs.git>
>>>         Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
>>>         GlusterFS comes with ABSOLUTELY NO WARRANTY.
>>>         It is licensed to you under your choice of the GNU Lesser
>>>         General Public License, version 3 or any later version (LGPLv3
>>>         or later), or the GNU General Public License, version 2 (GPLv2),
>>>         in all cases as published by the Free Software Foundation.
>>>
>>>
>>>         *dave at beauty:~$ uname -r*
>>>         3.15.4-1-ARCH
>>>
>>>
>>>         *dave at beauty:~$ sudo gluster volume info *
>>>         Volume Name: data
>>>         Type: Distribute
>>>         Volume ID: 1d5948c7-9b7a-40ca-8aa7-85c74bcef3bc
>>>         Status: Started
>>>         Number of Bricks: 3
>>>         Transport-type: tcp
>>>         Bricks:
>>>         Brick1: beauty:/export/beauty
>>>         Brick2: beast:/export/beast
>>>         Brick3: benji:/export/benji
>>>         Options Reconfigured:
>>>         performance.cache-size: 32MB
>>>         performance.write-behind-window-size: 1MB
>>>         auth.allow:
>>>         172.30.25.173,172.30.25.158,172.30.25.234,172.30.26.76,172.30.26.77,192.168.0.1,192.168.1.1,192.168.1.2,192.168.2.2,192.168.3.2,192.168.4.1,192.168.4.2,192.168.5.1,192.168.5.2
>>>         nfs.disable: off
>>>         diagnostics.brick-log-level: ERROR
>>>         diagnostics.client-log-level: ERROR
>>>         server.root-squash: enable
>>>
>>>
>>>
>>
>>         Hi Dave,
>>
>>         Was rebalance running when you did above operations?
>>
>>
>>         Regards,
>>         Raghavendra Bhat
>>
>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>         On 15 July 2014 15:29, Raghavendra Bhat <rabhat at redhat.com
>>>         <mailto:rabhat at redhat.com>> wrote:
>>>
>>>             On Monday 14 July 2014 09:10 PM, Pranith Kumar Karampuri
>>>             wrote:
>>>>             CCed Raghavendra Bhat who may know about the issue
>>>>
>>>>             Pranith
>>>>             On 07/14/2014 08:01 PM, Joe Julian wrote:
>>>>>             https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
>>>>>
>>>>>             Please file a bug report.
>>>>>
>>>>>             On July 14, 2014 12:38:11 AM PDT, David Raffelt
>>>>>             <d.raffelt at brain.org.au>
>>>>>             <mailto:d.raffelt at brain.org.au> wrote:
>>>>>
>>>>>                 Hi All,
>>>>>                 After a recent update to gluster 3.5 we are having
>>>>>                 some issues renaming files when root squashing is
>>>>>                 enabled and the folder group permissions are not
>>>>>                 set to write.
>>>>>
>>>>>                 For example if I create a folder with the
>>>>>                 following permissions
>>>>>                 $ mkdir test
>>>>>                 $ chmod g-w test
>>>>>                 $ ls -l
>>>>>                 drwxr-xr-x  2 dave dave  22 Jul 14 17:16 test
>>>>>
>>>>>                 When I create a file /within/ this folder, and try
>>>>>                 to rename it I get a file permissions error.
>>>>>
>>>>>                 $ cd test
>>>>>                 $ touch asdf
>>>>>                 $ mv asdf asdf2
>>>>>                 mv: cannot move 'asdf' to 'asdf2': Permission denied
>>>>>
>>>>>                 A strace on the mv command reveals the rename
>>>>>                 system call fails with:
>>>>>                 rename("asdf", "asdf2") = -1 EACCES (Permission
>>>>>                 denied)
>>>>>
>>>>>                 However I can copy the file and delete the old one
>>>>>                 fine.
>>>>>
>>>>>                 If I either disable gluster root squashing, or
>>>>>                 change the test_dir folder group permission to
>>>>>                 write then I can rename the file without any problems.
>>>>>
>>>>>                 System details are:
>>>>>                 Arch linux
>>>>>                 System umask is set to 002
>>>>>                 Distributed volume, 3 peers, 1 brick per peer.
>>>>>
>>>>>                 Any help is much appreciated!
>>>>>                 Dave
>>>>>
>>>>>
>>>
>>>             Hi Dave,
>>>
>>>             Can you please provide the brick and client log files?
>>>             Which client you were using? fuse or nfs?
>>>
>>>             Regards,
>>>             Raghavendra Bhat
>>>
>>>
>>>>>                 ------------------------------------------------------------------------
>>>>>
>>>>>                 Gluster-users mailing list
>>>>>                 Gluster-users at gluster.org  <mailto:Gluster-users at gluster.org>
>>>>>                 http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>>
>>>>>             -- 
>>>>>             Sent from my Android device with K-9 Mail. Please
>>>>>             excuse my brevity.
>>>>>
>>>>>
>>>>>             _______________________________________________
>>>>>             Gluster-users mailing list
>>>>>             Gluster-users at gluster.org  <mailto:Gluster-users at gluster.org>
>>>>>             http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>>
>>>>
>>>>             _______________________________________________
>>>>             Gluster-users mailing list
>>>>             Gluster-users at gluster.org  <mailto:Gluster-users at gluster.org>
>>>>             http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>             _______________________________________________
>>>             Gluster-users mailing list
>>>             Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>             http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>
>>>
>>>         -- 
>>>         *David Raffelt (PhD)*
>>>         Postdoctoral Fellow
>>>
>>>         The Florey Institute of Neuroscience and Mental Health
>>>         Melbourne Brain Centre - Austin Campus
>>>         245 Burgundy Street
>>>         Heidelberg Vic 3084
>>>         Ph: +61 3 9035 7024
>>>         www.florey.edu.au
>>
>>
>>         _______________________________________________
>>         Gluster-users mailing list
>>         Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>         http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>>
>>     -- 
>>     *David Raffelt (PhD)*
>>     Postdoctoral Fellow
>>
>>     The Florey Institute of Neuroscience and Mental Health
>>     Melbourne Brain Centre - Austin Campus
>>     245 Burgundy Street
>>     Heidelberg Vic 3084
>>     Ph: +61 3 9035 7024
>>     www.florey.edu.au
>
>
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>     http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> -- 
> *David Raffelt (PhD)*
> Postdoctoral Fellow
>
> The Florey Institute of Neuroscience and Mental Health
> Melbourne Brain Centre - Austin Campus
> 245 Burgundy Street
> Heidelberg Vic 3084
> Ph: +61 3 9035 7024
> www.florey.edu.au
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140718/a16a1c50/attachment.html>


More information about the Gluster-users mailing list