[Gluster-users] Cannot rename files with root squashing and r-x folder group permissions

Saurabh Jain saujain at redhat.com
Fri Jul 18 06:50:42 UTC 2014 

Hello Dave, 

  It is a known issue from 3.4.0. One of the reasons behind the issue is this,
"It is failing only when the rename operation will cause the file to move from one brick to another due to DHT. Actually the file is not moved physically but a link file is created in the target brick."

Thanks,
Saurabh

----- Original Message -----
> From: "Raghavendra Bhat" <rabhat at redhat.com>
> To: gluster-users at gluster.org
> Sent: Friday, July 18, 2014 10:32:24 AM
> Subject: Re: [Gluster-users] Cannot rename files with root squashing and r-x folder group permissions
> 
> On Thursday 17 July 2014 03:46 PM, David Raffelt wrote:
> 
> 
> 
> Hi Raghavendra,
> I'm don't quite understand the issue. Yes, a rebalance was was performed in
> December last year when I added a brick. However, we have only just come
> across this (reproducible) problem upon upgrading to 3.5.
> 
> Is there anything I can do to try and correct the issue? Perhaps turn off
> root squashing while running " gluster volume rebalance VOLNAME fix-layout
> start"?
> 
> Cheers,
> Dave
> 
> 
> Hi Dave,
> 
> For now, you can turn off root-squashing. I am still trying to root cause the
> issue. Will update ASAP with my finding.
> 
> Regards,
> Raghavendra Bhat
> 
> 
> 
> 
> 
> 
> 
> On 17 July 2014 17:31, Raghavendra Bhat < rabhat at redhat.com > wrote:
> 
> 
> 
> On Wednesday 16 July 2014 10:18 AM, David Raffelt wrote:
> 
> 
> 
> Hi Raghavendra,
> No
> Thanks
> Dave
> 
> 
> 
> As per the cmd_log_history file (a hidden file present in the log directory
> which stores the CLI commands executed on that peer), rebalance seems to be
> running (or was run).
> 
> [2013-12-17 03:08:59.081232] : volume rebalance data start : SUCCESS
> [2013-12-17 03:09:14.631826] : volume rebalance data status : SUCCESS
> [2013-12-17 03:09:22.761097] : volume rebalance data status : SUCCESS
> [2013-12-17 03:09:27.748014] : volume rebalance data status : SUCCESS
> [2013-12-17 03:09:28.839242] : volume rebalance data status : SUCCESS
> [2013-12-17 03:10:39.982747] : volume rebalance data status : SUCCESS
> [2013-12-17 03:14:30.919676] : volume rebalance data status : SUCCESS
> [2013-12-17 03:14:33.772300] : volume rebalance data status : SUCCESS
> [2013-12-17 03:29:14.467954] : volume rebalance data status : SUCCESS
> [2013-12-17 03:29:43.303852] : volume rebalance data status : SUCCESS
> [2013-12-17 03:30:04.309054] : volume rebalance data status : SUCCESS
> [2013-12-17 04:35:45.631119] : volume rebalance data status : SUCCESS
> 
> 
> I think this is what has happened. As part of rebalance layout might have
> changed for some directories and distribute tries to repair it by doing a
> self-heal when a lookup is performed on the directory. Distribute performs
> self-heal as root. But when the requests from that client comes to brick
> process, the requests from root are changed by default to nfsnobody (uid:
> 65534) and that uid does not have permissions to do some modifications (in
> this case self-heal) on the directory which brick thinks is owned by root.
> So self-heal does not happen properly and because of that some operations
> performed (in this case rename of a file within that directory)
> fails.
> 
> Dave,
> Please let me know if I have missed anything. This is my observation based on
> the log files.
> 
> CCing Raghavendra G who might be able to clarify whether this is what
> happened.
> 
> Regards,
> Raghavendra Bhat
> 
> 
> 
> 
> 
> On 16 July 2014 14:47, Raghavendra Bhat < rabhat at redhat.com > wrote:
> 
> 
> 
> On Tuesday 15 July 2014 01:57 PM, David Raffelt wrote:
> 
> 
> 
> Hi Raghavendra,
> Thanks for looking into this. Attached are the log files from the 3 peers.
> The glusterfs server is running on "Beauty". All 3 peers mount the native
> gluster client on /home. Each peer has a direct connection to each other,
> addressable via the /etc/hosts file.
> 
> Note that I do not see any new output in the log when this error occurs. Also
> note that I tried to replicate this issue on Ubuntu 14.04 with a single
> brick and could not replicate it.
> 
> Below is some more output that might help.
> Thanks!
> Dave
> 
> 
> 
> dave at beauty:~$ glusterfs --version
> glusterfs 3.5git built on Jun 30 2014 15:58:19
> Repository revision: git:// git.gluster.com/glusterfs.git
> Copyright (c) 2006-2013 Red Hat, Inc. < http://www.redhat.com/ >
> GlusterFS comes with ABSOLUTELY NO WARRANTY.
> It is licensed to you under your choice of the GNU Lesser
> General Public License, version 3 or any later version (LGPLv3
> or later), or the GNU General Public License, version 2 (GPLv2),
> in all cases as published by the Free Software Foundation.
> 
> 
> dave at beauty:~$ uname -r
> 3.15.4-1-ARCH
> 
> 
> dave at beauty:~$ sudo gluster volume info
> Volume Name: data
> Type: Distribute
> Volume ID: 1d5948c7-9b7a-40ca-8aa7-85c74bcef3bc
> Status: Started
> Number of Bricks: 3
> Transport-type: tcp
> Bricks:
> Brick1: beauty:/export/beauty
> Brick2: beast:/export/beast
> Brick3: benji:/export/benji
> Options Reconfigured:
> performance.cache-size: 32MB
> performance.write-behind-window-size: 1MB
> auth.allow:
> 172.30.25.173,172.30.25.158,172.30.25.234,172.30.26.76,172.30.26.77,192.168.0.1,192.168.1.1,192.168.1.2,192.168.2.2,192.168.3.2,192.168.4.1,192.168.4.2,192.168.5.1,192.168.5.2
> nfs.disable: off
> diagnostics.brick-log-level: ERROR
> diagnostics.client-log-level: ERROR
> server.root-squash: enable
> 
> 
> 
> 
> Hi Dave,
> 
> Was rebalance running when you did above operations?
> 
> 
> Regards,
> Raghavendra Bhat
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On 15 July 2014 15:29, Raghavendra Bhat < rabhat at redhat.com > wrote:
> 
> 
> 
> On Monday 14 July 2014 09:10 PM, Pranith Kumar Karampuri wrote:
> 
> 
> CCed Raghavendra Bhat who may know about the issue
> 
> Pranith
> On 07/14/2014 08:01 PM, Joe Julian wrote:
> 
> 
> https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
> 
> Please file a bug report.
> 
> On July 14, 2014 12:38:11 AM PDT, David Raffelt <d.raffelt at brain.org.au>
> wrote:
> 
> 
> Hi All,
> After a recent update to gluster 3.5 we are having some issues renaming files
> when root squashing is enabled and the folder group permissions are not set
> to write.
> 
> For example if I create a folder with the following permissions
> $ mkdir test
> $ chmod g-w test
> $ ls -l
> drwxr-xr-x 2 dave dave 22 Jul 14 17:16 test
> 
> When I create a file within this folder, and try to rename it I get a file
> permissions error.
> 
> $ cd test
> $ touch asdf
> $ mv asdf asdf2
> mv: cannot move ‘asdf’ to ‘asdf2’: Permission denied
> 
> A strace on the mv command reveals the rename system call fails with:
> rename("asdf", "asdf2") = -1 EACCES (Permission denied)
> 
> However I can copy the file and delete the old one fine.
> 
> If I either disable gluster root squashing, or change the test_dir folder
> group permission to write then I can rename the file without any problems.
> 
> System details are:
> Arch linux
> System umask is set to 002
> Distributed volume, 3 peers, 1 brick per peer.
> 
> Any help is much appreciated!
> Dave
> 
> 
> 
> Hi Dave,
> 
> Can you please provide the brick and client log files? Which client you were
> using? fuse or nfs?
> 
> Regards,
> Raghavendra Bhat
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Gluster-users mailing list Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> 
> _______________________________________________
> Gluster-users mailing list Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 
> _______________________________________________
> Gluster-users mailing list Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 
> --
> David Raffelt (PhD)
> Postdoctoral Fellow
> 
> The Florey Institute of Neuroscience and Mental Health
> Melbourne Brain Centre - Austin Campus
> 245 Burgundy Street
> Heidelberg Vic 3084
> Ph: +61 3 9035 7024
> www.florey.edu.au
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 
> --
> David Raffelt (PhD)
> Postdoctoral Fellow
> 
> The Florey Institute of Neuroscience and Mental Health
> Melbourne Brain Centre - Austin Campus
> 245 Burgundy Street
> Heidelberg Vic 3084
> Ph: +61 3 9035 7024
> www.florey.edu.au
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 
> --
> David Raffelt (PhD)
> Postdoctoral Fellow
> 
> The Florey Institute of Neuroscience and Mental Health
> Melbourne Brain Centre - Austin Campus
> 245 Burgundy Street
> Heidelberg Vic 3084
> Ph: +61 3 9035 7024
> www.florey.edu.au
> 
> 
> _______________________________________________
> Gluster-users mailing list Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

More information about the Gluster-users mailing list