[Gluster-users] group write permissions not being respected

Pat Haley phaley at mit.edu
Wed Sep 7 02:09:31 UTC 2016


Using the gluster client rather than NFS seems to fix the problem


On 09/01/2016 02:35 PM, Pat Haley wrote:
>
>
> Hi Pranith,
>
> In attached file capture.pcap
>
>
> On 09/01/2016 01:01 PM, Pranith Kumar Karampuri wrote:
>> You need to capture the file so that we can see the tcpdump in 
>> wireshark to inspect the uid/gid etc that are going out the wire.
>>
>> On Thu, Sep 1, 2016 at 10:04 PM, Pat Haley <phaley at mit.edu 
>> <mailto:phaley at mit.edu>> wrote:
>>
>>
>>     Hi Pranith,
>>
>>     Here is the output when I'm trying a touch command that fails
>>     with "Permission denied"
>>
>>     [root at compute-11-10 ~]# tcpdump -nnSs 0 host 10.1.1.4
>>     tcpdump: verbose output suppressed, use -v or -vv for full
>>     protocol decode
>>     listening on eth1, link-type EN10MB (Ethernet), capture size
>>     65535 bytes
>>     12:30:46.248293 IP 10.255.255.124.4215828946 > 10.1.1.4.2049: 208
>>     getattr fh 0,0/22
>>     12:30:46.252509 IP 10.1.1.4.2049 > 10.255.255.124.4215828946:
>>     reply ok 240 getattr NON 3 ids 0/3 sz 0
>>     12:30:46.252596 IP 10.255.255.124.4232606162 <tel:4232606162> >
>>     10.1.1.4.2049: 300 getattr fh 0,0/22
>>     12:30:46.253308 IP 10.1.1.4.2049 > 10.255.255.124.4232606162:
>>     reply ok 52 getattr ERROR: Permission denied
>>     12:30:46.253358 IP 10.255.255.124.4249383378 <tel:4249383378> >
>>     10.1.1.4.2049: 216 getattr fh 0,0/22
>>     12:30:46.260347 IP 10.1.1.4.2049 > 10.255.255.124.4249383378:
>>     reply ok 52 getattr ERROR: No such file or directory
>>     12:30:46.300306 IP 10.255.255.124.931 > 10.1.1.4.2049: Flags [.],
>>     ack 1979284005, win 501, options [nop,nop,TS val 490628016 ecr
>>     75449144], length 0
>>     ^C
>>     7 packets captured
>>     7 packets received by filter
>>     0 packets dropped by kernel
>>
>>
>>     On 09/01/2016 03:31 AM, Pranith Kumar Karampuri wrote:
>>>     hi Pat,
>>>            I think the other thing we should probably look for would
>>>     be to see the tcp dump of what uid/gid parameters are sent over
>>>     network when this command is executed.
>>>
>>>     On Thu, Sep 1, 2016 at 7:14 AM, Pat Haley <phaley at mit.edu
>>>     <mailto:phaley at mit.edu>> wrote:
>>>
>>>         --------------------------------------------------------------------------------------------
>>>>         hi Pat,
>>>>               Are you seeing this issue only after migration or
>>>>         even before? May be we should look at the gid numbers on
>>>>         the disk and the ones that are coming from client for the
>>>>         given user to see if they match or not?
>>>         -------------------------------------------------------------------------------------------------
>>>         This issue was not being seen before the migration.  We have
>>>         copied the /etc/passwd and /etc/group files from the
>>>         front-end machine (the client) to the data server, so they
>>>         all match
>>>         -------------------------------------------------------------------------------------------------
>>>>         Could you give stat output of the directory in question
>>>>         from both the brick and the nfs client
>>>>
>>>         --------------------------------------------------------------------------------------------------
>>>         From the server for gluster:
>>>         [root at mseas-data2 ~]# stat /gdata/projects/nsf_alpha
>>>           File: `/gdata/projects/nsf_alpha'
>>>           Size: 4096          Blocks: 8          IO Block: 131072
>>>         directory
>>>         Device: 13h/19d    Inode: 13094773206281819436  Links: 13
>>>         Access: (2775/drwxrwsr-x)  Uid: (    0/    root)   Gid: (
>>>         598/nsf_alpha)
>>>         Access: 2016-08-31 19:08:59.735990904 -0400
>>>         Modify: 2016-08-31 16:37:09.048997167 -0400
>>>         Change: 2016-08-31 16:37:41.315997148 -0400
>>>
>>>         From the server for first underlying brick
>>>         [root at mseas-data2 ~]# stat /mnt/brick1/projects/nsf_alpha/
>>>           File: `/mnt/brick1/projects/nsf_alpha/'
>>>           Size: 4096          Blocks: 8 IO Block: 4096   directory
>>>         Device: 800h/2048d    Inode: 185630 Links: 13
>>>         Access: (2775/drwxrwsr-x)  Uid: ( 0/    root)   Gid: ( 
>>>         598/nsf_alpha)
>>>         Access: 2016-08-31 19:08:59.669990907 -0400
>>>         Modify: 2016-08-31 16:37:09.048997167 -0400
>>>         Change: 2016-08-31 16:37:41.315997148 -0400
>>>
>>>         From the server for second underlying brick
>>>         [root at mseas-data2 ~]# stat /mnt/brick2/projects/nsf_alpha/
>>>           File: `/mnt/brick2/projects/nsf_alpha/'
>>>           Size: 4096          Blocks: 8 IO Block: 4096   directory
>>>         Device: 810h/2064d    Inode: 24085468 Links: 13
>>>         Access: (2775/drwxrwsr-x)  Uid: ( 0/    root)   Gid: ( 
>>>         598/nsf_alpha)
>>>         Access: 2016-08-31 19:08:59.735990904 -0400
>>>         Modify: 2016-08-03 14:01:52.000000000 -0400
>>>         Change: 2016-08-31 16:37:41.315997148 -0400
>>>
>>>         From the client
>>>         [root at mseas FixOwn]# stat /gdata/projects/nsf_alpha
>>>           File: `/gdata/projects/nsf_alpha'
>>>           Size: 4096          Blocks: 8 IO Block: 1048576 directory
>>>         Device: 23h/35d    Inode: 13094773206281819436  Links: 13
>>>         Access: (2775/drwxrwsr-x)  Uid: ( 0/    root)   Gid: ( 
>>>         598/nsf_alpha)
>>>         Access: 2016-08-31 19:08:59.735990904 -0400
>>>         Modify: 2016-08-31 16:37:09.048997167 -0400
>>>         Change: 2016-08-31 16:37:41.315997148 -0400
>>>
>>>         ------------------------------------------------------------------------------------------------
>>>>         Could you also let us know version of gluster you are using
>>>>         -------------------------------------------------------------------------------------------------
>>>
>>>>         [root at mseas-data2 ~]# gluster --version
>>>>         glusterfs 3.7.11 built on Apr 27 2016 14:09:22
>>>>
>>>         [root at mseas-data2 ~]# gluster volume info
>>>
>>>         Volume Name: data-volume
>>>         Type: Distribute
>>>         Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
>>>         Status: Started
>>>         Number of Bricks: 2
>>>         Transport-type: tcp
>>>         Bricks:
>>>         Brick1: mseas-data2:/mnt/brick1
>>>         Brick2: mseas-data2:/mnt/brick2
>>>         Options Reconfigured:
>>>         performance.readdir-ahead: on
>>>         nfs.disable: on
>>>         nfs.export-volumes: off
>>>
>>>         [root at mseas-data2 ~]# gluster volume status
>>>         Status of volume: data-volume
>>>         Gluster process TCP Port  RDMA Port  Online  Pid
>>>         ------------------------------------------------------------------------------
>>>         Brick mseas-data2:/mnt/brick1 49154     0          Y       5005
>>>         Brick mseas-data2:/mnt/brick2 49155     0          Y       5010
>>>
>>>         Task Status of Volume data-volume
>>>         ------------------------------------------------------------------------------
>>>         Task                 : Rebalance
>>>         ID                   : 892d9e3a-b38c-4971-b96a-8e4a496685ba
>>>         Status               : completed
>>>
>>>
>>>         [root at mseas-data2 ~]# gluster peer status
>>>         Number of Peers: 0
>>>
>>>>
>>>>         -------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>>         On Thu, Sep 1, 2016 at 2:46 AM, Pat Haley <phaley at mit.edu
>>>>         <mailto:phaley at mit.edu>> wrote:
>>>>
>>>>
>>>>             Hi,
>>>>
>>>>             Another piece of data. There are 2 distinct volumes on
>>>>             the file server
>>>>
>>>>              1. a straight nfs partition
>>>>              2. a gluster volume (served over nfs)
>>>>
>>>>             The straight nfs partition does respect the group write
>>>>             permissions, while the gluster volume does not. Any
>>>>             suggestions on how to debug this or what additional
>>>>             information would be helpful would be greatly appreciated
>>>>
>>>>             Thanks
>>>>
>>>>             On 08/30/2016 06:01 PM, Pat Haley wrote:
>>>>>
>>>>>             Hi
>>>>>
>>>>>             We have just migrated our data to a new file server
>>>>>             (more space, old server was showing its age). We have
>>>>>             a volume for collaborative use, based on group
>>>>>             membership.  In our new server, the group write
>>>>>             permissions are not being respected (e.g. the owner of
>>>>>             a directory can still write to that directory but any
>>>>>             other member of the associated group cannot, even
>>>>>             though the directory clearly has group write
>>>>>             permissions set).  This is occurring regardless of how
>>>>>             many groups the user is a member of (i.e. users that
>>>>>             are members of fewer then 16 groups are still affected).
>>>>>
>>>>>             the relevant fstab line from the server looks like
>>>>>             localhost:/data-volume /gdata glusterfs       defaults
>>>>>             0 0
>>>>>
>>>>>             and for a client:
>>>>>             mseas-data2:/gdata /gdata      nfs defaults        0 0
>>>>>
>>>>>             Any help would be greatly appreciated.
>>>>>
>>>>>             Thanks
>>>>>
>>>>
>>>>             -- 
>>>>
>>>>             -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>             Pat Haley                          Email:phaley at mit.edu <mailto:phaley at mit.edu>
>>>>             Center for Ocean Engineering       Phone:(617) 253-6824 <tel:%28617%29%20253-6824>
>>>>             Dept. of Mechanical Engineering    Fax:(617) 253-8125 <tel:%28617%29%20253-8125>
>>>>             MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>             77 Massachusetts Avenue
>>>>             Cambridge, MA  02139-4301
>>>>
>>>>             _______________________________________________
>>>>             Gluster-users mailing list Gluster-users at gluster.org
>>>>             <mailto:Gluster-users at gluster.org>
>>>>             http://www.gluster.org/mailman/listinfo/gluster-users
>>>>             <http://www.gluster.org/mailman/listinfo/gluster-users> 
>>>>
>>>>         -- 
>>>>         Pranith
>>>
>>>         -- 
>>>
>>>         -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>         Pat Haley                          Email:phaley at mit.edu <mailto:phaley at mit.edu>
>>>         Center for Ocean Engineering       Phone:(617) 253-6824 <tel:%28617%29%20253-6824>
>>>         Dept. of Mechanical Engineering    Fax:(617) 253-8125 <tel:%28617%29%20253-8125>
>>>         MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>         77 Massachusetts Avenue
>>>         Cambridge, MA  02139-4301
>>>
>>>         _______________________________________________
>>>         Gluster-users mailing list Gluster-users at gluster.org
>>>         <mailto:Gluster-users at gluster.org>
>>>         http://www.gluster.org/mailman/listinfo/gluster-users
>>>         <http://www.gluster.org/mailman/listinfo/gluster-users> 
>>>
>>>     -- 
>>>     Pranith
>>     -- 
>>
>>     -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>     Pat Haley                          Email:phaley at mit.edu <mailto:phaley at mit.edu>
>>     Center for Ocean Engineering       Phone:(617) 253-6824 <tel:%28617%29%20253-6824>
>>     Dept. of Mechanical Engineering    Fax:(617) 253-8125 <tel:%28617%29%20253-8125>
>>     MIT, Room 5-213http://web.mit.edu/phaley/www/
>>     77 Massachusetts Avenue
>>     Cambridge, MA  02139-4301
>>
>>     _______________________________________________ Gluster-users
>>     mailing list Gluster-users at gluster.org
>>     <mailto:Gluster-users at gluster.org>
>>     http://www.gluster.org/mailman/listinfo/gluster-users
>>     <http://www.gluster.org/mailman/listinfo/gluster-users> 
>>
>> -- 
>> Pranith
> -- 
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley                          Email:phaley at mit.edu
> Center for Ocean Engineering       Phone:  (617) 253-6824
> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
> MIT, Room 5-213http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA  02139-4301
-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160906/d5c2e1b9/attachment.html>


More information about the Gluster-users mailing list