[Gluster-users] GlusterFS 3.7.2 and ACL
Jüri Palis
jyri.palis at gmail.com
Thu Jul 30 13:38:10 UTC 2015
[2015-07-30 13:16:01.002296] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: ACL3 - SETACL for 10.1.1.32:742
[2015-07-30 13:16:01.002325] T [MSGID: 0] [acl3.c:672:acl3svc_setacl] 0-nfs-ACL: FH to Volume: acltest
[2015-07-30 13:16:01.004287] T [rpcsvc.c:1319:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x16185ddc, Program: ACL3, ProgVers: 3, Proc: 2) to rpc-transport (socket.nfs-server)
[2015-07-30 13:16:22.823894] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: ACL3 - GETACL for 10.1.1.32:742
[2015-07-30 13:16:22.823900] T [MSGID: 0] [acl3.c:532:acl3svc_getacl] 0-nfs-ACL: FH to Volume: acltest
[2015-07-30 13:16:22.824218] D [MSGID: 0] [client-rpc-fops.c:1156:client3_3_getxattr_cbk] 0-acltest-client-1: remote operation failed: No data available. Path: <gfid:12f02b4f-a181-47d4-9b5b-69e889483570> (12f02b4f-a181-47d4-9b5b-69e889483570). Key: system.posix_acl_default
[2015-07-30 13:16:22.825675] D [MSGID: 0] [client-rpc-fops.c:1156:client3_3_getxattr_cbk] 0-acltest-client-0: remote operation failed: No data available. Path: <gfid:12f02b4f-a181-47d4-9b5b-69e889483570> (12f02b4f-a181-47d4-9b5b-69e889483570). Key: system.posix_acl_default
[2015-07-30 13:16:22.825713] T [rpcsvc.c:1319:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x63815edc, Program: ACL3, ProgVers: 3, Proc: 1) to rpc-transport (socket.nfs-server)
[2015-07-30 13:16:22.828243] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: ACL3 - SETACL for 10.1.1.32:742
[2015-07-30 13:16:22.828266] T [MSGID: 0] [acl3.c:672:acl3svc_setacl] 0-nfs-ACL: FH to Volume: acltest
[2015-07-30 13:16:22.829931] T [rpcsvc.c:1319:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x75815edc, Program: ACL3, ProgVers: 3, Proc: 2) to rpc-transport (socket.nfs-server)
Enabled trace for few moments and tried to make any sense of it by searching for lines containing ‘acl’ according to this everything kind of works except lines which state that “remote operation failed” GlusterFS failed to replicate or commit acl changes?
>
>
> On 07/30/2015 06:22 PM, Jüri Palis wrote:
>> Hi,
>>
>> Thanks Niels, your hints about those two options did the trick although
>> I had to enable both of them and I had to add nscd (sssd provides user
>> identities) to this mix as well.
>>
>> Now back to the problem with ACL’s. Is your test setup something like
>> this: GlusterFS 3.7.2 replicated volume on Centos/RHEL 7 and client or
>> clients accessing GlusterFS volumes by NFS protocol, correct?
>>
> As Jiffin had suggested, did you try the same command on GlusterFS Native mount?
>
> Log levels can be increased to TRACE/DEBUG mode using the command 'gluster vol set <volname> diagnostics.client-log-level [TRACE,DEBUG]'
>
> Also please capture a packet trace on the server-side using the command - 'tcpdump -i any -s 0 -w /var/tmp/nfs-acl.pcap tcp and not port 22'
>
> Verify the packets sent by Gluster-NFS process to the brick process to set the ACL.
>
> Thanks,
> Soumya
>
>> # gluster volume info acltest
>> Volume Name: acltest
>> Type: Replicate
>> Volume ID: 9e0de3f5-45ba-4612-a4f1-16bc5d1eb985
>> Status: Started
>> Number of Bricks: 1 x 2 = 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: vfs-node-01:/data/gfs/acltest/brick0/brick
>> Brick2: vfs-node-02:/data/gfs/acltest/brick0/brick
>> Options Reconfigured:
>> server.manage-gids: on
>> nfs.server-aux-gids: on
>> performance.readdir-ahead: on
>> server.event-threads: 32
>> performance.cache-size: 2GB
>> storage.linux-aio: on
>> nfs.disable: off
>> performance.write-behind-window-size: 1GB
>> performance.nfs.io-cache: on
>> performance.nfs.write-behind-window-size: 250MB
>> performance.nfs.stat-prefetch: on
>> performance.nfs.read-ahead: on
>> performance.nfs.io-threads: on
>> cluster.readdir-optimize: on
>> network.remote-dio: on
>> auth.allow: 10.1.1.32,10.1.1.42
>> diagnostics.latency-measurement: on
>> diagnostics.count-fop-hits: on
>> nfs.rpc-auth-allow: 10.1.1.32,10.1.1.42
>> nfs.trusted-sync: on
>>
>> Maybe there is a way to increase verbosity of nfs server which could
>> help me to trace this problem. I did not find any good hints for
>> increasing verbosity of nfs server in documentation.
>>
>> Regards,
>> J.
>>
>> On 30 Jul 2015, at 10:09, Jiffin Tony Thottan <jthottan at redhat.com
>> <mailto:jthottan at redhat.com>> wrote:
>>
>>>
>>>
>>> On 29/07/15 20:14, Niels de Vos wrote:
>>>> On Wed, Jul 29, 2015 at 05:22:31PM +0300, Jüri Palis wrote:
>>>>> Hi,
>>>>>
>>>>> Another issue with NFS and sec=sys mode. As we all know there is a
>>>>> limit of 15 security ids involved when running NFS in sec=sys mode.
>>>>> This limit makes effective and granular usage of ACL assigned through
>>>>> groups almost unusable. One way to overcome this limit is to use
>>>>> kerberised NFS but GlusterFS does not natively support this access
>>>>> mode . Another option, at least according to one email thread, states
>>>>> that GlusterFS has an option server.manage-gids which should mitigate
>>>>> this limit and raise it to 90 something. Is this the option, which
>>>>> can be used for increasing sec=sys limit. Sadly documentation does not
>>>>> have clear description about this option, what exactly this option
>>>>> does and how it should be used.
>>>> server.manage-gids is an option to resolve the groups of a uid in the
>>>> brick process. You probably need to also use the nfs.server-aux-gids
>>>> option so that the NFS-server resolves the gids of the uid accessing the
>>>> NFS-server.
>>>>
>>>> The nfs.server-aux-gids option is used to overcome the
>>>> AUTH_SYS/AUTH_UNIX limit of (I thought 32?) groups.
>>>>
>>>> The server.manage-gids option is used to overcome the GlusterFS protocol
>>>> limit of ~93 groups.
>>>>
>>>> If your users do not belong to 90+ groups, you would not need to set the
>>>> server.manage-gids option, and nfs.server-aux-gids might be sufficient.
>>>>
>>>> HTH,
>>>> Niels
>>>>
>>>>> J.
>>>>>
>>>>>
>>>>> On 29 Jul 2015, at 16:16, Jiffin Tony Thottan <jthottan at redhat.com
>>>>> <mailto:jthottan at redhat.com>> wrote:
>>>>>
>>>>>>
>>>>>> On 29/07/15 18:04, Jüri Palis wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> setfacl for dir on local filesystem:
>>>>>>>
>>>>>>> 1. set acl setfacl -m g:x_meie_sec-test02:rx test
>>>>>>> 2. get acl
>>>>>>>
>>>>>>> # getfacl test
>>>>>>> user::rwx
>>>>>>> group::r-x
>>>>>>> group:x_meie_sec-test02:r-x
>>>>>>> mask::r-x
>>>>>>> other::r-x
>>>>>>>
>>>>>>> setfacl for dir on GlusterFS volume which is NFS mounted to client
>>>>>>> system
>>>>>>>
>>>>>>> 1. same command is used for setting ACE, no error is returned by
>>>>>>> that command
>>>>>>> 2. get acl
>>>>>>>
>>>>>>> #getfacl test
>>>>>>> user::rwx
>>>>>>> group::r-x
>>>>>>> other::---
>>>>>>>
>>>>>>>
>>>>>>> If I use ordinary file as a target on GlusterFS like this
>>>>>>>
>>>>>>> setfacl -m g:x_meie_sec-test02:rw dummy
>>>>>>>
>>>>>>> then ACE entry is set for file dummy stored on GlusterFS
>>>>>>>
>>>>>>> # getfacl dummy
>>>>>>> user::rw-
>>>>>>> group::r--
>>>>>>> group:x_meie_sec-test02:rw-
>>>>>>> mask::rw-
>>>>>>> other::—
>>>>>>>
>>>>>>> So, as you can see setting ACLs for files works but does not work
>>>>>>> for directories.
>>>>>>>
>>>>>>> This all is happening on CentOS7, running GlusterFS 3.7.2
>>>>>> Hi Jyri,
>>>>>>
>>>>>> It seems there are couple of issues ,
>>>>>>
>>>>>> 1.) when u set a named group acl for file/directory, it clears the
>>>>>> permission of others too.
>>>>>> 2.) named group acl is not working properly for directories ,
>>>>>>
>>>>>> I will try the same on my setup and share my findings.
>>>>>> --
>>>>>> Jiffin
>>>
>>> In my setup (glusterfs 3.7.2 and RHEL 7.1 client) it worked properly
>>>
>>> I followed the same steps mentioned by you.
>>> #cd /mnt
>>> # mkdir dir
>>> # touch file
>>> # getfacl file
>>> # file: file
>>> # owner: root
>>> # group: root
>>> user::rw-
>>> group::r--
>>> other::r--
>>>
>>> # getfacl dir
>>> # file: dir
>>> # owner: root
>>> # group: root
>>> user::rwx
>>> group::r-x
>>> other::r-x
>>>
>>> # setfacl -m g:gluster:rw file
>>> # getfacl file
>>> # file: file
>>> # owner: root
>>> # group: root
>>> user::rw-
>>> group::r--
>>> group:gluster:rw-
>>> mask::rw-
>>> other::r--
>>>
>>> setfacl -m g:gluster:r-x dir
>>> getfacl dir
>>> # file: dir
>>> # owner: root
>>> # group: root
>>> user::rwx
>>> group::r-x
>>> group:gluster:r-x
>>> mask::r-x
>>> other::r-x
>>>
>>>
>>> So can u share the following information from the server.
>>> 1.) gluster vol info
>>> 2.) nfs.log (nfs-server log)
>>> 3.) brick logs
>>>
>>> and also can u try the same on fuse mount(gluster native mount).
>>>
>>> --
>>> Jiffin
>>>
>>>>>>> J.
>>>>>>> On 29 Jul 2015, at 15:16, Jiffin Thottan <jthottan at redhat.com
>>>>>>> <mailto:jthottan at redhat.com>> wrote:
>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>> From: "Jüri Palis" <jyri.palis at gmail.com
>>>>>>>> <mailto:jyri.palis at gmail.com>>
>>>>>>>> To: gluster-users at gluster.org <mailto:gluster-users at gluster.org>
>>>>>>>> Sent: Wednesday, July 29, 2015 4:19:20 PM
>>>>>>>> Subject: [Gluster-users] GlusterFS 3.7.2 and ACL
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> Setup:
>>>>>>>> GFS 3.7.2, NFS is used for host access
>>>>>>>>
>>>>>>>> Problem:
>>>>>>>> POSIX ACL work correctly when ACLs are applied to files but do
>>>>>>>> not work when ACLs are applied to directories on GFS volumes.
>>>>>>>>
>>>>>>>> How can I debug this issue more deeply?
>>>>>>>>
>>>>>>>> Can you please explain the issue with more details, i.e what
>>>>>>>> exactly not working properly , is it setting acl or any
>>>>>>>> functionality issue, in which client?
>>>>>>>> __
>>>>>>>> Jiffin
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Jyri
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150730/b2076b4e/attachment.html>
More information about the Gluster-users
mailing list