[Gluster-users] Slow write times to gluster disk

[Soumya Koduri]

On 06/30/2017 07:56 PM, Pat Haley wrote:
>
> Hi,
>
> I was wondering if there were any additional test we could perform to
> help debug the group write-permissions issue?

Sorry for the delay. Please find response inline --

>
> Thanks
>
> Pat
>
>
> On 06/27/2017 12:29 PM, Pat Haley wrote:
>>
>> Hi Soumya,
>>
>> One example, we have a common working directory dri_fleat in the
>> gluster volume
>>
>> drwxrwsr-x 22 root dri_fleat     4.0K May  1 15:14 dri_fleat
>>
>> my user (phaley) does not own that directory but is a member of the
>> group  dri_fleat and should have write permissions.  When I go to the
>> nfs-mounted version and try to use the touch command I get the following
>>
>> ibfdr-compute-0-4(dri_fleat)% touch dum
>> touch: cannot touch `dum': Permission denied
>>
>> One of the sub-directories under dri_fleat is "test" which phaley owns
>>
>> drwxrwsr-x  2 phaley   dri_fleat 4.0K May  1 15:16 test
>>
>> Under this directory (mounted via nfs) user phaley can write
>>
>> ibfdr-compute-0-4(test)% touch dum
>> ibfdr-compute-0-4(test)%
>>
>> I have put the packet captures in
>>
>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
>>
>> capture_nfsfail.pcap   has the results from the failed touch experiment
>> capture_nfssucceed.pcap  has the results from the successful touch
>> experiment
>>
>> The command I used for these was
>>
>> tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w /root/capture_nfstest.pcap

I hope these pkts were captured on the node where NFS server is running. 
Could you please use '-i any' as I do not see glusterfs traffic in the 
tcpdump.

Also looks like NFS v4 is used between client & nfs server. Are you 
using kernel-NFS here (i.e, kernel-NFS exporting fuse mounted gluster 
volume)?
If that is the case please capture fuse-mnt logs as well. This error may 
well be coming from kernel-NFS itself before the request is sent to 
fuse-mnt process.

FWIW, we have below option -

Option: server.manage-gids
Default Value: off
Description: Resolve groups on the server-side.

I haven't looked into what this option exactly does. But it may worth 
testing with this option on.

Thanks,
Soumya


>>
>> The brick log files are also in the above link.  If I read them
>> correctly they both funny times.  Specifically I see entries from
>> around 2017-06-27 14:02:37.404865  even though the system time was
>> 2017-06-27 12:00:00.
>>
>> One final item, another reply to my post had a link for possible
>> problems that could arise from users belonging to too many group. We
>> have seen the above problem even with a user belonging to only 4 groups.
>>
>> Let me know what additional information I can provide.

>>
>> Thanks
>>
>> Pat
>>
>>
>> On 06/27/2017 02:45 AM, Soumya Koduri wrote:
>>>
>>>
>>> On 06/27/2017 10:17 AM, Pranith Kumar Karampuri wrote:
>>>> The only problem with using gluster mounted via NFS is that it does not
>>>> respect the group write permissions which we need.
>>>>
>>>> We have an exercise coming up in the a couple of weeks.  It seems to me
>>>> that in order to improve our write times before then, it would be good
>>>> to solve the group write permissions for gluster mounted via NFS now.
>>>> We can then revisit gluster mounted via FUSE afterwards.
>>>>
>>>> What information would you need to help us force gluster mounted via
>>>> NFS
>>>> to respect the group write permissions?
>>>
>>> Is this owning group or one of the auxiliary groups whose write
>>> permissions are not considered? AFAIK, there are no special
>>> permission checks done by gNFS server when compared to gluster native
>>> client.
>>>
>>> Could you please provide simple steps to reproduce the issue and
>>> collect pkt trace and nfs/brick logs as well.
>>>
>>> Thanks,
>>> Soumya
>>
>