[Gluster-users] Slow write times to gluster disk

[Soumya Koduri]

On 07/03/2017 09:01 PM, Pat Haley wrote:
>
> Hi Soumya,
>
> When I originally did the tests I ran tcpdump on the client.
>
> I have rerun the tests, doing tcpdump on the server
>
> tcpdump -i any -nnSs 0 host 172.16.1.121 -w /root/capture_nfsfail.pcap
>
> The results are in the same place
>
> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
>
> capture_nfsfail.pcap   has the results from the failed touch experiment
> capture_nfssucceed.pcap  has the results from the successful touch
> experiment
>
> The brick log files are there too.

Thanks for sharing. Looks like the error is not generated 
@gluster-server side. The permission denied error was caused by either 
kNFS or by fuse-mnt process or probably by the combination.

To check fuse-mnt logs, please look at 
/var/log/glusterfs/<fuse_mnt_direcotry>.log

For eg.: if you have fuse mounted the gluster volume at /mnt/fuse-mnt 
and exported it via kNFS, the log location for that fuse_mnt shall be at 
/var/log/glusterfs/mnt-fuse-mnt.log


Also why not switch to either gluster-NFS native server or NFS-Ganesha 
instead of using kNFS, as they are recommended NFS servers to use with 
gluster?

Thanks,
Soumya

>
> I believe we are using kernel-NFS exporting a fuse mounted gluster
> volume.  I am having Steve confirm this.  I tried to find the fuse-mnt
> logs but failed.  Where should I look for them?
>
> Thanks
>
> Pat
>
>
>
> On 07/03/2017 07:58 AM, Soumya Koduri wrote:
>>
>>
>> On 06/30/2017 07:56 PM, Pat Haley wrote:
>>>
>>> Hi,
>>>
>>> I was wondering if there were any additional test we could perform to
>>> help debug the group write-permissions issue?
>>
>> Sorry for the delay. Please find response inline --
>>
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>>
>>> On 06/27/2017 12:29 PM, Pat Haley wrote:
>>>>
>>>> Hi Soumya,
>>>>
>>>> One example, we have a common working directory dri_fleat in the
>>>> gluster volume
>>>>
>>>> drwxrwsr-x 22 root dri_fleat     4.0K May  1 15:14 dri_fleat
>>>>
>>>> my user (phaley) does not own that directory but is a member of the
>>>> group  dri_fleat and should have write permissions.  When I go to the
>>>> nfs-mounted version and try to use the touch command I get the
>>>> following
>>>>
>>>> ibfdr-compute-0-4(dri_fleat)% touch dum
>>>> touch: cannot touch `dum': Permission denied
>>>>
>>>> One of the sub-directories under dri_fleat is "test" which phaley owns
>>>>
>>>> drwxrwsr-x  2 phaley   dri_fleat 4.0K May  1 15:16 test
>>>>
>>>> Under this directory (mounted via nfs) user phaley can write
>>>>
>>>> ibfdr-compute-0-4(test)% touch dum
>>>> ibfdr-compute-0-4(test)%
>>>>
>>>> I have put the packet captures in
>>>>
>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
>>>>
>>>> capture_nfsfail.pcap   has the results from the failed touch experiment
>>>> capture_nfssucceed.pcap  has the results from the successful touch
>>>> experiment
>>>>
>>>> The command I used for these was
>>>>
>>>> tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w /root/capture_nfstest.pcap
>>
>> I hope these pkts were captured on the node where NFS server is
>> running. Could you please use '-i any' as I do not see glusterfs
>> traffic in the tcpdump.
>>
>> Also looks like NFS v4 is used between client & nfs server. Are you
>> using kernel-NFS here (i.e, kernel-NFS exporting fuse mounted gluster
>> volume)?
>> If that is the case please capture fuse-mnt logs as well. This error
>> may well be coming from kernel-NFS itself before the request is sent
>> to fuse-mnt process.
>>
>> FWIW, we have below option -
>>
>> Option: server.manage-gids
>> Default Value: off
>> Description: Resolve groups on the server-side.
>>
>> I haven't looked into what this option exactly does. But it may worth
>> testing with this option on.
>>
>> Thanks,
>> Soumya
>>
>>
>>>>
>>>> The brick log files are also in the above link.  If I read them
>>>> correctly they both funny times.  Specifically I see entries from
>>>> around 2017-06-27 14:02:37.404865  even though the system time was
>>>> 2017-06-27 12:00:00.
>>>>
>>>> One final item, another reply to my post had a link for possible
>>>> problems that could arise from users belonging to too many group. We
>>>> have seen the above problem even with a user belonging to only 4
>>>> groups.
>>>>
>>>> Let me know what additional information I can provide.
>>
>>>>
>>>> Thanks
>>>>
>>>> Pat
>>>>
>>>>
>>>> On 06/27/2017 02:45 AM, Soumya Koduri wrote:
>>>>>
>>>>>
>>>>> On 06/27/2017 10:17 AM, Pranith Kumar Karampuri wrote:
>>>>>> The only problem with using gluster mounted via NFS is that it
>>>>>> does not
>>>>>> respect the group write permissions which we need.
>>>>>>
>>>>>> We have an exercise coming up in the a couple of weeks. It seems
>>>>>> to me
>>>>>> that in order to improve our write times before then, it would be
>>>>>> good
>>>>>> to solve the group write permissions for gluster mounted via NFS now.
>>>>>> We can then revisit gluster mounted via FUSE afterwards.
>>>>>>
>>>>>> What information would you need to help us force gluster mounted via
>>>>>> NFS
>>>>>> to respect the group write permissions?
>>>>>
>>>>> Is this owning group or one of the auxiliary groups whose write
>>>>> permissions are not considered? AFAIK, there are no special
>>>>> permission checks done by gNFS server when compared to gluster native
>>>>> client.
>>>>>
>>>>> Could you please provide simple steps to reproduce the issue and
>>>>> collect pkt trace and nfs/brick logs as well.
>>>>>
>>>>> Thanks,
>>>>> Soumya
>>>>
>>>
>