[Gluster-users] Slow write times to gluster disk

Soumya Koduri skoduri at redhat.com
Fri Jul 7 17:31:42 UTC 2017 

Hi,

On 07/07/2017 06:16 AM, Pat Haley wrote:
>
> Hi All,
>
> A follow-up question.  I've been looking at various pages on nfs-ganesha
> & gluster.  Is there a version of nfs-ganesha that is recommended for
> use with
>
> glusterfs 3.7.11 built on Apr 27 2016 14:09:22
> CentOS release 6.8 (Final)

For glusterfs 3.7, nfs-ganesha-2.3-* version can be used.

I see the packages built in centos7 storage sig [1] but not for centos6. 
Request Niels to comment.


>
> Thanks
>
> Pat
>
>
> On 07/05/2017 11:36 AM, Pat Haley wrote:
>>
>> Hi Soumya,
>>
>> (1) In http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
>> I've placed the following 2 log files
>>
>> etc-glusterfs-glusterd.vol.log
>> gdata.log
>>
>> The first has repeated messages about nfs disconnects.  The second had
>> the <fuse_mnt_direcotry>.log name (but not much information).
>>

Hmm yeah..weird ..there are not much logs in fuse mnt log file.

>> (2) About the gluster-NFS native server:  do you know where we can
>> find documentation on how to use/install it?  We haven't had success
>> in our searches.
>>

Till glusterfs-3.7, gluster-NFS (gNFS) gets enabled by default. The only 
requirement is that kernel-NFS has to be disabled for gluster-NFS to 
come up. Please disable kernel-NFS server and restart glusterd to start 
gNFS. In case of any issues with starting gNFS server, please look at 
/var/log/glusterfs/nfs.log.

Thanks,
Soumya


[1] https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.7/
[2] https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/

>> Thanks
>>
>> Pat
>>
>>
>> On 07/04/2017 05:01 AM, Soumya Koduri wrote:
>>>
>>>
>>> On 07/03/2017 09:01 PM, Pat Haley wrote:
>>>>
>>>> Hi Soumya,
>>>>
>>>> When I originally did the tests I ran tcpdump on the client.
>>>>
>>>> I have rerun the tests, doing tcpdump on the server
>>>>
>>>> tcpdump -i any -nnSs 0 host 172.16.1.121 -w /root/capture_nfsfail.pcap
>>>>
>>>> The results are in the same place
>>>>
>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
>>>>
>>>> capture_nfsfail.pcap   has the results from the failed touch experiment
>>>> capture_nfssucceed.pcap  has the results from the successful touch
>>>> experiment
>>>>
>>>> The brick log files are there too.
>>>
>>> Thanks for sharing. Looks like the error is not generated
>>> @gluster-server side. The permission denied error was caused by
>>> either kNFS or by fuse-mnt process or probably by the combination.
>>>
>>> To check fuse-mnt logs, please look at
>>> /var/log/glusterfs/<fuse_mnt_direcotry>.log
>>>
>>> For eg.: if you have fuse mounted the gluster volume at /mnt/fuse-mnt
>>> and exported it via kNFS, the log location for that fuse_mnt shall be
>>> at /var/log/glusterfs/mnt-fuse-mnt.log
>>>
>>>
>>> Also why not switch to either gluster-NFS native server or
>>> NFS-Ganesha instead of using kNFS, as they are recommended NFS
>>> servers to use with gluster?
>>>
>>> Thanks,
>>> Soumya
>>>
>>>>
>>>> I believe we are using kernel-NFS exporting a fuse mounted gluster
>>>> volume.  I am having Steve confirm this.  I tried to find the fuse-mnt
>>>> logs but failed.  Where should I look for them?
>>>>
>>>> Thanks
>>>>
>>>> Pat
>>>>
>>>>
>>>>
>>>> On 07/03/2017 07:58 AM, Soumya Koduri wrote:
>>>>>
>>>>>
>>>>> On 06/30/2017 07:56 PM, Pat Haley wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I was wondering if there were any additional test we could perform to
>>>>>> help debug the group write-permissions issue?
>>>>>
>>>>> Sorry for the delay. Please find response inline --
>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Pat
>>>>>>
>>>>>>
>>>>>> On 06/27/2017 12:29 PM, Pat Haley wrote:
>>>>>>>
>>>>>>> Hi Soumya,
>>>>>>>
>>>>>>> One example, we have a common working directory dri_fleat in the
>>>>>>> gluster volume
>>>>>>>
>>>>>>> drwxrwsr-x 22 root dri_fleat     4.0K May  1 15:14 dri_fleat
>>>>>>>
>>>>>>> my user (phaley) does not own that directory but is a member of the
>>>>>>> group  dri_fleat and should have write permissions. When I go to the
>>>>>>> nfs-mounted version and try to use the touch command I get the
>>>>>>> following
>>>>>>>
>>>>>>> ibfdr-compute-0-4(dri_fleat)% touch dum
>>>>>>> touch: cannot touch `dum': Permission denied
>>>>>>>
>>>>>>> One of the sub-directories under dri_fleat is "test" which phaley
>>>>>>> owns
>>>>>>>
>>>>>>> drwxrwsr-x  2 phaley   dri_fleat 4.0K May  1 15:16 test
>>>>>>>
>>>>>>> Under this directory (mounted via nfs) user phaley can write
>>>>>>>
>>>>>>> ibfdr-compute-0-4(test)% touch dum
>>>>>>> ibfdr-compute-0-4(test)%
>>>>>>>
>>>>>>> I have put the packet captures in
>>>>>>>
>>>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
>>>>>>>
>>>>>>> capture_nfsfail.pcap   has the results from the failed touch
>>>>>>> experiment
>>>>>>> capture_nfssucceed.pcap  has the results from the successful touch
>>>>>>> experiment
>>>>>>>
>>>>>>> The command I used for these was
>>>>>>>
>>>>>>> tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w
>>>>>>> /root/capture_nfstest.pcap
>>>>>
>>>>> I hope these pkts were captured on the node where NFS server is
>>>>> running. Could you please use '-i any' as I do not see glusterfs
>>>>> traffic in the tcpdump.
>>>>>
>>>>> Also looks like NFS v4 is used between client & nfs server. Are you
>>>>> using kernel-NFS here (i.e, kernel-NFS exporting fuse mounted gluster
>>>>> volume)?
>>>>> If that is the case please capture fuse-mnt logs as well. This error
>>>>> may well be coming from kernel-NFS itself before the request is sent
>>>>> to fuse-mnt process.
>>>>>
>>>>> FWIW, we have below option -
>>>>>
>>>>> Option: server.manage-gids
>>>>> Default Value: off
>>>>> Description: Resolve groups on the server-side.
>>>>>
>>>>> I haven't looked into what this option exactly does. But it may worth
>>>>> testing with this option on.
>>>>>
>>>>> Thanks,
>>>>> Soumya
>>>>>
>>>>>
>>>>>>>
>>>>>>> The brick log files are also in the above link.  If I read them
>>>>>>> correctly they both funny times.  Specifically I see entries from
>>>>>>> around 2017-06-27 14:02:37.404865  even though the system time was
>>>>>>> 2017-06-27 12:00:00.
>>>>>>>
>>>>>>> One final item, another reply to my post had a link for possible
>>>>>>> problems that could arise from users belonging to too many group. We
>>>>>>> have seen the above problem even with a user belonging to only 4
>>>>>>> groups.
>>>>>>>
>>>>>>> Let me know what additional information I can provide.
>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Pat
>>>>>>>
>>>>>>>
>>>>>>> On 06/27/2017 02:45 AM, Soumya Koduri wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 06/27/2017 10:17 AM, Pranith Kumar Karampuri wrote:
>>>>>>>>> The only problem with using gluster mounted via NFS is that it
>>>>>>>>> does not
>>>>>>>>> respect the group write permissions which we need.
>>>>>>>>>
>>>>>>>>> We have an exercise coming up in the a couple of weeks. It seems
>>>>>>>>> to me
>>>>>>>>> that in order to improve our write times before then, it would be
>>>>>>>>> good
>>>>>>>>> to solve the group write permissions for gluster mounted via
>>>>>>>>> NFS now.
>>>>>>>>> We can then revisit gluster mounted via FUSE afterwards.
>>>>>>>>>
>>>>>>>>> What information would you need to help us force gluster
>>>>>>>>> mounted via
>>>>>>>>> NFS
>>>>>>>>> to respect the group write permissions?
>>>>>>>>
>>>>>>>> Is this owning group or one of the auxiliary groups whose write
>>>>>>>> permissions are not considered? AFAIK, there are no special
>>>>>>>> permission checks done by gNFS server when compared to gluster
>>>>>>>> native
>>>>>>>> client.
>>>>>>>>
>>>>>>>> Could you please provide simple steps to reproduce the issue and
>>>>>>>> collect pkt trace and nfs/brick logs as well.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Soumya
>>>>>>>
>>>>>>
>>>>
>>
>

More information about the Gluster-users mailing list