[Gluster-users] Slow write times to gluster disk
Soumya Koduri
skoduri at redhat.com
Fri Jul 14 05:04:13 UTC 2017
On 07/14/2017 06:40 AM, Pat Haley wrote:
>
> Hi Soumya,
>
> I just noticed some of the notes at the bottom. In particular
>
> * Till glusterfs-3.7, gluster-NFS (gNFS) gets enabled by default. The
> only requirement is that kernel-NFS has to be disabled for
> gluster-NFS to come up. Please disable kernel-NFS server and restart
> glusterd to start gNFS. In case of any issues with starting gNFS
> server, please look at /var/log/glusterfs/nfs.log.
>
> If we disable the kernel-NFS on our server and restart glusterd to start
> gNFS will that affect the NFS file system also being served by that
> server (i.e. the single server serves both a glusterFS area and an NFS
> area)? clients
Thats right. When you restart glusterd, it tries to spawn (provided
nfs.disable option is set to off for any volume) a new glusterfs client
process which acts like NFS-server as well.
Would we also have to disable the kernel-NFS for NFS-ganesha?
yes.
>
> My second question concerns NFS-ganesha (v 2.3.x) for CentOS 6.8 and
> gluster 3.7.11. I think I see a couple of possibilities
>
> 1. I see one possible rpm for version 2.3.3 in
> https://mirror.chpc.utah.edu/pub/vault.centos.org/centos/6.8/storage/Source/gluster-3.8/
> The other rpm's seem to be for gluster 3.8 packages, so I'm
> wondering if there is a concern for conflict
AFAIK, nfs-ganesha-2.3.3 should work with both 3.8 & 3.7 gluster.
> 2. In one of the links you sent
> (https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/)
> I see an rpm for glusterfs-ganesha-3.7.11 . Is this a specific
> gluster package for compatibility with ganesha or a ganesha package
> for gluster?
This is to be compatible with gluster-3.7* package.
>
> Does either possibility seem more likely to be what I need than the other?
The current stable/maintained/tested combination is nfs-ganesha2.4/2.5 +
glusterfs-3.8/3.10. But however incase you cannot upgrade, you can still
use nfs-ganesha-2.3* with glusterfs-3.8/3.7
Hope it is clear.
Thanks,
Soumya
>
> Pat
>
>
> On 07/07/2017 01:31 PM, Soumya Koduri wrote:
>> Hi,
>>
>> On 07/07/2017 06:16 AM, Pat Haley wrote:
>>>
>>> Hi All,
>>>
>>> A follow-up question. I've been looking at various pages on nfs-ganesha
>>> & gluster. Is there a version of nfs-ganesha that is recommended for
>>> use with
>>>
>>> glusterfs 3.7.11 built on Apr 27 2016 14:09:22
>>> CentOS release 6.8 (Final)
>>
>> For glusterfs 3.7, nfs-ganesha-2.3-* version can be used.
>>
>> I see the packages built in centos7 storage sig [1] but not for
>> centos6. Request Niels to comment.
>>
>>
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>>
>>> On 07/05/2017 11:36 AM, Pat Haley wrote:
>>>>
>>>> Hi Soumya,
>>>>
>>>> (1) In http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
>>>> I've placed the following 2 log files
>>>>
>>>> etc-glusterfs-glusterd.vol.log
>>>> gdata.log
>>>>
>>>> The first has repeated messages about nfs disconnects. The second had
>>>> the <fuse_mnt_direcotry>.log name (but not much information).
>>>>
>>
>> Hmm yeah..weird ..there are not much logs in fuse mnt log file.
>>
>>>> (2) About the gluster-NFS native server: do you know where we can
>>>> find documentation on how to use/install it? We haven't had success
>>>> in our searches.
>>>>
>>
>> Till glusterfs-3.7, gluster-NFS (gNFS) gets enabled by default. The
>> only requirement is that kernel-NFS has to be disabled for gluster-NFS
>> to come up. Please disable kernel-NFS server and restart glusterd to
>> start gNFS. In case of any issues with starting gNFS server, please
>> look at /var/log/glusterfs/nfs.log.
>>
>> Thanks,
>> Soumya
>>
>>
>> [1] https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.7/
>> [2] https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/
>>
>>>> Thanks
>>>>
>>>> Pat
>>>>
>>>>
>>>> On 07/04/2017 05:01 AM, Soumya Koduri wrote:
>>>>>
>>>>>
>>>>> On 07/03/2017 09:01 PM, Pat Haley wrote:
>>>>>>
>>>>>> Hi Soumya,
>>>>>>
>>>>>> When I originally did the tests I ran tcpdump on the client.
>>>>>>
>>>>>> I have rerun the tests, doing tcpdump on the server
>>>>>>
>>>>>> tcpdump -i any -nnSs 0 host 172.16.1.121 -w
>>>>>> /root/capture_nfsfail.pcap
>>>>>>
>>>>>> The results are in the same place
>>>>>>
>>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
>>>>>>
>>>>>> capture_nfsfail.pcap has the results from the failed touch
>>>>>> experiment
>>>>>> capture_nfssucceed.pcap has the results from the successful touch
>>>>>> experiment
>>>>>>
>>>>>> The brick log files are there too.
>>>>>
>>>>> Thanks for sharing. Looks like the error is not generated
>>>>> @gluster-server side. The permission denied error was caused by
>>>>> either kNFS or by fuse-mnt process or probably by the combination.
>>>>>
>>>>> To check fuse-mnt logs, please look at
>>>>> /var/log/glusterfs/<fuse_mnt_direcotry>.log
>>>>>
>>>>> For eg.: if you have fuse mounted the gluster volume at /mnt/fuse-mnt
>>>>> and exported it via kNFS, the log location for that fuse_mnt shall be
>>>>> at /var/log/glusterfs/mnt-fuse-mnt.log
>>>>>
>>>>>
>>>>> Also why not switch to either gluster-NFS native server or
>>>>> NFS-Ganesha instead of using kNFS, as they are recommended NFS
>>>>> servers to use with gluster?
>>>>>
>>>>> Thanks,
>>>>> Soumya
>>>>>
>>>>>>
>>>>>> I believe we are using kernel-NFS exporting a fuse mounted gluster
>>>>>> volume. I am having Steve confirm this. I tried to find the
>>>>>> fuse-mnt
>>>>>> logs but failed. Where should I look for them?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Pat
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 07/03/2017 07:58 AM, Soumya Koduri wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 06/30/2017 07:56 PM, Pat Haley wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I was wondering if there were any additional test we could
>>>>>>>> perform to
>>>>>>>> help debug the group write-permissions issue?
>>>>>>>
>>>>>>> Sorry for the delay. Please find response inline --
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Pat
>>>>>>>>
>>>>>>>>
>>>>>>>> On 06/27/2017 12:29 PM, Pat Haley wrote:
>>>>>>>>>
>>>>>>>>> Hi Soumya,
>>>>>>>>>
>>>>>>>>> One example, we have a common working directory dri_fleat in the
>>>>>>>>> gluster volume
>>>>>>>>>
>>>>>>>>> drwxrwsr-x 22 root dri_fleat 4.0K May 1 15:14 dri_fleat
>>>>>>>>>
>>>>>>>>> my user (phaley) does not own that directory but is a member of
>>>>>>>>> the
>>>>>>>>> group dri_fleat and should have write permissions. When I go
>>>>>>>>> to the
>>>>>>>>> nfs-mounted version and try to use the touch command I get the
>>>>>>>>> following
>>>>>>>>>
>>>>>>>>> ibfdr-compute-0-4(dri_fleat)% touch dum
>>>>>>>>> touch: cannot touch `dum': Permission denied
>>>>>>>>>
>>>>>>>>> One of the sub-directories under dri_fleat is "test" which phaley
>>>>>>>>> owns
>>>>>>>>>
>>>>>>>>> drwxrwsr-x 2 phaley dri_fleat 4.0K May 1 15:16 test
>>>>>>>>>
>>>>>>>>> Under this directory (mounted via nfs) user phaley can write
>>>>>>>>>
>>>>>>>>> ibfdr-compute-0-4(test)% touch dum
>>>>>>>>> ibfdr-compute-0-4(test)%
>>>>>>>>>
>>>>>>>>> I have put the packet captures in
>>>>>>>>>
>>>>>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
>>>>>>>>>
>>>>>>>>> capture_nfsfail.pcap has the results from the failed touch
>>>>>>>>> experiment
>>>>>>>>> capture_nfssucceed.pcap has the results from the successful touch
>>>>>>>>> experiment
>>>>>>>>>
>>>>>>>>> The command I used for these was
>>>>>>>>>
>>>>>>>>> tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w
>>>>>>>>> /root/capture_nfstest.pcap
>>>>>>>
>>>>>>> I hope these pkts were captured on the node where NFS server is
>>>>>>> running. Could you please use '-i any' as I do not see glusterfs
>>>>>>> traffic in the tcpdump.
>>>>>>>
>>>>>>> Also looks like NFS v4 is used between client & nfs server. Are you
>>>>>>> using kernel-NFS here (i.e, kernel-NFS exporting fuse mounted
>>>>>>> gluster
>>>>>>> volume)?
>>>>>>> If that is the case please capture fuse-mnt logs as well. This error
>>>>>>> may well be coming from kernel-NFS itself before the request is sent
>>>>>>> to fuse-mnt process.
>>>>>>>
>>>>>>> FWIW, we have below option -
>>>>>>>
>>>>>>> Option: server.manage-gids
>>>>>>> Default Value: off
>>>>>>> Description: Resolve groups on the server-side.
>>>>>>>
>>>>>>> I haven't looked into what this option exactly does. But it may
>>>>>>> worth
>>>>>>> testing with this option on.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Soumya
>>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>> The brick log files are also in the above link. If I read them
>>>>>>>>> correctly they both funny times. Specifically I see entries from
>>>>>>>>> around 2017-06-27 14:02:37.404865 even though the system time was
>>>>>>>>> 2017-06-27 12:00:00.
>>>>>>>>>
>>>>>>>>> One final item, another reply to my post had a link for possible
>>>>>>>>> problems that could arise from users belonging to too many
>>>>>>>>> group. We
>>>>>>>>> have seen the above problem even with a user belonging to only 4
>>>>>>>>> groups.
>>>>>>>>>
>>>>>>>>> Let me know what additional information I can provide.
>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Pat
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 06/27/2017 02:45 AM, Soumya Koduri wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 06/27/2017 10:17 AM, Pranith Kumar Karampuri wrote:
>>>>>>>>>>> The only problem with using gluster mounted via NFS is that it
>>>>>>>>>>> does not
>>>>>>>>>>> respect the group write permissions which we need.
>>>>>>>>>>>
>>>>>>>>>>> We have an exercise coming up in the a couple of weeks. It seems
>>>>>>>>>>> to me
>>>>>>>>>>> that in order to improve our write times before then, it
>>>>>>>>>>> would be
>>>>>>>>>>> good
>>>>>>>>>>> to solve the group write permissions for gluster mounted via
>>>>>>>>>>> NFS now.
>>>>>>>>>>> We can then revisit gluster mounted via FUSE afterwards.
>>>>>>>>>>>
>>>>>>>>>>> What information would you need to help us force gluster
>>>>>>>>>>> mounted via
>>>>>>>>>>> NFS
>>>>>>>>>>> to respect the group write permissions?
>>>>>>>>>>
>>>>>>>>>> Is this owning group or one of the auxiliary groups whose write
>>>>>>>>>> permissions are not considered? AFAIK, there are no special
>>>>>>>>>> permission checks done by gNFS server when compared to gluster
>>>>>>>>>> native
>>>>>>>>>> client.
>>>>>>>>>>
>>>>>>>>>> Could you please provide simple steps to reproduce the issue and
>>>>>>>>>> collect pkt trace and nfs/brick logs as well.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Soumya
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley Email: phaley at mit.edu
> Center for Ocean Engineering Phone: (617) 253-6824
> Dept. of Mechanical Engineering Fax: (617) 253-8125
> MIT, Room 5-213 http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA 02139-4301
>
More information about the Gluster-users
mailing list