<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
Hi Soumya,<br>
<br>
I just noticed some of the notes at the bottom. In particular<br>
<ul>
<li>Till glusterfs-3.7, gluster-NFS (gNFS) gets enabled by
default. The only requirement is that kernel-NFS has to be
disabled for gluster-NFS to come up. Please disable kernel-NFS
server and restart glusterd to start gNFS. In case of any issues
with starting gNFS server, please look at
/var/log/glusterfs/nfs.log.
</li>
</ul>
If we disable the kernel-NFS on our server and restart glusterd to
start gNFS will that affect the NFS file system also being served by
that server (i.e. the single server serves both a glusterFS area and
an NFS area)? Would we also have to disable the kernel-NFS for
NFS-ganesha?<br>
<br>
My second question concerns NFS-ganesha (v 2.3.x) for CentOS 6.8 and
gluster 3.7.11. I think I see a couple of possibilities<br>
<ol>
<li>I see one possible rpm for version 2.3.3 in
<a class="moz-txt-link-freetext" href="https://mirror.chpc.utah.edu/pub/vault.centos.org/centos/6.8/storage/Source/gluster-3.8/">https://mirror.chpc.utah.edu/pub/vault.centos.org/centos/6.8/storage/Source/gluster-3.8/</a>
The other rpm's seem to be for gluster 3.8 packages, so I'm
wondering if there is a concern for conflict<br>
</li>
<li>In one of the links you sent
(<a class="moz-txt-link-freetext" href="https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/">https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/</a>)
I see an rpm for glusterfs-ganesha-3.7.11 . Is this a specific
gluster package for compatibility with ganesha or a ganesha
package for gluster?<br>
</li>
</ol>
Does either possibility seem more likely to be what I need than the
other?<br>
<br>
Pat<br>
<br>
<br>
<div class="moz-cite-prefix">On 07/07/2017 01:31 PM, Soumya Koduri
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:4a73b085-7879-c4a8-944e-a4c8b396e792@redhat.com">Hi,
<br>
<br>
On 07/07/2017 06:16 AM, Pat Haley wrote:
<br>
<blockquote type="cite">
<br>
Hi All,
<br>
<br>
A follow-up question. I've been looking at various pages on
nfs-ganesha
<br>
& gluster. Is there a version of nfs-ganesha that is
recommended for
<br>
use with
<br>
<br>
glusterfs 3.7.11 built on Apr 27 2016 14:09:22
<br>
CentOS release 6.8 (Final)
<br>
</blockquote>
<br>
For glusterfs 3.7, nfs-ganesha-2.3-* version can be used.
<br>
<br>
I see the packages built in centos7 storage sig [1] but not for
centos6. Request Niels to comment.
<br>
<br>
<br>
<blockquote type="cite">
<br>
Thanks
<br>
<br>
Pat
<br>
<br>
<br>
On 07/05/2017 11:36 AM, Pat Haley wrote:
<br>
<blockquote type="cite">
<br>
Hi Soumya,
<br>
<br>
(1) In
<a class="moz-txt-link-freetext" href="http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/">http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/</a>
<br>
I've placed the following 2 log files
<br>
<br>
etc-glusterfs-glusterd.vol.log
<br>
gdata.log
<br>
<br>
The first has repeated messages about nfs disconnects. The
second had
<br>
the <fuse_mnt_direcotry>.log name (but not much
information).
<br>
<br>
</blockquote>
</blockquote>
<br>
Hmm yeah..weird ..there are not much logs in fuse mnt log file.
<br>
<br>
<blockquote type="cite">
<blockquote type="cite">(2) About the gluster-NFS native
server: do you know where we can
<br>
find documentation on how to use/install it? We haven't had
success
<br>
in our searches.
<br>
<br>
</blockquote>
</blockquote>
<br>
Till glusterfs-3.7, gluster-NFS (gNFS) gets enabled by default.
The only requirement is that kernel-NFS has to be disabled for
gluster-NFS to come up. Please disable kernel-NFS server and
restart glusterd to start gNFS. In case of any issues with
starting gNFS server, please look at /var/log/glusterfs/nfs.log.
<br>
<br>
Thanks,
<br>
Soumya
<br>
<br>
<br>
[1]
<a class="moz-txt-link-freetext" href="https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.7/">https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.7/</a>
<br>
[2]
<a class="moz-txt-link-freetext" href="https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/">https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/</a>
<br>
<br>
<blockquote type="cite">
<blockquote type="cite">Thanks
<br>
<br>
Pat
<br>
<br>
<br>
On 07/04/2017 05:01 AM, Soumya Koduri wrote:
<br>
<blockquote type="cite">
<br>
<br>
On 07/03/2017 09:01 PM, Pat Haley wrote:
<br>
<blockquote type="cite">
<br>
Hi Soumya,
<br>
<br>
When I originally did the tests I ran tcpdump on the
client.
<br>
<br>
I have rerun the tests, doing tcpdump on the server
<br>
<br>
tcpdump -i any -nnSs 0 host 172.16.1.121 -w
/root/capture_nfsfail.pcap
<br>
<br>
The results are in the same place
<br>
<br>
<a class="moz-txt-link-freetext" href="http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/">http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/</a>
<br>
<br>
capture_nfsfail.pcap has the results from the failed
touch experiment
<br>
capture_nfssucceed.pcap has the results from the
successful touch
<br>
experiment
<br>
<br>
The brick log files are there too.
<br>
</blockquote>
<br>
Thanks for sharing. Looks like the error is not generated
<br>
@gluster-server side. The permission denied error was caused
by
<br>
either kNFS or by fuse-mnt process or probably by the
combination.
<br>
<br>
To check fuse-mnt logs, please look at
<br>
/var/log/glusterfs/<fuse_mnt_direcotry>.log
<br>
<br>
For eg.: if you have fuse mounted the gluster volume at
/mnt/fuse-mnt
<br>
and exported it via kNFS, the log location for that fuse_mnt
shall be
<br>
at /var/log/glusterfs/mnt-fuse-mnt.log
<br>
<br>
<br>
Also why not switch to either gluster-NFS native server or
<br>
NFS-Ganesha instead of using kNFS, as they are recommended
NFS
<br>
servers to use with gluster?
<br>
<br>
Thanks,
<br>
Soumya
<br>
<br>
<blockquote type="cite">
<br>
I believe we are using kernel-NFS exporting a fuse mounted
gluster
<br>
volume. I am having Steve confirm this. I tried to find
the fuse-mnt
<br>
logs but failed. Where should I look for them?
<br>
<br>
Thanks
<br>
<br>
Pat
<br>
<br>
<br>
<br>
On 07/03/2017 07:58 AM, Soumya Koduri wrote:
<br>
<blockquote type="cite">
<br>
<br>
On 06/30/2017 07:56 PM, Pat Haley wrote:
<br>
<blockquote type="cite">
<br>
Hi,
<br>
<br>
I was wondering if there were any additional test we
could perform to
<br>
help debug the group write-permissions issue?
<br>
</blockquote>
<br>
Sorry for the delay. Please find response inline --
<br>
<br>
<blockquote type="cite">
<br>
Thanks
<br>
<br>
Pat
<br>
<br>
<br>
On 06/27/2017 12:29 PM, Pat Haley wrote:
<br>
<blockquote type="cite">
<br>
Hi Soumya,
<br>
<br>
One example, we have a common working directory
dri_fleat in the
<br>
gluster volume
<br>
<br>
drwxrwsr-x 22 root dri_fleat 4.0K May 1 15:14
dri_fleat
<br>
<br>
my user (phaley) does not own that directory but is
a member of the
<br>
group dri_fleat and should have write permissions.
When I go to the
<br>
nfs-mounted version and try to use the touch command
I get the
<br>
following
<br>
<br>
ibfdr-compute-0-4(dri_fleat)% touch dum
<br>
touch: cannot touch `dum': Permission denied
<br>
<br>
One of the sub-directories under dri_fleat is "test"
which phaley
<br>
owns
<br>
<br>
drwxrwsr-x 2 phaley dri_fleat 4.0K May 1 15:16
test
<br>
<br>
Under this directory (mounted via nfs) user phaley
can write
<br>
<br>
ibfdr-compute-0-4(test)% touch dum
<br>
ibfdr-compute-0-4(test)%
<br>
<br>
I have put the packet captures in
<br>
<br>
<a class="moz-txt-link-freetext" href="http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/">http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/</a>
<br>
<br>
capture_nfsfail.pcap has the results from the
failed touch
<br>
experiment
<br>
capture_nfssucceed.pcap has the results from the
successful touch
<br>
experiment
<br>
<br>
The command I used for these was
<br>
<br>
tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w
<br>
/root/capture_nfstest.pcap
<br>
</blockquote>
</blockquote>
<br>
I hope these pkts were captured on the node where NFS
server is
<br>
running. Could you please use '-i any' as I do not see
glusterfs
<br>
traffic in the tcpdump.
<br>
<br>
Also looks like NFS v4 is used between client & nfs
server. Are you
<br>
using kernel-NFS here (i.e, kernel-NFS exporting fuse
mounted gluster
<br>
volume)?
<br>
If that is the case please capture fuse-mnt logs as
well. This error
<br>
may well be coming from kernel-NFS itself before the
request is sent
<br>
to fuse-mnt process.
<br>
<br>
FWIW, we have below option -
<br>
<br>
Option: server.manage-gids
<br>
Default Value: off
<br>
Description: Resolve groups on the server-side.
<br>
<br>
I haven't looked into what this option exactly does. But
it may worth
<br>
testing with this option on.
<br>
<br>
Thanks,
<br>
Soumya
<br>
<br>
<br>
<blockquote type="cite">
<blockquote type="cite">
<br>
The brick log files are also in the above link. If
I read them
<br>
correctly they both funny times. Specifically I see
entries from
<br>
around 2017-06-27 14:02:37.404865 even though the
system time was
<br>
2017-06-27 12:00:00.
<br>
<br>
One final item, another reply to my post had a link
for possible
<br>
problems that could arise from users belonging to
too many group. We
<br>
have seen the above problem even with a user
belonging to only 4
<br>
groups.
<br>
<br>
Let me know what additional information I can
provide.
<br>
</blockquote>
</blockquote>
<br>
<blockquote type="cite">
<blockquote type="cite">
<br>
Thanks
<br>
<br>
Pat
<br>
<br>
<br>
On 06/27/2017 02:45 AM, Soumya Koduri wrote:
<br>
<blockquote type="cite">
<br>
<br>
On 06/27/2017 10:17 AM, Pranith Kumar Karampuri
wrote:
<br>
<blockquote type="cite">The only problem with
using gluster mounted via NFS is that it
<br>
does not
<br>
respect the group write permissions which we
need.
<br>
<br>
We have an exercise coming up in the a couple of
weeks. It seems
<br>
to me
<br>
that in order to improve our write times before
then, it would be
<br>
good
<br>
to solve the group write permissions for gluster
mounted via
<br>
NFS now.
<br>
We can then revisit gluster mounted via FUSE
afterwards.
<br>
<br>
What information would you need to help us force
gluster
<br>
mounted via
<br>
NFS
<br>
to respect the group write permissions?
<br>
</blockquote>
<br>
Is this owning group or one of the auxiliary
groups whose write
<br>
permissions are not considered? AFAIK, there are
no special
<br>
permission checks done by gNFS server when
compared to gluster
<br>
native
<br>
client.
<br>
<br>
Could you please provide simple steps to reproduce
the issue and
<br>
collect pkt trace and nfs/brick logs as well.
<br>
<br>
Thanks,
<br>
Soumya
<br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
</blockquote>
<br>
</blockquote>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</body>
</html>