<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p><br>
</p>
<p>Hi Pranith,</p>
<p>In attached file capture.pcap<br>
</p>
<br>
<div class="moz-cite-prefix">On 09/01/2016 01:01 PM, Pranith Kumar
Karampuri wrote:<br>
</div>
<blockquote
cite="mid:CAOgeEnbHo1f7AoKwBVEVLdbT4SaVqz0zDwri4fTDobmnVhxY1w@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr">You need to capture the file so that we can see the
tcpdump in wireshark to inspect the uid/gid etc that are going
out the wire.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Sep 1, 2016 at 10:04 PM, Pat
Haley <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p><br>
</p>
<p>Hi Pranith,</p>
<p>Here is the output when I'm trying a touch command that
fails with "Permission denied"<br>
</p>
<p><tt>[root@compute-11-10 ~]# tcpdump -nnSs 0 host
10.1.1.4</tt><tt><br>
</tt><tt>tcpdump: verbose output suppressed, use -v or
-vv for full protocol decode</tt><tt><br>
</tt><tt>listening on eth1, link-type EN10MB (Ethernet),
capture size 65535 bytes</tt><tt><br>
</tt><tt>12:30:46.248293 IP 10.255.255.124.4215828946
> 10.1.1.4.2049: 208 getattr fh 0,0/22</tt><tt><br>
</tt><tt>12:30:46.252509 IP 10.1.1.4.2049 >
10.255.255.124.4215828946: reply ok 240 getattr NON 3
ids 0/3 sz 0</tt><tt><br>
</tt><tt>12:30:46.252596 IP 10.255.255.124.<a
moz-do-not-send="true" href="tel:4232606162"
value="+14232606162" target="_blank">4232606162</a>
> 10.1.1.4.2049: 300 getattr fh 0,0/22</tt><tt><br>
</tt><tt>12:30:46.253308 IP 10.1.1.4.2049 >
10.255.255.124.4232606162: reply ok 52 getattr ERROR:
Permission denied</tt><tt><br>
</tt><tt>12:30:46.253358 IP 10.255.255.124.<a
moz-do-not-send="true" href="tel:4249383378"
value="+14249383378" target="_blank">4249383378</a>
> 10.1.1.4.2049: 216 getattr fh 0,0/22</tt><tt><br>
</tt><tt>12:30:46.260347 IP 10.1.1.4.2049 >
10.255.255.124.4249383378: reply ok 52 getattr ERROR:
No such file or directory</tt><tt><br>
</tt><tt>12:30:46.300306 IP 10.255.255.124.931 >
10.1.1.4.2049: Flags [.], ack 1979284005, win 501,
options [nop,nop,TS val 490628016 ecr 75449144],
length 0</tt><tt><br>
</tt><tt>^C</tt><tt><br>
</tt><tt>7 packets captured</tt><tt><br>
</tt><tt>7 packets received by filter</tt><tt><br>
</tt><tt>0 packets dropped by kernel</tt><tt><br>
</tt><br>
</p>
<div>
<div class="h5"> <br>
<div>On 09/01/2016 03:31 AM, Pranith Kumar Karampuri
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">hi Pat,<br>
I think the other thing we should probably
look for would be to see the tcp dump of what
uid/gid parameters are sent over network when this
command is executed.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Sep 1, 2016 at
7:14 AM, Pat Haley <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--<span><br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>hi Pat,<br>
</div>
Are you seeing this issue only
after migration or even before? May
be we should look at the gid numbers
on the disk and the ones that are
coming from client for the given
user to see if they match or not?<br>
</div>
</div>
</blockquote>
</span> ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------<br>
This issue was not being seen before the
migration. We have copied the /etc/passwd
and /etc/group files from the front-end
machine (the client) to the data server, so
they all match<br>
------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------<span><br>
<blockquote type="cite">
<div dir="ltr">Could you give stat
output of the directory in question
from both the brick and the nfs client<br>
<div><br>
</div>
</div>
</blockquote>
</span> ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------<br>
From the server for gluster:<br>
<tt>[root@mseas-data2 ~]# stat
/gdata/projects/nsf_alpha</tt><tt><br>
</tt><tt> File: `/gdata/projects/nsf_alpha'</tt><tt><br>
</tt><tt> Size: 4096 Blocks:
8 IO Block: 131072 directory</tt><tt><br>
</tt><tt>Device: 13h/19d Inode:
13094773206281819436 Links: 13</tt><tt><br>
</tt><tt>Access: (2775/drwxrwsr-x) Uid:
( 0/ root) Gid: ( 598/nsf_alpha)</tt><tt><br>
</tt><tt>Access: 2016-08-31
19:08:59.735990904 -0400</tt><tt><br>
</tt><tt>Modify: 2016-08-31
16:37:09.048997167 -0400</tt><tt><br>
</tt><tt>Change: 2016-08-31
16:37:41.315997148 -0400</tt><tt><br>
</tt><tt><br>
From the server for first underlying brick<br>
[root@mseas-data2 ~]# stat
/mnt/brick1/projects/nsf_alpha<wbr>/<br>
File: `/mnt/brick1/projects/nsf_alph<wbr>a/'<br>
Size: 4096 Blocks: 8
IO Block: 4096 directory<br>
Device: 800h/2048d Inode: 185630
Links: 13<br>
Access: (2775/drwxrwsr-x) Uid: ( 0/
root) Gid: ( 598/nsf_alpha)<br>
Access: 2016-08-31 19:08:59.669990907
-0400<br>
Modify: 2016-08-31 16:37:09.048997167
-0400<br>
Change: 2016-08-31 16:37:41.315997148
-0400<br>
</tt><br>
<tt>From the server for second underlying
brick<br>
[root@mseas-data2 ~]# stat
/mnt/brick2/projects/nsf_alpha<wbr>/<br>
File: `/mnt/brick2/projects/nsf_alph<wbr>a/'<br>
Size: 4096 Blocks: 8
IO Block: 4096 directory<br>
Device: 810h/2064d Inode: 24085468
Links: 13<br>
Access: (2775/drwxrwsr-x) Uid: ( 0/
root) Gid: ( 598/nsf_alpha)<br>
Access: 2016-08-31 19:08:59.735990904
-0400<br>
Modify: 2016-08-03 14:01:52.000000000
-0400<br>
Change: 2016-08-31 16:37:41.315997148
-0400<br>
<br>
From the client<br>
[root@mseas FixOwn]# stat
/gdata/projects/nsf_alpha<br>
File: `/gdata/projects/nsf_alpha'<br>
Size: 4096 Blocks: 8
IO Block: 1048576 directory<br>
Device: 23h/35d Inode:
13094773206281819436 Links: 13<br>
Access: (2775/drwxrwsr-x) Uid: ( 0/
root) Gid: ( 598/nsf_alpha)<br>
Access: 2016-08-31 19:08:59.735990904
-0400<br>
Modify: 2016-08-31 16:37:09.048997167
-0400<br>
Change: 2016-08-31 16:37:41.315997148
-0400<br>
<br>
</tt><tt> </tt>
<blockquote type="cite"> </blockquote>
------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------<span><br>
<blockquote type="cite"> </blockquote>
<blockquote type="cite">
<div dir="ltr">
<div>Could you also let us know
version of gluster you are using</div>
</div>
</blockquote>
</span>
<blockquote type="cite">
<div dir="ltr">
<div>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------<br>
</div>
</div>
</blockquote>
<br>
<blockquote type="cite">
<div dir="ltr">
<div> <tt>[root@mseas-data2 ~]# gluster
--version</tt><tt><br>
</tt><tt>glusterfs 3.7.11 built on Apr
27 2016 14:09:22</tt><tt><br>
</tt><br>
</div>
</div>
</blockquote>
<tt>[root@mseas-data2 ~]# gluster volume
info</tt><tt><br>
</tt><tt> </tt><tt><br>
</tt><tt>Volume Name: data-volume</tt><tt><br>
</tt><tt>Type: Distribute</tt><tt><br>
</tt><tt>Volume ID:
c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18</tt><tt><br>
</tt><tt>Status: Started</tt><tt><br>
</tt><tt>Number of Bricks: 2</tt><tt><br>
</tt><tt>Transport-type: tcp</tt><tt><br>
</tt><tt>Bricks:</tt><tt><br>
</tt><tt>Brick1: mseas-data2:/mnt/brick1</tt><tt><br>
</tt><tt>Brick2: mseas-data2:/mnt/brick2</tt><tt><br>
</tt><tt>Options Reconfigured:</tt><tt><br>
</tt><tt>performance.readdir-ahead: on</tt><tt><br>
</tt><tt>nfs.disable: on</tt><tt><br>
</tt><tt>nfs.export-volumes: off</tt><tt><br>
</tt><br>
<tt>[root@mseas-data2 ~]# gluster volume
status </tt><tt><br>
</tt><tt>Status of volume: data-volume</tt><tt><br>
</tt><tt>Gluster
process <wbr>
TCP Port RDMA Port Online Pid</tt><tt><br>
</tt><tt>------------------------------<wbr>------------------------------<wbr>------------------</tt><tt><br>
</tt><tt>Brick
mseas-data2:/mnt/brick1 <wbr>
49154 0 Y 5005 </tt><tt><br>
</tt><tt>Brick
mseas-data2:/mnt/brick2 <wbr>
49155 0 Y 5010 </tt><tt><br>
</tt><tt> </tt><tt><br>
</tt><tt>Task Status of Volume data-volume</tt><tt><br>
</tt><tt>------------------------------<wbr>------------------------------<wbr>------------------</tt><tt><br>
</tt><tt>Task :
Rebalance </tt><tt><br>
</tt><tt>ID :
892d9e3a-b38c-4971-b96a-8e4a49<wbr>6685ba</tt><tt><br>
</tt><tt>Status : completed
<br>
<br>
</tt><tt><br>
</tt><tt>[root@mseas-data2 ~]# gluster peer
status</tt><tt><br>
</tt><tt>Number of Peers: 0</tt><tt><br>
</tt><br>
<blockquote type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------
<div>
<div><br>
On Thu, Sep 1, 2016 at 2:46 AM,
Pat Haley <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:phaley@mit.edu"
target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000">
<p><br>
</p>
<p>Hi,</p>
<p>Another piece of data.
There are 2 distinct volumes
on the file server</p>
<ol>
<li>a straight nfs partition</li>
<li>a gluster volume (served
over nfs)<br>
</li>
</ol>
The straight nfs partition
does respect the group write
permissions, while the gluster
volume does not. Any
suggestions on how to debug
this or what additional
information would be helpful
would be greatly appreciated<br>
<br>
Thanks<span><br>
<br>
<div>On 08/30/2016 06:01 PM,
Pat Haley wrote:<br>
</div>
<blockquote type="cite"> <br>
Hi <br>
<br>
We have just migrated our
data to a new file server
(more space, old server
was showing its age). We
have a volume for
collaborative use, based
on group membership. In
our new server, the group
write permissions are not
being respected (e.g. the
owner of a directory can
still write to that
directory but any other
member of the associated
group cannot, even though
the directory clearly has
group write permissions
set). This is occurring
regardless of how many
groups the user is a
member of (i.e. users that
are members of fewer then
16 groups are still
affected). <br>
<br>
the relevant fstab line
from the server looks like
<br>
localhost:/data-volume
/gdata glusterfs
defaults 0 0 <br>
<br>
and for a client: <br>
mseas-data2:/gdata
/gdata nfs
defaults 0 0 <br>
<br>
Any help would be greatly
appreciated. <br>
<br>
Thanks <br>
<br>
</blockquote>
<br>
<pre cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: <a moz-do-not-send="true" href="tel:%28617%29%20253-6824" value="+16172536824" target="_blank">(617) 253-6824</a>
Dept. of Mechanical Engineering Fax: <a moz-do-not-send="true" href="tel:%28617%29%20253-8125" value="+16172538125" target="_blank">(617) 253-8125</a>
MIT, Room 5-213 <a moz-do-not-send="true" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</span></div>
______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a>
</blockquote></div></div></div><span><font color="#888888">
--
<div data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</font></span></div>
</blockquote><div><div>
<pre cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: <a moz-do-not-send="true" href="tel:%28617%29%20253-6824" value="+16172536824" target="_blank">(617) 253-6824</a>
Dept. of Mechanical Engineering Fax: <a moz-do-not-send="true" href="tel:%28617%29%20253-8125" value="+16172538125" target="_blank">(617) 253-8125</a>
MIT, Room 5-213 <a moz-do-not-send="true" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></div></div></div>
______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a>
</blockquote></div>
--
<div data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>
</blockquote>
<pre cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: <a moz-do-not-send="true" href="tel:%28617%29%20253-6824" value="+16172536824" target="_blank">(617) 253-6824</a>
Dept. of Mechanical Engineering Fax: <a moz-do-not-send="true" href="tel:%28617%29%20253-8125" value="+16172538125" target="_blank">(617) 253-8125</a>
MIT, Room 5-213 <a moz-do-not-send="true" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></div></div></div>
______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a>
</blockquote></div>
--
<div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></body></html>