[Gluster-users] nfs acces denied
Carlos Capriotti
capriotti.carlos at gmail.com
Thu Apr 3 09:27:50 UTC 2014
Try mounting like this:
mount -t nfs -o
rw,async,vers=3,rsize=65536,wsize=65536,noatime,nodiratime
localhost:/caviar_data11
/media
Try to test with fresh files, and not necessarily already existing ones.
Then the existing ones.
Let's see what it tells us.
On Thu, Apr 3, 2014 at 11:05 AM, VAN CAUSBROECK Wannes <
Wannes.VANCAUSBROECK at onprvp.fgov.be> wrote:
> Hello Carlos,
>
>
>
> I created a new disk formatted xfs with an inode size of 512.
>
> I created a new gluster volume and migrated the data. Again, when I mount
> it as a gluster volume, everything works fine.
>
> On nfs (mounted on the same server that's running the gluster volume) I
> get the following weirdness:
>
>
>
> [root at lpr-nas01 /]# mount -t nfs -o mountproto=tcp,vers=3
> lpr-nas01:/caviar_data11 /media
>
> [root at lpr-nas01 /]# ll /media/*/
>
>
>
> ....
>
> /media/2012/201206:
>
> ls: /media/2012/201206/20120621: No such file or directory
>
> total 0
>
> drwxrwsr-x 3 960 1003 15 Jun 15 2011 20120621
>
> ....
>
> drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 38
>
> drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 39
>
> drwxrwsr-x 2 nfsnobody 1003 138 Jun 6 2011 40
>
> drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 42
>
> drwxrwsr-x 2 nfsnobody 1003 171 Jun 6 2011 43
>
> drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 45
>
> drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 46
>
> drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 47
>
> drwxrwsr-x 2 nfsnobody 1003 369 Jun 6 2011 48
>
> ...
>
> ls: cannot access /media/2011/201106/20110606/81: Invalid argument
>
> ls: cannot access /media/2011/201106/20110606/55: Invalid argument
>
> ls: cannot access /media/2011/201106/20110606/30: Invalid argument
>
> ls: cannot access /media/2011/201106/20110606/90: Invalid argument
>
> total 12
>
> d????????? ? ? ? ? ? 00
>
> d????????? ? ? ? ? ? 01
>
> d????????? ? ? ? ? ? 02
>
> d????????? ? ? ? ? ? 03
>
> d????????? ? ? ? ? ? 04
>
> d????????? ? ? ? ? ? 05
>
> d????????? ? ? ? ? ? 06
>
> d????????? ? ? ? ? ? 07
>
> d????????? ? ? ? ? ? 08
>
> d????????? ? ? ? ? ? 09
>
> d????????? ? ? ? ? ? 10
>
> ....
>
>
>
>
>
> In the nfs log, I get the following errors:
>
> ...
>
> [2014-04-03 08:50:13.624517] E
> [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup
> failed: <gfid:a1acf77c-2b81-4b5f-a113-521c6ab8fd23>: Invalid argument
>
> [2014-04-03 08:50:13.624547] E [acl3.c:334:acl3_getacl_resume] 0-nfs-ACL:
> Unable to resolve FH: (192.168.151.21:954) caviar_data11 :
> a1acf77c-2b81-4b5f-a113-521c6ab8fd23
>
> [2014-04-03 08:50:13.624562] E [acl3.c:342:acl3_getacl_resume] 0-nfs-ACL:
> unable to open_and_resume
>
> [2014-04-03 08:50:13.624914] E
> [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup
> failed: <gfid:980a29f7-29f4-4c88-9896-2b1a549370e2>: Invalid argument
>
> [2014-04-03 08:50:13.624960] E [acl3.c:334:acl3_getacl_resume] 0-nfs-ACL:
> Unable to resolve FH: (192.168.151.21:954) caviar_data11 :
> 980a29f7-29f4-4c88-9896-2b1a549370e2
>
> [2014-04-03 08:50:13.624970] E [acl3.c:342:acl3_getacl_resume] 0-nfs-ACL:
> unable to open_and_resume
>
> [2014-04-03 08:50:13.625290] E
> [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup
> failed: <gfid:b67993d2-c647-4493-aa78-64033614dc33>: Invalid argument
>
> [2014-04-03 08:50:13.625322] E [nfs3.c:755:nfs3_getattr_resume]
> 0-nfs-nfsv3: Unable to resolve FH: (192.168.151.21:954) caviar_data11 :
> b67993d2-c647-4493-aa78-64033614dc33
>
> [2014-04-03 08:50:13.625335] W [nfs3-helpers.c:3380:nfs3_log_common_res]
> 0-nfs-nfsv3: XID: c838d715, GETATTR: NFS: 22(Invalid argument for
> operation), POSIX: 14(Bad address)
>
> [2014-04-03 08:50:13.626268] I [dht-layout.c:638:dht_layout_normalize]
> 0-caviar_data11-dht: found anomalies in /2011/201108/20110802. holes=1
> overlaps=0
>
> [2014-04-03 08:50:13.627701] I [dht-layout.c:638:dht_layout_normalize]
> 0-caviar_data11-dht: found anomalies in /2011/201108/20110803. holes=1
> overlaps=0
>
> [2014-04-03 08:50:13.628839] E
> [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup
> failed: <gfid:bb587d95-ffa3-42f3-9b27-b1cf0c5c05cb>: Invalid argument
>
> ....
>
> [2014-04-03 08:50:13.706106] W [nfs3-helpers.c:3380:nfs3_log_common_res]
> 0-nfs-nfsv3: XID: 6639d715, GETATTR: NFS: 22(Invalid argument for
> operation), POSIX: 14(Bad address)
>
> [2014-04-03 08:50:13.706585] E
> [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup
> failed: <gfid:897bf28d-41a3-4cc2-a120-8374656be858>: Invalid argument
>
> [2014-04-03 08:50:13.706626] E [nfs3.c:755:nfs3_getattr_resume]
> 0-nfs-nfsv3: Unable to resolve FH: (192.168.151.21:954) caviar_data11 :
> 897bf28d-41a3-4cc2-a120-8374656be858
>
> [2014-04-03 08:50:13.706647] W [nfs3-helpers.c:3380:nfs3_log_common_res]
> 0-nfs-nfsv3: XID: 6739d715, GETATTR: NFS: 22(Invalid argument for
> operation), POSIX: 14(Bad address)
>
> [2014-04-03 08:50:13.707113] E
> [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup
> failed: <gfid:ded9c3fe-171b-4104-bc0e-e64c0d3e18e2>: Invalid argument
>
> [2014-04-03 08:50:13.707158] E [nfs3.c:755:nfs3_getattr_resume]
> 0-nfs-nfsv3: Unable to resolve FH: (192.168.151.21:954) caviar_data11 :
> ded9c3fe-171b-4104-bc0e-e64c0d3e18e2
>
> [2014-04-03 08:50:13.707174] W [nfs3-helpers.c:3380:nfs3_log_common_res]
> 0-nfs-nfsv3: XID: 6839d715, GETATTR: NFS: 22(Invalid argument for
> operation), POSIX: 14(Bad address)
>
> [2014-04-03 08:50:13.707597] E
> [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup
> failed: <gfid:c5ebeb07-ee2e-4161-9e65-65d65f614628>: Invalid argument
>
> [2014-04-03 08:50:13.707644] E [nfs3.c:755:nfs3_getattr_resume]
> 0-nfs-nfsv3: Unable to resolve FH: (192.168.151.21:954) caviar_data11 :
> c5ebeb07-ee2e-4161-9e65-65d65f614628
>
> [2014-04-03 08:50:13.707707] W [nfs3-helpers.c:3380:nfs3_log_common_res]
> 0-nfs-nfsv3: XID: 6939d715, GETATTR: NFS: 22(Invalid argument for
> operation), POSIX: 14(Bad address)
>
> [2014-04-03 08:50:13.708357] E
> [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup
> failed: <gfid:da363aa4-6d43-47c0-90fe-edead3689064>: Invalid argument
>
> [2014-04-03 08:50:13.708407] E [nfs3.c:755:nfs3_getattr_resume]
> 0-nfs-nfsv3: Unable to resolve FH: (192.168.151.21:954) caviar_data11 :
> da363aa4-6d43-47c0-90fe-edead3689064
>
> [2014-04-03 08:50:13.708432] W [nfs3-helpers.c:3380:nfs3_log_common_res]
> 0-nfs-nfsv3: XID: 6a39d715, GETATTR: NFS: 22(Invalid argument for
> operation), POSIX: 14(Bad address)
>
> [2014-04-03 08:50:13.708796] E [dht-helper.c:429:dht_subvol_get_hashed]
> (-->/usr/lib64/glusterfs/3.4.2/xlator/debug/io-stats.so(io_stats_lookup+0x157)
> [0x7f4465bf52e7] (-->/usr/lib64/libglusterfs.so.0(default_lookup+0x6d)
> [0x3dfe01c03d]
> (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup+0xa7e)
> [0x7f4466037f2e]))) 0-caviar_data11-dht: invalid argument: loc->parent
>
> [2014-04-03 08:50:13.709066] W
> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-caviar_data11-client-0:
> remote operation failed: Invalid argument. Path:
> <gfid:00000000-0000-0000-0000-000000000000>
> (00000000-0000-0000-0000-000000000000)
>
> [2014-04-03 08:50:13.709128] E [acl3.c:334:acl3_getacl_resume] 0-nfs-ACL:
> Unable to resolve FH: (192.168.151.21:954) caviar_data11 :
> 00000000-0000-0000-0000-000000000000
>
> [2014-04-03 08:50:13.709159] E [acl3.c:342:acl3_getacl_resume] 0-nfs-ACL:
> unable to open_and_resume
>
> [2014-04-03 08:50:13.709741] E
> [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup
> failed: <gfid:db0eec80-e122-4751-9695-d903a6e6f29e>: Invalid argument
>
> ....
>
>
>
>
>
> So..... any ideas?
>
>
>
>
>
> *From:* Carlos Capriotti [mailto:capriotti.carlos at gmail.com]
> *Sent:* maandag 31 maart 2014 18:03
> *To:* VAN CAUSBROECK Wannes
> *Cc:* gluster-users at gluster.org
>
> *Subject:* Re: [Gluster-users] nfs acces denied
>
>
>
> maybe it would be nice to see your volume info for affected volumes.
>
>
>
> Also, on the server side, what happens if you mount the share using
> glusterfs instead of nfs ?
>
>
>
> any change the native nfs server is running on your server ?
>
>
>
> Are there any auto-heal processes running ?
>
>
>
> There are a few name resolution messages on your logs, that seem to refer
> to the nodes themselves. Any DNS conflicts ? Maybe add the names of servers
> to the hosts file ?
>
>
>
> You MS client seems to be having issues with user/group translation. It
> seems to create files with gid 1003. (I could be wrong).
>
>
>
> Again, is SElinux/ACLs/iptables disabled ?
>
>
>
> All is very inconclusive os far.
>
>
>
> On Mon, Mar 31, 2014 at 5:26 PM, VAN CAUSBROECK Wannes <
> Wannes.VANCAUSBROECK at onprvp.fgov.be> wrote:
>
> Well, with 'client' i do actually mean the server itself.
>
> i've tried forcing linux and windows to nfs V3 and tcp, and on windows i
> played around with the uid and gid, but the result is always the same
>
>
>
>
> On 31 Mar 2014, at 17:22, "Carlos Capriotti" <capriotti.carlos at gmail.com>
> wrote:
>
> Well, saying your client-side is "linux" does not help much. Distro,
> flavor, etc helps a lot, but I'll take a wild guess here.
>
>
>
> First, force your NFS mount (client) to use nfs version 3.
>
>
>
> The same for Microsoft. (It is fair to say I have no idea if the MS client
> supports v4 or not).
>
>
>
> Additionally, check that firewalls are disabled on both sides, just for
> testing. The same goes for SElinux.
>
>
>
> Windows and ACL, and user mapping is something that might be in your way
> too. There is a Technet document that describes how to handle this mapping
> if I am not wrong.
>
> Just for testing, mount your nfs share you your own server, using
> localhost:/nfs_share and see how it goes.
>
>
>
> It is a good start.
>
>
>
> Kr,
>
>
>
> Carlos
>
>
>
> On Mon, Mar 31, 2014 at 3:58 PM, VAN CAUSBROECK Wannes <
> Wannes.VANCAUSBROECK at onprvp.fgov.be> wrote:
>
> Hello all,
>
>
>
> I've already tried to post this, but i'm unsure it arrived to the mailing
> list.
>
>
>
> I have some issues regarding my nfs mounts. My setup is as follows:
>
> Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached to a
> san. I have one disk on which are all the bricks (formatted ext4 in 64 bit
> mode) of 25TB.
>
> On the gluster side of things, everything works without issues. The
> trouble starts when I mount a volume as an nfs mount.
>
> Lots of volumes work without issues, but others behave strangely. The
> volumes that act weird generally contain many files (can be accidental?).
>
> The volumes in question mount without issues, but when I try to go into
> any subdirectory sometimes it works, sometimes I get errors.
>
>
>
> On windows with nfs client: access denied
>
>
>
> In nfslog:
>
> [2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize]
> 0-caviar_data11-dht: found anomalies in
> <gfid:c8d94120-6851-46ea-9f28-c629a44b1015>. holes=1 overlaps=0
>
> [2014-03-31 13:57:58.771348] E
> [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup
> failed: <gfid:c8d94120-6851-46ea-9f28-c629a44b1015>: Invalid argument
>
> [2014-03-31 13:57:58.771380] E [nfs3.c:1380:nfs3_lookup_resume]
> 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 :
> c8d94120-6851-46ea-9f28-c629a44b1015
>
> [2014-03-31 13:57:58.771819] W [nfs3-helpers.c:3380:nfs3_log_common_res]
> 0-nfs-nfsv3: XID: 1ec28530, LOOKUP: NFS: 22(Invalid argument for
> operation), POSIX: 14(Bad address)
>
> [2014-03-31 13:57:58.798967] I [dht-layout.c:638:dht_layout_normalize]
> 0-caviar_data11-dht: found anomalies in
> <gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b>. holes=1 overlaps=0
>
> [2014-03-31 13:57:58.799039] E
> [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup
> failed: <gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b>: Invalid argument
>
> [2014-03-31 13:57:58.799056] E [nfs3.c:1380:nfs3_lookup_resume]
> 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 :
> 14972193-1039-4d7a-aed5-0d7e7eccf57b
>
> [2014-03-31 13:57:58.799088] W [nfs3-helpers.c:3380:nfs3_log_common_res]
> 0-nfs-nfsv3: XID: 1ec28531, LOOKUP: NFS: 22(Invalid argument for
> operation), POSIX: 14(Bad address)
>
> ....
>
>
>
>
>
> On linux:
>
> [root at lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/
>
> ls: /media/2011/201105/20110530/37: No such file or directory
>
> total 332
>
> ...
>
> drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 32
>
> drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 34
>
> drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 35
>
> drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 36
>
> drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 37
>
> ...
>
>
>
> [root at lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/37
>
> ls: /media/2011/201105/20110530/37/NN.0000073824357.00001.tif: No such
> file or directory
>
> ls: /media/2011/201105/20110530/37/NN.0000073824357.00003.tif: No such
> file or directory
>
> total 54
>
> -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.0000073824357.00001.tif
>
> -rwxrwxr-x 1 nfsnobody 1003 35312 Jun 6 2011 NN.0000073824357.00002.tif
>
> -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.0000073824357.00003.tif
>
>
>
>
>
> I see in the nfslog:
>
> ...
>
> [2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize]
> 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1
> overlaps=0
>
> [2014-03-31 12:44:18.958078] I [dht-layout.c:638:dht_layout_normalize]
> 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/30. holes=1
> overlaps=0
>
> [2014-03-31 12:44:18.959980] I [dht-layout.c:638:dht_layout_normalize]
> 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/90. holes=1
> overlaps=0
>
> [2014-03-31 12:44:18.961094] E [dht-helper.c:429:dht_subvol_get_hashed]
> (-->/usr/lib64/glusterfs/3.4.2/xlator/debug/io-stats.so(io_stats_lookup+0x157)
> [0x7fd6a61282e7] (-->/usr/lib64/libglusterfs.so.0(default_lookup+0x6d)
> [0x3dfe01c03d]
> (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup+0xa7e)
> [0x7fd6a656af2e]))) 0-caviar_data11-dht: invalid argument: loc->parent
>
> [2014-03-31 12:44:18.961283] W
> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-caviar_data11-client-0:
> remote operation failed: Invalid argument. Path:
> <gfid:00000000-0000-0000-0000-000000000000>
> (00000000-0000-0000-0000-000000000000)
>
> [2014-03-31 12:44:18.961319] E [acl3.c:334:acl3_getacl_resume] 0-nfs-ACL:
> Unable to resolve FH: (192.168.151.21:740) caviar_data11 :
> 00000000-0000-0000-0000-000000000000
>
> [2014-03-31 12:44:18.961338] E [acl3.c:342:acl3_getacl_resume] 0-nfs-ACL:
> unable to open_and_resume
>
> ...
>
>
>
> The weirdest thing is it changes from time to time which files and
> directories work and which don't
>
> Any ideas?
>
>
>
> Thanks!
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140403/f3d9682f/attachment.html>
More information about the Gluster-users
mailing list