[Gluster-users] Some files inaccessible via NFS

[Jan Písačka (@nsa.gov)]

Hi everybody,

we are experiencing random problems accessing some files on a volume via NFS. Any help is highly 
appreciated.

The setup:
CentOS release 6.5
glusterfs-libs-3.4.7-1.el6.x86_64
glusterfs-fuse-3.4.7-1.el6.x86_64
glusterfs-server-3.4.7-1.el6.x86_64
glusterfs-3.4.7-1.el6.x86_64
glusterfs-cli-3.4.7-1.el6.x86_64

Volume Name: repsilo
Type: Distributed-Replicate
Volume ID: 94d8f999-bf1e-4f99-a96f-5bb6564222d9
Status: Started
Number of Bricks: 10 x 2 = 20
Transport-type: tcp
Bricks:
Brick1: gls07.srv:/mnt/glusterRawL
Brick2: gls08.srv:/mnt/glusterRawL
Brick3: gls07.srv:/mnt/glusterRawR
Brick4: gls08.srv:/mnt/glusterRawR
Brick5: gls14.srv:/mnt/glusterRawL
Brick6: gls15.srv:/mnt/glusterRawL
Brick7: gls14.srv:/mnt/glusterRawR
Brick8: gls15.srv:/mnt/glusterRawR
Brick9: gls12.srv:/mnt/glusterRawL
Brick10: gls13.srv:/mnt/glusterRawL
Brick11: gls12.srv:/mnt/glusterRawR
Brick12: gls13.srv:/mnt/glusterRawR
Brick13: gls01.srv:/mnt/glusterRawL
Brick14: gls02.srv:/mnt/glusterRawL
Brick15: gls01.srv:/mnt/glusterRawR
Brick16: gls02.srv:/mnt/glusterRawR
Brick17: gls03.srv:/mnt/glusterRawL
Brick18: gls04.srv:/mnt/glusterRawL
Brick19: gls03.srv:/mnt/glusterRawR
Brick20: gls04.srv:/mnt/glusterRawR

While the files are readable through the native client, trying to read them via NFS fails on some 
servers. This is an example - relevant content of nfs.log on the gls04 server:

[2015-09-02 08:54:51.916981] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-repsilo-client-18: 
remote operation
failed: Permission denied. Path: 
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackReq
uest.1.h5 (e3a07a5a-9ef4-4bbf-a2a9-86b1dd078a12)
[2015-09-02 08:54:51.917051] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-repsilo-client-19: 
remote operation
failed: Permission denied. Path: 
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 
(e3a07a5a-9ef4-4bbf-a2a9-86b1dd078a12)
[2015-09-02 08:54:51.917109] E [dht-helper.c:1259:dht_inode_ctx_get] 
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x6d) 
[0x7fbfd8398f6d] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_layout_preset+0x5e) 
[0x7fbfd837be6e] 
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x34) 
[0x7fbfd837de84]))) 0-repsilo-dht: invalid argument: inode
[2015-09-02 08:54:51.917140] E [dht-helper.c:1278:dht_inode_ctx_set] 
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x6d) 
[0x7fbfd8398f6d] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_layout_preset+0x5e) 
[0x7fbfd837be6e] 
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x52) 
[0x7fbfd837dea2]))) 0-repsilo-dht: invalid argument: inode
[2015-09-02 08:54:51.918177] W [nfs3.c:726:nfs3svc_getattr_stat_cbk] 0-nfs: a4fcea5f: 
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 => -1 
(Invalid argument)
[2015-09-02 08:54:51.918225] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: a4fcea5f, 
GETATTR: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2015-09-02 08:54:51.920121] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-repsilo-client-19: 
remote operation failed: Permission denied. Path: 
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 
(e3a07a5a-9ef4-4bbf-a2a9-86b1dd078a12)
[2015-09-02 08:54:51.920208] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-repsilo-client-18: 
remote operation failed: Permission denied. Path: 
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 
(e3a07a5a-9ef4-4bbf-a2a9-86b1dd078a12)
[2015-09-02 08:54:51.920267] E [dht-helper.c:1259:dht_inode_ctx_get] 
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x6d) 
[0x7fbfd8398f6d] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_layout_preset+0x5e) 
[0x7fbfd837be6e] 
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x34) 
[0x7fbfd837de84]))) 0-repsilo-dht: invalid argument: inode
[2015-09-02 08:54:51.920298] E [dht-helper.c:1278:dht_inode_ctx_set] 
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x6d) 
[0x7fbfd8398f6d] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_layout_preset+0x5e) 
[0x7fbfd837be6e] 
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x52) 
[0x7fbfd837dea2]))) 0-repsilo-dht: invalid argument: inode
[2015-09-02 08:54:51.920679] W [nfs3.c:1512:nfs3svc_access_cbk] 0-nfs: a6fcea5f: 
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 => -1 
(Invalid argument)
[2015-09-02 08:54:51.920703] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: a6fcea5f, 
ACCESS: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2015-09-02 08:54:51.921374] W [nfs3.c:1512:nfs3svc_access_cbk] 0-nfs: a8fcea5f: 
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 => -1 
(Invalid argument)
[2015-09-02 08:54:51.921391] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: a8fcea5f, 
ACCESS: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2015-09-02 08:55:07.041430] W [nfs3.c:726:nfs3svc_getattr_stat_cbk] 0-nfs: a9fcea5f: 
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 => -1 
(Invalid argument)
[2015-09-02 08:55:07.041493] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: a9fcea5f, 
GETATTR: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)


gluster volume heal repsilo info heal-failed
and
gluster volume heal repsilo info split-brain

both give Number of entries: 0 on all bricks

Thanks.
Regards,
Jan