[Gluster-devel] Remote operation failed: Stale NFS file handle

Neil Van Lysel van-lyse at cs.wisc.edu
Tue Oct 15 20:44:45 UTC 2013


Hello!

Many of our Gluster client nodes are seeing a lot of these errors in 
their log files:

[2013-10-15 06:48:59.467263] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-6: remote 
operation failed: Stale NFS file handle. Path: /path 
(3cfbebf4-40e4-4300-aa6e-bd43b4310b94)
[2013-10-15 06:48:59.467331] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-7: remote 
operation failed: Stale NFS file handle. Path: /path 
(3cfbebf4-40e4-4300-aa6e-bd43b4310b94)
[2013-10-15 06:48:59.470554] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-0: remote 
operation failed: Stale NFS file handle. Path: /path 
(d662e7db-7864-4b18-b587-bdc5e8756076)
[2013-10-15 06:48:59.470624] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-1: remote 
operation failed: Stale NFS file handle. Path: /path 
(d662e7db-7864-4b18-b587-bdc5e8756076)
[2013-10-15 06:49:04.537548] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-3: remote 
operation failed: Stale NFS file handle. Path: /path 
(a4ea32e0-25f8-440d-b258-23430490624d)
[2013-10-15 06:49:04.537651] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-2: remote 
operation failed: Stale NFS file handle. Path: /path 
(a4ea32e0-25f8-440d-b258-23430490624d)
[2013-10-15 06:49:14.380551] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-0: remote 
operation failed: Stale NFS file handle. Path: /path 
(669a2d6b-2998-48b2-8f3f-93d5f65cdd87)
[2013-10-15 06:49:14.380663] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-1: remote 
operation failed: Stale NFS file handle. Path: /path 
(669a2d6b-2998-48b2-8f3f-93d5f65cdd87)
[2013-10-15 06:49:14.386390] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-4: remote 
operation failed: Stale NFS file handle. Path: /path 
(016aafa9-35ac-4f6f-90bd-b4ac5d435ad0)
[2013-10-15 06:49:14.386471] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-5: remote 
operation failed: Stale NFS file handle. Path: /path 
(016aafa9-35ac-4f6f-90bd-b4ac5d435ad0)
[2013-10-15 18:28:10.630357] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-2: remote 
operation failed: Stale NFS file handle. Path: /path 
(5d6153cc-64b3-4151-85cd-2646c33c6918)
[2013-10-15 18:28:10.630425] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-3: remote 
operation failed: Stale NFS file handle. Path: /path 
(5d6153cc-64b3-4151-85cd-2646c33c6918)
[2013-10-15 18:28:10.636301] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-4: remote 
operation failed: Stale NFS file handle. Path: /path 
(2f64b9fe-02a0-408b-9edb-0c5e5bf0ed0e)
[2013-10-15 18:28:10.636377] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-5: remote 
operation failed: Stale NFS file handle. Path: /path 
(2f64b9fe-02a0-408b-9edb-0c5e5bf0ed0e)
[2013-10-15 18:28:10.638574] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-5: remote 
operation failed: Stale NFS file handle. Path: /path 
(990de721-1fc9-461d-8412-8c17c23ebbbd)
[2013-10-15 18:28:10.638647] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-4: remote 
operation failed: Stale NFS file handle. Path: /path 
(990de721-1fc9-461d-8412-8c17c23ebbbd)
[2013-10-15 18:28:10.645043] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-7: remote 
operation failed: Stale NFS file handle. Path: /path 
(0d8d3c5a-d26e-4c15-a8d5-987a4033a6d0)
[2013-10-15 18:28:10.645157] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-6: remote 
operation failed: Stale NFS file handle. Path: /path 
(0d8d3c5a-d26e-4c15-a8d5-987a4033a6d0)
[2013-10-15 18:28:10.648126] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-6: remote 
operation failed: Stale NFS file handle. Path: /path 
(c1c84d57-f54d-4dc1-a5df-9be563da78fb)
[2013-10-15 18:28:10.648276] W 
[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-7: remote 
operation failed: Stale NFS file handle. Path: /path 
(c1c84d57-f54d-4dc1-a5df-9be563da78fb)


How can I resolve these errors?


*gluster --version:
glusterfs 3.4.0 built on Jul 25 2013 04:12:27


*gluster volume info:
Volume Name: scratch
Type: Distributed-Replicate
Volume ID: 198b9d77-96e6-4c7f-9f0c-3618cbcaa940
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: 10.129.40.21:/data/glusterfs/brick1/scratch
Brick2: 10.129.40.22:/data/glusterfs/brick1/scratch
Brick3: 10.129.40.23:/data/glusterfs/brick1/scratch
Brick4: 10.129.40.24:/data/glusterfs/brick1/scratch
Brick5: 10.129.40.21:/data/glusterfs/brick2/scratch
Brick6: 10.129.40.22:/data/glusterfs/brick2/scratch
Brick7: 10.129.40.23:/data/glusterfs/brick2/scratch
Brick8: 10.129.40.24:/data/glusterfs/brick2/scratch
Options Reconfigured:
features.quota: off

Volume Name: home
Type: Distributed-Replicate
Volume ID: 0d8ebafc-471e-4b16-a4a9-787ce8616225
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: 10.129.40.21:/data/glusterfs/brick1/home
Brick2: 10.129.40.22:/data/glusterfs/brick1/home
Brick3: 10.129.40.23:/data/glusterfs/brick1/home
Brick4: 10.129.40.24:/data/glusterfs/brick1/home
Brick5: 10.129.40.21:/data/glusterfs/brick2/home
Brick6: 10.129.40.22:/data/glusterfs/brick2/home
Brick7: 10.129.40.23:/data/glusterfs/brick2/home
Brick8: 10.129.40.24:/data/glusterfs/brick2/home
Options Reconfigured:
features.quota: off


*gluster volume status:
Status of volume: scratch
Gluster process                                         Port Online  Pid
------------------------------------------------------------------------------
Brick 10.129.40.21:/data/glusterfs/brick1/scratch       49154 Y       7536
Brick 10.129.40.22:/data/glusterfs/brick1/scratch       49154 Y       27976
Brick 10.129.40.23:/data/glusterfs/brick1/scratch       49154 Y       7436
Brick 10.129.40.24:/data/glusterfs/brick1/scratch       49154 Y       19773
Brick 10.129.40.21:/data/glusterfs/brick2/scratch       49155 Y       7543
Brick 10.129.40.22:/data/glusterfs/brick2/scratch       49155 Y       27982
Brick 10.129.40.23:/data/glusterfs/brick2/scratch       49155 Y       7442
Brick 10.129.40.24:/data/glusterfs/brick2/scratch       49155 Y       19778
NFS Server on localhost                                 2049 Y       7564
Self-heal Daemon on localhost                           N/A Y       7569
NFS Server on 10.129.40.24                              2049 Y       19788
Self-heal Daemon on 10.129.40.24                        N/A Y       19792
NFS Server on 10.129.40.23                              2049 Y       7464
Self-heal Daemon on 10.129.40.23                        N/A Y       7468
NFS Server on 10.129.40.22                              2049 Y       28004
Self-heal Daemon on 10.129.40.22                        N/A Y       28008

There are no active volume tasks
Status of volume: home
Gluster process                                         Port Online  Pid
------------------------------------------------------------------------------
Brick 10.129.40.21:/data/glusterfs/brick1/home          49152 Y       7549
Brick 10.129.40.22:/data/glusterfs/brick1/home          49152 Y       27989
Brick 10.129.40.23:/data/glusterfs/brick1/home          49152 Y       7449
Brick 10.129.40.24:/data/glusterfs/brick1/home          49152 Y       19760
Brick 10.129.40.21:/data/glusterfs/brick2/home          49153 Y       7554
Brick 10.129.40.22:/data/glusterfs/brick2/home          49153 Y       27994
Brick 10.129.40.23:/data/glusterfs/brick2/home          49153 Y       7454
Brick 10.129.40.24:/data/glusterfs/brick2/home          49153 Y       19766
NFS Server on localhost                                 2049 Y       7564
Self-heal Daemon on localhost                           N/A Y       7569
NFS Server on 10.129.40.24                              2049 Y       19788
Self-heal Daemon on 10.129.40.24                        N/A Y       19792
NFS Server on 10.129.40.22                              2049 Y       28004
Self-heal Daemon on 10.129.40.22                        N/A Y       28008
NFS Server on 10.129.40.23                              2049 Y       7464
Self-heal Daemon on 10.129.40.23                        N/A Y       7468

There are no active volume tasks


*The gluster volumes are mounted using the glusterfs-fuse package 
(glusterfs-fuse-3.4.0-3.el6.x86_64) on the clients like so:
/sbin/mount.glusterfs 10.129.40.21:home /home
/sbin/mount.glusterfs 10.129.40.21:scratch /scratch


*Gluster packages on Gluster servers:
glusterfs-server-3.4.0-3.el6.x86_64
glusterfs-libs-3.4.0-8.el6.x86_64
glusterfs-3.4.0-3.el6.x86_64
glusterfs-geo-replication-3.4.0-3.el6.x86_64
glusterfs-fuse-3.4.0-3.el6.x86_64
glusterfs-rdma-3.4.0-3.el6.x86_64


*Gluster packages on clients:
glusterfs-fuse-3.4.0-3.el6.x86_64
glusterfs-3.4.0-3.el6.x86_64


All clients and servers are running the same OS and kernel:

*uname -a:
Linux <hostname> 2.6.32-358.6.1.el6.x86_64 #1 SMP Tue Apr 23 16:15:13 
CDT 2013 x86_64 x86_64 x86_64 GNU/Linux

*cat /etc/redhat-release :
Scientific Linux release 6.3 (Carbon)


Thanks for your help,

Neil Van Lysel
UNIX Systems Administrator
Center for High Throughput Computing
University of Wisconsin - Madison


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3528 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20131015/cc6e29da/attachment-0001.p7s>


More information about the Gluster-devel mailing list