[Gluster-devel] nfs mount point hanging when gluster nfs server restart with a huge network flow.
christ1016 at sina.com
christ1016 at sina.com
Thu Sep 10 00:09:06 UTC 2015
Hello, everyone:
My name is louis, and I am a newcomer of glusterfs. Nowadays, I have confused by a classic TCP connection issue, which is called "TCP cheat dead".
In my cluster, I have some bricks in different cluster nodes. And I mount the export volname dir of gluster in Windows 2008 by using glusterfs nfs server.
Then I have being written a huge network flow to the mount dir. In the condition, I restart glusterfs nfs server in the cluster node sometimes, and it occurred that the mount dir can not be accessed (operation hang) in Windows.
How to reproduce this issue with a very low probability:
1. we start the gluster, and export the volume (volume1) by using gluster nfs server. like this:
showmount -e 173.16.10.1
Export list for 173.16.10.1:
/volume1 *
2. Then mount the export volume in the windows 2008, like this:
mount -o sec=sys \\173.16.10.1\volume1 Z:
3. Then writting a huge network flow to the mount point (Z:).
4. restart the gluster nfs server in the cluster node (173.16.10.1).
5. it will occur that the mount point (Z:) hanging in windows 2008, and can not recover access again, of cause the network flow had been interrupted.
Of cause, it occurred in a low probability. After this issuse occurred, the nfs server and nfs client tcp connection status like below:
In the cluster node (gluster nfs server 2049) the netstat info of nfs tcp connections:
[root at vx1600-cjf50 ~]# netstat -anp | grep 173.16.10.1
tcp 0 0 173.16.10.1:111 173.16.10.45:50292 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51380 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52643 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49841 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49461 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49907 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:25620 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:25892 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49966 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51329 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52619 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51341 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49877 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49949 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49806 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51371 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49790 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49812 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:26136 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51356 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50306 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51354 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49852 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52621 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49931 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:25927 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52651 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50972 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49963 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51311 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52653 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:2049 173.16.10.45:50360 ESTABLISHED 24892/ucsfs
tcp 0 0 173.16.10.1:111 173.16.10.45:50326 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:25973 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:26149 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49953 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49885 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49831 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49929 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51315 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49827 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49893 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49879 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49856 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49794 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51321 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49865 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50286 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49829 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49833 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49883 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49835 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50304 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49913 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49846 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52629 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50290 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50345 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50009 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49955 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:26152 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51301 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49810 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49780 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49895 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:25747 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50339 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49951 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50294 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49778 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:25584 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49919 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51337 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51335 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51339 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52655 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50316 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50298 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:25701 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49822 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52623 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:38468 173.16.10.45:906 ESTABLISHED 24892/ucsfs
tcp 0 0 173.16.10.1:111 173.16.10.45:50318 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51313 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50302 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52637 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:26308 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50353 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51389 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51362 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49897 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51377 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49849 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50361 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51292 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51305 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49911 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49862 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49792 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51331 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51350 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49968 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52631 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51344 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50355 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50330 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49788 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50357 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:2049 173.16.10.45:50362 ESTABLISHED 24892/ucsfs
tcp 0 0 173.16.10.1:111 173.16.10.45:51325 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49939 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49891 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49873 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52649 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49927 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49925 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51327 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49869 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52641 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51365 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49947 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50300 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49881 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:25782 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49889 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49808 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52615 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49917 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50322 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52647 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:26009 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52639 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51317 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49933 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49839 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51299 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52611 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49804 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51369 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49786 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:25538 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51323 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51333 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52627 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49937 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49935 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50349 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50334 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51348 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49871 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52617 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50312 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50347 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:25665 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51367 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50328 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52633 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51352 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:26164 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50314 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49867 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51319 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:26353 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51375 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49901 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50320 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50351 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:26055 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49816 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52635 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49909 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49837 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49818 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49784 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51303 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49782 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51360 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50359 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:26237 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49796 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50296 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49915 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50337 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52625 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49887 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51297 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49858 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51309 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49903 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49798 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51346 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49844 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49899 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:26188 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50332 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49854 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:26224 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49943 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:25795 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49860 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52613 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.11:111 173.16.10.53:52645 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49959 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:25830 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49800 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49921 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50310 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49961 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50341 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49905 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49875 ESTABLISHED 3157/rpcbind
tcp 310980 0 173.16.10.1:2049 173.16.10.45:901 ESTABLISHED 24892/ucsfs
tcp 0 0 173.16.10.1:38465 173.16.10.45:955 ESTABLISHED 24892/ucsfs
tcp 0 0 173.16.10.1:111 173.16.10.48:26100 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51358 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:25856 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49941 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49825 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51307 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49923 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49957 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50324 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50288 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49820 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50343 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.48:26272 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51373 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49945 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49802 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51397 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51290 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:51295 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:49814 ESTABLISHED 3157/rpcbind
tcp 0 0 173.16.10.1:111 173.16.10.45:50308 ESTABLISHED 3157/rpcbind
udp 0 0 173.16.10.11:137 0.0.0.0:* 12177/nmbd
udp 0 0 173.16.10.1:137 0.0.0.0:* 12177/nmbd
udp 0 0 173.16.10.11:138 0.0.0.0:* 12177/nmbd
udp 0 0 173.16.10.1:138 0.0.0.0:* 12177/nmbd
we can see from the above tcp connection status, one of a tcp connection had received 310980 packets in receive buffer. And it can not ever send packet any more.
That is to say the tcp connection is hang yet. And the windows mount point can not recover access (hang too).
the windows netstat info of nfs related tcp connetion like below:
TCP 173.16.10.55:944 173.16.10.5:38468 ESTABLISHED 2492
TCP 173.16.10.55:967 173.16.10.5:38465 ESTABLISHED 2380
TCP 173.16.10.55:976 173.16.10.5:2049 ESTABLISHED 2492
TCP 173.16.10.55:49457 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49459 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49460 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49461 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49462 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49463 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49464 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49465 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49466 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49467 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49473 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49474 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49785 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49786 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49787 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49788 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49789 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49790 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49791 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49792 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49793 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49794 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49795 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49796 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49797 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49798 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49799 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49800 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49802 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49803 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49804 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49805 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49806 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49807 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49808 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49809 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49810 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49811 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49812 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49813 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49814 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49815 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49816 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49817 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49818 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49819 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49820 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49821 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49822 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49823 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49824 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49825 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49826 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49827 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49828 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49829 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49830 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49831 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49832 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49833 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49834 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49835 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49836 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49837 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49838 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49839 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49840 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49841 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49842 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49843 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49844 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49845 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49846 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49847 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49848 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49849 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49850 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49851 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49852 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49853 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49854 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49855 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49856 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49857 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49858 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49859 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49860 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49861 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49862 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49863 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49865 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49866 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49867 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49868 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49869 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49870 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49871 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49872 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49873 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49874 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49875 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49876 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49877 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49878 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49879 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49880 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49881 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49882 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49883 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49884 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49885 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49886 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49887 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49888 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49889 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49890 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49891 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49892 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49893 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49894 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49895 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49896 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49897 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49898 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49899 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49900 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49901 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49902 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49903 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49904 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49905 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49906 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49907 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49908 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49909 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49910 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49911 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49912 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49913 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49914 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49915 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49916 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49917 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49918 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49919 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49920 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49921 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49922 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49923 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49924 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49925 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49926 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49928 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49929 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49930 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49931 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49932 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49933 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49934 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49935 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49936 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49937 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49939 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49940 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49941 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49942 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49943 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49944 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49945 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49946 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49947 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49948 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49949 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49950 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49951 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49952 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49953 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49954 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49955 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49956 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49957 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49958 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:49959 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:49960 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50270 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50271 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50278 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50279 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50280 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50281 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50282 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50283 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50284 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50285 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50286 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50287 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50288 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50289 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50290 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50291 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50292 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50293 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50294 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50295 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50296 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50297 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50298 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50299 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50300 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50301 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50302 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50303 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50304 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50305 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50306 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50307 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50308 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50309 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50310 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50311 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50312 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50313 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50314 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50315 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50316 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50317 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50318 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50319 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50320 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50321 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50322 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50323 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50324 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50325 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50326 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50327 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50328 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50329 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50330 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50331 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50332 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50333 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50334 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50335 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50336 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50337 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50338 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50339 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50340 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50341 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50342 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50343 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50344 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50345 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50346 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50347 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50348 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50349 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50350 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50351 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50352 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50353 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50354 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50355 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50356 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50357 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50358 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50359 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50360 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50361 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50362 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50363 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50364 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50365 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50366 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50367 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50368 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50369 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50370 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50371 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50372 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50373 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50374 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50375 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50376 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50377 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50378 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50379 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50380 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50381 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50382 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50383 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50384 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50385 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50386 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50387 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50388 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50389 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50390 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50391 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50392 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50393 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50394 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50395 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50396 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50397 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50398 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50399 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50400 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50401 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50402 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50403 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50404 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50405 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50406 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50407 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50408 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50409 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50410 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50411 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50412 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50413 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50414 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50415 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50416 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50417 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50418 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50419 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50420 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50421 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50423 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50424 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50426 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50427 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50428 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50429 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50430 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50431 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50432 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50433 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50434 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50435 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50436 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50437 173.16.10.5:2049 CLOSE_WAIT 1560
TCP 173.16.10.55:50439 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50440 173.16.10.5:2049 ESTABLISHED 1560
TCP 173.16.10.55:50442 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50443 173.16.10.5:2049 ESTABLISHED 1560
TCP 173.16.10.55:50445 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50446 173.16.10.5:2049 ESTABLISHED 1560
TCP 173.16.10.55:50447 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50448 173.16.10.5:2049 ESTABLISHED 1560
TCP 173.16.10.55:50450 173.16.10.5:111 ESTABLISHED 1560
TCP 173.16.10.55:50451 173.16.10.5:2049 ESTABLISHED 1560
we can see from the above status of nfs related tcp connections in windows 2008, after the issue occurred, there were a lot of CLOSE_WAIT status tcp connections in port 2049(nfs service port).
After reading the TCP/IP illustrated volume1, it describes we can use the tcp KeepAlive function to solve this classic issuse. Can I use the glusterfs KeepAlive function in the socket to solve this issuse, to avoid the nfs mount point hanging in windows when I restart gluster nfs server with a huge network flow writting. I just set like below:
First, I set the TCP parameters in the Windows 2008 (by using regedit):
HKEY_LOCAL_MACHINE/CurrentControlSet/Services/Tcpip/Parameters:
KeepAliveInterval set the value : 1000
KeepAliveTime set the value : 300000 (ms)
TcpMaxDataRetransmissions set the value : 5
Second, I set the volume keepalive switch to "on", like this:
gluster volume set volume1 transport.keepalive on
and the keepalive parameters like this:
keepaliveintvl = 2
keepaliveidle = 20
After take the operations above, I am not sure it is really solve this nfs hang issue, because it just occurs in a very low probabllity. If it can not solve this issue, how could I do.
I am very appreciate for your help, and thanks very much.
Best Regards
Louis
2015/09/10
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20150910/5138d8a3/attachment-0001.html>
More information about the Gluster-devel
mailing list