We were able to stop the lockd error after rebooted the server.
However still encountering another strange issue with sqlplus.
I sent another email to the group on it.
Thanks
Peter
________________________________________
From: gluster-users-bounces at gluster.org [gluster-users-bounces at gluster.org] on behalf of Peter Auyeung [pauyeung at connexity.com]
Sent: Tuesday, January 27, 2015 2:35 PM
To: Niels de Vos
Cc: gluster-users at gluster.org; gluster-devel at gluster.org
Subject: Re: [Gluster-users] [Gluster-devel] lockd: server not responding, timed out
Hi Niels,
I see no kernel NFS service running on the gluster node.
Here are the output
root at glusterprod001:~# rpcinfo
program version netid address service owner
100000 4 tcp6 ::.0.111 portmapper superuser
100000 3 tcp6 ::.0.111 portmapper superuser
100000 4 udp6 ::.0.111 portmapper superuser
100000 3 udp6 ::.0.111 portmapper superuser
100000 4 tcp 0.0.0.0.0.111 portmapper superuser
100000 3 tcp 0.0.0.0.0.111 portmapper superuser
100000 2 tcp 0.0.0.0.0.111 portmapper superuser
100000 4 udp 0.0.0.0.0.111 portmapper superuser
100000 3 udp 0.0.0.0.0.111 portmapper superuser
100000 2 udp 0.0.0.0.0.111 portmapper superuser
100000 4 local /run/rpcbind.sock portmapper superuser
100000 3 local /run/rpcbind.sock portmapper superuser
100005 3 tcp 0.0.0.0.150.65 mountd superuser
100005 1 tcp 0.0.0.0.150.66 mountd superuser
100003 3 tcp 0.0.0.0.8.1 nfs superuser
100021 4 tcp 0.0.0.0.150.68 nlockmgr superuser
100227 3 tcp 0.0.0.0.8.1 - superuser
100021 1 udp 0.0.0.0.2.215 nlockmgr superuser
100021 1 tcp 0.0.0.0.2.217 nlockmgr superuser
100024 1 udp 0.0.0.0.136.211 status 105
100024 1 tcp 0.0.0.0.170.60 status 105
100024 1 udp6 ::.182.65 status 105
100024 1 tcp6 ::.172.250 status 105
root at glusterprod001:~# ss
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 10.101.165.61:1015 10.101.165.63:24007
ESTAB 0 0 10.101.165.61:936 10.101.165.61:49156
ESTAB 0 0 10.101.165.61:1012 10.101.165.66:24007
ESTAB 0 0 10.101.165.61:49157 10.101.165.64:987
ESTAB 0 0 10.101.165.61:999 10.101.165.62:49153
ESTAB 0 0 10.101.165.61:49157 10.101.165.66:988
ESTAB 0 0 10.101.165.61:49155 10.101.165.62:834
ESTAB 0 0 10.101.165.61:49156 10.101.165.63:998
ESTAB 0 0 10.101.165.61:912 10.101.165.65:49153
ESTAB 0 0 127.0.0.1:982 127.0.0.1:24007
ESTAB 0 0 10.101.165.61:49156 10.101.165.65:997
ESTAB 0 0 10.101.165.61:49155 10.101.165.61:850
ESTAB 0 0 10.101.165.61:922 10.101.165.62:49154
ESTAB 0 0 10.101.165.61:896 10.101.165.65:49154
ESTAB 0 0 10.101.165.61:1010 10.101.165.61:24007
ESTAB 0 0 10.101.165.61:imaps 10.101.165.61:49156
ESTAB 0 0 10.101.165.61:49155 10.101.165.63:981
ESTAB 0 0 10.101.165.61:930 10.101.165.64:49155
ESTAB 0 0 10.101.165.61:4379 10.101.165.65:44899
ESTAB 0 0 10.101.165.61:983 10.101.165.61:49157
ESTAB 0 0 10.101.165.61:49156 10.101.165.64:1007
ESTAB 0 0 10.101.165.61:935 10.101.165.61:49155
ESTAB 0 0 10.101.165.61:49155 10.101.165.62:900
ESTAB 0 0 10.101.165.97:nfs 10.102.69.226:861
ESTAB 0 0 10.101.165.61:4379 10.101.165.64:46473
ESTAB 0 0 10.101.165.61:860 10.101.165.65:49155
ESTAB 125 0 10.101.165.61:55453 10.101.161.104:4242
ESTAB 0 0 10.101.165.61:897 10.101.165.66:49153
ESTAB 0 0 10.101.165.61:49156 10.101.165.61:imaps
ESTAB 0 0 10.101.165.61:49156 10.101.165.62:999
ESTAB 0 0 10.101.165.61:49156 10.101.165.66:ftps
ESTAB 0 0 10.101.165.61:49155 10.101.165.62:958
ESTAB 0 0 10.101.165.61:49154 10.101.165.65:976
ESTAB 0 0 10.101.165.61:24007 10.101.195.108:1023
ESTAB 0 0 10.101.165.61:49156 10.101.165.66:982
ESTAB 0 0 10.101.165.61:49156 10.101.165.61:936
ESTAB 0 0 10.101.165.61:49155 10.101.195.108:1005
ESTAB 0 0 10.101.165.61:855 10.101.165.64:49154
ESTAB 0 0 10.101.165.61:49155 10.101.165.65:974
ESTAB 0 0 10.101.165.61:49157 10.101.165.66:973
ESTAB 0 0 10.101.165.61:49156 10.101.165.65:982
ESTAB 0 0 10.101.165.61:941 10.101.165.66:49154
ESTAB 0 0 10.101.165.61:ftps 10.101.165.64:49155
ESTAB 0 0 10.101.165.61:1004 10.101.165.66:49152
ESTAB 0 0 10.101.165.61:818 10.101.165.65:49152
ESTAB 0 0 10.101.165.61:49157 10.101.165.65:996
ESTAB 0 0 10.101.165.61:891 10.101.165.65:49153
ESTAB 0 0 10.101.165.61:49156 10.101.165.62:943
ESTAB 0 0 10.101.165.61:49157 10.101.165.65:ftps
ESTAB 0 0 10.101.165.61:913 10.101.165.64:49154
ESTAB 0 0 10.101.165.61:996 10.101.165.66:49153
ESTAB 0 0 10.101.165.61:49157 10.101.165.64:telnets
ESTAB 0 0 10.101.165.61:946 10.101.165.64:49154
ESTAB 0 0 10.101.165.61:49155 10.101.166.51:1011
ESTAB 0 0 10.101.165.61:849 10.101.165.62:49152
ESTAB 0 0 127.0.0.1:983 127.0.0.1:24007
ESTAB 0 0 10.101.165.61:24007 10.101.165.63:1021
ESTAB 0 0 10.101.165.61:893 10.101.165.63:49153
ESTAB 0 0 10.101.165.61:49155 10.101.165.64:ftps
ESTAB 0 0 10.101.165.61:59797 10.101.165.64:4379
ESTAB 0 0 10.101.165.61:928 10.101.165.65:49155
ESTAB 0 0 10.101.165.61:49154 10.101.165.65:ftps-data
ESTAB 0 0 10.101.165.61:851 10.101.165.63:49152
ESTAB 0 0 10.101.165.61:4379 10.101.165.63:46734
ESTAB 0 0 10.101.165.61:49157 10.101.165.61:858
ESTAB 0 0 10.101.165.61:817 10.101.165.63:49153
ESTAB 0 0 10.101.165.61:4379 10.101.165.62:53373
ESTAB 0 0 10.101.165.61:49154 10.101.165.63:ircs
ESTAB 0 0 10.101.165.61:24007 10.101.165.61:1009
ESTAB 0 0 10.101.165.61:927 10.101.165.66:49155
ESTAB 0 0 10.101.165.61:857 10.101.165.62:49154
ESTAB 0 0 10.101.165.61:49156 10.101.165.62:824
ESTAB 0 0 10.101.165.61:49155 10.101.165.61:914
ESTAB 0 0 10.101.165.61:861 10.101.165.64:49155
ESTAB 0 0 10.101.165.61:49154 10.101.165.62:ftps
ESTAB 0 0 10.101.165.61:49154 10.101.165.64:ircs
ESTAB 0 0 10.101.165.61:49154 10.101.166.51:1017
ESTAB 0 0 10.101.165.61:939 10.101.165.61:49154
ESTAB 0 0 10.101.165.61:49156 10.101.165.63:1011
ESTAB 0 0 10.101.165.61:856 10.101.165.63:49154
ESTAB 0 0 10.101.165.61:24007 10.101.165.61:1008
ESTAB 0 0 10.101.165.61:49157 10.101.165.63:997
ESTAB 0 0 10.101.165.61:934 10.101.165.62:49155
ESTAB 0 0 10.101.165.61:822 10.101.165.64:49153
ESTAB 0 0 10.101.165.61:55997 10.101.165.66:4379
ESTAB 0 0 10.101.165.61:24007 10.101.166.51:1023
ESTAB 0 0 10.101.165.61:929 10.101.165.66:49152
ESTAB 0 0 10.101.165.61:49155 10.101.165.66:966
ESTAB 0 0 10.101.165.61:49154 10.101.165.61:939
ESTAB 0 0 10.101.165.61:49156 10.101.165.64:984
ESTAB 0 0 10.101.165.61:49157 10.101.195.108:1006
ESTAB 0 0 10.101.165.61:41681 10.101.165.63:4379
ESTAB 0 0 10.101.165.61:997 10.101.165.64:49153
ESTAB 0 0 10.101.165.61:850 10.101.165.61:49155
ESTAB 0 0 10.101.165.61:49154 10.101.165.66:987
ESTAB 0 0 10.101.165.61:49154 10.101.165.63:976
ESTAB 0 0 10.101.165.61:894 10.101.165.64:49153
ESTAB 0 0 10.101.165.61:49155 10.101.165.66:969
ESTAB 0 0 10.101.165.61:49157 10.101.165.63:983
ESTAB 0 0 10.101.165.61:43557 10.101.165.65:4379
ESTAB 0 0 10.101.165.61:1017 10.101.165.61:49155
ESTAB 0 0 10.101.165.61:1013 10.101.165.65:24007
ESTAB 0 0 10.101.165.61:49156 10.101.165.63:985
ESTAB 0 0 10.101.165.61:918 10.101.165.63:49154
ESTAB 0 0 10.101.165.61:1007 10.101.165.63:49152
ESTAB 0 0 10.101.165.61:24007 10.101.165.61:1011
ESTAB 0 0 10.101.165.61:932 10.101.165.63:49155
ESTAB 0 0 10.101.165.61:1008 10.101.165.61:24007
ESTAB 0 0 10.101.165.61:49155 10.101.165.64:986
ESTAB 0 0 10.101.165.61:914 10.101.165.61:49155
ESTAB 0 0 10.101.165.61:981 10.101.165.63:49154
ESTAB 0 0 127.0.0.1:24007 127.0.0.1:981
ESTAB 0 0 10.101.165.61:1018 10.101.165.64:24007
ESTAB 0 0 10.101.165.61:49154 10.101.165.63:982
ESTAB 0 0 10.101.165.61:816 10.101.165.66:49153
ESTAB 0 0 10.101.165.61:49156 10.101.165.66:customs
ESTAB 0 0 10.101.165.61:998 10.101.165.63:49153
ESTAB 0 0 10.101.165.61:49157 10.101.165.66:981
ESTAB 0 0 10.101.165.61:telnets 10.101.165.62:49155
ESTAB 0 0 10.101.165.61:925 10.101.165.61:49157
ESTAB 0 0 10.101.165.61:1014 10.101.165.62:24007
ESTAB 0 0 10.101.165.61:49154 10.101.165.66:962
ESTAB 0 6800 10.101.165.61:ssh 10.40.152.72:62912
ESTAB 0 0 10.101.165.61:24007 10.101.165.62:1023
ESTAB 0 0 127.0.0.1:24007 127.0.0.1:982
ESTAB 0 0 10.101.165.61:852 10.101.165.61:49154
ESTAB 0 0 10.101.165.61:49154 10.101.165.66:970
ESTAB 0 0 10.101.165.61:824 10.101.165.64:49152
ESTAB 0 0 10.101.165.61:991 10.101.165.63:49155
ESTAB 0 0 10.101.165.61:49155 10.101.165.62:987
ESTAB 0 0 10.101.165.61:24007 10.101.165.61:1010
ESTAB 0 0 10.101.165.61:49154 10.101.165.61:852
ESTAB 0 0 10.101.165.61:938 10.101.165.62:49152
ESTAB 0 0 10.101.165.61:823 10.101.165.66:49152
ESTAB 0 0 10.101.165.61:49155 10.101.165.63:973
ESTAB 0 0 10.101.165.61:920 10.101.165.62:49153
ESTAB 0 0 10.101.165.61:854 10.101.165.65:49154
ESTAB 0 0 10.101.165.61:40487 10.101.165.62:4379
ESTAB 0 0 10.101.165.61:858 10.101.165.61:49157
ESTAB 0 0 10.101.165.61:49154 10.101.165.64:1008
ESTAB 0 0 10.101.165.61:49154 10.101.165.64:991
ESTAB 0 0 10.101.165.61:937 10.101.165.64:49152
ESTAB 0 0 10.101.165.61:24007 10.101.165.61:1002
ESTAB 0 0 10.101.165.61:49155 10.101.165.61:1017
ESTAB 0 0 10.101.165.61:923 10.101.165.64:49153
ESTAB 0 0 10.101.165.61:49156 10.101.195.108:customs
ESTAB 0 0 10.101.165.61:988 10.101.165.66:49155
ESTAB 0 0 10.101.165.61:862 10.101.165.63:49155
ESTAB 0 0 10.101.165.61:49154 10.101.165.61:1023
ESTAB 0 0 10.101.165.61:1011 10.101.165.61:24007
ESTAB 0 0 10.101.165.61:1023 10.101.165.61:49154
ESTAB 0 0 10.101.165.61:49155 10.101.165.64:imaps
ESTAB 0 0 10.101.165.61:1002 10.101.165.61:24007
ESTAB 0 0 10.101.165.61:1006 10.101.165.62:49152
ESTAB 0 0 127.0.0.1:24007 127.0.0.1:1022
ESTAB 0 0 10.101.165.61:944 10.101.165.65:49154
ESTAB 0 0 10.101.165.61:916 10.101.165.63:49153
ESTAB 0 0 10.101.165.61:ftps-data 10.101.165.65:49155
ESTAB 0 0 10.101.165.61:49157 10.101.165.62:924
ESTAB 0 0 10.101.165.61:864 10.101.165.61:49156
ESTAB 0 0 10.101.165.61:1000 10.101.165.65:49153
ESTAB 0 0 10.101.165.61:811 10.101.165.65:49153
ESTAB 0 0 10.101.165.61:931 10.101.165.65:49152
ESTAB 0 0 10.101.165.61:49155 10.101.165.65:988
ESTAB 0 0 10.101.165.61:982 10.101.165.62:49154
ESTAB 0 0 127.0.0.1:1022 127.0.0.1:24007
ESTAB 0 0 10.101.165.61:892 10.101.165.66:49153
ESTAB 0 0 10.101.165.61:49155 10.101.165.61:935
ESTAB 0 0 10.101.165.61:1009 10.101.165.61:24007
ESTAB 0 0 10.101.165.61:49154 10.101.165.62:883
ESTAB 0 0 10.101.165.61:49154 10.101.165.62:812
ESTAB 0 0 10.101.165.61:49154 10.101.195.108:1004
ESTAB 0 0 10.101.165.61:49157 10.101.165.61:983
ESTAB 0 0 10.101.165.61:49157 10.101.165.61:925
ESTAB 0 0 10.101.165.61:4379 10.101.165.66:35729
ESTAB 0 0 127.0.0.1:981 127.0.0.1:24007
ESTAB 0 0 10.101.165.61:49155 10.101.165.63:975
ESTAB 0 0 10.101.165.61:859 10.101.165.66:49155
ESTAB 0 0 10.101.165.61:49157 10.101.165.63:977
ESTAB 0 0 10.101.165.61:933 10.101.165.63:49152
ESTAB 0 0 10.101.165.61:49155 10.101.165.66:980
ESTAB 0 0 10.101.165.61:49155 10.101.165.66:986
ESTAB 0 0 10.101.165.61:863 10.101.165.62:49155
ESTAB 0 0 10.101.165.61:49157 10.101.165.62:891
ESTAB 0 0 10.101.165.61:24007 10.101.165.66:1022
ESTAB 0 0 127.0.0.1:24007 127.0.0.1:983
ESTAB 0 0 10.101.165.61:49156 10.101.165.64:996
ESTAB 0 0 10.101.165.61:49156 10.101.165.65:customs
ESTAB 0 0 10.101.165.61:815 10.101.165.62:49153
ESTAB 0 0 127.0.0.1:24007 127.0.0.1:1023
ESTAB 0 0 10.101.165.61:49154 10.101.165.65:979
ESTAB 0 0 10.101.165.61:853 10.101.165.66:49154
ESTAB 0 0 10.101.165.61:49157 10.101.165.62:817
ESTAB 0 0 10.101.165.61:49155 10.101.165.64:1004
ESTAB 0 0 10.101.165.61:49157 10.101.165.64:pop3s
ESTAB 0 0 10.101.165.61:898 10.101.165.66:49154
ESTAB 0 0 10.101.165.61:24007 10.101.165.65:1021
ESTAB 0 0 127.0.0.1:1023 127.0.0.1:24007
ESTAB 0 0 10.101.165.61:49155 10.101.165.63:ftps-data
ESTAB 0 0 10.101.165.61:1003 10.101.165.65:49152
ESTAB 0 0 10.101.165.61:24007 10.101.165.64:1019
ESTAB 0 0 10.101.165.61:49157 10.101.165.65:980
ESTAB 0 0 10.101.165.61:49155 10.101.165.65:pop3s
ESTAB 0 0 10.101.165.61:49155 10.101.165.65:978
ESTAB 0 0 10.101.165.61:1005 10.101.165.64:49152
ESTAB 0 0 10.101.165.61:895 10.101.165.62:49153
ESTAB 0 0 10.101.165.61:49156 10.101.165.61:864
root at glusterprod001:~# netstat -s
Ip:
3836362639 total packets received
0 forwarded
0 incoming packets discarded
3836362639 incoming packets delivered
2715649214 requests sent out
1 outgoing packets dropped
5 dropped because of missing route
Icmp:
73531 ICMP messages received
11 input ICMP message failed.
ICMP input histogram:
destination unreachable: 212
echo requests: 73317
echo replies: 2
73700 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 381
echo request: 2
echo replies: 73317
IcmpMsg:
InType0: 2
InType3: 212
InType8: 73317
OutType0: 73317
OutType3: 381
OutType8: 2
Tcp:
3414520 active connections openings
50916 passive connection openings
2236146 failed connection attempts
22393 connection resets received
207 connections established
3831070976 segments received
2012419479 segments send out
11779093 segments retransmited
46 bad segments received.
2530424 resets sent
Udp:
5217439 packets received
348 packets to unknown port received.
0 packet receive errors
8087732 packets sent
UdpLite:
TcpExt:
2 resets received for embryonic SYN_RECV sockets
1050 packets pruned from receive queue because of socket buffer overrun
204156 TCP sockets finished time wait in fast timer
12026710 delayed acks sent
18993 delayed acks further delayed because of locked socket
Quick ack mode was activated 14237571 times
24958868 packets directly queued to recvmsg prequeue.
27395 bytes directly in process context from backlog
1731434 bytes directly received in process context from prequeue
935889540 packet headers predicted
111 packets header predicted and directly queued to user
933778742 acknowledgments not containing data payload received
2389068782 predicted acknowledgments
6497616 times recovered from packet loss by selective acknowledgements
219 bad SACK blocks received
Detected reordering 14 times using FACK
Detected reordering 29211 times using SACK
Detected reordering 3704 times using time stamp
188312 congestion windows fully recovered without slow start
262526 congestion windows partially recovered using Hoe heuristic
3712075 congestion windows recovered without slow start by DSACK
902 congestion windows recovered without slow start after partial ack
625 TCP data loss events
TCPLostRetransmit: 1
21 timeouts after SACK recovery
4547218 fast retransmits
7227549 forward retransmits
702 retransmits in slow start
1654 other TCP timeouts
40 SACK retransmits failed
49154 packets collapsed in receive queue due to low socket buffer
14237653 DSACKs sent for old packets
1 DSACKs sent for out of order packets
8963734 DSACKs received
121369 connections reset due to unexpected data
5968 connections reset due to early user close
34 connections aborted due to timeout
TCPSACKDiscard: 253
TCPDSACKIgnoredOld: 64
TCPDSACKIgnoredNoUndo: 12840
TCPSpuriousRTOs: 14
TCPSackShifted: 48580479
TCPSackMerged: 43924691
TCPSackShiftFallback: 159482792
TCPBacklogDrop: 521
TCPChallengeACK: 4858
TCPSYNChallenge: 53
IpExt:
InBcastPkts: 344
InOctets: -1991304967
OutOctets: 1560295186
InBcastOctets: 142592
________________________________________
From: Niels de Vos [ndevos at redhat.com]
Sent: Monday, January 26, 2015 4:37 AM
To: Peter Auyeung
Cc: gluster-users at gluster.org; gluster-devel at gluster.org
Subject: Re: [Gluster-devel] [Gluster-users] lockd: server not responding, timed out
On Mon, Jan 26, 2015 at 12:26:53AM +0000, Peter Auyeung wrote:
> Hi Niels,
>
> The question if we keep getting the lockd error even after restart and
> rebooted the NFS client..
This particular error would only occur when the NFS-server could not
register the nlockmgr RPC-program to rpcbind/portmapper. The most likely
scenario where this fails, is where there is an NFS-client (or service)
on the storage server that conflicts with the Gluster/NFS service.
If there are conflicting RPC services in rpcbind/portmapper, you may be
able check and remove those with the 'rpcinfo' command. Ports that are
listed in te output, but are not listed in netstat/ss are in used by
kernel services (like the lockd kernel module).
In order to restore the NLM function of Gluster/NFS, you can take these
steps:
1. ensure that there are no other NFS-services (server or client)
running on the Gluster storage server. Gluster/NFS should be the only
service which does some NFS on the server.
2. stop the rpcbind service
3. clear the rpcbind-cache (rm /var/lib/rpcbind/portmap.xdr)
4. start the rpcbind service
5. restart the Gluster/NFS service
In case your NFS-client got connected to the incorrect NLM service on
your storage server, you would need to unmount and mount the export
again.
Niels
>
> Peter
> ________________________________________
> From: Niels de Vos [ndevos at redhat.com]
> Sent: Saturday, January 24, 2015 3:26 AM
> To: Peter Auyeung
> Cc: gluster-users at gluster.org; gluster-devel at gluster.org
> Subject: Re: [Gluster-devel] [Gluster-users] lockd: server not responding, timed out
>
> On Fri, Jan 23, 2015 at 11:50:26PM +0000, Peter Auyeung wrote:
> > We have a 6 nodes gluster running ubuntu on xfs sharing gluster
> > volumes over NFS been running fine for 3 months.
> > We restarted glusterfs-server on one of the node and all NFS clients
> > start getting the " lockd: server not responding, timed out" on
> > /var/log/messages
> >
> > We are still able to read write but seems like process that require a
> > persistent file lock failed like database exports.
> >
> > We have an interim fix to remount the NFS with nolock option but need
> > to know why that is necessary all in a sudden after a service
> > glusterfs-server restart on one of the gluster node
>
> The cause that you need to mount wiht 'nolock' is that one server can
> only have one NLM-service active. The Linux NFS-client uses the 'lockd'
> kernel module, and the Gluster/NFS server provides its own lock manager.
> To be able to use a lock manager, it needs to be registered at
> rpcbind/portmapper. Only one lock manager can be registered at a time,
> the 2nd one that tries to register will fail. In case the NFS-client has
> registered the lockd kernel module as lock manager, any locking requests
> to the Gluster/NFS service will fail and you will see those messages in
> /var/log/messages.
>
> This is one of the main reasons why it is not advised to access volumes
> over NFS on a Gluster storage server. You should rather use the
> GlusterFS protocol for mounting volumes locally. (Or even better,
> seperate your storage servers from the application servers.)
>
> HTH,
> Niels
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users