[Gluster-devel] [Gluster-users] lockd: server not responding, timed out

Peter Auyeung pauyeung at connexity.com
Thu Jan 29 01:48:53 UTC 2015


We were able to stop the lockd error after rebooted the server.

However still encountering another strange issue with sqlplus.

I sent another email to the group on it.

Thanks
Peter
________________________________________
From: gluster-users-bounces at gluster.org [gluster-users-bounces at gluster.org] on behalf of Peter Auyeung [pauyeung at connexity.com]
Sent: Tuesday, January 27, 2015 2:35 PM
To: Niels de Vos
Cc: gluster-users at gluster.org; gluster-devel at gluster.org
Subject: Re: [Gluster-users] [Gluster-devel]  lockd: server  not responding, timed out

Hi Niels,

I see no kernel NFS service running on the gluster node.

Here are the output

root at glusterprod001:~# rpcinfo
   program version netid     address                service    owner
    100000    4    tcp6      ::.0.111               portmapper superuser
    100000    3    tcp6      ::.0.111               portmapper superuser
    100000    4    udp6      ::.0.111               portmapper superuser
    100000    3    udp6      ::.0.111               portmapper superuser
    100000    4    tcp       0.0.0.0.0.111          portmapper superuser
    100000    3    tcp       0.0.0.0.0.111          portmapper superuser
    100000    2    tcp       0.0.0.0.0.111          portmapper superuser
    100000    4    udp       0.0.0.0.0.111          portmapper superuser
    100000    3    udp       0.0.0.0.0.111          portmapper superuser
    100000    2    udp       0.0.0.0.0.111          portmapper superuser
    100000    4    local     /run/rpcbind.sock      portmapper superuser
    100000    3    local     /run/rpcbind.sock      portmapper superuser
    100005    3    tcp       0.0.0.0.150.65         mountd     superuser
    100005    1    tcp       0.0.0.0.150.66         mountd     superuser
    100003    3    tcp       0.0.0.0.8.1            nfs        superuser
    100021    4    tcp       0.0.0.0.150.68         nlockmgr   superuser
    100227    3    tcp       0.0.0.0.8.1            -          superuser
    100021    1    udp       0.0.0.0.2.215          nlockmgr   superuser
    100021    1    tcp       0.0.0.0.2.217          nlockmgr   superuser
    100024    1    udp       0.0.0.0.136.211        status     105
    100024    1    tcp       0.0.0.0.170.60         status     105
    100024    1    udp6      ::.182.65              status     105
    100024    1    tcp6      ::.172.250             status     105

root at glusterprod001:~# ss
State       Recv-Q Send-Q                                        Local Address:Port                                            Peer Address:Port
ESTAB       0      0                                             10.101.165.61:1015                                           10.101.165.63:24007
ESTAB       0      0                                             10.101.165.61:936                                            10.101.165.61:49156
ESTAB       0      0                                             10.101.165.61:1012                                           10.101.165.66:24007
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.64:987
ESTAB       0      0                                             10.101.165.61:999                                            10.101.165.62:49153
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.66:988
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.62:834
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.63:998
ESTAB       0      0                                             10.101.165.61:912                                            10.101.165.65:49153
ESTAB       0      0                                                 127.0.0.1:982                                                127.0.0.1:24007
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.65:997
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.61:850
ESTAB       0      0                                             10.101.165.61:922                                            10.101.165.62:49154
ESTAB       0      0                                             10.101.165.61:896                                            10.101.165.65:49154
ESTAB       0      0                                             10.101.165.61:1010                                           10.101.165.61:24007
ESTAB       0      0                                             10.101.165.61:imaps                                          10.101.165.61:49156
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.63:981
ESTAB       0      0                                             10.101.165.61:930                                            10.101.165.64:49155
ESTAB       0      0                                             10.101.165.61:4379                                           10.101.165.65:44899
ESTAB       0      0                                             10.101.165.61:983                                            10.101.165.61:49157
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.64:1007
ESTAB       0      0                                             10.101.165.61:935                                            10.101.165.61:49155
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.62:900
ESTAB       0      0                                             10.101.165.97:nfs                                            10.102.69.226:861
ESTAB       0      0                                             10.101.165.61:4379                                           10.101.165.64:46473
ESTAB       0      0                                             10.101.165.61:860                                            10.101.165.65:49155
ESTAB       125    0                                             10.101.165.61:55453                                         10.101.161.104:4242
ESTAB       0      0                                             10.101.165.61:897                                            10.101.165.66:49153
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.61:imaps
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.62:999
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.66:ftps
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.62:958
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.65:976
ESTAB       0      0                                             10.101.165.61:24007                                         10.101.195.108:1023
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.66:982
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.61:936
ESTAB       0      0                                             10.101.165.61:49155                                         10.101.195.108:1005
ESTAB       0      0                                             10.101.165.61:855                                            10.101.165.64:49154
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.65:974
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.66:973
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.65:982
ESTAB       0      0                                             10.101.165.61:941                                            10.101.165.66:49154
ESTAB       0      0                                             10.101.165.61:ftps                                           10.101.165.64:49155
ESTAB       0      0                                             10.101.165.61:1004                                           10.101.165.66:49152
ESTAB       0      0                                             10.101.165.61:818                                            10.101.165.65:49152
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.65:996
ESTAB       0      0                                             10.101.165.61:891                                            10.101.165.65:49153
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.62:943
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.65:ftps
ESTAB       0      0                                             10.101.165.61:913                                            10.101.165.64:49154
ESTAB       0      0                                             10.101.165.61:996                                            10.101.165.66:49153
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.64:telnets
ESTAB       0      0                                             10.101.165.61:946                                            10.101.165.64:49154
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.166.51:1011
ESTAB       0      0                                             10.101.165.61:849                                            10.101.165.62:49152
ESTAB       0      0                                                 127.0.0.1:983                                                127.0.0.1:24007
ESTAB       0      0                                             10.101.165.61:24007                                          10.101.165.63:1021
ESTAB       0      0                                             10.101.165.61:893                                            10.101.165.63:49153
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.64:ftps
ESTAB       0      0                                             10.101.165.61:59797                                          10.101.165.64:4379
ESTAB       0      0                                             10.101.165.61:928                                            10.101.165.65:49155
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.65:ftps-data
ESTAB       0      0                                             10.101.165.61:851                                            10.101.165.63:49152
ESTAB       0      0                                             10.101.165.61:4379                                           10.101.165.63:46734
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.61:858
ESTAB       0      0                                             10.101.165.61:817                                            10.101.165.63:49153
ESTAB       0      0                                             10.101.165.61:4379                                           10.101.165.62:53373
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.63:ircs
ESTAB       0      0                                             10.101.165.61:24007                                          10.101.165.61:1009
ESTAB       0      0                                             10.101.165.61:927                                            10.101.165.66:49155
ESTAB       0      0                                             10.101.165.61:857                                            10.101.165.62:49154
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.62:824
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.61:914
ESTAB       0      0                                             10.101.165.61:861                                            10.101.165.64:49155
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.62:ftps
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.64:ircs
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.166.51:1017
ESTAB       0      0                                             10.101.165.61:939                                            10.101.165.61:49154
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.63:1011
ESTAB       0      0                                             10.101.165.61:856                                            10.101.165.63:49154
ESTAB       0      0                                             10.101.165.61:24007                                          10.101.165.61:1008
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.63:997
ESTAB       0      0                                             10.101.165.61:934                                            10.101.165.62:49155
ESTAB       0      0                                             10.101.165.61:822                                            10.101.165.64:49153
ESTAB       0      0                                             10.101.165.61:55997                                          10.101.165.66:4379
ESTAB       0      0                                             10.101.165.61:24007                                          10.101.166.51:1023
ESTAB       0      0                                             10.101.165.61:929                                            10.101.165.66:49152
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.66:966
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.61:939
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.64:984
ESTAB       0      0                                             10.101.165.61:49157                                         10.101.195.108:1006
ESTAB       0      0                                             10.101.165.61:41681                                          10.101.165.63:4379
ESTAB       0      0                                             10.101.165.61:997                                            10.101.165.64:49153
ESTAB       0      0                                             10.101.165.61:850                                            10.101.165.61:49155
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.66:987
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.63:976
ESTAB       0      0                                             10.101.165.61:894                                            10.101.165.64:49153
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.66:969
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.63:983
ESTAB       0      0                                             10.101.165.61:43557                                          10.101.165.65:4379
ESTAB       0      0                                             10.101.165.61:1017                                           10.101.165.61:49155
ESTAB       0      0                                             10.101.165.61:1013                                           10.101.165.65:24007
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.63:985
ESTAB       0      0                                             10.101.165.61:918                                            10.101.165.63:49154
ESTAB       0      0                                             10.101.165.61:1007                                           10.101.165.63:49152
ESTAB       0      0                                             10.101.165.61:24007                                          10.101.165.61:1011
ESTAB       0      0                                             10.101.165.61:932                                            10.101.165.63:49155
ESTAB       0      0                                             10.101.165.61:1008                                           10.101.165.61:24007
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.64:986
ESTAB       0      0                                             10.101.165.61:914                                            10.101.165.61:49155
ESTAB       0      0                                             10.101.165.61:981                                            10.101.165.63:49154
ESTAB       0      0                                                 127.0.0.1:24007                                              127.0.0.1:981
ESTAB       0      0                                             10.101.165.61:1018                                           10.101.165.64:24007
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.63:982
ESTAB       0      0                                             10.101.165.61:816                                            10.101.165.66:49153
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.66:customs
ESTAB       0      0                                             10.101.165.61:998                                            10.101.165.63:49153
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.66:981
ESTAB       0      0                                             10.101.165.61:telnets                                        10.101.165.62:49155
ESTAB       0      0                                             10.101.165.61:925                                            10.101.165.61:49157
ESTAB       0      0                                             10.101.165.61:1014                                           10.101.165.62:24007
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.66:962
ESTAB       0      6800                                          10.101.165.61:ssh                                             10.40.152.72:62912
ESTAB       0      0                                             10.101.165.61:24007                                          10.101.165.62:1023
ESTAB       0      0                                                 127.0.0.1:24007                                              127.0.0.1:982
ESTAB       0      0                                             10.101.165.61:852                                            10.101.165.61:49154
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.66:970
ESTAB       0      0                                             10.101.165.61:824                                            10.101.165.64:49152
ESTAB       0      0                                             10.101.165.61:991                                            10.101.165.63:49155
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.62:987
ESTAB       0      0                                             10.101.165.61:24007                                          10.101.165.61:1010
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.61:852
ESTAB       0      0                                             10.101.165.61:938                                            10.101.165.62:49152
ESTAB       0      0                                             10.101.165.61:823                                            10.101.165.66:49152
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.63:973
ESTAB       0      0                                             10.101.165.61:920                                            10.101.165.62:49153
ESTAB       0      0                                             10.101.165.61:854                                            10.101.165.65:49154
ESTAB       0      0                                             10.101.165.61:40487                                          10.101.165.62:4379
ESTAB       0      0                                             10.101.165.61:858                                            10.101.165.61:49157
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.64:1008
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.64:991
ESTAB       0      0                                             10.101.165.61:937                                            10.101.165.64:49152
ESTAB       0      0                                             10.101.165.61:24007                                          10.101.165.61:1002
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.61:1017
ESTAB       0      0                                             10.101.165.61:923                                            10.101.165.64:49153
ESTAB       0      0                                             10.101.165.61:49156                                         10.101.195.108:customs
ESTAB       0      0                                             10.101.165.61:988                                            10.101.165.66:49155
ESTAB       0      0                                             10.101.165.61:862                                            10.101.165.63:49155
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.61:1023
ESTAB       0      0                                             10.101.165.61:1011                                           10.101.165.61:24007
ESTAB       0      0                                             10.101.165.61:1023                                           10.101.165.61:49154
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.64:imaps
ESTAB       0      0                                             10.101.165.61:1002                                           10.101.165.61:24007
ESTAB       0      0                                             10.101.165.61:1006                                           10.101.165.62:49152
ESTAB       0      0                                                 127.0.0.1:24007                                              127.0.0.1:1022
ESTAB       0      0                                             10.101.165.61:944                                            10.101.165.65:49154
ESTAB       0      0                                             10.101.165.61:916                                            10.101.165.63:49153
ESTAB       0      0                                             10.101.165.61:ftps-data                                        10.101.165.65:49155
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.62:924
ESTAB       0      0                                             10.101.165.61:864                                            10.101.165.61:49156
ESTAB       0      0                                             10.101.165.61:1000                                           10.101.165.65:49153
ESTAB       0      0                                             10.101.165.61:811                                            10.101.165.65:49153
ESTAB       0      0                                             10.101.165.61:931                                            10.101.165.65:49152
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.65:988
ESTAB       0      0                                             10.101.165.61:982                                            10.101.165.62:49154
ESTAB       0      0                                                 127.0.0.1:1022                                               127.0.0.1:24007
ESTAB       0      0                                             10.101.165.61:892                                            10.101.165.66:49153
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.61:935
ESTAB       0      0                                             10.101.165.61:1009                                           10.101.165.61:24007
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.62:883
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.62:812
ESTAB       0      0                                             10.101.165.61:49154                                         10.101.195.108:1004
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.61:983
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.61:925
ESTAB       0      0                                             10.101.165.61:4379                                           10.101.165.66:35729
ESTAB       0      0                                                 127.0.0.1:981                                                127.0.0.1:24007
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.63:975
ESTAB       0      0                                             10.101.165.61:859                                            10.101.165.66:49155
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.63:977
ESTAB       0      0                                             10.101.165.61:933                                            10.101.165.63:49152
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.66:980
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.66:986
ESTAB       0      0                                             10.101.165.61:863                                            10.101.165.62:49155
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.62:891
ESTAB       0      0                                             10.101.165.61:24007                                          10.101.165.66:1022
ESTAB       0      0                                                 127.0.0.1:24007                                              127.0.0.1:983
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.64:996
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.65:customs
ESTAB       0      0                                             10.101.165.61:815                                            10.101.165.62:49153
ESTAB       0      0                                                 127.0.0.1:24007                                              127.0.0.1:1023
ESTAB       0      0                                             10.101.165.61:49154                                          10.101.165.65:979
ESTAB       0      0                                             10.101.165.61:853                                            10.101.165.66:49154
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.62:817
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.64:1004
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.64:pop3s
ESTAB       0      0                                             10.101.165.61:898                                            10.101.165.66:49154
ESTAB       0      0                                             10.101.165.61:24007                                          10.101.165.65:1021
ESTAB       0      0                                                 127.0.0.1:1023                                               127.0.0.1:24007
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.63:ftps-data
ESTAB       0      0                                             10.101.165.61:1003                                           10.101.165.65:49152
ESTAB       0      0                                             10.101.165.61:24007                                          10.101.165.64:1019
ESTAB       0      0                                             10.101.165.61:49157                                          10.101.165.65:980
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.65:pop3s
ESTAB       0      0                                             10.101.165.61:49155                                          10.101.165.65:978
ESTAB       0      0                                             10.101.165.61:1005                                           10.101.165.64:49152
ESTAB       0      0                                             10.101.165.61:895                                            10.101.165.62:49153
ESTAB       0      0                                             10.101.165.61:49156                                          10.101.165.61:864

root at glusterprod001:~# netstat -s
Ip:
    3836362639 total packets received
    0 forwarded
    0 incoming packets discarded
    3836362639 incoming packets delivered
    2715649214 requests sent out
    1 outgoing packets dropped
    5 dropped because of missing route
Icmp:
    73531 ICMP messages received
    11 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 212
        echo requests: 73317
        echo replies: 2
    73700 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 381
        echo request: 2
        echo replies: 73317
IcmpMsg:
        InType0: 2
        InType3: 212
        InType8: 73317
        OutType0: 73317
        OutType3: 381
        OutType8: 2
Tcp:
    3414520 active connections openings
    50916 passive connection openings
    2236146 failed connection attempts
    22393 connection resets received
    207 connections established
    3831070976 segments received
    2012419479 segments send out
    11779093 segments retransmited
    46 bad segments received.
    2530424 resets sent
Udp:
    5217439 packets received
    348 packets to unknown port received.
    0 packet receive errors
    8087732 packets sent
UdpLite:
TcpExt:
    2 resets received for embryonic SYN_RECV sockets
    1050 packets pruned from receive queue because of socket buffer overrun
    204156 TCP sockets finished time wait in fast timer
    12026710 delayed acks sent
    18993 delayed acks further delayed because of locked socket
    Quick ack mode was activated 14237571 times
    24958868 packets directly queued to recvmsg prequeue.
    27395 bytes directly in process context from backlog
    1731434 bytes directly received in process context from prequeue
    935889540 packet headers predicted
    111 packets header predicted and directly queued to user
    933778742 acknowledgments not containing data payload received
    2389068782 predicted acknowledgments
    6497616 times recovered from packet loss by selective acknowledgements
    219 bad SACK blocks received
    Detected reordering 14 times using FACK
    Detected reordering 29211 times using SACK
    Detected reordering 3704 times using time stamp
    188312 congestion windows fully recovered without slow start
    262526 congestion windows partially recovered using Hoe heuristic
    3712075 congestion windows recovered without slow start by DSACK
    902 congestion windows recovered without slow start after partial ack
    625 TCP data loss events
    TCPLostRetransmit: 1
    21 timeouts after SACK recovery
    4547218 fast retransmits
    7227549 forward retransmits
    702 retransmits in slow start
    1654 other TCP timeouts
    40 SACK retransmits failed
    49154 packets collapsed in receive queue due to low socket buffer
    14237653 DSACKs sent for old packets
    1 DSACKs sent for out of order packets
    8963734 DSACKs received
    121369 connections reset due to unexpected data
    5968 connections reset due to early user close
    34 connections aborted due to timeout
    TCPSACKDiscard: 253
    TCPDSACKIgnoredOld: 64
    TCPDSACKIgnoredNoUndo: 12840
    TCPSpuriousRTOs: 14
    TCPSackShifted: 48580479
    TCPSackMerged: 43924691
    TCPSackShiftFallback: 159482792
    TCPBacklogDrop: 521
    TCPChallengeACK: 4858
    TCPSYNChallenge: 53
IpExt:
    InBcastPkts: 344
    InOctets: -1991304967
    OutOctets: 1560295186
    InBcastOctets: 142592
________________________________________
From: Niels de Vos [ndevos at redhat.com]
Sent: Monday, January 26, 2015 4:37 AM
To: Peter Auyeung
Cc: gluster-users at gluster.org; gluster-devel at gluster.org
Subject: Re: [Gluster-devel] [Gluster-users] lockd: server  not responding, timed out

On Mon, Jan 26, 2015 at 12:26:53AM +0000, Peter Auyeung wrote:
> Hi Niels,
>
> The question if we keep getting the lockd error even after restart and
> rebooted the NFS client..

This particular error would only occur when the NFS-server could not
register the nlockmgr RPC-program to rpcbind/portmapper. The most likely
scenario where this fails, is where there is an NFS-client (or service)
on the storage server that conflicts with the Gluster/NFS service.

If there are conflicting RPC services in rpcbind/portmapper, you may be
able check and remove those with the 'rpcinfo' command. Ports that are
listed in te output, but are not listed in netstat/ss are in used by
kernel services (like the lockd kernel module).

In order to restore the NLM function of Gluster/NFS, you can take these
steps:

1. ensure that there are no other NFS-services (server or client)
   running on the Gluster storage server. Gluster/NFS should be the only
   service which does some NFS on the server.
2. stop the rpcbind service
3. clear the rpcbind-cache (rm /var/lib/rpcbind/portmap.xdr)
4. start the rpcbind service
5. restart the Gluster/NFS service


In case your NFS-client got connected to the incorrect NLM service on
your storage server, you would need to unmount and mount the export
again.

Niels

>
> Peter
> ________________________________________
> From: Niels de Vos [ndevos at redhat.com]
> Sent: Saturday, January 24, 2015 3:26 AM
> To: Peter Auyeung
> Cc: gluster-users at gluster.org; gluster-devel at gluster.org
> Subject: Re: [Gluster-devel] [Gluster-users] lockd: server  not responding, timed out
>
> On Fri, Jan 23, 2015 at 11:50:26PM +0000, Peter Auyeung wrote:
> > We have a 6 nodes gluster running ubuntu on xfs sharing gluster
> > volumes over NFS been running fine for 3 months.
> > We restarted glusterfs-server on one of the node and all NFS clients
> > start getting the " lockd: server  not responding, timed out" on
> > /var/log/messages
> >
> > We are still able to read write but seems like process that require a
> > persistent file lock failed like database exports.
> >
> > We have an interim fix to remount the NFS with nolock option but need
> > to know why that is necessary all in a sudden after a service
> > glusterfs-server restart on one of the gluster node
>
> The cause that you need to mount wiht 'nolock' is that one server can
> only have one NLM-service active. The Linux NFS-client uses the 'lockd'
> kernel module, and the Gluster/NFS server provides its own lock manager.
> To be able to use a lock manager, it needs to be registered at
> rpcbind/portmapper. Only one lock manager can be registered at a time,
> the 2nd one that tries to register will fail. In case the NFS-client has
> registered the lockd kernel module as lock manager, any locking requests
> to the Gluster/NFS service will fail and you will see those messages in
> /var/log/messages.
>
> This is one of the main reasons why it is not advised to access volumes
> over NFS on a Gluster storage server. You should rather use the
> GlusterFS protocol for mounting volumes locally. (Or even better,
> seperate your storage servers from the application servers.)
>
> HTH,
> Niels
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


More information about the Gluster-devel mailing list