[Gluster-users] After upgrade from 3.5 to 3.7 gluster local NFS is not starting on one of the servers

Yaroslav Molochko onorua at gmail.com
Mon Sep 14 17:09:03 UTC 2015


any ideas?

2015-09-13 20:41 GMT+08:00 Yaroslav Molochko <onorua at gmail.com>:

> So, I've done:
> root at PSC01SERV008:/var/log# tail -f syslog | grep -Ev
> 'docker|kubelet|kube-proxy'
> Sep 13 12:18:16 psc01serv008 systemd[1]: Stopped GlusterFS an clustered
> file-system server.
> Sep 13 12:19:21 psc01serv008 systemd[1]: Reloading.
> Sep 13 12:19:35 psc01serv008 systemd[1]: message repeated 3 times: [
> Reloading.]
> Sep 13 12:20:10 psc01serv008 systemd[1]: Starting GlusterFS an clustered
> file-system server...
> Sep 13 12:20:12 psc01serv008 systemd[1]: Started GlusterFS an clustered
> file-system server.
>
> and stopped glusterfs, it said it was stopped but processes where there,
> and I killed them manually, maybe something wrong with the system unit
> file, but it was working with 3.5, so don't know. Then I disabled my
> "homemade" glusterfs service and enabled /etc/init.d/glusterfs-server and
> it got the same problem, I could not restart the glusterfs processes from
> "init" doesn't matter what init file I try.
> And when I kill processes by hands, it starts up but, as you can see there
> is no reports of any problems with starting up the NFS or any blocking
> port. There is no firewalld running on my host, and the problem is that I
> have 2 hosts identical to peer with, one is working and one is not.
> dmesg is attached as well as my "handmade" glusterfs systemd service, just
> in case I start it wrongly.
>
> 2015-09-13 19:00 GMT+08:00 Soumya Koduri <skoduri at redhat.com>:
>
>>
>>
>> On 09/13/2015 09:38 AM, Yaroslav Molochko wrote:
>>
>>> I wish this could be that simple:
>>> root at PSC01SERV008:/var/lib# netstat -nap | grep 38465
>>> root at PSC01SERV008:/var/lib# ss -n  | grep 38465
>>> root at PSC01SERV008:/var/lib#
>>>
>>> 2015-09-13 1:34 GMT+08:00 Atin Mukherjee <atin.mukherjee83 at gmail.com
>>> <mailto:atin.mukherjee83 at gmail.com>>:
>>>
>>>     By any chance is your Gluster NFS server is already running? Output
>>>     of netstat -nap | grep 38465 might give some clue?
>>>
>>>     -Atin
>>>     Sent from one plus one
>>>
>>>     On Sep 12, 2015 10:54 PM, "Yaroslav Molochko" <onorua at gmail.com
>>>     <mailto:onorua at gmail.com>> wrote:
>>>
>>>         Hello,
>>>
>>>         I have a problem reported in logs:
>>>         ==================
>>>         [2015-09-12 13:56:06.271644] I [MSGID: 100030]
>>>         [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running
>>>         /usr/sbin/glusterfs version 3.7.4 (args: /usr/sbin/glusterfs -s
>>>         localhost --volfile-id gluster/nfs -p
>>>         /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log
>>>         -S /var/run/gluster/cb186678589f28e74c67da70fd06e736.socket)
>>>         [2015-09-12 13:56:06.277921] I [MSGID: 101190]
>>>         [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started
>>>         thread with index 1
>>>         [2015-09-12 13:56:07.284888] I
>>>         [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service:
>>>         Configured rpc.outstanding-rpc-limit with value 16
>>>         [2015-09-12 13:56:07.292484] W [MSGID: 112153]
>>>         [mount3.c:3910:mnt3svc_init] 0-nfs-mount: Exports auth has been
>>>         disabled!
>>>         [2015-09-12 13:56:07.294357] E
>>>         [rpcsvc.c:1370:rpcsvc_program_register_portmap] 0-rpc-service:
>>>         Could not register with portmap 100005 3 38465
>>>
>>
>> Port registration failed. Could you check '/var/log/messages' and dmesg
>> to see if there are any errors logged? Is firewalld running on your system.
>> Verify if the port is open to be used.
>>
>> Thanks,
>> Soumya
>>
>>>         [2015-09-12 13:56:07.294398] E [MSGID: 112088]
>>>         [nfs.c:341:nfs_init_versions] 0-nfs: Required program  MOUNT3
>>>         registration failed
>>>         [2015-09-12 13:56:07.294413] E [MSGID: 112109] [nfs.c:1482:init]
>>>         0-nfs: Failed to initialize protocols
>>>         [2015-09-12 13:56:07.294426] E [MSGID: 101019]
>>>         [xlator.c:428:xlator_init] 0-nfs-server: Initialization of
>>>         volume 'nfs-server' failed, review your volfile again
>>>         [2015-09-12 13:56:07.294438] E
>>>         [graph.c:322:glusterfs_graph_init] 0-nfs-server: initializing
>>>         translator failed
>>>         [2015-09-12 13:56:07.294448] E
>>>         [graph.c:661:glusterfs_graph_activate] 0-graph: init failed
>>>         [2015-09-12 13:56:07.294781] W
>>>         [glusterfsd.c:1219:cleanup_and_exit]
>>>         (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x11a) [0x7fbe9c754b7a]
>>>         -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x123)
>>>         [0x7fbe9c74fcb3] -->/usr/sbin/glusterfs(cleanup_and_exit+0x59)
>>>         [0x7fbe9c74f329] ) 0-: received signum (0), shutting down
>>>         ===================
>>>
>>>         I've checked the page:
>>>
>>> http://www.gluster.org/community/documentation/index.php/Gluster_3.1:_NFS_Frequently_Asked_Questions
>>>
>>>         I've found report in RedHat that it's necessary to remove -w
>>>         from rpcbind becuse some times it causes problems.
>>>         I did all that but still no luck on one of the servers, what is
>>>         interesting, the other server (peered) is working fine without
>>>         any problems.
>>>
>>>         root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status nfs
>>>         ● nfs.service
>>>             Loaded: not-found (Reason: No such file or directory)
>>>             Active: inactive (dead)
>>>
>>>         root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status
>>> rpcbind
>>>         ● rpcbind.service - RPC bind portmap service
>>>             Loaded: loaded (/etc/systemd/system/rpcbind.service;
>>>         enabled; vendor preset: enabled)
>>>            Drop-In: /run/systemd/generator/rpcbind.service.d
>>>                     └─50-rpcbind-$portmap.conf
>>>             Active: active (running) since Sat 2015-09-12 13:55:07 UTC;
>>>         6min ago
>>>           Main PID: 9796 (rpcbind)
>>>             Memory: 428.0K
>>>             CGroup: /system.slice/rpcbind.service
>>>                     └─9796 /sbin/rpcbind
>>>
>>>         Sep 12 13:55:07 PSC01SERV008 systemd[1]: Starting RPC bind
>>>         portmap service...
>>>         Sep 12 13:55:07 PSC01SERV008 systemd[1]: Started RPC bind
>>>         portmap service.
>>>         root at PSC01SERV008:/var/lib/glusterd/nfs# rpcinfo -p
>>>             program vers proto   port  service
>>>              100000    4   tcp    111  portmapper
>>>              100000    3   tcp    111  portmapper
>>>              100000    2   tcp    111  portmapper
>>>              100000    4   udp    111  portmapper
>>>              100000    3   udp    111  portmapper
>>>              100000    2   udp    111  portmapper
>>>
>>>         I've tried to reinstall it agaon and again - but there is no
>>> luck.
>>>
>>>         What I have:
>>>         cat /etc/lsb-release
>>>         DISTRIB_ID=Ubuntu
>>>         DISTRIB_RELEASE=15.04
>>>         DISTRIB_CODENAME=vivid
>>>         DISTRIB_DESCRIPTION="Ubuntu 15.04"
>>>
>>>         ii  glusterfs-client
>>>         3.7.4-ubuntu1~vivid1              amd64        clustered
>>>         file-system (client package)
>>>         ii  glusterfs-common
>>>         3.7.4-ubuntu1~vivid1              amd64        GlusterFS common
>>>         libraries and translator modules
>>>         ii  glusterfs-server
>>>         3.7.4-ubuntu1~vivid1              amd64        clustered
>>>         file-system (server package)
>>>
>>>         What else can I check? How can I fix it, what is most important
>>> :)
>>>         Thanks in advance!
>>>
>>>         _______________________________________________
>>>         Gluster-users mailing list
>>>         Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>         http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150915/b6d758a3/attachment.html>


More information about the Gluster-users mailing list