[Gluster-users] [ovirt-users] Centos 7.1 failed to start glusterd after upgrading to ovirt 3.6
Kaushal M
kshlmster at gmail.com
Tue Nov 10 07:02:20 UTC 2015
On Mon, Nov 9, 2015 at 9:06 PM, Stefano Danzi <s.danzi at hawai.it> wrote:
> Here output from systemd-analyze critical-chain and systemd-analyze blame.
> I think that now glusterd start too early (before networking)
You are nearly right. GlusterD did start too early. GlusterD is
configured to start after network.target. But network.target in
systemd only guarantees that the network management stack is up; it
doesn't guarantee that the network devices have been configured and
are usable (Ref [1]). This means that when GlusterD starts, the
network is still not up and hence GlusterD will fail to resolve
bricks.
While we could start GlusterD after network-online.target, it would
break GlusterFS mounts configured in /etc/fstab with _netdev option.
Systemd automatically schedules _netdev mounts to be done after
network-online.target. (Ref [1] network-online.target). This could
allow the GlusterFS mounts to be done before GlusterD is up, causing
them to fail. This can be done using systemd-220 [2] which introduced
support for `x-systemd.requires` option for fstab, which can be used
to order mounts after specific services, but is not possible with el7
which has systemd-208.
[1]: https://wiki.freedesktop.org/www/Software/systemd/NetworkTarget/
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=812826
>
> [root at ovirt01 tmp]# systemd-analyze critical-chain
> The time after the unit is active or started is printed after the "@"
> character.
> The time the unit takes to start is printed after the "+" character.
>
> multi-user.target @17.148s
> └─ovirt-ha-agent.service @17.021s +127ms
> └─vdsmd.service @15.871s +1.148s
> └─vdsm-network.service @11.495s +4.373s
> └─libvirtd.service @11.238s +254ms
> └─iscsid.service @11.228s +8ms
> └─network.target @11.226s
> └─network.service @6.748s +4.476s
> └─iptables.service @6.630s +117ms
> └─basic.target @6.629s
> └─paths.target @6.629s
> └─brandbot.path @6.629s
> └─sysinit.target @6.615s
> └─systemd-update-utmp.service @6.610s +4ms
> └─auditd.service @6.450s +157ms
> └─systemd-tmpfiles-setup.service @6.369s +77ms
> └─rhel-import-state.service @6.277s +88ms
> └─local-fs.target @6.275s
> └─home-glusterfs-data.mount @5.805s +470ms
> └─home.mount @3.946s +1.836s
> └─systemd-fsck at dev-mapper-centos_ovirt01\x2dhome.service @3.937s +7ms
> └─dev-mapper-centos_ovirt01\x2dhome.device @3.936s
>
>
>
> [root at ovirt01 tmp]# systemd-analyze blame
> 4.476s network.service
> 4.373s vdsm-network.service
> 2.318s glusterd.service
> 2.076s postfix.service
> 1.836s home.mount
> 1.651s lvm2-monitor.service
> 1.258s lvm2-pvscan at 9:1.service
> 1.211s systemd-udev-settle.service
> 1.148s vdsmd.service
> 1.079s dmraid-activation.service
> 1.046s boot.mount
> 904ms kdump.service
> 779ms multipathd.service
> 657ms var-lib-nfs-rpc_pipefs.mount
> 590ms
> systemd-fsck at dev-disk-by\x2duuid-e185849f\x2d2c82\x2d4eb2\x2da215\x2d97340e90c93e.service
> 547ms tuned.service
> 481ms kmod-static-nodes.service
> 470ms home-glusterfs-data.mount
> 427ms home-glusterfs-engine.mount
> 422ms sys-kernel-debug.mount
> 411ms dev-hugepages.mount
> 411ms dev-mqueue.mount
> 278ms systemd-fsck-root.service
> 263ms systemd-readahead-replay.service
> 254ms libvirtd.service
> 243ms systemd-tmpfiles-setup-dev.service
> 216ms systemd-modules-load.service
> 209ms rhel-readonly.service
> 195ms wdmd.service
> 192ms sanlock.service
> 191ms gssproxy.service
> 186ms systemd-udev-trigger.service
> 157ms auditd.service
> 151ms plymouth-quit-wait.service
> 151ms plymouth-quit.service
> 132ms proc-fs-nfsd.mount
> 127ms ovirt-ha-agent.service
> 117ms iptables.service
> 110ms ovirt-ha-broker.service
> 96ms avahi-daemon.service
> 89ms systemd-udevd.service
> 88ms rhel-import-state.service
> 77ms systemd-tmpfiles-setup.service
> 71ms sysstat.service
> 71ms microcode.service
> 71ms chronyd.service
> 69ms systemd-readahead-collect.service
> 68ms systemd-sysctl.service
> 65ms systemd-logind.service
> 61ms rsyslog.service
> 58ms systemd-remount-fs.service
> 46ms rpcbind.service
> 46ms nfs-config.service
> 45ms systemd-tmpfiles-clean.service
> 41ms rhel-dmesg.service
> 37ms dev-mapper-centos_ovirt01\x2dswap.swap
> 29ms systemd-vconsole-setup.service
> 26ms plymouth-read-write.service
> 26ms systemd-random-seed.service
> 24ms netcf-transaction.service
> 22ms mdmonitor.service
> 20ms systemd-machined.service
> 14ms plymouth-start.service
> 12ms systemd-update-utmp-runlevel.service
> 11ms
> systemd-fsck at dev-mapper-centos_ovirt01\x2dglusterOVEngine.service
> 8ms iscsid.service
> 7ms systemd-fsck at dev-mapper-centos_ovirt01\x2dhome.service
> 7ms systemd-readahead-done.service
> 7ms
> systemd-fsck at dev-mapper-centos_ovirt01\x2dglusterOVData.service
> 6ms sys-fs-fuse-connections.mount
> 4ms systemd-update-utmp.service
> 4ms glusterfsd.service
> 4ms rpc-statd-notify.service
> 3ms iscsi-shutdown.service
> 3ms systemd-journal-flush.service
> 2ms sys-kernel-config.mount
> 1ms systemd-user-sessions.service
>
>
> Il 06/11/2015 9.27, Stefano Danzi ha scritto:
>>
>> Hi!
>> I have only one node (Test system) and I don't chage any ip address and
>> the entry is on /etc/hosts.
>> I think that now gluster start before networking
>>
>>
>> Il 06/11/2015 6.32, Atin Mukherjee ha scritto:
>>>>>
>>>>> [glusterd-store.c:4243:glusterd_resolve_all_bricks] 0-glusterd:
>>>>> resolve brick failed in restore
>>>
>>> The above log is the culprit here. Generally this function fails when
>>> GlusterD fails to resolve the associated host of a brick. Has any of the
>>> node undergone an IP change during the upgrade process?
>>>
>>> ~Atin
>>>
>>> On 11/06/2015 09:59 AM, Sahina Bose wrote:
>>>>
>>>> Did you upgrade all the nodes too?
>>>> Are some of your nodes not-reachable?
>>>>
>>>> Adding gluster-users for glusterd error.
>>>>
>>>> On 11/06/2015 12:00 AM, Stefano Danzi wrote:
>>>>>
>>>>> After upgrading oVirt from 3.5 to 3.6, glusterd fail to start when the
>>>>> host boot.
>>>>> Manual start of service after boot works fine.
>>>>>
>>>>> gluster log:
>>>>>
>>>>> [2015-11-04 13:37:55.360876] I [MSGID: 100030]
>>>>> [glusterfsd.c:2318:main] 0-/usr/sbin/glusterd: Started running
>>>>> /usr/sbin/glusterd version 3.7.5 (args: /usr/sbin/glusterd -p
>>>>> /var/run/glusterd.pid)
>>>>> [2015-11-04 13:37:55.447413] I [MSGID: 106478] [glusterd.c:1350:init]
>>>>> 0-management: Maximum allowed open file descriptors set to 65536
>>>>> [2015-11-04 13:37:55.447477] I [MSGID: 106479] [glusterd.c:1399:init]
>>>>> 0-management: Using /var/lib/glusterd as working directory
>>>>> [2015-11-04 13:37:55.464540] W [MSGID: 103071]
>>>>> [rdma.c:4592:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event
>>>>> channel creation failed [Nessun device corrisponde]
>>>>> [2015-11-04 13:37:55.464559] W [MSGID: 103055] [rdma.c:4899:init]
>>>>> 0-rdma.management: Failed to initialize IB Device
>>>>> [2015-11-04 13:37:55.464566] W
>>>>> [rpc-transport.c:359:rpc_transport_load] 0-rpc-transport: 'rdma'
>>>>> initialization failed
>>>>> [2015-11-04 13:37:55.464616] W [rpcsvc.c:1597:rpcsvc_transport_create]
>>>>> 0-rpc-service: cannot create listener, initing the transport failed
>>>>> [2015-11-04 13:37:55.464624] E [MSGID: 106243] [glusterd.c:1623:init]
>>>>> 0-management: creation of 1 listeners failed, continuing with
>>>>> succeeded transport
>>>>> [2015-11-04 13:37:57.663862] I [MSGID: 106513]
>>>>> [glusterd-store.c:2036:glusterd_restore_op_version] 0-glusterd:
>>>>> retrieved op-version: 30600
>>>>> [2015-11-04 13:37:58.284522] I [MSGID: 106194]
>>>>> [glusterd-store.c:3465:glusterd_store_retrieve_missed_snaps_list]
>>>>> 0-management: No missed snaps list.
>>>>> [2015-11-04 13:37:58.287477] E [MSGID: 106187]
>>>>> [glusterd-store.c:4243:glusterd_resolve_all_bricks] 0-glusterd:
>>>>> resolve brick failed in restore
>>>>> [2015-11-04 13:37:58.287505] E [MSGID: 101019]
>>>>> [xlator.c:428:xlator_init] 0-management: Initialization of volume
>>>>> 'management' failed, review your volfile again
>>>>> [2015-11-04 13:37:58.287513] E [graph.c:322:glusterfs_graph_init]
>>>>> 0-management: initializing translator failed
>>>>> [2015-11-04 13:37:58.287518] E [graph.c:661:glusterfs_graph_activate]
>>>>> 0-graph: init failed
>>>>> [2015-11-04 13:37:58.287799] W [glusterfsd.c:1236:cleanup_and_exit]
>>>>> (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xfd) [0x7f29b876524d]
>>>>> -->/usr/sbin/glusterd(glusterfs_process_volfp+0x126) [0x7f29b87650f6]
>>>>> -->/usr/sbin/glusterd(cleanup_and_exit+0x69) [0x7f29b87646d9] ) 0-:
>>>>> received signum (0), shutting down
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list