[Bugs] [Bug 1672318] "failed to fetch volume file" when trying to activate host in DC with glusterfs 3.12 domains
bugzilla at redhat.com
bugzilla at redhat.com
Tue Apr 9 10:43:38 UTC 2019
https://bugzilla.redhat.com/show_bug.cgi?id=1672318
Netbulae <info at netbulae.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Flags|needinfo?(info at netbulae.com |
|) |
--- Comment #28 from Netbulae <info at netbulae.com> ---
(In reply to Atin Mukherjee from comment #27)
I tried the sequence you posted but currently we have *.16 and ssd5 as the
master storage domain as I removed ssd9
Also upgraded to ovirt node 4.3.3 rc2 because of several fixes.
But I don't get the 'failed to fetch volume file' this way, it appears the
backup volfile option is not working:
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:10:09,186::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/sbin/gluster --mode=script volume info --remote-host=*.*.*.16 ssd5 --xml (cwd None)
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:10:09,297::commands::219::root::(execCmd) FAILED: <err> = ''; <rc> = 1
>MainProcess|jsonrpc/6::ERROR::2019-04-09 12:10:09,297::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in volumeInfo
>Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 101, in wrapper
> res = func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 529, in volumeInfo
> xmltree = _execGlusterXml(command)
> File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 131, in _execGlusterXml
> return _getTree(rc, out, err)
> File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 112, in _getTree
> raise ge.GlusterCmdExecFailedException(rc, out, err)
>GlusterCmdExecFailedException: Command execution failed: rc=1 out='Connection failed. Please check if gluster daemon is operational.\n' err=''
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:10:09,298::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call mount with (<vdsm.supervdsm_server._SuperVdsm object at 0x7f10804e2990>, u'*.*.*.16:ssd5', u'/rhev/data-center/mnt/glusterSD/*.*.*.16:ssd5') {'vfstype': u'glusterfs', 'mntOpts': u'backup-volfile-servers=1*.*.*.15:*.*.*.14', 'cgroup': 'vdsm-glusterfs'}
When I leave the glusterfs process running, I also don't get it anymore in
4.3.3, but I get "'Error : Request timed out\n' err=''" and "MountError: (32,
';umount: /rhev/data-center/mnt/glusterSD/*.*.*.15:ssd4: mountpoint not
found\n')"
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:25:54,918::commands::219::root::(execCmd) FAILED: <err> = 'Running scope as unit run-10178.scope.\nMount failed. Please check the log file for more details.\n'; <rc> = 1
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:25:54,918::logutils::319::root::(_report_stats) ThreadedHandler is ok in the last 765 seconds (max pending: 2)
>MainProcess|jsonrpc/6::ERROR::2019-04-09 12:25:54,918::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in mount
>Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 101, in wrapper
> res = func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 143, in mount
> cgroup=cgroup)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 277, in _mount
> _runcmd(cmd)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 305, in _runcmd
> raise MountError(rc, b";".join((out, err)))
>MountError: (1, ';Running scope as unit run-10178.scope.\nMount failed. Please check the log file for more details.\n')
>MainProcess|jsonrpc/1::DEBUG::2019-04-09 12:30:49,720::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call network_caps with () {}
>MainProcess|jsonrpc/1::DEBUG::2019-04-09 12:30:49,720::logutils::319::root::(_report_stats) ThreadedHandler is ok in the last 294 seconds (max pending: 2)
>MainProcess|jsonrpc/1::DEBUG::2019-04-09 12:30:49,802::routes::110::root::(get_gateway) The gateway *.*.*.3 is duplicated for the device ovirtmgmt
>MainProcess|jsonrpc/1::DEBUG::2019-04-09 12:30:49,807::routes::110::root::(get_gateway) The gateway *.*.*.3 is duplicated for the device ovirtmgmt
>MainProcess|jsonrpc/1::DEBUG::2019-04-09 12:30:49,809::cmdutils::133::root::(exec_cmd) /sbin/tc qdisc show (cwd None)
>MainProcess|jsonrpc/1::DEBUG::2019-04-09 12:30:49,860::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
>MainProcess|jsonrpc/1::DEBUG::2019-04-09 12:30:49,866::cmdutils::133::root::(exec_cmd) /sbin/tc class show dev bond0 classid 1389:1388 (cwd None)
>MainProcess|jsonrpc/1::DEBUG::2019-04-09 12:30:49,919::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
>MainProcess|jsonrpc/1::DEBUG::2019-04-09 12:30:50,005::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface
>MainProcess|jsonrpc/1::DEBUG::2019-04-09 12:30:50,006::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
>MainProcess|jsonrpc/1::DEBUG::2019-04-09 12:30:50,038::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
>
>MainProcess|jsonrpc/2::DEBUG::2019-04-09 12:37:02,472::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/umount -f -l /rhev/data-center/mnt/ovirt.netbulae.mgmt:_var_lib_exports_iso (cwd None)
>MainProcess|jsonrpc/2::DEBUG::2019-04-09 12:37:02,502::commands::219::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>MainProcess|jsonrpc/2::DEBUG::2019-04-09 12:37:02,503::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return umount with None
>MainProcess|jsonrpc/2::DEBUG::2019-04-09 12:37:02,529::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call umount with (<vdsm.supervdsm_server._SuperVdsm object at 0x7f10804e2990>, u'/rhev/data-center/mnt/*.*.*.15:_data_ovirt') >{'force': True, 'lazy': True, 'freeloop': False}
>MainProcess|jsonrpc/2::DEBUG::2019-04-09 12:37:02,529::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/umount -f -l /rhev/data-center/mnt/*.*.*.15:_data_ovirt (cwd None)
>MainProcess|jsonrpc/2::DEBUG::2019-04-09 12:37:02,595::commands::219::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>MainProcess|jsonrpc/2::DEBUG::2019-04-09 12:37:02,596::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return umount with None
>MainProcess|jsonrpc/2::DEBUG::2019-04-09 12:37:02,738::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call hbaRescan with (<vdsm.supervdsm_server._SuperVdsm object at 0x7f10804e2990>,) {}
>MainProcess|jsonrpc/2::DEBUG::2019-04-09 12:37:02,738::commands::198::storage.HBA::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/libexec/vdsm/fc-scan (cwd None)
>MainProcess|jsonrpc/2::DEBUG::2019-04-09 12:37:02,804::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return hbaRescan with None
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,841::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call umount with (<vdsm.supervdsm_server._SuperVdsm object at 0x7f10804e2990>, u'/rhev/data-center/mnt/glusterSD/*.*.*.15:ssd4') {'force': True, 'lazy': True, 'freeloop': False}
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,841::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/umount -f -l /rhev/data-center/mnt/glusterSD/*.*.*.15:ssd4 (cwd None)
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,849::commands::219::root::(execCmd) FAILED: <err> = 'umount: /rhev/data-center/mnt/glusterSD/*.*.*.15:ssd4: mountpoint not found\n'; <rc> = 32
>MainProcess|jsonrpc/6::ERROR::2019-04-09 12:37:02,850::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in umount
>Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 101, in wrapper
> res = func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 147, in umount
> mount._umount(fs_file, force=force, lazy=lazy, freeloop=freeloop)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 296, in _umount
> _runcmd(cmd)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 305, in _runcmd
> raise MountError(rc, b";".join((out, err)))
>MountError: (32, ';umount: /rhev/data-center/mnt/glusterSD/*.*.*.15:ssd4: mountpoint not found\n')
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,850::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call umount with (<vdsm.supervdsm_server._SuperVdsm object at 0x7f10804e2990>, u'/rhev/data-center/mnt/glusterSD/*.*.*.16:ssd5') {'force': True, 'lazy': True, 'freeloop': False}
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,850::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/umount -f -l /rhev/data-center/mnt/glusterSD/*.*.*.16:ssd5 (cwd None)
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,875::commands::219::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,876::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return umount with None
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,889::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call umount with (<vdsm.supervdsm_server._SuperVdsm object at 0x7f10804e2990>, u'/rhev/data-center/mnt/glusterSD/*.*.*.14:ssd6') {'force': True, 'lazy': True, 'freeloop': False}
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,889::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/umount -f -l /rhev/data-center/mnt/glusterSD/*.*.*.14:ssd6 (cwd None)
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,895::commands::219::root::(execCmd) FAILED: <err> = 'umount: /rhev/data-center/mnt/glusterSD/*.*.*.14:ssd6: mountpoint not found\n'; <rc> = 32
>MainProcess|jsonrpc/6::ERROR::2019-04-09 12:37:02,895::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in umount
>Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 101, in wrapper
> res = func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 147, in umount
> mount._umount(fs_file, force=force, lazy=lazy, freeloop=freeloop)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 296, in _umount
> _runcmd(cmd)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 305, in _runcmd
> raise MountError(rc, b";".join((out, err)))
>MountError: (32, ';umount: /rhev/data-center/mnt/glusterSD/*.*.*.14:ssd6: mountpoint not found\n')
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,895::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call umount with (<vdsm.supervdsm_server._SuperVdsm object at 0x7f10804e2990>, u'/rhev/data-center/mnt/glusterSD/*.*.*.14:_hdd2') {'force': True, 'lazy': True, 'freeloop': False}
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,896::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/umount -f -l /rhev/data-center/mnt/glusterSD/*.*.*.14:_hdd2 (cwd None)
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,902::commands::219::root::(execCmd) FAILED: <err> = 'umount: /rhev/data-center/mnt/glusterSD/*.*.*.14:_hdd2: mountpoint not found\n'; <rc> = 32
>MainProcess|jsonrpc/6::ERROR::2019-04-09 12:37:02,902::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in umount
>Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 101, in wrapper
> res = func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 147, in umount
> mount._umount(fs_file, force=force, lazy=lazy, freeloop=freeloop)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 296, in _umount
> _runcmd(cmd)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 305, in _runcmd
> raise MountError(rc, b";".join((out, err)))
>MountError: (32, ';umount: /rhev/data-center/mnt/glusterSD/*.*.*.14:_hdd2: mountpoint not found\n')
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,903::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call umount with (<vdsm.supervdsm_server._SuperVdsm object at 0x7f10804e2990>, u'/rhev/data-center/mnt/glusterSD/*.*.*.14:_ssd3') {'force': True, 'lazy': True, 'freeloop': False}
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,903::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/umount -f -l /rhev/data-center/mnt/glusterSD/*.*.*.14:_ssd3 (cwd None)
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,910::commands::219::root::(execCmd) FAILED: <err> = 'umount: /rhev/data-center/mnt/glusterSD/*.*.*.14:_ssd3: mountpoint not found\n'; <rc> = 32
>MainProcess|jsonrpc/6::ERROR::2019-04-09 12:37:02,910::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in umount
>Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 101, in wrapper
> res = func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 147, in umount
> mount._umount(fs_file, force=force, lazy=lazy, freeloop=freeloop)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 296, in _umount
> _runcmd(cmd)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 305, in _runcmd
> raise MountError(rc, b";".join((out, err)))
>MountError: (32, ';umount: /rhev/data-center/mnt/glusterSD/*.*.*.14:_ssd3: mountpoint not found\n')
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,911::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call umount with (<vdsm.supervdsm_server._SuperVdsm object at 0x7f10804e2990>, u'/rhev/data-center/mnt/glusterSD/*.*.*.14:_sdd8') {'force': True, 'lazy': True, 'freeloop': False}
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,911::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/umount -f -l /rhev/data-center/mnt/glusterSD/*.*.*.14:_sdd8 (cwd None)
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:02,917::commands::219::root::(execCmd) FAILED: <err> = 'umount: /rhev/data-center/mnt/glusterSD/*.*.*.14:_sdd8: mountpoint not found\n'; <rc> = 32
>MainProcess|jsonrpc/6::ERROR::2019-04-09 12:37:02,917::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in umount
>Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 101, in wrapper
> res = func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 147, in umount
> mount._umount(fs_file, force=force, lazy=lazy, freeloop=freeloop)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 296, in _umount
> _runcmd(cmd)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 305, in _runcmd
> raise MountError(rc, b";".join((out, err)))
>MountError: (32, ';umount: /rhev/data-center/mnt/glusterSD/*.*.*.14:_sdd8: mountpoint not found\n')
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:03,042::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call hbaRescan with (<vdsm.supervdsm_server._SuperVdsm object at 0x7f10804e2990>,) {}
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:03,042::commands::198::storage.HBA::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/libexec/vdsm/fc-scan (cwd None)
>MainProcess|jsonrpc/6::DEBUG::2019-04-09 12:37:03,107::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return hbaRescan with None
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list