[Bugs] [Bug 1672318] "failed to fetch volume file" when trying to activate host in DC with glusterfs 3.12 domains
bugzilla at redhat.com
bugzilla at redhat.com
Wed Apr 10 08:06:11 UTC 2019
https://bugzilla.redhat.com/show_bug.cgi?id=1672318
--- Comment #30 from Netbulae <info at netbulae.com> ---
The volume info is in comment 4
I get the same with the brick running I see now:
>MainProcess|jsonrpc/0::DEBUG::2019-04-10 09:51:27,437::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call volumeInfo with (u'ssd5', u'*.*.*.16') {}
>MainProcess|jsonrpc/0::DEBUG::2019-04-10 09:51:27,437::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/sbin/gluster --mode=script volume info --remote-host=*.*.*.16 ssd5 --xml (cwd None)
>MainProcess|jsonrpc/0::DEBUG::2019-04-10 09:53:27,553::commands::219::root::(execCmd) FAILED: <err> = ''; <rc> = 1
>MainProcess|jsonrpc/0::DEBUG::2019-04-10 09:53:27,553::logutils::319::root::(_report_stats) ThreadedHandler is ok in the last 123 seconds (max pending: 1)
>MainProcess|jsonrpc/0::ERROR::2019-04-10 09:53:27,553::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in volumeInfo
>Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 101, in wrapper
> res = func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 529, in volumeInfo
> xmltree = _execGlusterXml(command)
> File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 131, in _execGlusterXml
> return _getTree(rc, out, err)
> File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 112, in _getTree
> raise ge.GlusterCmdExecFailedException(rc, out, err)
>GlusterCmdExecFailedException: Command execution failed: rc=1 out='Error : Request timed out\n' err=''
>MainProcess|jsonrpc/0::DEBUG::2019-04-10 09:53:27,555::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call mount with (<vdsm.supervdsm_server._SuperVdsm object at 0x7f10804e2990>, u'*.*.*.16:ssd5', u'/rhev/data-center/mnt/glusterSD/*.*.*.16:ssd5') {'vfstype': u'glusterfs', 'mntOpts': u'backup-volfile-servers=*.*.*.15:*.*.*.14', 'cgroup': 'vdsm-glusterfs'}
>MainProcess|jsonrpc/0::DEBUG::2019-04-10 09:53:27,555::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount -t glusterfs -o backup-volfile-servers=*.*.*.15:*.*.*.14 *.*.*.16:ssd5 /rhev/data-center/mnt/glusterSD/*.*.*.16:ssd5 (cwd None)
>MainProcess|jsonrpc/1::DEBUG::2019-04-10 09:54:27,765::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call hbaRescan with (<vdsm.supervdsm_server._SuperVdsm object at 0x7f10804e2990>,) {}
But the port is reachable from the node:
>Brick *.*.*.16:/data/ssd5/brick1 49156 0 Y 15429
>[root at node9 ~]# telnet *.*.*.16 49156
>Trying *.*.*.16...
>Connected to *.*.*.16.
>Escape character is '^]'.
And in /var/log/messages:
>Apr 10 09:56:27 node9 vdsm[56358]: WARN Worker blocked: <Worker name=jsonrpc/0 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'mnt_options': u'backup-volfile-servers=*.*.*.15:*.*.*.14', u'id': u'b036005a-d44d-4689-a8c3-13e1bbf55af7', u'connection': u'*.*.*.16:ssd5', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], u'storagepoolID': >u'00000001-0001-0001-0001-000000000043', u'domainType': 7}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'fbedfc27-e064-48d5-b332-daeeccdd1cf4'} at 0x7f77c4589210> timeout=60, duration=300.00 at 0x7f77c4589810> >task#=674 at 0x7f77e406f790>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 >self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, >**kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = >method(**params)#012File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1095, in connectStorageServer#012 >connectionParams)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper#012 result = ctask.prepare(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper#012 return m(self, *a, **kw)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in prepare#012 result = self._run(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in connectStorageServer#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method#012 ret = func(*args, **kwargs)#012File: >"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2411, in connectStorageServer#012 conObj.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 172, in connect#012 >self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in mount#012 cgroup=cgroup)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv()
>Apr 10 09:56:27 node9 vdsm[56358]: WARN Worker blocked: <Worker name=jsonrpc/1 running <Task <JsonRpcTask {'params': {u'masterVersion': 10, u'domainDict': {u'33f1ba00-6a16-4e58-b4c5-94426f1c4482': u'active', u'782a61af-a520-44c4-8845-74bf92888552': u'active', u'b30bab9d-9a66-44ce-ad17-2eb4ee858d8f': u'active', u'09959920-a31b-42c2-a547-e50b73602c96': u'active', u'f5d2f7c6-093f-46d6-a844-224d92db5ef9': u'active', u'515166a4-4b2f-402d-bf37-95f3c59635cb': >u'active', u'1ed0a635-67ee-4255-aad9-b70822350706': u'active', u'95b4e5d2-2974-4d5f-91e4-351f75a15435': u'active', u'84f2ff4a-2ec5-42dc-807d-bd12745f387d': u'active'}, u'storagepoolID': u'00000001-0001-0001-0001-000000000043', u'scsiKey': >u'00000001-0001-0001-0001-000000000043', u'masterSdUUID': u'09959920-a31b-42c2-a547-e50b73602c96', u'hostID': 12}, 'jsonrpc': '2.0', 'method': u'StoragePool.connect', 'id': u'95545e2f-6b72-4107-bc19-8de3e549aba0'} at 0x7f77c458d310> >timeout=60, duration=120.01 at 0x7f77c458d0d0> task#=677 at 0x7f77e406f990>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", >line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line >195, in >run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 >task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(self._ctx, >self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1091, in >connect#012 self._UUID, hostID, masterSdUUID, masterVersion, domainDict)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper#012 result = ctask.prepare(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper#012 return m(self, *a, **kw)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in prepare#012 result = self._run(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in connectStoragePool#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", >line 50, in method#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1034, in connectStoragePool#012 spUUID, hostID, msdUUID, masterVersion, domainsMap)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1096, in _connectStoragePool#012 res = pool.connect(hostID, msdUUID, masterVersion)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 700, in connect#012 >self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1274, in __rebuild#012 self.setMasterDomain(msdUUID, masterVersion)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1491, in setMasterDomain#012 domain = sdCache.produce(msdUUID)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce#012 domain.getRealDomain()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain#012 return self._cache._realProduce(self._sdUUID)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce#012 domain = >self._findDomain(sdUUID)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain#012 return findMethod(sdUUID)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 169, in >_findUnfetchedDomain#012 return mod.findDomain(sdUUID)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 146, in findDomain#012 return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 130, in findDomainPath#012 for tmpSdUUID, domainPath in fileSD.scanDomains("*"):#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 893, in scanDomains#012 for >res in misc.itmap(collectMetaFiles, mntList, oop.HELPERS_PER_DOMAIN):#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/misc.py", line 538, in itmap#012 yield respQueue.get()#012File: "/usr/lib64/python2.7/Queue.py", line 168, in >get#012 self.not_empty.wait()#012File: "/usr/lib/python2.7/site-packages/pthreading.py", line 127, in wait#012 return self.__cond.wait()#012File: "/usr/lib/python2.7/site-packages/pthread.py", line 131, in wait#012 return >_libpthread.pthread_cond_wait(self._cond, m.mutex())
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list