[Bugs] [Bug 1694002] New: Geo-re: Geo replication failing in "cannot allocate memory"
bugzilla at redhat.com
bugzilla at redhat.com
Fri Mar 29 09:04:51 UTC 2019
https://bugzilla.redhat.com/show_bug.cgi?id=1694002
Bug ID: 1694002
Summary: Geo-re: Geo replication failing in "cannot allocate
memory"
Product: GlusterFS
Version: 6
Hardware: x86_64
OS: Linux
Status: NEW
Component: geo-replication
Keywords: ZStream
Severity: medium
Priority: medium
Assignee: bugs at gluster.org
Reporter: khiremat at redhat.com
CC: abhishku at redhat.com, avishwan at redhat.com,
bkunal at redhat.com, bugs at gluster.org, csaba at redhat.com,
khiremat at redhat.com, rhinduja at redhat.com,
rhs-bugs at redhat.com, sankarshan at redhat.com,
skandark at redhat.com, smulay at redhat.com,
storage-qa-internal at redhat.com, sunkumar at redhat.com
Depends On: 1670429, 1693648
Target Milestone: ---
Classification: Community
Description of the Problem:
Geo-rep is 'Faulty' and not syncing
Slave worker crash:
[2019-01-21 14:46:36.338450] I [resource(slave):1422:connect] GLUSTER: Mounting
gluster volume locally...
[2019-01-21 14:46:47.581492] I [resource(slave):1435:connect] GLUSTER: Mounted
gluster volume duration=11.2428
[2019-01-21 14:46:47.582036] I [resource(slave):905:service_loop] GLUSTER:
slave listening
[2019-01-21 14:47:36.831804] E [repce(slave):117:worker] <top>: call failed:
Traceback (most recent call last):
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 113, in worker
res = getattr(self.obj, rmeth)(*in_data[2:])
File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 756, in
entry_ops
[ESTALE, EINVAL, EBUSY])
File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 553, in
errno_wrap
return call(*arg)
File "/usr/libexec/glusterfs/python/syncdaemon/libcxattr.py", line 79, in
lsetxattr
cls.raise_oserr()
File "/usr/libexec/glusterfs/python/syncdaemon/libcxattr.py", line 37, in
raise_oserr
raise OSError(errn, os.strerror(errn))
OSError: [Errno 12] Cannot allocate memory
Master worker crash:
[2019-01-21 14:46:36.7253] I
[resource(/glusterfs/glprd01-vsb-pil-modshape000/brick1):1700:connect_remote]
SSH: Initializing SSH connection between master and slave...
[2019-01-21 14:46:36.7440] I
[changelogagent(/glusterfs/glprd01-vsb-pil-modshape000/brick1):73:__init__]
ChangelogAgent: Agent listining...
[2019-01-21 14:46:47.585638] I
[resource(/glusterfs/glprd01-vsb-pil-modshape000/brick1):1707:connect_remote]
SSH: SSH connection between master and slave established. duration=11.5781
[2019-01-21 14:46:47.585905] I
[resource(/glusterfs/glprd01-vsb-pil-modshape000/brick1):1422:connect] GLUSTER:
Mounting gluster volume locally...
[2019-01-21 14:46:48.650470] I
[resource(/glusterfs/glprd01-vsb-pil-modshape000/brick1):1435:connect] GLUSTER:
Mounted gluster volume duration=1.0644
[2019-01-21 14:46:48.650816] I
[gsyncd(/glusterfs/glprd01-vsb-pil-modshape000/brick1):803:main_i] <top>:
Worker spawn successful. Acknowledging back to monitor
[2019-01-21 14:46:50.675277] I
[master(/glusterfs/glprd01-vsb-pil-modshape000/brick1):1583:register] _GMaster:
Working dir
path=/var/lib/misc/glusterfsd/pil-vbs-modshape/ssh%3A%2F%2Fgeoaccount%40172.21.142.
33%3Agluster%3A%2F%2F127.0.0.1%3Apil-vbs-modshape/5eaac78a29ba1e2e24b401621c5240c3
[2019-01-21 14:46:50.675633] I
[resource(/glusterfs/glprd01-vsb-pil-modshape000/brick1):1582:service_loop]
GLUSTER: Register time time=1548082010
[2019-01-21 14:46:50.690826] I
[master(/glusterfs/glprd01-vsb-pil-modshape000/brick1):482:mgmt_lock] _GMaster:
Didn't get lock Becoming PASSIVE
brick=/glusterfs/glprd01-vsb-pil-modshape000/brick1
[2019-01-21 14:46:50.703552] I
[gsyncdstatus(/glusterfs/glprd01-vsb-pil-modshape000/brick1):282:set_passive]
GeorepStatus: Worker Status Change status=Passive
[2019-01-21 14:47:35.797741] I
[master(/glusterfs/glprd01-vsb-pil-modshape000/brick1):436:mgmt_lock] _GMaster:
Got lock Becoming ACTIVE brick=/glusterfs/glprd01-vsb-pil-modshape000/brick1
[2019-01-21 14:47:35.802330] I
[gsyncdstatus(/glusterfs/glprd01-vsb-pil-modshape000/brick1):276:set_active]
GeorepStatus: Worker Status Change status=Active
[2019-01-21 14:47:35.804092] I
[gsyncdstatus(/glusterfs/glprd01-vsb-pil-modshape000/brick1):248:set_worker_crawl_status]
GeorepStatus: Crawl Status Change status=History Crawl
[2019-01-21 14:47:35.804485] I
[master(/glusterfs/glprd01-vsb-pil-modshape000/brick1):1497:crawl] _GMaster:
starting history crawl turns=1 stime=(1548059316, 0)
entry_stime=(1548059310, 0) etime=15480
82055
[2019-01-21 14:47:36.808142] I
[master(/glusterfs/glprd01-vsb-pil-modshape000/brick1):1526:crawl] _GMaster:
slave's time stime=(1548059316, 0)
[2019-01-21 14:47:36.833885] E
[repce(/glusterfs/glprd01-vsb-pil-modshape000/brick1):209:__call__]
RepceClient: call failed call=32116:139676615182144:1548082056.82
method=entry_ops error=OSError
[2019-01-21 14:47:36.834212] E
[syncdutils(/glusterfs/glprd01-vsb-pil-modshape000/brick1):349:log_raise_exception]
<top>: FAIL:
Traceback (most recent call last):
File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 210, in main
main_i()
File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 805, in
main_i
local.service_loop(*[r for r in [remote] if r])
File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1588, in
service_loop
g3.crawlwrap(oneshot=True)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 597, in
crawlwrap
self.crawl()
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1535, in
crawl
self.changelogs_batch_process(changes)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1435, in
changelogs_batch_process
self.process(batch)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1269, in
process
self.process_change(change, done, retry)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1165, in
process_change
failures = self.slave.server.entry_ops(entries)
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 228, in
__call__
return self.ins(self.meth, *a)
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 210, in
__call__
raise res
OSError: [Errno 12] Cannot allocate memory
[2019-01-21 14:47:36.846298] I
[syncdutils(/glusterfs/glprd01-vsb-pil-modshape000/brick1):289:finalize] <top>:
exiting.
[2019-01-21 14:47:36.849236] I
[repce(/glusterfs/glprd01-vsb-pil-modshape000/brick1):92:service_loop]
RepceServer: terminating
--- Additional comment from Worker Ant on 2019-03-29 07:24:23 UTC ---
REVIEW: https://review.gluster.org/22438 (geo-rep: Fix syncing multiple rename
of symlink) merged (#2) on master by Amar Tumballi
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1693648
[Bug 1693648] Geo-re: Geo replication failing in "cannot allocate memory"
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list