[Bugs] [Bug 1435587] New: [GSS]geo-replication faulty
bugzilla at redhat.com
bugzilla at redhat.com
Fri Mar 24 10:10:35 UTC 2017
https://bugzilla.redhat.com/show_bug.cgi?id=1435587
Bug ID: 1435587
Summary: [GSS]geo-replication faulty
Product: Red Hat Gluster Storage
Version: 3.0
Component: geo-replication
Keywords: Triaged
Severity: high
Assignee: avishwan at redhat.com
Reporter: rnalakka at redhat.com
QA Contact: rhinduja at redhat.com
CC: avishwan at redhat.com, bugs at gluster.org,
csaba at redhat.com, ihkim at osci.kr, khoj at osci.kr,
rhs-bugs at redhat.com, sarumuga at redhat.com,
storage-qa-internal at redhat.com
Depends On: 1261689
+++ This bug was initially created as a clone of Bug #1261689 +++
Description of problem:
geo-replication passive faulty
Version-Release number of selected component (if applicable):
glusterfs-3.6.3
How reproducible:
didn't try to reproduce
Steps to Reproduce:
1. create glusterfs distributed-replication
2. create glusterfs geo-replication
3.
Actual results:
Expected results:
Additional info:
ssh%3A%2F%2Fgeorepuser1%4052.74.184.17%3Agluster%3A%2F%2F127.0.0.1%3Acnprddrnas.log
[2015-08-31 00:31:18.726161] E
[syncdutils(/estore_disk02):246:log_raise_exception] <top>: connection to peer
is broken
[2015-08-31 00:31:18.726675] I [syncdutils(/estore_disk02):214:finalize] <top>:
exiting.
[2015-08-31 00:31:18.728502] I [repce(agent):92:service_loop] RepceServer:
terminating on reaching EOF.
[2015-08-31 00:31:18.729090] I [syncdutils(agent):214:finalize] <top>: exiting.
[2015-08-31 00:31:18.801343] I [monitor(monitor):141:set_state] Monitor: new
state: faulty
[2015-08-31 00:31:48.324974] I [monitor(monitor):215:monitor] Monitor:
------------------------------------------------------------
[2015-08-31 00:31:48.325562] I [monitor(monitor):216:monitor] Monitor: starting
gsyncd worker
[2015-08-31 00:31:48.540290] I [gsyncd(/estore_disk02):633:main_i] <top>:
syncing: gluster://localhost:cnprdnas ->
ssh://georepuser1@sg1-cndr-fst01:gluster://localhost:cnprddrnas
[2015-08-31 00:31:48.542256] I [changelogagent(agent):72:__init__]
ChangelogAgent: Agent listining...
[2015-08-31 00:32:17.621978] I [master(/estore_disk01):514:crawlwrap] _GMaster:
0 crawls, 0 turns
[2015-08-31 00:32:48.387411] I [monitor(monitor):281:monitor] Monitor:
worker(/estore_disk02) not confirmed in 60 sec, aborting it
[2015-08-31 00:32:48.389297] I [repce(agent):92:service_loop] RepceServer:
terminating on reaching EOF.
[2015-08-31 00:32:48.389467] I [syncdutils(agent):214:finalize] <top>: exiting.
[2015-08-31 00:32:59.851869] I [monitor(monitor):215:monitor] Monitor:
------------------------------------------------------------
[2015-08-31 00:32:59.852358] I [monitor(monitor):216:monitor] Monitor: starting
gsyncd worker
[2015-08-31 00:32:59.914304] I [gsyncd(/estore_disk02):633:main_i] <top>:
syncing: gluster://localhost:cnprdnas ->
ssh://georepuser1@sg1-cndr-fst01:gluster://localhost:cnprddrnas
[2015-08-31 00:32:59.915914] I [changelogagent(agent):72:__init__]
ChangelogAgent: Agent listining...
[2015-08-31 00:33:17.995424] I [master(/estore_disk01):514:crawlwrap] _GMaster:
0 crawls, 0 turns
[2015-08-31 00:33:59.911894] I [monitor(monitor):281:monitor] Monitor:
worker(/estore_disk02) not confirmed in 60 sec, aborting it
[2015-08-31 00:33:59.914036] I [repce(agent):92:service_loop] RepceServer:
terminating on reaching EOF.
[2015-08-31 00:33:59.914247] I [syncdutils(agent):214:finalize] <top>: exiting.
[2015-08-31 00:34:18.636343] I [master(/estore_disk01):514:crawlwrap] _GMaster:
0 crawls, 0 turns
[2015-08-31 00:34:57.938266] I [monitor(monitor):215:monitor] Monitor:
------------------------------------------------------------
[2015-08-31 00:34:57.938764] I [monitor(monitor):216:monitor] Monitor: starting
gsyncd worker
[2015-08-31 00:34:58.2572] I [gsyncd(/estore_disk02):633:main_i] <top>:
syncing: gluster://localhost:cnprdnas ->
ssh://georepuser1@sg1-cndr-fst01:gluster://localhost:cnprddrnas
[2015-08-31 00:34:58.3433] I [changelogagent(agent):72:__init__]
ChangelogAgent: Agent listining...
[2015-08-31 00:35:19.155519] I [master(/estore_disk01):514:crawlwrap] _GMaster:
0 crawls, 0 turns
[2015-08-31 00:35:58.533] I [monitor(monitor):281:monitor] Monitor:
worker(/estore_disk02) not confirmed in 60 sec, aborting it
[2015-08-31 00:35:58.2600] I [repce(agent):92:service_loop] RepceServer:
terminating on reaching EOF.
[2015-08-31 00:35:58.2794] I [syncdutils(agent):214:finalize] <top>: exiting.
[2015-08-31 00:36:08.497905] I [monitor(monitor):215:monitor] Monitor:
------------------------------------------------------------
[2015-08-31 00:36:08.498428] I [monitor(monitor):216:monitor] Monitor: starting
gsyncd worker
[2015-08-31 00:36:08.559051] I [gsyncd(/estore_disk02):633:main_i] <top>:
syncing: gluster://localhost:cnprdnas ->
ssh://georepuser1@sg1-cndr-fst01:gluster://localhost:cnprddrnas
[2015-08-31 00:36:08.559923] I [changelogagent(agent):72:__init__]
ChangelogAgent: Agent listining...
[2015-08-31 00:36:19.614520] I [master(/estore_disk01):514:crawlwrap] _GMaster:
0 crawls, 0 turns
[2015-08-31 00:37:08.559808] I [monitor(monitor):281:monitor] Monitor:
worker(/estore_disk02) not confirmed in 60 sec, aborting it
[2015-08-31 00:37:08.561731] I [repce(agent):92:service_loop] RepceServer:
terminating on reaching EOF.
[2015-08-31 00:37:08.561908] I [syncdutils(agent):214:finalize] <top>: exiting.
[2015-08-31 00:37:20.161235] I [master(/estore_disk01):514:crawlwrap] _GMaster:
0 crawls, 0 turns
[2015-08-31 00:37:34.803252] I [monitor(monitor):215:monitor] Monitor:
------------------------------------------------------------
[2015-08-31 00:37:34.803802] I [monitor(monitor):216:monitor] Monitor: starting
gsyncd worker
[2015-08-31 00:37:34.866637] I [gsyncd(/estore_disk02):633:main_i] <top>:
syncing: gluster://localhost:cnprdnas ->
ssh://georepuser1@sg1-cndr-fst01:gluster://localhost:cnprddrnas
[2015-08-31 00:37:34.867227] I [changelogagent(agent):72:__init__]
ChangelogAgent: Agent listining...
[2015-08-31 00:38:20.776204] I [master(/estore_disk01):514:crawlwrap] _GMaster:
0 crawls, 0 turns
[2015-08-31 00:38:34.865513] I [monitor(monitor):281:monitor] Monitor:
worker(/estore_disk02) not confirmed in 60 sec, aborting it
[2015-08-31 00:38:34.867564] I [repce(agent):92:service_loop] RepceServer:
terminating on reaching EOF.
[2015-08-31 00:38:34.867752] I [syncdutils(agent):214:finalize] <top>: exiting.
[2015-08-31 00:39:21.417596] I [master(/estore_disk01):514:crawlwrap] _GMaster:
0 crawls, 0 turns
[2015-08-31 00:39:37.110416] I [monitor(monitor):215:monitor] Monitor:
------------------------------------------------------------
[2015-08-31 00:39:37.110945] I [monitor(monitor):216:monitor] Monitor: starting
gsyncd worker
[2015-08-31 00:39:37.173197] I [gsyncd(/estore_disk02):633:main_i] <top>:
syncing: gluster://localhost:cnprdnas ->
ssh://georepuser1@sg1-cndr-fst01:gluster://localhost:cnprddrnas
[2015-08-31 00:39:37.173186] I [changelogagent(agent):72:__init__]
ChangelogAgent: Agent listining...
[2015-08-31 00:40:21.983525] I [master(/estore_disk01):514:crawlwrap] _GMaster:
0 crawls, 0 turns
[2015-08-31 00:40:37.118439] I [monitor(monitor):281:monitor] Monitor:
worker(/estore_disk02) not confirmed in 60 sec, aborting it
[2015-08-31 00:40:37.120224] I [repce(agent):92:service_loop] RepceServer:
terminating on reaching EOF.
[2015-08-31 00:40:37.120392] I [syncdutils(agent):214:finalize] <top>: exiting.
[2015-08-31 00:40:48.605340] I [monitor(monitor):215:monitor] Monitor:
------------------------------------------------------------
[2015-08-31 00:40:48.605844] I [monitor(monitor):216:monitor] Monitor: starting
gsyncd worker
[2015-08-31 00:40:48.668005] I [changelogagent(agent):72:__init__]
ChangelogAgent: Agent listining...
[2015-08-31 00:40:48.668332] I [gsyncd(/estore_disk02):633:main_i] <top>:
syncing: gluster://localhost:cnprdnas ->
ssh://georepuser1@sg1-cndr-fst01:gluster://localhost:cnprddrnas
[2015-08-31 00:41:22.499874] I [master(/estore_disk01):514:crawlwrap] _GMaster:
0 crawls, 0 turns
[2015-08-31 00:41:48.667587] I [monitor(monitor):281:monitor] Monitor:
worker(/estore_disk02) not confirmed in 60 sec, aborting it
[2015-08-31 00:41:48.669570] I [repce(agent):92:service_loop] RepceServer:
terminating on reaching EOF.
[2015-08-31 00:41:48.669785] I [syncdutils(agent):214:finalize] <top>: exiting.
[2015-08-31 00:42:13.974661] I [monitor(monitor):215:monitor] Monitor:
------------------------------------------------------------
[2015-08-31 00:42:13.975204] I [monitor(monitor):216:monitor] Monitor: starting
gsyncd worker
[2015-08-31 00:42:14.37354] I [gsyncd(/estore_disk02):633:main_i] <top>:
syncing: gluster://localhost:cnprdnas ->
ssh://georepuser1@sg1-cndr-fst01:gluster://localhost:cnprddrnas
[2015-08-31 00:42:14.37564] I [changelogagent(agent):72:__init__]
ChangelogAgent: Agent listining...
[2015-08-31 00:42:22.983730] I [master(/estore_disk01):514:crawlwrap] _GMaster:
0 crawls, 0 turns
[2015-08-31 00:43:14.36992] I [monitor(monitor):281:monitor] Monitor:
worker(/estore_disk02) not confirmed in 60 sec, aborting it
[2015-08-31 00:43:14.38670] I [repce(agent):92:service_loop] RepceServer:
terminating on reaching EOF.
[2015-08-31 00:43:14.38840] I [syncdutils(agent):214:finalize] <top>: exiting.
[2015-08-31 00:43:23.795301] I [master(/estore_disk01):514:crawlwrap] _GMaster:
0 crawls, 0 turns
[2015-08-31 00:44:24.395556] I [master(/estore_disk01):514:crawlwrap] _GMaster:
0 crawls, 0 turns
[2015-08-31 00:45:24.291826] E [resource(monitor):221:errlog] Popen: command
"gluster --xml --remote-host=sg1-cndr-fst01 volume status cnprddrnas detail"
returned with 146
[2015-08-31 00:45:24.292504] I [syncdutils(monitor):214:finalize] <top>:
exiting.
ssh%3A%2F%2Fgeorepuser1%4052.74.184.17%3Agluster%3A%2F%2F127.0.0.1%3Acnprddrnas.%2Festore_disk02.gluster.log
[2015-08-31 00:31:18.728293] I [fuse-bridge.c:4921:fuse_thread_proc] 0-fuse:
unmounting /tmp/gsyncd-aux-mount-BehNtW
[2015-08-31 00:31:18.797937] W [glusterfsd.c:1194:cleanup_and_exit] (--> 0-:
received signum (15), shutting down
[2015-08-31 00:31:18.797957] I [fuse-bridge.c:5599:fini] 0-fuse: Unmounting
'/tmp/gsyncd-aux-mount-BehNtW'.
file attach
sosreport, glusterfs log
--- Additional comment from hojin kim on 2015-09-15 22:22:36 EDT ---
Hi, Venky. the configuration is as below..
--------------------------------------------------------------
mount volume
server1: CN1-PRD-FS01 CN2-PRD-FS01 ==> replicated volume0
| |
distributed distributed
| |
server2: CN1-PRD-FS02 CN2-PRD-FS02 ==> replicated volume1
|
|
|
|
georeplicate cndr
we restart the geo-replication and the problem is solved.
but we want to know which cause fall into the faulty state.
Thank for advance...
--- Additional comment from hojin kim on 2015-09-17 03:08:52 EDT ---
we met same error again. It fell into faulty status,,,we restart by force
option..
please let me know the reason, asap...Thks..
--- Additional comment from Aravinda VK on 2016-08-19 07:30:43 EDT ---
GlusterFS-3.6 is nearing its End-Of-Life, only important security bugs still
make a chance on getting fixed. Moving this to the mainline 'version'. If this
needs to get fixed in 3.7 or 3.8 this bug should get cloned.
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1261689
[Bug 1261689] geo-replication faulty
--
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=tsPQKEQmbX&a=cc_unsubscribe
More information about the Bugs
mailing list