<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 09/29/2017 09:30 PM, rick sanchez
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAP-FWeBJrvsTQbo2A+_bWYH+7+brm+ec=BGy6GfHS6aU1t2M_g@mail.gmail.com">
<div dir="ltr">I am trying to get up geo replication between two
gluster volumes
<div><br>
</div>
<div>I have set up two replica 2 arbiter 1 volumes with 9 bricks</div>
<div><br>
</div>
<div>
<div>[root@gfs1 ~]# gluster volume info</div>
<div> </div>
<div>Volume Name: gfsvol</div>
<div>Type: Distributed-Replicate</div>
<div>Volume ID: c2fb4365-480b-4d37-8c7d-c3046bca7306</div>
<div>Status: Started</div>
<div>Snapshot Count: 0</div>
<div>Number of Bricks: 3 x (2 + 1) = 9</div>
<div>Transport-type: tcp</div>
<div>Bricks:</div>
<div>Brick1: gfs2:/gfs/brick1/gv0</div>
<div>Brick2: gfs3:/gfs/brick1/gv0</div>
<div>Brick3: gfs1:/gfs/arbiter/gv0 (arbiter)</div>
<div>Brick4: gfs1:/gfs/brick1/gv0</div>
<div>Brick5: gfs3:/gfs/brick2/gv0</div>
<div>Brick6: gfs2:/gfs/arbiter/gv0 (arbiter)</div>
<div>Brick7: gfs1:/gfs/brick2/gv0</div>
<div>Brick8: gfs2:/gfs/brick2/gv0</div>
<div>Brick9: gfs3:/gfs/arbiter/gv0 (arbiter)</div>
<div>Options Reconfigured:</div>
<div>nfs.disable: on</div>
<div>transport.address-family: inet</div>
<div>geo-replication.indexing: on</div>
<div>geo-replication.ignore-pid-check: on</div>
<div>changelog.changelog: on</div>
</div>
<div><br>
</div>
<div>
<div>[root@gfs4 ~]# gluster volume info</div>
<div> </div>
<div>Volume Name: gfsvol_rep</div>
<div>Type: Distributed-Replicate</div>
<div>Volume ID: 42bfa062-ad0d-4242-a813-63389be1c404</div>
<div>Status: Started</div>
<div>Snapshot Count: 0</div>
<div>Number of Bricks: 3 x (2 + 1) = 9</div>
<div>Transport-type: tcp</div>
<div>Bricks:</div>
<div>Brick1: gfs5:/gfs/brick1/gv0</div>
<div>Brick2: gfs6:/gfs/brick1/gv0</div>
<div>Brick3: gfs4:/gfs/arbiter/gv0 (arbiter)</div>
<div>Brick4: gfs4:/gfs/brick1/gv0</div>
<div>Brick5: gfs6:/gfs/brick2/gv0</div>
<div>Brick6: gfs5:/gfs/arbiter/gv0 (arbiter)</div>
<div>Brick7: gfs4:/gfs/brick2/gv0</div>
<div>Brick8: gfs5:/gfs/brick2/gv0</div>
<div>Brick9: gfs6:/gfs/arbiter/gv0 (arbiter)</div>
<div>Options Reconfigured:</div>
<div>nfs.disable: on</div>
<div>transport.address-family: inet</div>
</div>
<div><br>
</div>
<div>I set up passwordless ssh login from all the master servers
to all the slave servers then created and started the geo
replicated volume</div>
</div>
</blockquote>
Passwordless SSH not required for all nodes, it is required from one
of the master node to one of the slave node. (From the master node
where you want to run create command to the slave node which will be
specified in the Create command)<br>
<br>
Alternatively you can use a tool called "gluster-georep-setup",
which doesn't require initial passwordless step.<br>
<br>
<a class="moz-txt-link-freetext" href="http://aravindavk.in/blog/gluster-georep-tools/">http://aravindavk.in/blog/gluster-georep-tools/</a><br>
<a class="moz-txt-link-freetext" href="https://github.com/aravindavk/gluster-georep-tools">https://github.com/aravindavk/gluster-georep-tools</a><br>
<br>
<blockquote type="cite"
cite="mid:CAP-FWeBJrvsTQbo2A+_bWYH+7+brm+ec=BGy6GfHS6aU1t2M_g@mail.gmail.com">
<div dir="ltr">
<div><br>
</div>
<div>I check the status and they switch between being active
with history crawl and faulty with n/a every few second</div>
<div>
<div>
<div>MASTER NODE MASTER VOL MASTER BRICK SLAVE
USER SLAVE SLAVE NODE
STATUS CRAWL STATUS LAST_SYNCED </div>
<div>-------------------------------------------------------------------------------------------------------------------------------------------------------------</div>
<div>gfs1 gfsvol /gfs/arbiter/gv0
geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A
Faulty N/A N/A
</div>
<div>gfs1 gfsvol /gfs/brick1/gv0
geo-rep-user geo-rep-user@gfs4::gfsvol_rep gfs6
Active History Crawl 2017-09-28 23:30:19
</div>
<div>gfs1 gfsvol /gfs/brick2/gv0
geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A
Faulty N/A N/A
</div>
<div>gfs3 gfsvol /gfs/brick1/gv0
geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A
Faulty N/A N/A
</div>
<div>gfs3 gfsvol /gfs/brick2/gv0
geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A
Faulty N/A N/A
</div>
<div>gfs3 gfsvol /gfs/arbiter/gv0
geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A
Faulty N/A N/A
</div>
<div>gfs2 gfsvol /gfs/brick1/gv0
geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A
Faulty N/A N/A
</div>
<div>gfs2 gfsvol /gfs/arbiter/gv0
geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A
Faulty N/A N/A
</div>
<div>gfs2 gfsvol /gfs/brick2/gv0
geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A
Faulty N/A N/A
</div>
</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Here is the output of the geo replication log file </div>
<div>
<div>[root@gfs1 ~]# tail -n 100 $(gluster volume
geo-replication gfsvol geo-rep-user@gfs4::gfsvol_rep config
log-file)</div>
<div>[2017-09-29 15:53:29.785386] I
[master(/gfs/brick2/gv0):1860:syncjob] Syncer: Sync Time
Taken<span style="white-space:pre"> </span>duration=0.0357<span style="white-space:pre"> </span>num_files=1<span style="white-space:pre"> </span>job=3<span style="white-space:pre"> </span>return_code=12</div>
<div>[2017-09-29 15:53:29.785615] E
[resource(/gfs/brick2/gv0):208:errlog] Popen: command
returned error<span style="white-space:pre"> </span>cmd=rsync
-aR0 --inplace --files-from=- --super --stats --numeric-ids
--no-implied-dirs --existing --xattrs --acls . -e ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22
-oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-fdyDHm/78cf8b204207154de59d7ac32eee737f.sock
--compress geo-rep-user@gfs6:/proc/17554/cwd<span style="white-space:pre"> </span>error=12</div>
<div>[2017-09-29 15:53:29.797259] I
[syncdutils(/gfs/brick2/gv0):271:finalize] <top>:
exiting.</div>
<div>[2017-09-29 15:53:29.799386] I
[repce(/gfs/brick2/gv0):92:service_loop] RepceServer:
terminating on reaching EOF.</div>
<div>[2017-09-29 15:53:29.799570] I
[syncdutils(/gfs/brick2/gv0):271:finalize] <top>:
exiting.</div>
<div>[2017-09-29 15:53:30.105407] I
[monitor(monitor):280:monitor] Monitor: starting gsyncd
worker<span style="white-space:pre"> </span>brick=/gfs/brick1/gv0<span style="white-space:pre"> </span>slave_node=<a class="moz-txt-link-freetext" href="ssh://geo-rep-user@gfs6:gluster://localhost:gfsvol_rep">ssh://geo-rep-user@gfs6:gluster://localhost:gfsvol_rep</a></div>
<div>[2017-09-29 15:53:30.232007] I
[resource(/gfs/brick1/gv0):1772:connect_remote] SSH:
Initializing SSH connection between master and slave...</div>
<div>[2017-09-29 15:53:30.232738] I
[changelogagent(/gfs/brick1/gv0):73:__init__]
ChangelogAgent: Agent listining...</div>
<div>[2017-09-29 15:53:30.248094] I
[monitor(monitor):363:monitor] Monitor: worker died in
startup phase<span style="white-space:pre"> </span>brick=/gfs/brick2/gv0</div>
<div>[2017-09-29 15:53:30.252793] I
[gsyncdstatus(monitor):242:set_worker_status] GeorepStatus:
Worker Status Change<span style="white-space:pre"> </span>status=Faulty</div>
<div>[2017-09-29 15:53:30.742058] I
[master(/gfs/arbiter/gv0):1515:register] _GMaster: Working
dir<span style="white-space:pre"> </span>path=/var/lib/misc/glusterfsd/gfsvol/ssh%3A%2F%2Fgeo-rep-user%4010.1.1.104%3Agluster%3A%2F%2F127.0.0.1%3Agfsvol_rep/40efd54bad1d5828a1221dd560de376f</div>
<div>[2017-09-29 15:53:30.742360] I
[resource(/gfs/arbiter/gv0):1654:service_loop] GLUSTER:
Register time<span style="white-space:pre"> </span>time=1506700410</div>
<div>[2017-09-29 15:53:30.754738] I
[gsyncdstatus(/gfs/arbiter/gv0):275:set_active]
GeorepStatus: Worker Status Change<span style="white-space:pre"> </span>status=Active</div>
<div>[2017-09-29 15:53:30.756040] I
[gsyncdstatus(/gfs/arbiter/gv0):247:set_worker_crawl_status]
GeorepStatus: Crawl Status Change<span style="white-space:pre"> </span>status=History
Crawl</div>
<div>[2017-09-29 15:53:30.756280] I
[master(/gfs/arbiter/gv0):1429:crawl] _GMaster: starting
history crawl<span style="white-space:pre"> </span>turns=1<span style="white-space:pre"> </span>stime=(1506637819,
0)<span style="white-space:pre"> </span>entry_stime=None<span style="white-space:pre"> </span>etime=1506700410</div>
<div>[2017-09-29 15:53:31.758335] I
[master(/gfs/arbiter/gv0):1458:crawl] _GMaster: slave's time<span style="white-space:pre"> </span>stime=(1506637819,
0)</div>
<div>[2017-09-29 15:53:31.939471] I
[resource(/gfs/brick1/gv0):1779:connect_remote] SSH: SSH
connection between master and slave established.<span style="white-space:pre"> </span>duration=1.7073</div>
<div>[2017-09-29 15:53:31.939665] I
[resource(/gfs/brick1/gv0):1494:connect] GLUSTER: Mounting
gluster volume locally...</div>
<div>[2017-09-29 15:53:32.284754] I
[master(/gfs/arbiter/gv0):1860:syncjob] Syncer: Sync Time
Taken<span style="white-space:pre"> </span>duration=0.0372<span style="white-space:pre"> </span>num_files=1<span style="white-space:pre"> </span>job=3<span style="white-space:pre"> </span>return_code=12</div>
<div>[2017-09-29 15:53:32.284996] E
[resource(/gfs/arbiter/gv0):208:errlog] Popen: command
returned error<span style="white-space:pre"> </span>cmd=rsync
-aR0 --inplace --files-from=- --super --stats --numeric-ids
--no-implied-dirs --existing --xattrs --acls . -e ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22
-oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-i_wIMu/5f1d38555e12d0018fb6ed1e6bd63023.sock
--compress geo-rep-user@gfs5:/proc/8334/cwd<span style="white-space:pre"> </span>error=12</div>
<div>[2017-09-29 15:53:32.300786] I
[syncdutils(/gfs/arbiter/gv0):271:finalize] <top>:
exiting.</div>
<div>[2017-09-29 15:53:32.303261] I
[repce(/gfs/arbiter/gv0):92:service_loop] RepceServer:
terminating on reaching EOF.</div>
<div>[2017-09-29 15:53:32.303452] I
[syncdutils(/gfs/arbiter/gv0):271:finalize] <top>:
exiting.</div>
<div>[2017-09-29 15:53:32.732858] I
[monitor(monitor):363:monitor] Monitor: worker died in
startup phase<span style="white-space:pre"> </span>brick=/gfs/arbiter/gv0</div>
<div>[2017-09-29 15:53:32.736538] I
[gsyncdstatus(monitor):242:set_worker_status] GeorepStatus:
Worker Status Change<span style="white-space:pre"> </span>status=Faulty</div>
<div>[2017-09-29 15:53:33.35219] I
[resource(/gfs/brick1/gv0):1507:connect] GLUSTER: Mounted
gluster volume<span style="white-space:pre"> </span>duration=1.0954</div>
<div>[2017-09-29 15:53:33.35403] I
[gsyncd(/gfs/brick1/gv0):799:main_i] <top>: Closing
feedback fd, waking up the monitor</div>
<div>[2017-09-29 15:53:35.50920] I
[master(/gfs/brick1/gv0):1515:register] _GMaster: Working
dir<span style="white-space:pre"> </span>path=/var/lib/misc/glusterfsd/gfsvol/ssh%3A%2F%2Fgeo-rep-user%4010.1.1.104%3Agluster%3A%2F%2F127.0.0.1%3Agfsvol_rep/f0393acbf9a1583960edbbd2f1dfb6b4</div>
<div>[2017-09-29 15:53:35.51227] I
[resource(/gfs/brick1/gv0):1654:service_loop] GLUSTER:
Register time<span style="white-space:pre"> </span>time=1506700415</div>
<div>[2017-09-29 15:53:35.64343] I
[gsyncdstatus(/gfs/brick1/gv0):275:set_active] GeorepStatus:
Worker Status Change<span style="white-space:pre"> </span>status=Active</div>
<div>[2017-09-29 15:53:35.65696] I
[gsyncdstatus(/gfs/brick1/gv0):247:set_worker_crawl_status]
GeorepStatus: Crawl Status Change<span style="white-space:pre"> </span>status=History
Crawl</div>
<div>[2017-09-29 15:53:35.65915] I
[master(/gfs/brick1/gv0):1429:crawl] _GMaster: starting
history crawl<span style="white-space:pre"> </span>turns=1<span style="white-space:pre"> </span>stime=(1506637819,
0)<span style="white-space:pre"> </span>entry_stime=None<span style="white-space:pre"> </span>etime=1506700415</div>
<div>[2017-09-29 15:53:36.68135] I
[master(/gfs/brick1/gv0):1458:crawl] _GMaster: slave's time<span style="white-space:pre"> </span>stime=(1506637819,
0)</div>
<div>[2017-09-29 15:53:36.578717] I
[master(/gfs/brick1/gv0):1860:syncjob] Syncer: Sync Time
Taken<span style="white-space:pre"> </span>duration=0.0376<span style="white-space:pre"> </span>num_files=1<span style="white-space:pre"> </span>job=1<span style="white-space:pre"> </span>return_code=12</div>
<div>[2017-09-29 15:53:36.578946] E
[resource(/gfs/brick1/gv0):208:errlog] Popen: command
returned error<span style="white-space:pre"> </span>cmd=rsync
-aR0 --inplace --files-from=- --super --stats --numeric-ids
--no-implied-dirs --existing --xattrs --acls . -e ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22
-oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-2pGnVA/78cf8b204207154de59d7ac32eee737f.sock
--compress geo-rep-user@gfs6:/proc/17648/cwd<span style="white-space:pre"> </span>error=12</div>
<div>[2017-09-29 15:53:36.590887] I
[syncdutils(/gfs/brick1/gv0):271:finalize] <top>:
exiting.</div>
<div>[2017-09-29 15:53:36.596421] I
[repce(/gfs/brick1/gv0):92:service_loop] RepceServer:
terminating on reaching EOF.</div>
<div>[2017-09-29 15:53:36.596635] I
[syncdutils(/gfs/brick1/gv0):271:finalize] <top>:
exiting.</div>
<div>[2017-09-29 15:53:37.41075] I
[monitor(monitor):363:monitor] Monitor: worker died in
startup phase<span style="white-space:pre"> </span>brick=/gfs/brick1/gv0</div>
<div>[2017-09-29 15:53:37.44637] I
[gsyncdstatus(monitor):242:set_worker_status] GeorepStatus:
Worker Status Change<span style="white-space:pre"> </span>status=Faulty</div>
<div>[2017-09-29 15:53:40.351263] I
[monitor(monitor):280:monitor] Monitor: starting gsyncd
worker<span style="white-space:pre"> </span>brick=/gfs/brick2/gv0<span style="white-space:pre"> </span>slave_node=<a class="moz-txt-link-freetext" href="ssh://geo-rep-user@gfs6:gluster://localhost:gfsvol_rep">ssh://geo-rep-user@gfs6:gluster://localhost:gfsvol_rep</a></div>
<div>[2017-09-29 15:53:40.484637] I
[resource(/gfs/brick2/gv0):1772:connect_remote] SSH:
Initializing SSH connection between master and slave...</div>
<div>[2017-09-29 15:53:40.497215] I
[changelogagent(/gfs/brick2/gv0):73:__init__]
ChangelogAgent: Agent listining...</div>
<div>[2017-09-29 15:53:42.278539] I
[resource(/gfs/brick2/gv0):1779:connect_remote] SSH: SSH
connection between master and slave established.<span style="white-space:pre"> </span>duration=1.7936</div>
<div>[2017-09-29 15:53:42.278747] I
[resource(/gfs/brick2/gv0):1494:connect] GLUSTER: Mounting
gluster volume locally...</div>
<div>[2017-09-29 15:53:42.851296] I
[monitor(monitor):280:monitor] Monitor: starting gsyncd
worker<span style="white-space:pre"> </span>brick=/gfs/arbiter/gv0<span style="white-space:pre"> </span>slave_node=<a class="moz-txt-link-freetext" href="ssh://geo-rep-user@gfs5:gluster://localhost:gfsvol_rep">ssh://geo-rep-user@gfs5:gluster://localhost:gfsvol_rep</a></div>
<div>[2017-09-29 15:53:42.985567] I
[resource(/gfs/arbiter/gv0):1772:connect_remote] SSH:
Initializing SSH connection between master and slave...</div>
<div>[2017-09-29 15:53:42.986390] I
[changelogagent(/gfs/arbiter/gv0):73:__init__]
ChangelogAgent: Agent listining...</div>
<div>[2017-09-29 15:53:43.377480] I
[resource(/gfs/brick2/gv0):1507:connect] GLUSTER: Mounted
gluster volume<span style="white-space:pre"> </span>duration=1.0986</div>
<div>[2017-09-29 15:53:43.377681] I
[gsyncd(/gfs/brick2/gv0):799:main_i] <top>: Closing
feedback fd, waking up the monitor</div>
<div>[2017-09-29 15:53:44.767873] I
[resource(/gfs/arbiter/gv0):1779:connect_remote] SSH: SSH
connection between master and slave established.<span style="white-space:pre"> </span>duration=1.7821</div>
<div>[2017-09-29 15:53:44.768059] I
[resource(/gfs/arbiter/gv0):1494:connect] GLUSTER: Mounting
gluster volume locally...</div>
<div>[2017-09-29 15:53:45.393150] I
[master(/gfs/brick2/gv0):1515:register] _GMaster: Working
dir<span style="white-space:pre"> </span>path=/var/lib/misc/glusterfsd/gfsvol/ssh%3A%2F%2Fgeo-rep-user%4010.1.1.104%3Agluster%3A%2F%2F127.0.0.1%3Agfsvol_rep/1eb15856c627f181513bf23f8bf2f9d0</div>
<div>[2017-09-29 15:53:45.393373] I
[resource(/gfs/brick2/gv0):1654:service_loop] GLUSTER:
Register time<span style="white-space:pre"> </span>time=1506700425</div>
<div>[2017-09-29 15:53:45.404992] I
[gsyncdstatus(/gfs/brick2/gv0):275:set_active] GeorepStatus:
Worker Status Change<span style="white-space:pre"> </span>status=Active</div>
<div>[2017-09-29 15:53:45.406404] I
[gsyncdstatus(/gfs/brick2/gv0):247:set_worker_crawl_status]
GeorepStatus: Crawl Status Change<span style="white-space:pre"> </span>status=History
Crawl</div>
<div>[2017-09-29 15:53:45.406660] I
[master(/gfs/brick2/gv0):1429:crawl] _GMaster: starting
history crawl<span style="white-space:pre"> </span>turns=1<span style="white-space:pre"> </span>stime=(1506637819,
0)<span style="white-space:pre"> </span>entry_stime=None<span style="white-space:pre"> </span>etime=1506700425</div>
<div>[2017-09-29 15:53:45.863256] I
[resource(/gfs/arbiter/gv0):1507:connect] GLUSTER: Mounted
gluster volume<span style="white-space:pre"> </span>duration=1.0950</div>
<div>[2017-09-29 15:53:45.863430] I
[gsyncd(/gfs/arbiter/gv0):799:main_i] <top>: Closing
feedback fd, waking up the monitor</div>
<div>[2017-09-29 15:53:46.408814] I
[master(/gfs/brick2/gv0):1458:crawl] _GMaster: slave's time<span style="white-space:pre"> </span>stime=(1506637819,
0)</div>
<div>[2017-09-29 15:53:46.920937] I
[master(/gfs/brick2/gv0):1860:syncjob] Syncer: Sync Time
Taken<span style="white-space:pre"> </span>duration=0.0363<span style="white-space:pre"> </span>num_files=1<span style="white-space:pre"> </span>job=3<span style="white-space:pre"> </span>return_code=12</div>
<div>[2017-09-29 15:53:46.921140] E
[resource(/gfs/brick2/gv0):208:errlog] Popen: command
returned error<span style="white-space:pre"> </span>cmd=rsync
-aR0 --inplace --files-from=- --super --stats --numeric-ids
--no-implied-dirs --existing --xattrs --acls . -e ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22
-oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-DCruqU/78cf8b204207154de59d7ac32eee737f.sock
--compress geo-rep-user@gfs6:/proc/17747/cwd<span style="white-space:pre"> </span>error=12</div>
<div>[2017-09-29 15:53:46.937288] I
[syncdutils(/gfs/brick2/gv0):271:finalize] <top>:
exiting.</div>
<div>[2017-09-29 15:53:46.940479] I
[repce(/gfs/brick2/gv0):92:service_loop] RepceServer:
terminating on reaching EOF.</div>
<div>[2017-09-29 15:53:46.940772] I
[syncdutils(/gfs/brick2/gv0):271:finalize] <top>:
exiting.</div>
<div>[2017-09-29 15:53:47.151477] I
[monitor(monitor):280:monitor] Monitor: starting gsyncd
worker<span style="white-space:pre"> </span>brick=/gfs/brick1/gv0<span style="white-space:pre"> </span>slave_node=<a class="moz-txt-link-freetext" href="ssh://geo-rep-user@gfs6:gluster://localhost:gfsvol_rep">ssh://geo-rep-user@gfs6:gluster://localhost:gfsvol_rep</a></div>
<div>[2017-09-29 15:53:47.303791] I
[resource(/gfs/brick1/gv0):1772:connect_remote] SSH:
Initializing SSH connection between master and slave...</div>
<div>[2017-09-29 15:53:47.316878] I
[changelogagent(/gfs/brick1/gv0):73:__init__]
ChangelogAgent: Agent listining...</div>
<div>[2017-09-29 15:53:47.382605] I
[monitor(monitor):363:monitor] Monitor: worker died in
startup phase<span style="white-space:pre"> </span>brick=/gfs/brick2/gv0</div>
<div>[2017-09-29 15:53:47.387926] I
[gsyncdstatus(monitor):242:set_worker_status] GeorepStatus:
Worker Status Change<span style="white-space:pre"> </span>status=Faulty</div>
<div>[2017-09-29 15:53:47.876825] I
[master(/gfs/arbiter/gv0):1515:register] _GMaster: Working
dir<span style="white-space:pre"> </span>path=/var/lib/misc/glusterfsd/gfsvol/ssh%3A%2F%2Fgeo-rep-user%4010.1.1.104%3Agluster%3A%2F%2F127.0.0.1%3Agfsvol_rep/40efd54bad1d5828a1221dd560de376f</div>
<div>[2017-09-29 15:53:47.877044] I
[resource(/gfs/arbiter/gv0):1654:service_loop] GLUSTER:
Register time<span style="white-space:pre"> </span>time=1506700427</div>
<div>[2017-09-29 15:53:47.888930] I
[gsyncdstatus(/gfs/arbiter/gv0):275:set_active]
GeorepStatus: Worker Status Change<span style="white-space:pre"> </span>status=Active</div>
<div>[2017-09-29 15:53:47.890043] I
[gsyncdstatus(/gfs/arbiter/gv0):247:set_worker_crawl_status]
GeorepStatus: Crawl Status Change<span style="white-space:pre"> </span>status=History
Crawl</div>
<div>[2017-09-29 15:53:47.890285] I
[master(/gfs/arbiter/gv0):1429:crawl] _GMaster: starting
history crawl<span style="white-space:pre"> </span>turns=1<span style="white-space:pre"> </span>stime=(1506637819,
0)<span style="white-space:pre"> </span>entry_stime=None<span style="white-space:pre"> </span>etime=1506700427</div>
<div>[2017-09-29 15:53:48.891966] I
[master(/gfs/arbiter/gv0):1458:crawl] _GMaster: slave's time<span style="white-space:pre"> </span>stime=(1506637819,
0)</div>
<div>[2017-09-29 15:53:48.998140] I
[resource(/gfs/brick1/gv0):1779:connect_remote] SSH: SSH
connection between master and slave established.<span style="white-space:pre"> </span>duration=1.6942</div>
<div>[2017-09-29 15:53:48.998330] I
[resource(/gfs/brick1/gv0):1494:connect] GLUSTER: Mounting
gluster volume locally...</div>
<div>[2017-09-29 15:53:49.406749] I
[master(/gfs/arbiter/gv0):1860:syncjob] Syncer: Sync Time
Taken<span style="white-space:pre"> </span>duration=0.0383<span style="white-space:pre"> </span>num_files=1<span style="white-space:pre"> </span>job=2<span style="white-space:pre"> </span>return_code=12</div>
<div>[2017-09-29 15:53:49.406999] E
[resource(/gfs/arbiter/gv0):208:errlog] Popen: command
returned error<span style="white-space:pre"> </span>cmd=rsync
-aR0 --inplace --files-from=- --super --stats --numeric-ids
--no-implied-dirs --existing --xattrs --acls . -e ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22
-oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-5VeNKp/5f1d38555e12d0018fb6ed1e6bd63023.sock
--compress geo-rep-user@gfs5:/proc/8448/cwd<span style="white-space:pre"> </span>error=12</div>
<div>[2017-09-29 15:53:49.426301] I
[syncdutils(/gfs/arbiter/gv0):271:finalize] <top>:
exiting.</div>
<div>[2017-09-29 15:53:49.428428] I
[repce(/gfs/arbiter/gv0):92:service_loop] RepceServer:
terminating on reaching EOF.</div>
<div>[2017-09-29 15:53:49.428618] I
[syncdutils(/gfs/arbiter/gv0):271:finalize] <top>:
exiting.</div>
<div>[2017-09-29 15:53:49.868974] I
[monitor(monitor):363:monitor] Monitor: worker died in
startup phase<span style="white-space:pre"> </span>brick=/gfs/arbiter/gv0</div>
<div>[2017-09-29 15:53:49.872705] I
[gsyncdstatus(monitor):242:set_worker_status] GeorepStatus:
Worker Status Change<span style="white-space:pre"> </span>status=Faulty</div>
<div>[2017-09-29 15:53:50.78377] I
[resource(/gfs/brick1/gv0):1507:connect] GLUSTER: Mounted
gluster volume<span style="white-space:pre"> </span>duration=1.0799</div>
<div>[2017-09-29 15:53:50.78643] I
[gsyncd(/gfs/brick1/gv0):799:main_i] <top>: Closing
feedback fd, waking up the monitor</div>
<div>[2017-09-29 15:53:52.93027] I
[master(/gfs/brick1/gv0):1515:register] _GMaster: Working
dir<span style="white-space:pre"> </span>path=/var/lib/misc/glusterfsd/gfsvol/ssh%3A%2F%2Fgeo-rep-user%4010.1.1.104%3Agluster%3A%2F%2F127.0.0.1%3Agfsvol_rep/f0393acbf9a1583960edbbd2f1dfb6b4</div>
<div>[2017-09-29 15:53:52.93331] I
[resource(/gfs/brick1/gv0):1654:service_loop] GLUSTER:
Register time<span style="white-space:pre"> </span>time=1506700432</div>
<div>[2017-09-29 15:53:52.107558] I
[gsyncdstatus(/gfs/brick1/gv0):275:set_active] GeorepStatus:
Worker Status Change<span style="white-space:pre"> </span>status=Active</div>
<div>[2017-09-29 15:53:52.108943] I
[gsyncdstatus(/gfs/brick1/gv0):247:set_worker_crawl_status]
GeorepStatus: Crawl Status Change<span style="white-space:pre"> </span>status=History
Crawl</div>
<div>[2017-09-29 15:53:52.109178] I
[master(/gfs/brick1/gv0):1429:crawl] _GMaster: starting
history crawl<span style="white-space:pre"> </span>turns=1<span style="white-space:pre"> </span>stime=(1506637819,
0)<span style="white-space:pre"> </span>entry_stime=None<span style="white-space:pre"> </span>etime=1506700432</div>
<div>[2017-09-29 15:53:53.111017] I
[master(/gfs/brick1/gv0):1458:crawl] _GMaster: slave's time<span style="white-space:pre"> </span>stime=(1506637819,
0)</div>
<div>[2017-09-29 15:53:53.622422] I
[master(/gfs/brick1/gv0):1860:syncjob] Syncer: Sync Time
Taken<span style="white-space:pre"> </span>duration=0.0369<span style="white-space:pre"> </span>num_files=1<span style="white-space:pre"> </span>job=2<span style="white-space:pre"> </span>return_code=12</div>
<div>[2017-09-29 15:53:53.622683] E
[resource(/gfs/brick1/gv0):208:errlog] Popen: command
returned error<span style="white-space:pre"> </span>cmd=rsync
-aR0 --inplace --files-from=- --super --stats --numeric-ids
--no-implied-dirs --existing --xattrs --acls . -e ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22
-oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-DBB9pL/78cf8b204207154de59d7ac32eee737f.sock
--compress geo-rep-user@gfs6:/proc/17837/cwd<span style="white-space:pre"> </span>error=12</div>
<div>[2017-09-29 15:53:53.635057] I
[syncdutils(/gfs/brick1/gv0):271:finalize] <top>:
exiting.</div>
<div>[2017-09-29 15:53:53.639909] I
[repce(/gfs/brick1/gv0):92:service_loop] RepceServer:
terminating on reaching EOF.</div>
<div>[2017-09-29 15:53:53.640172] I
[syncdutils(/gfs/brick1/gv0):271:finalize] <top>:
exiting.</div>
<div>[2017-09-29 15:53:54.85591] I
[monitor(monitor):363:monitor] Monitor: worker died in
startup phase<span style="white-space:pre"> </span>brick=/gfs/brick1/gv0</div>
<div>[2017-09-29 15:53:54.89509] I
[gsyncdstatus(monitor):242:set_worker_status] GeorepStatus:
Worker Status Change<span style="white-space:pre"> </span>status=Faulty</div>
</div>
<div><br>
</div>
<div>I think the error has to do with this part:</div>
<div>rsync -aR0 --inplace --files-from=- --super --stats
--numeric-ids --no-implied-dirs --existing --xattrs --acls .
-e ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no
-i /var/lib/glusterd/geo-replication/secret.pem -p 22
-oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-DBB9pL/78cf8b204207154de59d7ac32eee737f.sock
--compress geo-rep-user@gfs6:/proc/17837/cwd</div>
<div>especially the ssh part since I notice a lot of failed log
in attempts when geo replication is running</div>
</div>
</blockquote>
<br>
I suspect this is related to ssh keys, please let us know if resetup
with above mentioned steps helps.<br>
<br>
<blockquote type="cite"
cite="mid:CAP-FWeBJrvsTQbo2A+_bWYH+7+brm+ec=BGy6GfHS6aU1t2M_g@mail.gmail.com">
<div dir="ltr">
<div><br>
</div>
<div>Please can anybody help advise what to do in this
situation?</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users">http://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
regards
Aravinda VK
<a class="moz-txt-link-freetext" href="http://aravindavk.in">http://aravindavk.in</a>
</pre>
</body>
</html>