<div dir="ltr">Oh forgot about that. Just setup passwordless ssh and that particular node and try with default ssh pem key and remove /var/lib/glusterd/geo-replicationsecre.pem from the command line</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 27, 2019 at 12:43 PM Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com">jian.chern.tan@intel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-4653691950668460924WordSection1">
<p class="MsoNormal"><span lang="EN-MY">I’m getting this when I run that command so something’s wrong somewhere I guess.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY">[root@jfsotc22 mnt]# rsync -aR0 --inplace --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args file1 -e 'ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -p 22 -oControlMaster=auto
-i /var/lib/glusterd/geo-replication/secret.pem' root@pgsotc11.png.intel.com:/mnt/<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY">gsyncd sibling not found<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY">disallowed rsync invocation<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY">rsync: connection unexpectedly closed (0 bytes received so far) [sender]<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY">rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.3]<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY">[root@jfsotc22 mnt]#<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY"><u></u> <u></u></span></p>
<p class="MsoNormal"><b>From:</b> Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>>
<br>
<b>Sent:</b> Tuesday, November 26, 2019 7:22 PM<br>
<b>To:</b> Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>><br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Unable to setup geo replication<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Ok, Then it should work.<br>
Could you confirm rsync runs successfully when executed manually as below.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">1. On master node,<br>
a) # mkdir /mastermnt <br>
b) Mount master volume on /mastermnt<br>
b) # echo "test data" /master/file1<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2. On slave node<br>
a) # mkdir /slavemnt<br>
b) # Mount slave on /slavemnt<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> c) # touch /salvemnt/file1<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">3. On master node<br>
a) # cd /mastermnt<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> b) # rsync -aR0 --inplace --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args file1 -e 'ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -p 22 -oControlMaster=auto -i /var/lib/glusterd/geo-replication/secret.pem'
<a href="mailto:root@pgsotc11.png.intel.com:/slavemnt/" target="_blank">root@pgsotc11.png.intel.com:/slavemnt/</a><u></u><u></u></p>
</div>
<p class="MsoNormal">4. Check for content sync<u></u><u></u></p>
<div>
<p class="MsoNormal"> a) cat /slavemnt/file1<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, Nov 26, 2019 at 1:19 PM Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span lang="EN-MY">Rsync on both the slave and master are rsync version 3.1.3 protocol version 31, so both are up to date as far as I know.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">Gluster version on both machines are glusterfs 5.10</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">OS on both machines are Fedora 29 Server Edition</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY"> </span><u></u><u></u></p>
<p class="MsoNormal"><b>From:</b> Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>>
<br>
<b>Sent:</b> Tuesday, November 26, 2019 3:04 PM<br>
<b>To:</b> Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>><br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Unable to setup geo replication<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">The error code 14 related IPC where any of pipe/fork fails in rsync code.<br>
Please upgrade the rsync if not done. Also check the rsync versions between master and slave to be same.<u></u><u></u></p>
<div>
<p class="MsoNormal">Which version of gluster are you using?<br>
What's the host OS?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">What's the rsync version ?<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, Nov 26, 2019 at 11:34 AM Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-MY">I’m new to GlusterFS and trying to setup geo-replication with a master volume being mirrored to a slave volume on another machine. However I just can’t seem to
get it to work after starting the geo replication volume with the logs showing it failing rsync with error code 14. I can’t seem to find any info about this online so any help would be much appreciated.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.24706] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.24891] I [monitor(monitor):157:monitor] Monitor: starting gsyncd worker brick=/data/glusterimagebrick/jfsotc22-gv0 slave_node=<a href="http://pgsotc11.png.intel.com" target="_blank">pgsotc11.png.intel.com</a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.90935] I [gsyncd(agent /data/glusterimagebrick/jfsotc22-gv0):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/gsyncd.conf</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.92105] I [changelogagent(agent /data/glusterimagebrick/jfsotc22-gv0):72:__init__] ChangelogAgent: Agent listining...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.93148] I [gsyncd(worker /data/glusterimagebrick/jfsotc22-gv0):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/gsyncd.conf</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.102422] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1366:connect_remote] SSH: Initializing SSH connection between master and
slave...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:50.355233] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1413:connect_remote] SSH: SSH connection between master and slave established.
duration=19.2526</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:50.355583] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1085:connect] GLUSTER: Mounting gluster volume locally...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:51.404998] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1108:connect] GLUSTER: Mounted gluster volume duration=1.0492</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:51.405363] I [subcmds(worker /data/glusterimagebrick/jfsotc22-gv0):80:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.431502] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1603:register] _GMaster: Working dir path=/var/lib/misc/gluster/gsyncd/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/data-glusterimagebrick-jfsotc22-gv0</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.431846] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1271:service_loop] GLUSTER: Register time time=1574747213</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.445589] I [gsyncdstatus(worker /data/glusterimagebrick/jfsotc22-gv0):281:set_active] GeorepStatus: Worker Status Change status=Active</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.446184] I [gsyncdstatus(worker /data/glusterimagebrick/jfsotc22-gv0):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change
status=History Crawl</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.446367] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1517:crawl] _GMaster: starting history crawl turns=1 stime=(1574669325,
0) etime=1574747213 entry_stime=None</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:54.448994] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1546:crawl] _GMaster: slave's time stime=(1574669325, 0)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:54.928395] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1954:syncjob] Syncer: Sync Time Taken job=1 num_files=1 return_code=14
duration=0.0162</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:54.928607] E [syncdutils(worker /data/glusterimagebrick/jfsotc22-gv0):809:errlog] Popen: command returned error cmd=rsync -aR0 --inplace
--files-from=- --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args . -e ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-rgpu74f3/de0855b3336b4c3233934fcbeeb3674c.sock
pgsotc11.png.intel.com:/proc/29549/cwd error=14</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:54.935529] I [repce(agent /data/glusterimagebrick/jfsotc22-gv0):97:service_loop] RepceServer: terminating on reaching EOF.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:55.410444] I [monitor(monitor):278:monitor] Monitor: worker died in startup phase brick=/data/glusterimagebrick/jfsotc22-gv0</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:55.412591] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:47:05.631944] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">….</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY"> </span><u></u><u></u></p>
<p class="MsoNormal">Thanks!<u></u><u></u></p>
<p class="MsoNormal">Jian Chern<u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY"> </span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">________<br>
<br>
Community Meeting Calendar:<br>
<br>
APAC Schedule -<br>
Every 2nd and 4th Tuesday at 11:30 AM IST<br>
Bridge: <a href="https://bluejeans.com/441850968" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
NA/EMEA Schedule -<br>
Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
Bridge: <a href="https://bluejeans.com/441850968" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal">--
<u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Thanks and Regards,<u></u><u></u></p>
</div>
<p class="MsoNormal">Kotresh H R<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Thanks and Regards,<u></u><u></u></p>
</div>
<p class="MsoNormal">Kotresh H R<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>