<div dir="ltr">Ok, Then it should work.<br>Could you confirm rsync runs successfully when executed manually as below.<div><br></div><div>1. On master node,<br> a) # mkdir /mastermnt <br> b) Mount master volume on /mastermnt<br> b) # echo "test data" /master/file1</div><div>2. On slave node<br> a) # mkdir /slavemnt<br> b) # Mount slave on /slavemnt<br></div><div> c) # touch /salvemnt/file1</div><div>3. On master node<br> a) # cd /mastermnt</div><div> b) # rsync -aR0 --inplace --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args file1 -e 'ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -p 22 -oControlMaster=auto -i /var/lib/glusterd/geo-replication/secret.pem' root@pgsotc11.png.intel.com:/slavemnt/</div>4. Check for content sync<div> a) cat /slavemnt/file1</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 26, 2019 at 1:19 PM Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com">jian.chern.tan@intel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_3028080637249274742WordSection1">
<p class="MsoNormal"><span lang="EN-MY">Rsync on both the slave and master are rsync version 3.1.3 protocol version 31, so both are up to date as far as I know.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY">Gluster version on both machines are glusterfs 5.10<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY">OS on both machines are Fedora 29 Server Edition<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY"><u></u> <u></u></span></p>
<p class="MsoNormal"><b>From:</b> Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>>
<br>
<b>Sent:</b> Tuesday, November 26, 2019 3:04 PM<br>
<b>To:</b> Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>><br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Unable to setup geo replication<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">The error code 14 related IPC where any of pipe/fork fails in rsync code.<br>
Please upgrade the rsync if not done. Also check the rsync versions between master and slave to be same.<u></u><u></u></p>
<div>
<p class="MsoNormal">Which version of gluster are you using?<br>
What's the host OS?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">What's the rsync version ?<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, Nov 26, 2019 at 11:34 AM Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span lang="EN-MY">I’m new to GlusterFS and trying to setup geo-replication with a master volume being mirrored to a slave volume on another machine. However I just can’t seem to
get it to work after starting the geo replication volume with the logs showing it failing rsync with error code 14. I can’t seem to find any info about this online so any help would be much appreciated.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.24706] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.24891] I [monitor(monitor):157:monitor] Monitor: starting gsyncd worker brick=/data/glusterimagebrick/jfsotc22-gv0 slave_node=<a href="http://pgsotc11.png.intel.com" target="_blank">pgsotc11.png.intel.com</a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.90935] I [gsyncd(agent /data/glusterimagebrick/jfsotc22-gv0):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/gsyncd.conf</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.92105] I [changelogagent(agent /data/glusterimagebrick/jfsotc22-gv0):72:__init__] ChangelogAgent: Agent listining...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.93148] I [gsyncd(worker /data/glusterimagebrick/jfsotc22-gv0):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/gsyncd.conf</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.102422] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1366:connect_remote] SSH: Initializing SSH connection between master and
slave...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:50.355233] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1413:connect_remote] SSH: SSH connection between master and slave established.
duration=19.2526</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:50.355583] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1085:connect] GLUSTER: Mounting gluster volume locally...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:51.404998] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1108:connect] GLUSTER: Mounted gluster volume duration=1.0492</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:51.405363] I [subcmds(worker /data/glusterimagebrick/jfsotc22-gv0):80:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.431502] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1603:register] _GMaster: Working dir path=/var/lib/misc/gluster/gsyncd/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/data-glusterimagebrick-jfsotc22-gv0</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.431846] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1271:service_loop] GLUSTER: Register time time=1574747213</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.445589] I [gsyncdstatus(worker /data/glusterimagebrick/jfsotc22-gv0):281:set_active] GeorepStatus: Worker Status Change status=Active</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.446184] I [gsyncdstatus(worker /data/glusterimagebrick/jfsotc22-gv0):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change
status=History Crawl</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.446367] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1517:crawl] _GMaster: starting history crawl turns=1 stime=(1574669325,
0) etime=1574747213 entry_stime=None</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:54.448994] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1546:crawl] _GMaster: slave's time stime=(1574669325, 0)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:54.928395] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1954:syncjob] Syncer: Sync Time Taken job=1 num_files=1 return_code=14
duration=0.0162</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:54.928607] E [syncdutils(worker /data/glusterimagebrick/jfsotc22-gv0):809:errlog] Popen: command returned error cmd=rsync -aR0 --inplace
--files-from=- --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args . -e ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-rgpu74f3/de0855b3336b4c3233934fcbeeb3674c.sock
pgsotc11.png.intel.com:/proc/29549/cwd error=14</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:54.935529] I [repce(agent /data/glusterimagebrick/jfsotc22-gv0):97:service_loop] RepceServer: terminating on reaching EOF.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:55.410444] I [monitor(monitor):278:monitor] Monitor: worker died in startup phase brick=/data/glusterimagebrick/jfsotc22-gv0</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:55.412591] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:47:05.631944] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">….</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY"> </span><u></u><u></u></p>
<p class="MsoNormal">Thanks!<u></u><u></u></p>
<p class="MsoNormal">Jian Chern<u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY"> </span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">________<br>
<br>
Community Meeting Calendar:<br>
<br>
APAC Schedule -<br>
Every 2nd and 4th Tuesday at 11:30 AM IST<br>
Bridge: <a href="https://bluejeans.com/441850968" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
NA/EMEA Schedule -<br>
Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
Bridge: <a href="https://bluejeans.com/441850968" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Thanks and Regards,<u></u><u></u></p>
</div>
<p class="MsoNormal">Kotresh H R<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>