<div dir="ltr">Ok, Then it should work.<br>Could you confirm rsync runs successfully when executed manually as below.<div><br></div><div>1. On master node,<br>     a) # mkdir /mastermnt <br>     b) Mount master volume on /mastermnt<br>     b) # echo &quot;test data&quot; /master/file1</div><div>2. On slave node<br>     a) # mkdir /slavemnt<br>     b) # Mount slave on /slavemnt<br></div><div>     c) # touch /salvemnt/file1</div><div>3. On master node<br>     a) # cd /mastermnt</div><div>     b) # rsync -aR0 --inplace --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args file1 -e &#39;ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no  -p 22 -oControlMaster=auto -i /var/lib/glusterd/geo-replication/secret.pem&#39; root@pgsotc11.png.intel.com:/slavemnt/</div>4. Check for content sync<div>     a) cat /slavemnt/file1</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 26, 2019 at 1:19 PM Tan, Jian Chern &lt;<a href="mailto:jian.chern.tan@intel.com">jian.chern.tan@intel.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">





<div lang="EN-US">
<div class="gmail-m_3028080637249274742WordSection1">
<p class="MsoNormal"><span lang="EN-MY">Rsync on both the slave and master are rsync  version 3.1.3  protocol version 31, so both are up to date as far as I know.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY">Gluster version on both machines are glusterfs 5.10<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY">OS on both machines are Fedora 29 Server Edition<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-MY"><u></u> <u></u></span></p>
<p class="MsoNormal"><b>From:</b> Kotresh Hiremath Ravishankar &lt;<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>&gt;
<br>
<b>Sent:</b> Tuesday, November 26, 2019 3:04 PM<br>
<b>To:</b> Tan, Jian Chern &lt;<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>&gt;<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Unable to setup geo replication<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">The error code 14 related IPC where any of pipe/fork fails in rsync code.<br>
Please upgrade the rsync if not done. Also check the rsync versions between master and slave to be same.<u></u><u></u></p>
<div>
<p class="MsoNormal">Which version of gluster are you using?<br>
What&#39;s the host OS?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">What&#39;s the rsync version ?<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, Nov 26, 2019 at 11:34 AM Tan, Jian Chern &lt;<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>&gt; wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span lang="EN-MY">I’m new to GlusterFS and trying to setup geo-replication with a master volume being mirrored to a slave volume on another machine. However I just can’t seem to
 get it to work after starting the geo replication volume with the logs showing it failing rsync with error code 14. I can’t seem to find any info about this online so any help would be much appreciated.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.24706] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change  status=Initializing...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.24891] I [monitor(monitor):157:monitor] Monitor: starting gsyncd worker    brick=/data/glusterimagebrick/jfsotc22-gv0      slave_node=<a href="http://pgsotc11.png.intel.com" target="_blank">pgsotc11.png.intel.com</a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.90935] I [gsyncd(agent /data/glusterimagebrick/jfsotc22-gv0):308:main] &lt;top&gt;: Using session config file    path=/var/lib/glusterd/geo-replication/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/gsyncd.conf</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.92105] I [changelogagent(agent /data/glusterimagebrick/jfsotc22-gv0):72:__init__] ChangelogAgent: Agent listining...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.93148] I [gsyncd(worker /data/glusterimagebrick/jfsotc22-gv0):308:main] &lt;top&gt;: Using session config file   path=/var/lib/glusterd/geo-replication/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/gsyncd.conf</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:31.102422] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1366:connect_remote] SSH: Initializing SSH connection between master and
 slave...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:50.355233] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1413:connect_remote] SSH: SSH connection between master and slave established.   
 duration=19.2526</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:50.355583] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1085:connect] GLUSTER: Mounting gluster volume locally...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:51.404998] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1108:connect] GLUSTER: Mounted gluster volume     duration=1.0492</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:51.405363] I [subcmds(worker /data/glusterimagebrick/jfsotc22-gv0):80:subcmd_worker] &lt;top&gt;: Worker spawn successful. Acknowledging back to monitor</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.431502] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1603:register] _GMaster: Working dir        path=/var/lib/misc/gluster/gsyncd/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/data-glusterimagebrick-jfsotc22-gv0</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.431846] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1271:service_loop] GLUSTER: Register time time=1574747213</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.445589] I [gsyncdstatus(worker /data/glusterimagebrick/jfsotc22-gv0):281:set_active] GeorepStatus: Worker Status Change    status=Active</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.446184] I [gsyncdstatus(worker /data/glusterimagebrick/jfsotc22-gv0):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change       
 status=History Crawl</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:53.446367] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1517:crawl] _GMaster: starting history crawl        turns=1 stime=(1574669325,
 0)   etime=1574747213        entry_stime=None</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:54.448994] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1546:crawl] _GMaster: slave&#39;s time  stime=(1574669325, 0)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:54.928395] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1954:syncjob] Syncer: Sync Time Taken       job=1   num_files=1     return_code=14 
 duration=0.0162</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:54.928607] E [syncdutils(worker /data/glusterimagebrick/jfsotc22-gv0):809:errlog] Popen: command returned error       cmd=rsync -aR0 --inplace
 --files-from=- --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args . -e ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-rgpu74f3/de0855b3336b4c3233934fcbeeb3674c.sock
 pgsotc11.png.intel.com:/proc/29549/cwd      error=14</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:54.935529] I [repce(agent /data/glusterimagebrick/jfsotc22-gv0):97:service_loop] RepceServer: terminating on reaching EOF.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:55.410444] I [monitor(monitor):278:monitor] Monitor: worker died in startup phase     brick=/data/glusterimagebrick/jfsotc22-gv0</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:46:55.412591] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-11-26 05:47:05.631944] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY">….</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY"> </span><u></u><u></u></p>
<p class="MsoNormal">Thanks!<u></u><u></u></p>
<p class="MsoNormal">Jian Chern<u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-MY"> </span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">________<br>
<br>
Community Meeting Calendar:<br>
<br>
APAC Schedule -<br>
Every 2nd and 4th Tuesday at 11:30 AM IST<br>
Bridge: <a href="https://bluejeans.com/441850968" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
NA/EMEA Schedule -<br>
Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
Bridge: <a href="https://bluejeans.com/441850968" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Thanks and Regards,<u></u><u></u></p>
</div>
<p class="MsoNormal">Kotresh H R<u></u><u></u></p>
</div>
</div>
</div>
</div>

</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>