<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-MY">Still not working :(<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[jiancher@jfsotc22 mnt]$ sudo gluster vol geo-rep jfsotc22-gv0 pgsotc10.png.intel.com::pgsotc10-gv0 config | grep sync_xattrs<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY" style="background:yellow;mso-highlight:yellow">sync_xattrs:false</span><span lang="EN-MY"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">use_rsync_xattrs:false<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[jiancher@jfsotc22 mnt]$ tail -n 30 /var/log/glusterfs/geo-replication/jfsotc22-gv0_pgsotc10.png.intel.com_pgsotc10-gv0/gsyncd.log<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:06.407781] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:06.407979] I [monitor(monitor):157:monitor] Monitor: starting gsyncd worker brick=/data/glusterbrick/gv0 slave_node=pgsotc10.png.intel.com<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:06.474308] I [gsyncd(agent /data/glusterbrick/gv0):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/jfsotc22-gv0_pgsotc10.png.intel.com_pgsotc10-gv0/gsyncd.conf<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:06.475533] I [changelogagent(agent /data/glusterbrick/gv0):72:__init__] ChangelogAgent: Agent listining...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:06.477900] I [gsyncd(worker /data/glusterbrick/gv0):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/jfsotc22-gv0_pgsotc10.png.intel.com_pgsotc10-gv0/gsyncd.conf<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:06.487359] I [resource(worker /data/glusterbrick/gv0):1366:connect_remote] SSH: Initializing SSH connection between master and slave...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:10.901315] I [resource(worker /data/glusterbrick/gv0):1413:connect_remote] SSH: SSH connection between master and slave established. duration=4.4138<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:10.901653] I [resource(worker /data/glusterbrick/gv0):1085:connect] GLUSTER: Mounting gluster volume locally...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:11.948481] I [resource(worker /data/glusterbrick/gv0):1108:connect] GLUSTER: Mounted gluster volume duration=1.0466<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:11.948833] I [subcmds(worker /data/glusterbrick/gv0):80:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:13.969974] I [master(worker /data/glusterbrick/gv0):1603:register] _GMaster: Working dir path=/var/lib/misc/gluster/gsyncd/jfsotc22-gv0_pgsotc10.png.intel.com_pgsotc10-gv0/data-glusterbrick-gv0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:13.970411] I [resource(worker /data/glusterbrick/gv0):1271:service_loop] GLUSTER: Register time time=1575251713<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:13.984259] I [gsyncdstatus(worker /data/glusterbrick/gv0):281:set_active] GeorepStatus: Worker Status Change status=Active<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:13.984892] I [gsyncdstatus(worker /data/glusterbrick/gv0):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:13.985096] I [master(worker /data/glusterbrick/gv0):1517:crawl] _GMaster: starting history crawl turns=1 stime=None etime=1575251713 entry_stime=None<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:13.985174] I [resource(worker /data/glusterbrick/gv0):1287:service_loop] GLUSTER: No stime available, using xsync crawl<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:13.992273] I [master(worker /data/glusterbrick/gv0):1633:crawl] _GMaster: starting hybrid crawl stime=None<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:13.993591] I [gsyncdstatus(worker /data/glusterbrick/gv0):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change status=Hybrid Crawl<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:14.994798] I [master(worker /data/glusterbrick/gv0):1644:crawl] _GMaster: processing xsync changelog path=/var/lib/misc/gluster/gsyncd/jfsotc22-gv0_pgsotc10.png.intel.com_pgsotc10-gv0/data-glusterbrick-gv0/xsync/XSYNC-CHANGELOG.1575251713<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:15.470127] I [master(worker /data/glusterbrick/gv0):1954:syncjob] Syncer: Sync Time Taken job=1 num_files=2 return_code=14 duration=0.0165<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:15.470312] E [syncdutils(worker /data/glusterbrick/gv0):809:errlog] Popen: command returned error cmd=rsync -aR0 --inplace --files-from=- --super --stats --numeric-ids --no-implied-dirs --existing
--acls --ignore-missing-args . -e ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-47gkywzh/d91d12d424b6c44691090c3a561b932d.sock pgsotc10.png.intel.com:/proc/27159/cwd
error=14<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:15.477034] I [repce(agent /data/glusterbrick/gv0):97:service_loop] RepceServer: terminating on reaching EOF.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:15.953736] I [monitor(monitor):278:monitor] Monitor: worker died in startup phase brick=/data/glusterbrick/gv0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY">[2019-12-02 01:55:15.955809] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-MY"><o:p> </o:p></span></p>
<p class="MsoNormal"><b>From:</b> Kotresh Hiremath Ravishankar <khiremat@redhat.com>
<br>
<b>Sent:</b> Monday, December 2, 2019 2:37 AM<br>
<b>To:</b> Tan, Jian Chern <jian.chern.tan@intel.com><br>
<b>Cc:</b> gluster-users@gluster.org<br>
<b>Subject:</b> Re: [Gluster-users] Unable to setup geo replication<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Please try disabling xattr sync and see geo-rep works fine<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">gluster vol geo-rep <mastervol> <slavevol>::<slavevol> config sync_xattrs false<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Nov 28, 2019 at 1:29 PM Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com">jian.chern.tan@intel.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">Alright so it seems to work with some errors and this the output I’m getting.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[root@jfsotc22 mnt]# rsync -aR0 --inplace --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args file1 -e 'ssh -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -p 22 -oControlMaster=auto -i /var/lib/glusterd/geo-replication/secret.pem'
<a href="mailto:root@pgsotc10.png.intel.com:/mnt/">root@pgsotc10.png.intel.com:/mnt/</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">rsync: rsync_xal_set: lsetxattr("/mnt/file1","security.selinux") failed: Operation not supported (95)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">Number of files: 1 (reg: 1)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">Number of created files: 0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">Number of deleted files: 0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">Number of regular files transferred: 1</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">Total file size: 9 bytes</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">Total transferred file size: 9 bytes</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">Literal data: 9 bytes</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">Matched data: 0 bytes</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">File list size: 0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">File list generation time: 0.003 seconds</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">File list transfer time: 0.000 seconds</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">Total bytes sent: 152</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">Total bytes received: 141</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">sent 152 bytes received 141 bytes 65.11 bytes/sec</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">total size is 9 speedup is 0.03</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1189) [sender=3.1.3]</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">The data is synced over to the other machine when I view the file there</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[root@pgsotc10 mnt]# cat file1</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">testdata</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[root@pgsotc10 mnt]#</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>>
<br>
<b>Sent:</b> Wednesday, November 27, 2019 5:25 PM<br>
<b>To:</b> Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>><br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Unable to setup geo replication<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Oh forgot about that. Just setup passwordless ssh and that particular node and try with default ssh pem key and remove /var/lib/glusterd/geo-replicationsecre.pem from the command
line<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Wed, Nov 27, 2019 at 12:43 PM Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">I’m getting this when I run that command so something’s wrong somewhere I guess.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[root@jfsotc22 mnt]# rsync -aR0 --inplace --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args file1 -e 'ssh -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -p 22 -oControlMaster=auto -i /var/lib/glusterd/geo-replication/secret.pem'
<a href="mailto:root@pgsotc11.png.intel.com:/mnt/" target="_blank">root@pgsotc11.png.intel.com:/mnt/</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">gsyncd sibling not found</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">disallowed rsync invocation</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">rsync: connection unexpectedly closed (0 bytes received so far) [sender]</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.3]</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[root@jfsotc22 mnt]#</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>>
<br>
<b>Sent:</b> Tuesday, November 26, 2019 7:22 PM<br>
<b>To:</b> Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>><br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Unable to setup geo replication<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Ok, Then it should work.<br>
Could you confirm rsync runs successfully when executed manually as below.<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">1. On master node,<br>
a) # mkdir /mastermnt <br>
b) Mount master volume on /mastermnt<br>
b) # echo "test data" /master/file1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">2. On slave node<br>
a) # mkdir /slavemnt<br>
b) # Mount slave on /slavemnt<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> c) # touch /salvemnt/file1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">3. On master node<br>
a) # cd /mastermnt<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> b) # rsync -aR0 --inplace --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args file1 -e 'ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no
-p 22 -oControlMaster=auto -i /var/lib/glusterd/geo-replication/secret.pem' <a href="mailto:root@pgsotc11.png.intel.com:/slavemnt/" target="_blank">
root@pgsotc11.png.intel.com:/slavemnt/</a><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">4. Check for content sync<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> a) cat /slavemnt/file1<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Tue, Nov 26, 2019 at 1:19 PM Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">Rsync on both the slave and master are rsync version 3.1.3 protocol version 31, so both are up to date as far as I know.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">Gluster version on both machines are glusterfs 5.10</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">OS on both machines are Fedora 29 Server Edition</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>>
<br>
<b>Sent:</b> Tuesday, November 26, 2019 3:04 PM<br>
<b>To:</b> Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>><br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Unable to setup geo replication<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The error code 14 related IPC where any of pipe/fork fails in rsync code.<br>
Please upgrade the rsync if not done. Also check the rsync versions between master and slave to be same.<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Which version of gluster are you using?<br>
What's the host OS?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">What's the rsync version ?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Tue, Nov 26, 2019 at 11:34 AM Tan, Jian Chern <<a href="mailto:jian.chern.tan@intel.com" target="_blank">jian.chern.tan@intel.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">I’m new to GlusterFS and trying to setup geo-replication with a master volume being mirrored to a slave volume on another machine. However I just can’t seem to
get it to work after starting the geo replication volume with the logs showing it failing rsync with error code 14. I can’t seem to find any info about this online so any help would be much appreciated.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:31.24706] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing...</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:31.24891] I [monitor(monitor):157:monitor] Monitor: starting gsyncd worker brick=/data/glusterimagebrick/jfsotc22-gv0 slave_node=<a href="http://pgsotc11.png.intel.com" target="_blank">pgsotc11.png.intel.com</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:31.90935] I [gsyncd(agent /data/glusterimagebrick/jfsotc22-gv0):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/gsyncd.conf</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:31.92105] I [changelogagent(agent /data/glusterimagebrick/jfsotc22-gv0):72:__init__] ChangelogAgent: Agent listining...</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:31.93148] I [gsyncd(worker /data/glusterimagebrick/jfsotc22-gv0):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/gsyncd.conf</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:31.102422] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1366:connect_remote] SSH: Initializing SSH connection between master and
slave...</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:50.355233] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1413:connect_remote] SSH: SSH connection between master and slave established.
duration=19.2526</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:50.355583] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1085:connect] GLUSTER: Mounting gluster volume locally...</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:51.404998] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1108:connect] GLUSTER: Mounted gluster volume duration=1.0492</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:51.405363] I [subcmds(worker /data/glusterimagebrick/jfsotc22-gv0):80:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:53.431502] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1603:register] _GMaster: Working dir path=/var/lib/misc/gluster/gsyncd/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/data-glusterimagebrick-jfsotc22-gv0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:53.431846] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1271:service_loop] GLUSTER: Register time time=1574747213</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:53.445589] I [gsyncdstatus(worker /data/glusterimagebrick/jfsotc22-gv0):281:set_active] GeorepStatus: Worker Status Change status=Active</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:53.446184] I [gsyncdstatus(worker /data/glusterimagebrick/jfsotc22-gv0):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change
status=History Crawl</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:53.446367] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1517:crawl] _GMaster: starting history crawl turns=1 stime=(1574669325,
0) etime=1574747213 entry_stime=None</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:54.448994] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1546:crawl] _GMaster: slave's time stime=(1574669325, 0)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:54.928395] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1954:syncjob] Syncer: Sync Time Taken job=1 num_files=1 return_code=14
duration=0.0162</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:54.928607] E [syncdutils(worker /data/glusterimagebrick/jfsotc22-gv0):809:errlog] Popen: command returned error cmd=rsync -aR0 --inplace
--files-from=- --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args . -e ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-rgpu74f3/de0855b3336b4c3233934fcbeeb3674c.sock
pgsotc11.png.intel.com:/proc/29549/cwd error=14</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:54.935529] I [repce(agent /data/glusterimagebrick/jfsotc22-gv0):97:service_loop] RepceServer: terminating on reaching EOF.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:55.410444] I [monitor(monitor):278:monitor] Monitor: worker died in startup phase brick=/data/glusterimagebrick/jfsotc22-gv0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:46:55.412591] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">[2019-11-26 05:47:05.631944] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing...</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY">….</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks!<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jian Chern<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-MY"> </span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">________<br>
<br>
Community Meeting Calendar:<br>
<br>
APAC Schedule -<br>
Every 2nd and 4th Tuesday at 11:30 AM IST<br>
Bridge: <a href="https://bluejeans.com/441850968" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
NA/EMEA Schedule -<br>
Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
Bridge: <a href="https://bluejeans.com/441850968" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks and Regards,<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Kotresh H R<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks and Regards,<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Kotresh H R<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks and Regards,<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Kotresh H R<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">Thanks and Regards,<o:p></o:p></p>
</div>
<p class="MsoNormal">Kotresh H R<o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>