<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0cm;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
        {mso-style-name:msonormal;
        mso-margin-top-alt:auto;
        margin-right:0cm;
        mso-margin-bottom-alt:auto;
        margin-left:0cm;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
span.gmail-m-133563251013633903hoenzb
        {mso-style-name:gmail-m_-133563251013633903hoenzb;}
span.EmailStyle19
        {mso-style-type:personal-reply;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;}
@page WordSection1
        {size:612.0pt 792.0pt;
        margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
        {page:WordSection1;}
--></style>
</head>
<body lang="DE" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">&gt; There seems to be a bug, please raise a bug. For now as a work around add the following line at the end on all the master node's configuration with<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">&gt; any editor. After adding it on all master nodes, stop and start geo-rep.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">&gt; rsync-options = --ignore-missing-args<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">&gt; configuration file: /var/lib/glusterd/geo-replication/&lt;mastervol&gt;_&lt;slave_node&gt;_&lt;slave/vol&gt;gsyncd.conf<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I did that on the master and interim master. Nothing has changed.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Just to make sure. Here is what I did:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/var/lib/glusterd/geo-replication/</span><span lang="EN-US">/&lt;mastervol&gt;_&lt;slave_node&gt;_&lt;slave_vol&gt;</span><span lang="EN-US">/gsyncd.conf<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[vars]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">stime-xattr-prefix = trusted.glusterfs.d0b96093-d48d-4b92-bd91-7c88a3c33dcb.10dc1a40-70b2-4f3c-b874-2653fa778134<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">rsync-options = --ignore-missing-args<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">gluster volume geo-replication &lt;vol&gt; geoaccount@&lt;</span><span lang="EN-US"> slave_node</span><span lang="EN-US"> &gt;::&lt;vol&gt; stop<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">gluster volume geo-replication &lt;vol&gt; geoaccount@&lt;</span><span lang="EN-US"> slave_node</span><span lang="EN-US"> &gt;::&lt;vol&gt; start<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&gt;&gt; I read somewhere that if I delete the geo-replication with “reset-sync-time”, the changelogs are cleared, but this doesn’t happen.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">&gt; changelogs are not cleared, but in the new geo-rep session, the old changelogs are not used for syncing.&nbsp;
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">When I delete the geo-replication with “delete
</span><span lang="EN-GB">reset-sync-time” and then create it again, I have the same warnings and errors. However, the changelog names are different. It looks like every time a new changelog file is created and I already have hundreds of CHANGELOG.* files.
 Is this a normal behaviour?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"># gluster volume geo-replication &lt;vol&gt; geoaccount@&lt;slave_node&gt;::&lt;vol&gt; delete reset-sync-time<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Deleting geo-replication session between &lt;vol&gt; &amp; geoaccount@&lt;slave_node&gt;::&lt;vol&gt; has been successful<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"># gluster volume geo-replication &lt;vol&gt; geoaccount@&lt;slave_node&gt;::&lt;vol&gt; create<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Total available size of master is greater than available size of slave<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">&lt;slave_node&gt;::&lt;vol&gt; is not empty. Please delete existing files in &lt;slave_node&gt;::&lt;vol&gt; and retry, or use force to continue without deleting the existing files.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">geo-replication command failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"># gluster volume geo-replication &lt;vol&gt; geoaccount@&lt;slave_node&gt;::&lt;vol&gt; create force<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Creating geo-replication session between &lt;vol&gt; &amp; geoaccount@&lt;slave_node&gt;::&lt;vol&gt; has been successful<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"># gluster volume geo-replication &lt;vol&gt; geoaccount@&lt;slave_node&gt;::&lt;vol&gt; start<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Starting geo-replication session between &lt;vol&gt; &amp; geoaccount@&lt;slave_node&gt;::&lt;vol&gt; has been successful<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:42.957526] I [master(worker /bricks/brick1/brick):1460:crawl] _GMaster: slave's time&nbsp; stime=(1536749259, 0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:45.2098] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken duration=1.4931 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:45.2836] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp; files=['CHANGELOG.1536749320']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:47.48713] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.5327 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=1&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:47.50287] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536749320']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:49.110692] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4517 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=3&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:49.112823] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749320']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:51.505287] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4960 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:51.507086] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749320']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:53.557701] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.5014 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=1&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:53.559295] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749320']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:55.642981] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.5239 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=3&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:55.644567] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749320']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:58.44553] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.5315 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:48:58.46200] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536749320']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:00.74062] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.5080 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=1&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:00.75681] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536749320']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:02.155219] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.5041 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=3&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:02.156243] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749320']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:04.564528] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.5122 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:04.566395] E [master(worker /bricks/brick1/brick):1325:process] _GMaster: changelogs could not be processed completely - moving on... files=['CHANGELOG.1536749320']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:04.775491] I [master(worker /bricks/brick1/brick):1374:process] _GMaster: Entry Time Taken&nbsp;&nbsp;&nbsp; MKD=0&nbsp;&nbsp; MKN=0&nbsp;&nbsp; LIN=0&nbsp;&nbsp; SYM=0&nbsp;&nbsp; REN=0&nbsp;&nbsp; RMD=0&nbsp;&nbsp; CRE=0&nbsp;&nbsp; duration=0.0000 UNL=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:04.775701] I [master(worker /bricks/brick1/brick):1384:process] _GMaster: Data/Metadata Time Taken&nbsp;&nbsp;&nbsp; SETA=0&nbsp; SETX=0&nbsp; meta_duration=0.0000&nbsp;&nbsp;&nbsp; data_duration=1536749344.7757&nbsp;&nbsp; DATA=0&nbsp; XATT=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:04.776072] I [master(worker /bricks/brick1/brick):1394:process] _GMaster: Batch Completed&nbsp;&nbsp;&nbsp;&nbsp; changelog_end=1536749320&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; entry_stime=(1536747669, 0)&nbsp;&nbsp;&nbsp;&nbsp; changelog_start=1536749320&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; stime=(1536749319,
 0)&nbsp;&nbsp; duration=21.5680&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; num_changelogs=1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mode=live_changelog<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:44.858145] I [master(worker /bricks/brick1/brick):1460:crawl] _GMaster: slave's time&nbsp; stime=(1536749319, 0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:46.717991] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.5076 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=3&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:46.719527] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749380']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:49.182353] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.5217 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=1&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:49.183965] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749380']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:51.199156] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4758 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=3&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:51.200678] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749380']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:53.629234] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4621 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:53.630747] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749380']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:55.640120] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4507 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=1&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:55.641657] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749380']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:57.669339] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4623 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=3&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:49:57.670969] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749380']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:00.91667] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4546 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:00.93130] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536749380']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:02.129616] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4819 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=1&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:02.130310] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749380']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:04.134572] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4574 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=3&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:04.135277] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;files=['CHANGELOG.1536749380']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:06.488636] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.3901 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:06.489341] E [master(worker /bricks/brick1/brick):1325:process] _GMaster: changelogs could not be processed completely - moving on... files=['CHANGELOG.1536749380']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:06.599170] I [master(worker /bricks/brick1/brick):1374:process] _GMaster: Entry Time Taken&nbsp;&nbsp;&nbsp; MKD=0&nbsp;&nbsp; MKN=0&nbsp;&nbsp; LIN=0&nbsp;&nbsp; SYM=0&nbsp;&nbsp; REN=0&nbsp;&nbsp; RMD=0&nbsp;&nbsp; CRE=0&nbsp;&nbsp; duration=0.0000 UNL=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:06.599361] I [master(worker /bricks/brick1/brick):1384:process] _GMaster: Data/Metadata Time Taken&nbsp;&nbsp;&nbsp; SETA=0&nbsp; SETX=0&nbsp; meta_duration=0.0000&nbsp;&nbsp;&nbsp; data_duration=1536749406.5993&nbsp;&nbsp; DATA=0&nbsp; XATT=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:06.599673] I [master(worker /bricks/brick1/brick):1394:process] _GMaster: Batch Completed&nbsp;&nbsp;&nbsp;&nbsp; changelog_end=1536749380&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; entry_stime=(1536747669, 0)&nbsp;&nbsp;&nbsp;&nbsp; changelog_start=1536749380&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; stime=(1536749379,
 0)&nbsp;&nbsp; duration=21.4289&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; num_changelogs=1&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;mode=live_changelog<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:41.683792] I [master(worker /bricks/brick1/brick):1460:crawl] _GMaster: slave's time&nbsp; stime=(1536749379, 0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:43.507179] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;duration=1.4750 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:43.508754] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749440']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:45.637758] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4549 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=3&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:45.639220] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749440']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:47.641165] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4551 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=1&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:47.642779] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749440']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:49.972248] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4573 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:49.973787] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749440']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:52.134746] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4887 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=3&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:52.136266] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749440']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:54.152932] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.5009 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=1&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:54.154515] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749440']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:56.454425] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4743 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:56.456015] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749440']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:58.606976] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4643 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=3&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:50:58.608525] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749440']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:51:00.625092] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4642 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=1&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:51:00.626614] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536749440']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:51:02.913702] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4518 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:51:02.915240] E [master(worker /bricks/brick1/brick):1325:process] _GMaster: changelogs could not be processed completely - moving on... files=['CHANGELOG.1536749440']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:51:03.147001] I [master(worker /bricks/brick1/brick):1374:process] _GMaster: Entry Time Taken&nbsp;&nbsp;&nbsp; MKD=0&nbsp;&nbsp; MKN=0&nbsp;&nbsp; LIN=0&nbsp;&nbsp; SYM=0&nbsp;&nbsp; REN=0&nbsp;&nbsp; RMD=0&nbsp;&nbsp; CRE=0&nbsp;&nbsp; duration=0.0000 UNL=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:51:03.147419] I [master(worker /bricks/brick1/brick):1384:process] _GMaster: Data/Metadata Time Taken&nbsp;&nbsp;&nbsp; SETA=0&nbsp; SETX=0&nbsp; meta_duration=0.0000&nbsp;&nbsp;&nbsp; data_duration=1536749463.1473&nbsp;&nbsp; DATA=0&nbsp; XATT=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[2018-09-12 10:51:03.148145] I [master(worker /bricks/brick1/brick):1394:process] _GMaster: Batch Completed&nbsp;&nbsp;&nbsp;&nbsp; changelog_end=1536749440&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; entry_stime=(1536747669, 0)&nbsp;&nbsp;&nbsp;&nbsp; changelog_start=1536749440&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; stime=(1536749439,
 0)&nbsp;&nbsp; duration=21.2471&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; num_changelogs=1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mode=live_changelog<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&gt;&gt; ChangelogException: [Errno 2] No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">&gt; Please share the changelog log file to debug this. On the same node where you got this traceback, on same location, share the following log file<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">&gt; &quot;changes-bricks-brick1-brick.log&quot;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">I deleted the changelogs directory in that case. Looks like you shouldn’t do it and the changelog data is also stored somewhere else!?<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&gt;&gt; How can I reset the changelog without deleting all data?</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&gt;&nbsp;</span><span lang="EN-US">I didn't understand clearly what is the requirement here. Could you elaborate?
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">I can only get rid of those warnings and errors if I delete the gluster volume on the nodes including the .glusterfs directory. If I delete the changelogs inside .glusterfs, then I get the error mentioned above (</span><span lang="EN-GB">ChangelogException:
 [Errno 2] No such file or directory</span><span lang="EN-GB">).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Is there another way to start with a fresh geo-replication instead of deleting everything?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p>&nbsp;</o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-GB" style="color:black">Regards,<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-GB" style="color:black">Christian</span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p>&nbsp;</o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Kotresh Hiremath Ravishankar &lt;khiremat@redhat.com&gt;<br>
<b>Date: </b>Wednesday, 12. September 2018 at 07:31<br>
<b>To: </b>&quot;Kotte, Christian (Ext)&quot; &lt;christian.kotte@novartis.com&gt;<br>
<b>Cc: </b>&quot;gluster-users@gluster.org&quot; &lt;gluster-users@gluster.org&gt;<br>
<b>Subject: </b>Re: [Gluster-users] 4.1.x geo-replication &quot;changelogs could not be processed completely&quot; issue<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Answer inline.<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">On Tue, Sep 11, 2018 at 4:19 PM, Kotte, Christian (Ext) &lt;<a href="mailto:christian.kotte@novartis.com" target="_blank">christian.kotte@novartis.com</a>&gt; wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">Hi all,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">I use glusterfs 4.1.3 non-root user geo-replication in a cascading setup. The gsyncd.log on the master is fine, but I have some strange changelog warnings and
 errors on the interimmaster:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">gsyncd.log</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">…</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:35.575464] I [master(worker /bricks/brick1/brick):1460:crawl] _GMaster: slave's time&nbsp; stime=(1536662250, 0)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:37.126749] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4698 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=1&nbsp;&nbsp; return_code=23</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:37.128668] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536662311']</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:39.353209] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4057 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:39.354737] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536662311']</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:41.501187] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4781 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=3&nbsp;&nbsp; return_code=23</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:41.503048] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536662311']</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:43.575047] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4385 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=1&nbsp;&nbsp; return_code=23</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:43.576597] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536662311']</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:45.838089] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4765 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:45.840205] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536662311']</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:47.969033] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4602 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=3&nbsp;&nbsp; return_code=23</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:47.970118] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536662311']</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:50.54420] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4717 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=1&nbsp;&nbsp; return_code=23</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:50.56072] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536662311']</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:52.317955] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4711 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=2&nbsp;&nbsp; return_code=23</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:52.319642] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536662311']</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:54.448926] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4715 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=3&nbsp;&nbsp; return_code=23</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:54.451127] W [master(worker /bricks/brick1/brick):1346:process] _GMaster: incomplete sync, retrying changelogs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; files=['CHANGELOG.1536662311']</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:56.538007] I [master(worker /bricks/brick1/brick):1944:syncjob] Syncer: Sync Time Taken&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; duration=1.4759 num_files=1&nbsp;&nbsp;&nbsp;&nbsp; job=1&nbsp;&nbsp; return_code=23</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:56.538914] E [master(worker /bricks/brick1/brick):1325:process] _GMaster: changelogs could not be processed completely - moving on... files=['CHANGELOG.1536662311']</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:56.544816] I [master(worker /bricks/brick1/brick):1374:process] _GMaster: Entry Time Taken&nbsp;&nbsp;&nbsp; MKD=0&nbsp;&nbsp; MKN=0&nbsp;&nbsp; LIN=0&nbsp;&nbsp; SYM=0&nbsp;&nbsp; REN=0&nbsp;&nbsp; RMD=0&nbsp;&nbsp;
 CRE=0&nbsp;&nbsp; duration=0.0000 UNL=0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:56.545031] I [master(worker /bricks/brick1/brick):1384:process] _GMaster: Data/Metadata Time Taken&nbsp;&nbsp;&nbsp; SETA=0&nbsp; SETX=0&nbsp; meta_duration=0.0000&nbsp;&nbsp;&nbsp;
 data_duration=1536662336.5450&nbsp;&nbsp; DATA=0&nbsp; XATT=0</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:38:56.545356] I [master(worker /bricks/brick1/brick):1394:process] _GMaster: Batch Completed&nbsp;&nbsp;&nbsp;&nbsp; changelog_end=1536662311&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; entry_stime=None&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 changelog_start=1536662311&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; stime=(1536662310, 0)&nbsp;&nbsp; duration=20.9674&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; num_changelogs=1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mode=live_changelog</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">There seems to be a bug, please raise a bug.
</span>For now as a work around add the following line at the end on all the master node's configuration with<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">any editor. After adding it on all master nodes, stop and start geo-rep.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">rsync-options = --ignore-missing-args<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">configuration file: /var/lib/glusterd/geo-replication/&lt;mastervol&gt;_&lt;slave_node&gt;_&lt;slave/vol&gt;gsyncd.conf<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">I had those issues in the past with 4.1.2 as well. I could fix it only by deleting the geo-replication and the gluster volume and re-create everything.
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">If I delete the geo-replication and delete the changelogs directory or the CHANGELOG files, I get this error:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">gsyncd.log</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">…</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">[2018-09-11 10:26:44.928277] E [repce(agent /bricks/brick1/brick):105:worker] &lt;top&gt;: call failed:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">Traceback (most recent call last):</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp; File &quot;/usr/libexec/glusterfs/python/syncdaemon/repce.py&quot;, line 101, in worker</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;&nbsp;&nbsp; res = getattr(self.obj, rmeth)(*in_data[2:])</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp; File &quot;/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py&quot;, line 53, in history</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;&nbsp;&nbsp; num_parallel)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp; File &quot;/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py&quot;, line 101, in cl_history_changelog</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;&nbsp;&nbsp; cls.raise_changelog_err()</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp; File &quot;/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py&quot;, line 28, in raise_changelog_err</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;&nbsp;&nbsp; raise ChangelogException(errn, os.strerror(errn))</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">ChangelogException: [Errno 61] No data available</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">…</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">Or
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">Traceback (most recent call last):</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp; File &quot;/usr/libexec/glusterfs/python/syncdaemon/repce.py&quot;, line 101, in worker</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;&nbsp;&nbsp; res = getattr(self.obj, rmeth)(*in_data[2:])</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp; File &quot;/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py&quot;, line 53, in history</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;&nbsp;&nbsp; num_parallel)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp; File &quot;/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py&quot;, line 101, in cl_history_changelog</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;&nbsp;&nbsp; cls.raise_changelog_err()</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp; File &quot;/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py&quot;, line 28, in raise_changelog_err</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;&nbsp;&nbsp; raise ChangelogException(errn, os.strerror(errn))</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">ChangelogException: [Errno 2] No such file or directory</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">Please share the changelog log file to debug this. On the same node where you got this traceback, on same location, share the following log file<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">&quot;changes-bricks-brick1-brick.log&quot;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">I read somewhere that if I delete the geo-replication with “reset-sync-time”, the changelogs are cleared, but this doesn’t happen.</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">changelogs are not cleared, but in the new geo-rep session, the old changelogs are not used for syncing.&nbsp;
<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">How can I reset the changelog without deleting all data?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">&nbsp;</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">I didn't understand clearly what is the requirement here. Could you elaborate?
<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="color:black">Regards,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="color:black">&nbsp;</span><span style="color:#888888"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="color:black">Christian Kotte</span><span style="color:#888888"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gluster.org_mailman_listinfo_gluster-2Dusers&amp;d=DwMFaQ&amp;c=ZbgFmJjg4pdtrnL2HUJUDw&amp;r=faVOd9yfnSYhe2mQhqlDwcpXGm7x8HN1C9wPmFD3694&amp;m=zVKE4owXl2MJqjkOEtAywQY7sDA-0w4iVYz4wok3Krg&amp;s=oq-U86I4eNBDsHswBSEgeMAxXKvo0ndqWoxEmefz8NU&amp;e=" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">Thanks and Regards,<o:p></o:p></p>
</div>
<p class="MsoNormal">Kotresh H R<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>