<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Answer inline.<br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 11, 2018 at 4:19 PM, Kotte, Christian (Ext) <span dir="ltr"><<a href="mailto:christian.kotte@novartis.com" target="_blank">christian.kotte@novartis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="DE">
<div class="gmail-m_-133563251013633903m_-4368313752582772371WordSection1">
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">Hi all,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">I use glusterfs 4.1.3 non-root user geo-replication in a cascading setup. The gsyncd.log on the master is fine, but I have some strange changelog warnings and errors on the interimmaster:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">gsyncd.log<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">…<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:35.575464] I [master(worker /bricks/brick1/brick):1460:cra<wbr>wl] _GMaster: slave's time stime=(1536662250, 0)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:37.126749] I [master(worker /bricks/brick1/brick):1944:syn<wbr>cjob] Syncer: Sync Time Taken duration=1.4698 num_files=1 job=1 return_code=23<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:37.128668] W [master(worker /bricks/brick1/brick):1346:pro<wbr>cess] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536662311']<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:39.353209] I [master(worker /bricks/brick1/brick):1944:syn<wbr>cjob] Syncer: Sync Time Taken duration=1.4057 num_files=1 job=2 return_code=23<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:39.354737] W [master(worker /bricks/brick1/brick):1346:pro<wbr>cess] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536662311']<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:41.501187] I [master(worker /bricks/brick1/brick):1944:syn<wbr>cjob] Syncer: Sync Time Taken duration=1.4781 num_files=1 job=3 return_code=23<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:41.503048] W [master(worker /bricks/brick1/brick):1346:pro<wbr>cess] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536662311']<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:43.575047] I [master(worker /bricks/brick1/brick):1944:syn<wbr>cjob] Syncer: Sync Time Taken duration=1.4385 num_files=1 job=1 return_code=23<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:43.576597] W [master(worker /bricks/brick1/brick):1346:pro<wbr>cess] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536662311']<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:45.838089] I [master(worker /bricks/brick1/brick):1944:syn<wbr>cjob] Syncer: Sync Time Taken duration=1.4765 num_files=1 job=2 return_code=23<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:45.840205] W [master(worker /bricks/brick1/brick):1346:pro<wbr>cess] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536662311']<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:47.969033] I [master(worker /bricks/brick1/brick):1944:syn<wbr>cjob] Syncer: Sync Time Taken duration=1.4602 num_files=1 job=3 return_code=23<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:47.970118] W [master(worker /bricks/brick1/brick):1346:pro<wbr>cess] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536662311']<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:50.54420] I [master(worker /bricks/brick1/brick):1944:syn<wbr>cjob] Syncer: Sync Time Taken duration=1.4717 num_files=1 job=1 return_code=23<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:50.56072] W [master(worker /bricks/brick1/brick):1346:pro<wbr>cess] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536662311']<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:52.317955] I [master(worker /bricks/brick1/brick):1944:syn<wbr>cjob] Syncer: Sync Time Taken duration=1.4711 num_files=1 job=2 return_code=23<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:52.319642] W [master(worker /bricks/brick1/brick):1346:pro<wbr>cess] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536662311']<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:54.448926] I [master(worker /bricks/brick1/brick):1944:syn<wbr>cjob] Syncer: Sync Time Taken duration=1.4715 num_files=1 job=3 return_code=23<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:54.451127] W [master(worker /bricks/brick1/brick):1346:pro<wbr>cess] _GMaster: incomplete sync, retrying changelogs files=['CHANGELOG.1536662311']<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:56.538007] I [master(worker /bricks/brick1/brick):1944:syn<wbr>cjob] Syncer: Sync Time Taken duration=1.4759 num_files=1 job=1 return_code=23<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:56.538914] E [master(worker /bricks/brick1/brick):1325:pro<wbr>cess] _GMaster: changelogs could not be processed completely - moving on... files=['CHANGELOG.1536662311']<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:56.544816] I [master(worker /bricks/brick1/brick):1374:pro<wbr>cess] _GMaster: Entry Time Taken MKD=0 MKN=0 LIN=0 SYM=0 REN=0 RMD=0 CRE=0 duration=0.0000 UNL=0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:56.545031] I [master(worker /bricks/brick1/brick):1384:pro<wbr>cess] _GMaster: Data/Metadata Time Taken SETA=0 SETX=0 meta_duration=0.0000 data_duration=1536662336.5450 <wbr>
DATA=0 XATT=0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:38:56.545356] I [master(worker /bricks/brick1/brick):1394:pro<wbr>cess] _GMaster: Batch Completed changelog_end=1536662311 <wbr> entry_stime=None changelog_start=1536662311 <wbr>
stime=(1536662310, 0) duration=20.9674 num_changelogs=1 mode=live_changelog</span></p></div></div></blockquote><div><br></div><div><br></div><div>There seems to be a bug, please raise a bug. For now as a work around add the following line at the end on all the master node's configuration with<br></div><div>any editor. After adding it on all master nodes, stop and start geo-rep.<br></div><div><br></div><div>rsync-options = --ignore-missing-args</div><div><br></div><div>configuration file: /var/lib/glusterd/geo-replication/<mastervol>_<slave_node>_<slave/vol>gsyncd.conf</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="DE"><div class="gmail-m_-133563251013633903m_-4368313752582772371WordSection1"><p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">I had those issues in the past with 4.1.2 as well. I could fix it only by deleting the geo-replication and the gluster volume and re-create everything.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">If I delete the geo-replication and delete the changelogs directory or the CHANGELOG files, I get this error:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">gsyncd.log<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">…<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">[2018-09-11 10:26:44.928277] E [repce(agent /bricks/brick1/brick):105:work<wbr>er] <top>: call failed:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">Traceback (most recent call last):<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/repce.py", line 101, in worker<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> res = getattr(self.obj, rmeth)(*in_data[2:])<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/changelogagent.py"<wbr>, line 53, in history<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> num_parallel)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/libgfchangelog.py"<wbr>, line 101, in cl_history_changelog<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> cls.raise_changelog_err()<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/libgfchangelog.py"<wbr>, line 28, in raise_changelog_err<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> raise ChangelogException(errn, os.strerror(errn))<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">ChangelogException: [Errno 61] No data available<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">…<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">Or <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">Traceback (most recent call last):<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/repce.py", line 101, in worker<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> res = getattr(self.obj, rmeth)(*in_data[2:])<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/changelogagent.py"<wbr>, line 53, in history<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> num_parallel)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/libgfchangelog.py"<wbr>, line 101, in cl_history_changelog<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> cls.raise_changelog_err()<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/libgfchangelog.py"<wbr>, line 28, in raise_changelog_err<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"> raise ChangelogException(errn, os.strerror(errn))<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">ChangelogException: [Errno 2] No such file or directory</span></p></div></div></blockquote><div><br></div><div>Please share the changelog log file to debug this. On the same node where you got this traceback, on same location, share the following log file</div><div>"changes-bricks-brick1-brick.log"<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="DE"><div class="gmail-m_-133563251013633903m_-4368313752582772371WordSection1"><p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">I read somewhere that if I delete the geo-replication with “reset-sync-time”, the changelogs are cleared, but this doesn’t happen.</span></p></div></div></blockquote><div><br></div><div>changelogs are not cleared, but in the new geo-rep session, the old changelogs are not used for syncing. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="DE"><div class="gmail-m_-133563251013633903m_-4368313752582772371WordSection1"><p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB">How can I reset the changelog without deleting all data?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u> </span></p></div></div></blockquote><div>I didn't understand clearly what is the requirement here. Could you elaborate? <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="DE"><div class="gmail-m_-133563251013633903m_-4368313752582772371WordSection1"><p class="MsoNormal"><span style="font-size:11pt" lang="EN-GB"><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:black" lang="EN-GB">Regards,<span class="gmail-m_-133563251013633903HOEnZb"><font color="#888888"><u></u><u></u></font></span></span></p><span class="gmail-m_-133563251013633903HOEnZb"><font color="#888888">
<p class="MsoNormal"><span style="font-size:11pt;color:black" lang="EN-GB"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:black" lang="EN-GB">Christian Kotte</span><span style="font-size:11pt;color:black" lang="EN-GB"><u></u><u></u></span></p>
</font></span></div>
</div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail-m_-133563251013633903gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</div></div></div></div></div>