<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>Hi,</p>
<div class="moz-forward-container"><span class="VIiyi" lang="en"><span class="JLqJ4b ChMk0b" data-language-for-alternatives="en" data-language-to-translate-into="de" data-phrase-index="0"><span>I'm
having a problem with geo-replication.</span></span> <span class="JLqJ4b ChMk0b" data-language-for-alternatives="en" data-language-to-translate-into="de" data-phrase-index="1"><span>A
short summary...</span></span></span> </div>
<div class="moz-forward-container">About two month ago I have added
two further nodes to a distributed replicated volume. For that
purpose I have stopped the geo-replication, added two nodes on
mvol and svol and started a rebalance process on both sides. Once
the rebalance process was finished I startet the geo-replication
again.<br>
<br>
After a few days and beside some Unicode Errors the status of the
new added brick changed from hybrid crawl to history crawl. Since
then no progress, no files / directories have been created on svol
for a couple of days. <br>
</div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container">Looking for a possible reason I
recognized that there is was
/var/log/glusterfs/geo-replication-slaves/mvol1_gl-slave-01-int_svol1
directory on the new added slave nodes.</div>
<div class="moz-forward-container">Obviously I forgot to add the new
svol node IP addresses on all master's /etc/hosts. After fixing
that I did the '... execute gsec_create' and '...create push-pem
force' command again and corresponding directory were created.
Geo-replication started normal, all active sessions were in
history crawl (as shown below) and for a short while some data
were transfered to svol. But for about a week nothing had changed
on svol, 0 byte transferred.<br>
</div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container">Meanwhile i have deleted (without
reset-sync-time) and recreated the geo-rep session. the current
status is as shown below but without any last_synced date.</div>
<div class="moz-forward-container">an entry like
"last_synced_entry": 1609283145 is still visible in
/var/lib/glusterd/geo-replication/mvol1_gl-slave-01-int_svol1/*status
and changelog files are continously created in
/var/lib/misc/gluster/gsyncd/mvol1_gl-slave-01-int_svol1/<brick>/.processing.
<br>
</div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container">Short time ago i changed
log_level to DEBUG for a moment. Unfortunately I got an 'EOFError:
Ran out of input' in gsyncd.log and rebuild of .processing starts
from beginning. <br>
</div>
<div class="moz-forward-container">But one of the first very long
lines in gsyncd.log looks like :<br>
</div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container"><font face="monospace">[2021-03-03
11:59:39.503881] D [repce(worker /brick1/mvol1):215:__call__]
RepceClient: call 9163:139944064358208:1614772779.4982471
history_getchanges ->
['/var/lib/misc/gluster/gsyncd/mvol1_gl-slave-01-int_svol1/brick1-mvol1/.history/.processing/CHANGELOG.1609280278',...</font></div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;">1609280278 means Tuesday, December 29,
2020 10:17:58 PM and would somehow fit to the last_synced date.</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;"><br>
</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;">However, I got nearly 300k files in
<brick>/.history/.processing and in in log/trace it seems
that any file in <brick>/.history/.processing will be
processed and transferred to <brick>/.processing. <br>
</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;">My questions so far...</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;">first of all, is everything still ok
with this geo-replication ?</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;">do i have to wait until all changelog
files in <brick>/.history/.processing are processed until
transfers to svol start ?</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;">what happens if any other error
appears in geo-replication while these changelog files are
processed resp. crawl status is history crawl ... does the
entire process starts from the beginning ? would a checkpiont be
helpful...for future decisions...?</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;">is there any suitable setting in the
gluster-environment which would take influence on the speed of
the processing (current settings attached) ?</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;"><br>
</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;"><br>
</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;">I hope someone can help...</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;"><br>
</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;">best regards</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;">dietmar<br>
</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;"><br>
</span></div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container"> <font face="monospace">[
15:17:47 ] - root@gl-master-01
/var/lib/misc/gluster/gsyncd/mvol1_gl-slave-01-int_svol1/brick1-mvol1/.history
$ls .processing/ | wc -l<br>
294669</font></div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container"><font face="monospace">[ 12:56:31
] - root@gl-master-01 ~ $gluster volume geo-replication mvol1
gl-slave-01-int::svol1 status<br>
<br>
MASTER NODE MASTER VOL MASTER BRICK SLAVE USER
SLAVE SLAVE NODE STATUS CRAWL
STATUS LAST_SYNCED<br>
----------------------------------------------------------------------------------------------------------------------------------------------------<br>
gl-master-01-int mvol1 /brick1/mvol1 root
gl-slave-01-int::svol1 gl-slave-05-int Active History
Crawl 2020-12-29 23:00:48<br>
gl-master-01-int mvol1 /brick2/mvol1 root
gl-slave-01-int::svol1 gl-slave-03-int Active History
Crawl 2020-12-29 23:05:45<br>
gl-master-05-int mvol1 /brick1/mvol1 root
gl-slave-01-int::svol1 gl-slave-03-int Active History
Crawl 2021-02-20 17:38:38<br>
gl-master-06-int mvol1 /brick1/mvol1 root
gl-slave-01-int::svol1 gl-slave-06-int Passive
N/A N/A<br>
gl-master-03-int mvol1 /brick1/mvol1 root
gl-slave-01-int::svol1 gl-slave-05-int Passive
N/A N/A<br>
gl-master-03-int mvol1 /brick2/mvol1 root
gl-slave-01-int::svol1 gl-slave-04-int Active History
Crawl 2020-12-29 23:07:34<br>
gl-master-04-int mvol1 /brick1/mvol1 root
gl-slave-01-int::svol1 gl-slave-06-int Active History
Crawl 2020-12-29 23:07:22<br>
gl-master-04-int mvol1 /brick2/mvol1 root
gl-slave-01-int::svol1 gl-slave-01-int Passive
N/A N/A<br>
gl-master-02-int mvol1 /brick1/mvol1 root
gl-slave-01-int::svol1 gl-slave-01-int Passive
N/A N/A<br>
gl-master-02-int mvol1 /brick2/mvol1 root
gl-slave-01-int::svol1 gl-slave-06-int Passive
N/A N/A<br>
[ 13:14:47 ] - root@gl-master-01 ~ $</font></div>
<div class="moz-forward-container"><font face="monospace"><br>
</font>
<pre class="moz-signature">
</pre>
</div>
</body>
</html>