<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Dear Dietmar, <br>
</p>
<p><br>
</p>
<p>I am very interested in helping you with that geo-replication,
since we also have a setup with geo-replication that is crucial
for the <br>
</p>
<p>backup procedure. I just had a quick look at this and for the
moment, I just can suggest:</p>
<p><span style="color: rgb(0, 0, 0); font-family: "open
sans", "lucida grande", sans-serif; font-size:
15px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-thickness:
initial; text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"> </span></p>
<blockquote type="cite"><span style="color: rgb(0, 0, 0);
font-family: "open sans", "lucida grande",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;">is there any suitable setting in the
gluster-environment which would take influence on the speed of
the processing (current settings attached) ?</span></blockquote>
gluster volume geo-replication <font face="monospace">mvol1
gl-slave-05-int::svol config sync_jobs 9</font>
<p><span style="color: rgb(0, 0, 0); font-family: "open
sans", "lucida grande", sans-serif; font-size:
15px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-thickness:
initial; text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><font
face="monospace"><br>
</font></span></p>
<p><span style="color: rgb(0, 0, 0); font-family: "open
sans", "lucida grande", sans-serif; font-size:
15px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-thickness:
initial; text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><font
face="monospace">in order to increase the number of rsync
processes. <br>
</font></span></p>
<p><span style="color: rgb(0, 0, 0); font-family: "open
sans", "lucida grande", sans-serif; font-size:
15px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-thickness:
initial; text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><font
face="monospace">Furthermore, taken from
<a class="moz-txt-link-freetext" href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/recommended_practices3">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/recommended_practices3</a><br>
</font></span></p>
<p><span style="color: rgb(0, 0, 0); font-family: "open
sans", "lucida grande", sans-serif; font-size:
15px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-thickness:
initial; text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><font
face="monospace"><br>
</font></span></p>
<p><span style="color: rgb(0, 0, 0); font-family: "open
sans", "lucida grande", sans-serif; font-size:
15px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-thickness:
initial; text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><font
face="monospace"> </font></span></p>
<blockquote type="cite"><font face="monospace">
<div class="para">
<div class="title">Performance Tuning</div>
<p> When the following option is set, it has been observed
that there is an increase in geo-replication performance. On
the slave volume, run the following command: </p>
</div>
<pre class="screen"># <code class="command">gluster volume set <em class="replaceable">SLAVE_VOL</em> batch-fsync-delay-usec 0</code></pre>
</font></blockquote>
<p><span style="color: rgb(0, 0, 0); font-family: "open
sans", "lucida grande", sans-serif; font-size:
15px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-thickness:
initial; text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><font
face="monospace">Can you verify that the changelog-files are
consumed?</font></span></p>
<p><span style="color: rgb(0, 0, 0); font-family: "open
sans", "lucida grande", sans-serif; font-size:
15px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-thickness:
initial; text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><font
face="monospace"><br>
</font></span></p>
<p><span style="color: rgb(0, 0, 0); font-family: "open
sans", "lucida grande", sans-serif; font-size:
15px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-thickness:
initial; text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><font
face="monospace">Regards,</font></span></p>
<p><span style="color: rgb(0, 0, 0); font-family: "open
sans", "lucida grande", sans-serif; font-size:
15px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-thickness:
initial; text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><font
face="monospace">Felix</font></span></p>
<div class="moz-cite-prefix">On 03/03/2021 17:28, Dietmar Putz
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:fc56548d-708d-14d6-fc71-cc8946adba5d@3q.video">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>Hi,</p>
<div class="moz-forward-container"><span class="VIiyi" lang="en"><span
class="JLqJ4b ChMk0b" data-language-for-alternatives="en"
data-language-to-translate-into="de" data-phrase-index="0"><span>I'm
having a problem with geo-replication.</span></span> <span
class="JLqJ4b ChMk0b" data-language-for-alternatives="en"
data-language-to-translate-into="de" data-phrase-index="1"><span>A
short summary...</span></span></span> </div>
<div class="moz-forward-container">About two month ago I have
added two further nodes to a distributed replicated volume. For
that purpose I have stopped the geo-replication, added two nodes
on mvol and svol and started a rebalance process on both sides.
Once the rebalance process was finished I startet the
geo-replication again.<br>
<br>
After a few days and beside some Unicode Errors the status of
the new added brick changed from hybrid crawl to history crawl.
Since then no progress, no files / directories have been created
on svol for a couple of days. <br>
</div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container">Looking for a possible reason I
recognized that there is was
/var/log/glusterfs/geo-replication-slaves/mvol1_gl-slave-01-int_svol1
directory on the new added slave nodes.</div>
<div class="moz-forward-container">Obviously I forgot to add the
new svol node IP addresses on all master's /etc/hosts. After
fixing that I did the '... execute gsec_create' and '...create
push-pem force' command again and corresponding directory were
created. Geo-replication started normal, all active sessions
were in history crawl (as shown below) and for a short while
some data were transfered to svol. But for about a week nothing
had changed on svol, 0 byte transferred.<br>
</div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container">Meanwhile i have deleted
(without reset-sync-time) and recreated the geo-rep session. the
current status is as shown below but without any last_synced
date.</div>
<div class="moz-forward-container">an entry like
"last_synced_entry": 1609283145 is still visible in
/var/lib/glusterd/geo-replication/mvol1_gl-slave-01-int_svol1/*status
and changelog files are continously created in
/var/lib/misc/gluster/gsyncd/mvol1_gl-slave-01-int_svol1/<brick>/.processing.
<br>
</div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container">Short time ago i changed
log_level to DEBUG for a moment. Unfortunately I got an
'EOFError: Ran out of input' in gsyncd.log and rebuild of
.processing starts from beginning. <br>
</div>
<div class="moz-forward-container">But one of the first very long
lines in gsyncd.log looks like :<br>
</div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container"><font face="monospace">[2021-03-03
11:59:39.503881] D [repce(worker /brick1/mvol1):215:__call__]
RepceClient: call 9163:139944064358208:1614772779.4982471
history_getchanges ->
['/var/lib/misc/gluster/gsyncd/mvol1_gl-slave-01-int_svol1/brick1-mvol1/.history/.processing/CHANGELOG.1609280278',...</font></div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;">1609280278
means Tuesday, December 29, 2020 10:17:58 PM and would somehow
fit to the last_synced date.</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><br>
</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;">However, I
got nearly 300k files in <brick>/.history/.processing
and in in log/trace it seems that any file in
<brick>/.history/.processing will be processed and
transferred to <brick>/.processing. <br>
</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;">My
questions so far...</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;">first of
all, is everything still ok with this geo-replication ?</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;">do i have
to wait until all changelog files in
<brick>/.history/.processing are processed until
transfers to svol start ?</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;">what
happens if any other error appears in geo-replication while
these changelog files are processed resp. crawl status is
history crawl ... does the entire process starts from the
beginning ? would a checkpiont be helpful...for future
decisions...?</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;">is there
any suitable setting in the gluster-environment which would
take influence on the speed of the processing (current
settings attached) ?</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><br>
</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><br>
</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;">I hope
someone can help...</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><br>
</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;">best
regards</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;">dietmar<br>
</span></div>
<div class="moz-forward-container"><span style="color: rgb(0, 0,
0); font-family: "open sans", "lucida
grande", sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;"><br>
</span></div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container"> <font face="monospace">[
15:17:47 ] - root@gl-master-01
/var/lib/misc/gluster/gsyncd/mvol1_gl-slave-01-int_svol1/brick1-mvol1/.history
$ls .processing/ | wc -l<br>
294669</font></div>
<div class="moz-forward-container"><br>
</div>
<div class="moz-forward-container"><font face="monospace">[
12:56:31 ] - root@gl-master-01 ~ $gluster volume
geo-replication mvol1 gl-slave-01-int::svol1 status<br>
<br>
MASTER NODE MASTER VOL MASTER BRICK SLAVE USER
SLAVE SLAVE NODE STATUS CRAWL
STATUS LAST_SYNCED<br>
----------------------------------------------------------------------------------------------------------------------------------------------------<br>
gl-master-01-int mvol1 /brick1/mvol1 root
gl-slave-01-int::svol1 gl-slave-05-int Active
History Crawl 2020-12-29 23:00:48<br>
gl-master-01-int mvol1 /brick2/mvol1 root
gl-slave-01-int::svol1 gl-slave-03-int Active
History Crawl 2020-12-29 23:05:45<br>
gl-master-05-int mvol1 /brick1/mvol1 root
gl-slave-01-int::svol1 gl-slave-03-int Active
History Crawl 2021-02-20 17:38:38<br>
gl-master-06-int mvol1 /brick1/mvol1 root
gl-slave-01-int::svol1 gl-slave-06-int Passive
N/A N/A<br>
gl-master-03-int mvol1 /brick1/mvol1 root
gl-slave-01-int::svol1 gl-slave-05-int Passive
N/A N/A<br>
gl-master-03-int mvol1 /brick2/mvol1 root
gl-slave-01-int::svol1 gl-slave-04-int Active
History Crawl 2020-12-29 23:07:34<br>
gl-master-04-int mvol1 /brick1/mvol1 root
gl-slave-01-int::svol1 gl-slave-06-int Active
History Crawl 2020-12-29 23:07:22<br>
gl-master-04-int mvol1 /brick2/mvol1 root
gl-slave-01-int::svol1 gl-slave-01-int Passive
N/A N/A<br>
gl-master-02-int mvol1 /brick1/mvol1 root
gl-slave-01-int::svol1 gl-slave-01-int Passive
N/A N/A<br>
gl-master-02-int mvol1 /brick2/mvol1 root
gl-slave-01-int::svol1 gl-slave-06-int Passive
N/A N/A<br>
[ 13:14:47 ] - root@gl-master-01 ~ $</font></div>
<div class="moz-forward-container"><font face="monospace"><br>
</font> </div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: <a class="moz-txt-link-freetext" href="https://meet.google.com/cpu-eiue-hvk">https://meet.google.com/cpu-eiue-hvk</a>
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
</body>
</html>