<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi All,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I have run the following commands on master3, and that has added master3 to geo-replication.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(15, 92, 26);"><i>gluster system:: execute gsec_create</i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(15, 92, 26);"><i>gluster volume geo-replication tier1data drtier1data::drtier1data create push-pem force</i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(15, 92, 26);"><i>gluster volume geo-replication tier1data drtier1data::drtier1data stop</i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(15, 92, 26);"><i>gluster volume geo-replication tier1data drtier1data::drtier1data start</i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(15, 92, 26);"><i><br>
</i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Now I am able to start the geo-replication, but I am getting the same error.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:24.80892] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
Change [{status=Initializing...}]</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:24.81020] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/opt/tier1data2019/brick},
{slave_node=drtier1data}]</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:24.158021] I [resource(worker /opt/tier1data2019/brick):1387:connect_remote] SSH: Initializing
SSH connection between master and slave...</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:25.951998] I [resource(worker /opt/tier1data2019/brick):1436:connect_remote] SSH: SSH
connection between master and slave established. [{duration=1.7938}]</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:25.952292] I [resource(worker /opt/tier1data2019/brick):1116:connect] GLUSTER: Mounting
gluster volume locally...</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:26.986974] I [resource(worker /opt/tier1data2019/brick):1139:connect] GLUSTER: Mounted
gluster volume [{duration=1.0346}]</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:26.987137] I [subcmds(worker /opt/tier1data2019/brick):84:subcmd_worker] <top>: Worker
spawn successful. Acknowledging back to monitor</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:29.139131] I [master(worker /opt/tier1data2019/brick):1662:register] _GMaster: Working
dir [{path=/var/lib/misc/gluster/gsyncd/tier1data_drtier1data_drtier1data/opt-tier1data2019-brick}]</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:29.139531] I [resource(worker /opt/tier1data2019/brick):1292:service_loop] GLUSTER: Register
time [{time=1706125889}]</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:29.173877] I [gsyncdstatus(worker /opt/tier1data2019/brick):281:set_active] GeorepStatus:
Worker Status Change [{status=Active}]</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:29.174407] I [gsyncdstatus(worker /opt/tier1data2019/brick):253:set_worker_crawl_status]
GeorepStatus: Crawl Status Change [{status=History Crawl}]</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:29.174558] I [master(worker /opt/tier1data2019/brick):1576:crawl] _GMaster: starting
history crawl [{turns=1}, {stime=(1705935991, 0)}, {etime=1706125889}, {entry_stime=(1705935991, 0)}]</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:30.251965] I [master(worker /opt/tier1data2019/brick):1605:crawl] _GMaster: slave's time
[{stime=(1705935991, 0)}]</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:30.376715] E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>:
Gluster Mount process exited [{error=ENOTCONN}]</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:30.991856] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/opt/tier1data2019/brick}]</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);">[2024-01-24 19:51:30.993608] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
Change [{status=Faulty}]</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
<i><br>
</i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Any idea why it's stuck in this loop?</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Thanks,</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Anant</span></div>
<div id="appendonsend"></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div dir="ltr" id="divRplyFwdMsg"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Gluster-users <gluster-users-bounces@gluster.org> on behalf of Anant Saraswat <anant.saraswat@techblue.co.uk><br>
<b>Sent:</b> 22 January 2024 9:00 PM<br>
<b>To:</b> gluster-users@gluster.org <gluster-users@gluster.org><br>
<b>Subject:</b> [Gluster-users] Geo-replication status is getting Faulty after few seconds</span>
<div> </div>
</div>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="background-color: rgb(250, 247, 190);"><b>EXTERNAL: Do not click links or open attachments if you do not recognize the sender.</b></span></p>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi There,</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">We have a Gluster setup with three master nodes in replicated mode and one slave node
with geo-replication.</span></div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"># gluster volume info</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>Volume Name: tier1data</i></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>Type: Replicate</i></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>Volume ID: 93c45c14-f700-4d50-962b-7653be471e27</i></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>Status: Started</i></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>Snapshot Count: 0</i></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>Number of Bricks: 1 x 3 = 3</i></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>Transport-type: tcp</i></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>Bricks:</i></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>Brick1: master1:/opt/tier1data2019/brick</i></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>Brick2: master2:/opt/tier1data2019/brick</i></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>Brick3: master3:/opt/tier1data2019/brick</i></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">master1 |<br>
master2 | ------------------------------geo-replication----------------------------- | drtier1data<br>
master3 |</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">We added the master3 node a few months back, the initial setup consisted of 2 master
nodes and one geo-replicated slave(</span><span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-weight: 400;">drtier1data</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">).</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Our geo-replication was functioning well with the initial two master nodes (master1
and master2), where master1 was active and master2 was in passive mode. However, today, we started experiencing issues where geo-replication suddenly stopped and became stuck in a loop of Initializing..., Active.. Faulty on master1, while master2 remained
in passive mode.</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Upon checking the gsyncd.log on the master1 node,
we observed the following error (please refer to the attached logs for more details):<br>
<br>
</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b><i>E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>: Gluster Mount process
exited [{error=ENOTCONN}]</i></b></span></p>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b><i><br>
</i></b></span></p>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); font-weight: 400;"># gluster
volume geo-replication tier1data status</span></p>
<div style="direction: ltr; text-align: left; margin: 0px;"><span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19); font-weight: 400;"><i>MASTER
NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED </i></span></div>
<div style="direction: ltr; text-align: left; margin: 0px;"><span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19); font-weight: 400;"><i>-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------</i></span></div>
<div style="direction: ltr; text-align: left; margin: 0px;"><span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19); font-weight: 400;"><i>master1
tier1data /opt/tier1data2019/brick root ssh://drtier1data::drtier1data N/A Faulty N/A N/A </i></span></div>
<div style="direction: ltr; text-align: left; margin: 0px;"><span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19); font-weight: 400;"><i>master2
tier1data /opt/tier1data2019/brick root ssh://drtier1data::drtier1data Passive N/A N/A</i></span></div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<span style="font-size: 12pt;"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Suspecting an issue on the drtier1data(slave), I attempted to restart Gluster on the
slave node, also tried to restart </span><span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-weight: 400;">drtier1data</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> server
without any luck.</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">After that I tried the following command to get the Primary-log-file for geo-replication
on master1, and got the following error.</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"># gluster volume geo-replication tier1data drtier1data::drtier1data config log-file</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>Staging failed on master3. Error: Geo-replication session between tier1data
and drtier1data::drtier1data does not exist.</i></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>geo-replication command failed</i></span></div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Master3 was the new node added a few months back,
but geo-replication was working until today, and we never added this node under geo-replication.<br>
<br>
</span></p>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">After that, I forcefully stopped the geo-replication,
thinking that restarting geo-replication might fix the issue. However, now the geo-replication is not starting and is giving the same error.<br>
<br>
</span></p>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"># gluster volume geo-replication tier1data drtier1data::drtier1data start force</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>Staging failed on master3. Error: Geo-replication session between tier1data
and drtier1data::drtier1data does not exist.</i></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 10pt; color: rgb(200, 38, 19);"><i>geo-replication command failed</i></span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
<br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Can anyone please suggest what I should do next to resolve this issue? As there is
5TB of data in this volume, I don't want to resync the entire data to drtier1data. Instead, I want to resume the sync from where it last stopped.</span></div>
<div style="direction: ltr;"><span style="letter-spacing: normal; font-family: Arial, sans-serif; font-size: 14px; color: rgb(36, 36, 36); font-weight: 400;"><br>
</span></div>
<div style="direction: ltr;"><span style="letter-spacing: normal; font-family: Arial, sans-serif; font-size: 12pt; color: rgb(36, 36, 36); font-weight: 400;">Thanks in advance for any guidance/help.</span></div>
<div style="direction: ltr;"><span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><br>
</span></div>
<div style="direction: ltr;"><span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Kind regards,</span></div>
<div style="direction: ltr;"><span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Anant</span></div>
<div style="direction: ltr;"></div>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px; font-family: ARIAL; font-size: 10pt;">
DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential
information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this
email from your system.<br>
<br>
If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.</p>
<p style="FONT-SIZE: 10pt; FONT-FAMILY: ARIAL"><span style="FONT-FAMILY: Calibri Light"></p>
<p style="FONT-SIZE: 10pt; FONT-FAMILY: ARIAL">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please
notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if
you have received this email by mistake and delete this email from your system. <br>
<br>
If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.
</span></p>
</body>
</html>