<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi <a href="mailto:hunter86_bg@yahoo.com" id="OWAAM982404" class="tWKOu mention ms-bgc-nlr ms-fcl-b" data-loopstyle="linkonly">
@Strahil Nikolov</a>,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">I have checked the ssh connection from all the master servers and I can ssh
</span><span style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">drtier1data</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> from
master1 and master2 server(old master servers), but I am unable to ssh </span><span style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">drtier1data</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> from
master3 (new node).</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(237, 92, 87);"><i>[root@master3 ~]# ssh -i /var/lib/glusterd/geo-replication/secret.pem root@drtier1data</i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(237, 92, 87);"><i>Traceback (most recent call last):</i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(237, 92, 87);"><i> File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in <module></i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(237, 92, 87);"><i> main()</i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(237, 92, 87);"><i> File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 259, in main</i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(237, 92, 87);"><i> if args.subcmd in ("worker"):</i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(237, 92, 87);"><i>TypeError: 'in <string>' requires string as left operand, not NoneType</i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(237, 92, 87);"><i>Connection to drtier1data closed.</i></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><b>But</b> I am able to ssh
</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> </span><span style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">drtier1data</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> from
master3</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> without using the
</span><span style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">georep key.</span></div>
<div><span style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><br>
</span></div>
<div><span style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; color: rgb(12, 136, 42); background-color: rgb(255, 255, 255);">[root@master3 ~]# ssh root@drtier1data</span></div>
<div><span style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; color: rgb(12, 136, 42);">Last login: Sun Jan 28 01:16:25 2024 from 87.246.74.32</span></div>
<div><span style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; color: rgb(12, 136, 42); background-color: rgb(255, 255, 255);">[root@drtier1data ~]# </span></div>
<div><span style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><br>
</span></div>
<div><span style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Also, today I restarted the
gluster server on master1 as geo-replication is trying to be active from master1 server, and sometimes I am getting the following error in gsyncd.log</span></div>
<div><span style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><br>
</span></div>
<div><span style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; color: rgb(200, 38, 19); background-color: rgb(255, 255, 255);"><i>[2024-01-28 01:27:24.722663]
E [syncdutils(worker /opt/tier1data2019/brick):847:errlog] Popen: command returned error [{cmd=rsync -aR0 --inplace --files-from=- --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args . -e ssh -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-0exuoeg7/75785990b3233f5dbbab9f43cc3ed895.sock drtier1data:/proc/553418/cwd}, {error=3}]</i></span></div>
<div id="appendonsend"></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Many thanks,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Anant</div>
<hr style="display: inline-block; width: 98%;">
<div dir="ltr" id="divRplyFwdMsg"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Strahil Nikolov <hunter86_bg@yahoo.com><br>
<b>Sent:</b> 27 January 2024 5:25 AM<br>
<b>To:</b> gluster-users@gluster.org <gluster-users@gluster.org>; Anant Saraswat <anant.saraswat@techblue.co.uk><br>
<b>Subject:</b> Re: [Gluster-users] Geo-replication status is getting Faulty after few seconds</span>
<div> </div>
</div>
<div><span style="font-size: 11pt;">EXTERNAL: Do not click links or open attachments if you do not recognize the sender.<br>
<br>
Don't forget to test with the georep key. I think it was /var/lib/glusterd/geo-replication/secret.pem<br>
<br>
Best Regards,<br>
Strahil Nikolov<br>
<br>
<br>
В събота, 27 януари 2024 г. в 07:24:07 ч. Гринуич+2, Strahil Nikolov <hunter86_bg@yahoo.com> написа:<br>
<br>
<br>
<br>
<br>
<br>
Hi Anant,<br>
<br>
i would first start checking if you can do ssh from all masters to the slave node.If you haven't setup a dedicated user for the session, then gluster is using root.<br>
<br>
Best Regards,<br>
Strahil Nikolov<br>
<br>
<br>
<br>
<br>
<br>
<br>
В петък, 26 януари 2024 г. в 18:07:59 ч. Гринуич+2, Anant Saraswat <anant.saraswat@techblue.co.uk> написа:<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
Hi All,<br>
<br>
<br>
<br>
<br>
I have run the following commands on master3, and that has added master3 to geo-replication.<br>
<br>
<br>
<br>
<br>
gluster system:: execute gsec_create<br>
<br>
gluster volume geo-replication tier1data drtier1data::drtier1data create push-pem force<br>
<br>
gluster volume geo-replication tier1data drtier1data::drtier1data stop<br>
<br>
gluster volume geo-replication tier1data drtier1data::drtier1data start<br>
<br>
<br>
<br>
Now I am able to start the geo-replication, but I am getting the same error.<br>
<br>
<br>
<br>
[2024-01-24 19:51:24.80892] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]<br>
<br>
[2024-01-24 19:51:24.81020] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/opt/tier1data2019/brick}, {slave_node=drtier1data}]<br>
<br>
[2024-01-24 19:51:24.158021] I [resource(worker /opt/tier1data2019/brick):1387:connect_remote] SSH: Initializing SSH connection between master and slave...<br>
<br>
[2024-01-24 19:51:25.951998] I [resource(worker /opt/tier1data2019/brick):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.7938}]<br>
<br>
[2024-01-24 19:51:25.952292] I [resource(worker /opt/tier1data2019/brick):1116:connect] GLUSTER: Mounting gluster volume locally...<br>
<br>
[2024-01-24 19:51:26.986974] I [resource(worker /opt/tier1data2019/brick):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.0346}]<br>
<br>
[2024-01-24 19:51:26.987137] I [subcmds(worker /opt/tier1data2019/brick):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor<br>
<br>
[2024-01-24 19:51:29.139131] I [master(worker /opt/tier1data2019/brick):1662:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/tier1data_drtier1data_drtier1data/opt-tier1data2019-brick}]<br>
<br>
[2024-01-24 19:51:29.139531] I [resource(worker /opt/tier1data2019/brick):1292:service_loop] GLUSTER: Register time [{time=1706125889}]<br>
<br>
[2024-01-24 19:51:29.173877] I [gsyncdstatus(worker /opt/tier1data2019/brick):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]<br>
<br>
[2024-01-24 19:51:29.174407] I [gsyncdstatus(worker /opt/tier1data2019/brick):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]<br>
<br>
[2024-01-24 19:51:29.174558] I [master(worker /opt/tier1data2019/brick):1576:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1705935991, 0)}, {etime=1706125889}, {entry_stime=(1705935991, 0)}]<br>
<br>
[2024-01-24 19:51:30.251965] I [master(worker /opt/tier1data2019/brick):1605:crawl] _GMaster: slave's time [{stime=(1705935991, 0)}]<br>
<br>
[2024-01-24 19:51:30.376715] E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>: Gluster Mount process exited [{error=ENOTCONN}]<br>
<br>
[2024-01-24 19:51:30.991856] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/opt/tier1data2019/brick}]<br>
<br>
[2024-01-24 19:51:30.993608] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]<br>
<br>
Any idea why it's stuck in this loop?<br>
<br>
<br>
<br>
Thanks,<br>
<br>
Anant<br>
<br>
<br>
<br>
<br>
<br>
________________________________<br>
From: Gluster-users <gluster-users-bounces@gluster.org> on behalf of Anant Saraswat <anant.saraswat@techblue.co.uk><br>
Sent: 22 January 2024 9:00 PM<br>
To: gluster-users@gluster.org <gluster-users@gluster.org><br>
Subject: [Gluster-users] Geo-replication status is getting Faulty after few seconds<br>
<br>
<br>
<br>
EXTERNAL: Do not click links or open attachments if you do not recognize the sender.<br>
<br>
Hi There,<br>
<br>
<br>
<br>
<br>
We have a Gluster setup with three master nodes in replicated mode and one slave node with geo-replication.<br>
<br>
<br>
<br>
<br>
# gluster volume info<br>
<br>
Volume Name: tier1data<br>
<br>
Type: Replicate<br>
<br>
Volume ID: 93c45c14-f700-4d50-962b-7653be471e27<br>
<br>
Status: Started<br>
<br>
Snapshot Count: 0<br>
<br>
Number of Bricks: 1 x 3 = 3<br>
<br>
Transport-type: tcp<br>
<br>
Bricks:<br>
<br>
Brick1: master1:/opt/tier1data2019/brick<br>
<br>
Brick2: master2:/opt/tier1data2019/brick<br>
<br>
Brick3: master3:/opt/tier1data2019/brick<br>
<br>
<br>
<br>
<br>
<br>
master1 |master2 | ------------------------------geo-replication----------------------------- | drtier1datamaster3 |<br>
<br>
<br>
<br>
We added the master3 node a few months back, the initial setup consisted of 2 master nodes and one geo-replicated slave(drtier1data).<br>
<br>
<br>
<br>
Our geo-replication was functioning well with the initial two master nodes (master1 and master2), where master1 was active and master2 was in passive mode. However, today, we started experiencing issues where geo-replication suddenly stopped and became stuck
in a loop of Initializing..., Active.. Faulty on master1, while master2 remained in passive mode.<br>
<br>
<br>
<br>
Upon checking the gsyncd.log on the master1 node, we observed the following error (please refer to the attached logs for more details):<br>
<br>
E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>: Gluster Mount process exited [{error=ENOTCONN}]<br>
<br>
<br>
<br>
# gluster volume geo-replication tier1data status<br>
<br>
MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED<br>
<br>
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
<br>
master1 tier1data /opt/tier1data2019/brick root ssh://drtier1data::drtier1data N/A Faulty N/A N/A<br>
<br>
master2 tier1data /opt/tier1data2019/brick root ssh://drtier1data::drtier1data Passive N/A N/A<br>
<br>
<br>
<br>
Suspecting an issue on the drtier1data(slave), I attempted to restart Gluster on the slave node, also tried to restart drtier1data server without any luck.<br>
<br>
<br>
<br>
After that I tried the following command to get the Primary-log-file for geo-replication on master1, and got the following error.<br>
<br>
<br>
<br>
# gluster volume geo-replication tier1data drtier1data::drtier1data config log-file<br>
<br>
Staging failed on master3. Error: Geo-replication session between tier1data and drtier1data::drtier1data does not exist.<br>
<br>
geo-replication command failed<br>
<br>
<br>
<br>
<br>
Master3 was the new node added a few months back, but geo-replication was working until today, and we never added this node under geo-replication.<br>
<br>
After that, I forcefully stopped the geo-replication, thinking that restarting geo-replication might fix the issue. However, now the geo-replication is not starting and is giving the same error.<br>
<br>
# gluster volume geo-replication tier1data drtier1data::drtier1data start force<br>
<br>
Staging failed on master3. Error: Geo-replication session between tier1data and drtier1data::drtier1data does not exist.<br>
<br>
geo-replication command failed<br>
<br>
Can anyone please suggest what I should do next to resolve this issue? As there is 5TB of data in this volume, I don't want to resync the entire data to drtier1data. Instead, I want to resume the sync from where it last stopped.<br>
<br>
<br>
<br>
Thanks in advance for any guidance/help.<br>
<br>
<br>
<br>
Kind regards,<br>
<br>
Anant<br>
<br>
<br>
<br>
DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential
information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this
email from your system.<br>
<br>
If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.<br>
<br>
<br>
<br>
DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential
information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this
email from your system.<br>
<br>
If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.<br>
<br>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://urldefense.com/v3/__https://meet.google.com/cpu-eiue-hvk__;!!I_DbfM1H!FIFMVBFvoomIXp1pMhjtLbD-1B_qztpAUPBHP5MST7a1hcf3FP8o6GkbQwzQUnS2nT_YIQ1MF7GV_PtM0CAQoOCo4VSgaw$" id="OWAcb7dfb72-c5ee-0676-fcf9-b5afa9654cf8" class="OWAAutoLink" data-auth="NotApplicable" data-loopstyle="linkonly">
https://urldefense.com/v3/__https://meet.google.com/cpu-eiue-hvk__;!!I_DbfM1H!FIFMVBFvoomIXp1pMhjtLbD-1B_qztpAUPBHP5MST7a1hcf3FP8o6GkbQwzQUnS2nT_YIQ1MF7GV_PtM0CAQoOCo4VSgaw$</a><br>
Gluster-users mailing list<br>
Gluster-users@gluster.org<br>
<a href="https://urldefense.com/v3/__https://lists.gluster.org/mailman/listinfo/gluster-users__;!!I_DbfM1H!FIFMVBFvoomIXp1pMhjtLbD-1B_qztpAUPBHP5MST7a1hcf3FP8o6GkbQwzQUnS2nT_YIQ1MF7GV_PtM0CAQoOBrmvmlMg$" id="OWA0f68c6a0-78b2-a542-e1eb-289bec14e25d" class="OWAAutoLink" data-auth="NotApplicable" data-loopstyle="linkonly">https://urldefense.com/v3/__https://lists.gluster.org/mailman/listinfo/gluster-users__;!!I_DbfM1H!FIFMVBFvoomIXp1pMhjtLbD-1B_qztpAUPBHP5MST7a1hcf3FP8o6GkbQwzQUnS2nT_YIQ1MF7GV_PtM0CAQoOBrmvmlMg$</a></span></div>
<p style="FONT-SIZE: 10pt; FONT-FAMILY: ARIAL"><span style="FONT-FAMILY: Calibri Light"></p>
<p style="FONT-SIZE: 10pt; FONT-FAMILY: ARIAL">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please
notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if
you have received this email by mistake and delete this email from your system. <br>
<br>
If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.
</span></p>
</body>
</html>