<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><meta content="text/html;charset=UTF-8" http-equiv="Content-Type"></head><body ><div style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt;"><div>From the logs it looks like Geo-rep is unable to mount the Primary volume and getting ENOTCONN(Transport endpoint not connected) error. Check if you mount the Primary volume externally. </div><div><br></div><div>--</div><div id="Zm-_Id_-Sgn" data-sigid="3848334000000010003" data-zbluepencil-ignore="true"><div>Aravinda<br></div></div><div>Kadalu Technologies</div><div class="zmail_extra_hr" style="border-top: 1px solid rgb(204, 204, 204); height: 0px; margin-top: 10px; margin-bottom: 10px; line-height: 0px;"><br></div><div class="zmail_extra" data-zbluepencil-ignore="true"><div><br></div><div id="Zm-_Id_-Sgn1">---- On Tue, 06 Feb 2024 19:19:32 +0530 <b>Anant Saraswat <anant.saraswat@techblue.co.uk></b> wrote ---<br></div><div><br></div><blockquote id="blockquote_zmail" style="margin: 0px;"><div dir="ltr" class="zm_1419299185679087662_parse_8526798574434250048"><div class="x_255514570elementToProof" style="font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Thanks <a href="mailto:aravinda@kadalu.tech" id="x_255514570OWAAM806819" class="x_255514570tWKOu x_255514570mention x_255514570ms-bgc-nlr x_255514570ms-fcl-b" target="_blank"> @Aravinda</a>, Please check the logs from primary master node.<br></div><div class="x_255514570elementToProof" style="font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><div class="x_255514570elementToProof"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:10.150540] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus:
Worker Status Change [{status=Initializing...}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:10.150680] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/opt/tier1data2019/brick},
{slave_node=drtier1data}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:10.225295] I [resource(worker /opt/tier1data2019/brick):1387:connect_remote] SSH: Initializing
SSH connection between master and slave...</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:11.915725] I [resource(worker /opt/tier1data2019/brick):1436:connect_remote] SSH: SSH connection
between master and slave established. [{duration=1.6902}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:11.916033] I [resource(worker /opt/tier1data2019/brick):1116:connect] GLUSTER: Mounting gluster
volume locally...</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:12.950741] I [resource(worker /opt/tier1data2019/brick):1139:connect] GLUSTER: Mounted gluster
volume [{duration=1.0346}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:12.950921] I [subcmds(worker /opt/tier1data2019/brick):84:subcmd_worker] <top>: Worker spawn
successful. Acknowledging back to monitor</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:15.109059] I [master(worker /opt/tier1data2019/brick):1662:register] _GMaster: Working dir
[{path=/var/lib/misc/gluster/gsyncd/tier1data_drtier1data_drtier1data/opt-tier1data2019-brick}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:15.109421] I [resource(worker /opt/tier1data2019/brick):1292:service_loop] GLUSTER: Register
time [{time=1707077715}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:15.115965] I [gsyncdstatus(worker /opt/tier1data2019/brick):281:set_active] GeorepStatus:
Worker Status Change [{status=Active}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:15.116454] I [gsyncdstatus(worker /opt/tier1data2019/brick):253:set_worker_crawl_status]
GeorepStatus: Crawl Status Change [{status=History Crawl}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:15.116628] I [master(worker /opt/tier1data2019/brick):1576:crawl] _GMaster: starting history
crawl [{turns=1}, {stime=(1705935991, 0)}, {etime=1707077715}, {entry_stime=(1705935991, 0)}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:16.194210] I [master(worker /opt/tier1data2019/brick):1605:crawl] _GMaster: slave's time
[{stime=(1705935991, 0)}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:16.323585] E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>:
Gluster Mount process exited [{error=ENOTCONN}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:16.955697] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/opt/tier1data2019/brick}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:16.957722] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change
[{status=Faulty}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-04 20:15:26.993059] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change
[{status=Initializing...}]</span></span></span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div id="x_255514570appendonsend"><br></div><div style="font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><hr style="display : inline-block; width : 98%;"><div dir="ltr" id="x_255514570divRplyFwdMsg"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt"><b>From:</b> Aravinda <<a href="mailto:aravinda@kadalu.tech" target="_blank">aravinda@kadalu.tech</a>><br> <b>Sent:</b> 06 February 2024 1:43 PM<br> <b>To:</b> Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" target="_blank">anant.saraswat@techblue.co.uk</a>><br> <b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br> <b>Subject:</b> Re:_[Gluster-users]_Geo-replication status is getting Faulty after few seconds</span></span></span> <div> <br></div></div><p style="margin-top: 0; margin-bottom: 0;"><span class="highlight" style="background-color:rgb(250, 247, 190)"><b>EXTERNAL: Do not click links or open attachments if you do not recognize the sender.</b></span><br></p><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">Geo-rep delete and create will recreate the session and continue syncing from where it stopped last time. Last sync time is remembered in the xattrs of Brick root of Primary
volume (`trusted.glusterfs.<primary-volume-id>.<secondary-volume-id>.stime`).</span></span><br></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">If reset-sync-time option used with delete, then it will remove the above mentioned xattr. When Geo-rep session is created again, it will start a new session.</span></span><br></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">New session starts with Hybrid crawl then switches to History crawl and then switches to Changelog crawl. History crawl does file system crawl and all the files will be
identified as to be synced. Geo-rep uses rsync to sync the files (Change detection is done externally and rsync only gets the list of files to be synced), rsync will only sync the difference if any.</span></span><br></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">Only issue I see of using this option is it may take lot of time to reach the current time and start syncing the recent files.</span></span><br></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">> </span></span><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">I tried to remove the secret.pem.pub from master1 and create it again. Then, I ran georep create push pem,
which copied the missing entry to the drtier1data (slave) node. However, I am still getting the same error. I have also checked the logs on drtier1data (slave) node and found the following logs.</span></span></span><br></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">Please share the errors you see from Primary geo-rep logs.</span></span><br></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">--</span></span><br></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">Thanks and Regards</span></span><br></div><div id="x_Zm-_Id_-Sgn"><div>Aravinda<br></div><div>Kadalu Technologies<br></div></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></div><div style="line-height : 0px; margin-top : 10px; margin-bottom : 10px; border-top : 1px solid rgb(204, 204, 204); min-height: 0px;"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></div><div id="x_Zm-_Id_-Sgn1">---- On Tue, 06 Feb 2024 18:50:37 +0530 <b>Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" target="_blank">anant.saraswat@techblue.co.uk</a>></b> wrote ---<br></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Hi All,<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Does anyone know what will happen if I delete
the geo-replication with reset-sync-time and create it again, Will it copy the whole data again, or will it just check the files and then copy the new remaining files?</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px"><br></span></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px">Thanks,</span></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px">Anant</span></span></span></span><br></div><div id="x_x_872932236appendonsend"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><hr style="direction : ltr; margin-right : 0px; margin-left : 0px; display : inline-block; width : 98%;"><div id="x_x_872932236divRplyFwdMsg" dir="ltr"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt"><b>From:</b> Gluster-users <<a href="mailto:gluster-users-bounces@gluster.org" id="x_255514570OWAa12ba5de-bb5c-aa67-31b0-1874feda0419" class="x_255514570OWAAutoLink" target="_blank">gluster-users-bounces@gluster.org</a>>
on behalf of Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWA75f7b54d-168d-1314-1b7d-12c4af475a4c" class="x_255514570OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>><br> <b>Sent:</b> 04 February 2024 12:44 PM<br> <b>To:</b> <a href="mailto:gluster-users@gluster.org" id="x_255514570OWA8568aab0-047c-39de-efea-c1e6f09e5eda" class="x_255514570OWAAutoLink" target="_blank">gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" id="x_255514570OWAdd1d4ab4-fe5c-6478-ec9e-193242f938b1" class="x_255514570OWAAutoLink" target="_blank">gluster-users@gluster.org</a>><br> <b>Subject:</b> Re: [Gluster-users] Geo-replication status is getting Faulty after few seconds</span></span></span> <div> <br></div></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="highlight" style="background-color:rgb(250, 247, 190)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><b>EXTERNAL: Do not click links or open attachments if you do not recognize the sender.</b></span></span></span><br></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Hello Everyone,<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">I'm running out of options, I've already tried rebooting the physical servers
without any luck.</span></span></span><br></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Now, I am thinking, should I try to delete the geo-replication with reset-sync-time?
Will that resolve the issue? We have 6TB of data, so will running delete with reset-sync-time copy the whole 6TB of data again, or will it just check the files and then copy the new remaining files?</span></span></span><br></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">I have checked the documentation, and it's not clear what will happen with
the files already present on the slave volume. Will they be recopied, or just checked and skipped in case of deleting the session and creating it again?</span></span></span><br></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div id="x_x_appendonsend"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Anyone have any experience around this situation?
Any help is appreciated.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Many thanks,<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Anant <br></div><hr style="direction : ltr; margin-right : 0px; margin-left : 0px; display : inline-block; width : 98%;"><div dir="ltr" id="x_x_divRplyFwdMsg"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt"><b>From:</b> Gluster-users <<a href="mailto:gluster-users-bounces@gluster.org" id="x_255514570OWA32a90c8b-89d6-99cc-cb59-05db423400bf" class="x_255514570OWAAutoLink" target="_blank">gluster-users-bounces@gluster.org</a>>
on behalf of Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWA25e22d1b-7132-0d2f-7a89-7852e8053221" class="x_255514570OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>><br> <b>Sent:</b> 01 February 2024 6:30 PM<br> <b>To:</b> Aravinda <<a href="mailto:aravinda@kadalu.tech" id="x_255514570OWA4f9443c5-20e1-5dfb-71e1-62dcdf4565a2" class="x_255514570OWAAutoLink" target="_blank">aravinda@kadalu.tech</a>><br> <b>Cc:</b> <a href="mailto:gluster-users@gluster.org" id="x_255514570OWAeeb350cd-c76d-0eb6-ae42-9be2193c0dde" class="x_255514570OWAAutoLink" target="_blank">gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" id="x_255514570OWAfe454d8c-6ab6-8b2e-c189-cc82ae6c7b84" class="x_255514570OWAAutoLink" target="_blank">gluster-users@gluster.org</a>><br> <b>Subject:</b> Re: [Gluster-users] Geo-replication status is getting Faulty after few seconds</span></span></span> <div> <br></div></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="highlight" style="background-color:rgb(250, 247, 190)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><b>EXTERNAL: Do not click links or open attachments if you do not recognize the sender.</b></span></span></span><br></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Hi <a href="mailto:aravinda@kadalu.tech" id="x_255514570OWAa01d30b7-8b03-ba3e-f45a-ee7143526949" class="x_x_x_tWKOu x_x_x_mention x_x_x_ms-bgc-nlr x_x_x_ms-fcl-b OWAAutoLink" target="_blank"> @Aravinda</a>,<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Thanks for opening the gitlab issue.<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">I tried to remove the secret.pem.pub from master1
and create it again. Then, I ran georep create push pem, which copied the missing entry to the drtier1data (slave) node. However, I am still getting the same error. I have also checked the logs on drtier1data (slave) node and found the following logs.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">==> /var/log/glusterfs/geo-replication-slaves/tier1data_drtier1data_drtier1data/gsyncd.log
<==</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-01 18:00:02.93139] I [resource(slave
master1/opt/tier1data2019/brick):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.0831}]</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-01 18:00:02.129343] I [resource(slave
master1/opt/tier1data2019/brick):1166:service_loop] GLUSTER: slave listening</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-01 18:00:06.603335] I [repce(slave master1/opt/tier1data2019/brick):96:service_loop]
RepceServer: terminating on reaching EOF.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">==> /var/log/glusterfs/geo-replication-slaves/tier1data_drtier1data_drtier1data/mnt-master1-opt-tier1data2019-brick.log
<==</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-01 18:00:06.632144 +0000] I [fuse-bridge.c:6233:fuse_thread_proc]
0-fuse: initiating unmount of /tmp/gsyncd-aux-mount-j35hwrxj</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-01 18:00:06.633483 +0000] W [glusterfsd.c:1429:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x817a) [0x7f3797b7717a] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x5646034c5bfd] -->/usr/sbin/glusterfs(cleanup_and_exit+0x58) [0x5646034c5a48] ) 0-: received signum (15), shutting down </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-01 18:00:06.633589 +0000] I [fuse-bridge.c:7063:fini]
0-fuse: Unmounting '/tmp/gsyncd-aux-mount-j35hwrxj'.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-02-01 18:00:06.633622 +0000] I [fuse-bridge.c:7068:fini]
0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-j35hwrxj'.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Thanks,</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Anant</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><hr style="direction : ltr; margin-right : 0px; margin-left : 0px; display : inline-block; width : 98%;"><div dir="ltr" id="x_x_x_divRplyFwdMsg"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt"><b>From:</b> Aravinda <<a href="mailto:aravinda@kadalu.tech" id="x_255514570OWA434ae8eb-408d-089f-ec56-5131acf92867" class="x_255514570OWAAutoLink" target="_blank">aravinda@kadalu.tech</a>><br> <b>Sent:</b> 01 February 2024 8:58 AM<br> <b>To:</b> Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWA19f626d7-1ca6-6799-00a6-b8f712733292" class="x_255514570OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>><br> <b>Cc:</b> <a href="mailto:gluster-users@gluster.org" id="x_255514570OWA1a69c599-488b-976a-3051-cedacb4e8e3e" class="x_255514570OWAAutoLink" target="_blank">gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" id="x_255514570OWAad64cda5-2a11-f9da-6251-e40e845c611b" class="x_255514570OWAAutoLink" target="_blank">gluster-users@gluster.org</a>><br> <b>Subject:</b> Re: [Gluster-users] Geo-replication status is getting Faulty after few seconds</span></span></span> <div> <br></div></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="highlight" style="background-color:rgb(250, 247, 190)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><b>EXTERNAL: Do not click links or
open attachments if you do not recognize the sender.</b></span></span></span></span><br></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">> I think we can enhance Geo-rep to accept configuration option to select a worker as Active
manually. I will think about it and update here if possible.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">Opened a Github issue for the same.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><a href="https://urldefense.com/v3/__https://github.com/gluster/glusterfs/issues/4304__;!!I_DbfM1H!BhqOtEyWh2TghCsLjQAu6kNSYqtFoRKfU8nSfUnVX4ZZfmlMRVka_BTot4C1XsR1fBFnoBMgNlYUi-xIXH9J5jo8GPwD7Ao$" id="x_255514570OWA66592ded-1e8f-744d-275b-3a99fc85aaca" class="x_x_x_OWAAutoLink" target="_blank">https://github.com/gluster/glusterfs/issues/4304</a></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">--</span></span></span><br></div><div id="x_x_x_x_Zm-_Id_-Sgn"><div>Aravinda<br></div></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">Kadalu Technologies</span></span></span><br></div><div style="direction : ltr; line-height : 0px; margin : 10px 0px; border-top : 1px solid rgb(204, 204, 204); min-height : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div id="x_x_x_x_Zm-_Id_-Sgn1">---- On Thu, 01 Feb 2024 07:43:40 +0530 <b>Aravinda <<a href="mailto:aravinda@kadalu.tech" id="x_255514570OWAda8cef4c-eb69-df7e-618b-86ad94d5370d" class="x_255514570OWAAutoLink" target="_blank">aravinda@kadalu.tech</a>></b> wrote ---<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">Hi Anant,</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">> </span></span></span><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Still same thing happening, One thing I have noticed that everytime master1 is trying to be primary node,
how it's selected that which node will be primary node in the geo-replication?</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">Gsyncd worker from the first node of the Replica bricks will become Active and participate
in syncing. If the Active worker node goes down then the other worker becomes Active (Not when it is Faulty). Geo-rep can use Gluster meta volume to hold a lock and automatically switch when a Active worker is Faulty (Refer: Meta Volume config <a href="https://urldefense.com/v3/__https://docs.gluster.org/en/main/Administrator-Guide/Geo-Replication/*configurable-options__;Iw!!I_DbfM1H!BhqOtEyWh2TghCsLjQAu6kNSYqtFoRKfU8nSfUnVX4ZZfmlMRVka_BTot4C1XsR1fBFnoBMgNlYUi-xIXH9J5jo8MnMAKb4$" id="x_255514570OWAd95b10e7-e093-45e9-36d3-e19b0d963056" class="x_x_x_OWAAutoLink" target="_blank"> https://docs.gluster.org/en/main/Administrator-Guide/Geo-Replication/#configurable-options</a>).</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">I think we can enhance Geo-rep to accept configuration option to select a worker as Active
manually. I will think about it and update here if possible.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">> </span></span></span><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Second, I have noticed </span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">in </span></span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px">the ".ssh/authorized_keys" file on the secondry node, master2 and master3 </span></span></span></span><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">have 2 entries (command="/usr/libexec/glusterfs/gsyncd" AND command="tar ${SSH_ORIGINAL_COMMAND#* }", but
master1 node have only one entry with command="tar ${SSH_ORIGINAL_COMMAND#* }" , This indicates that there is a missing entry for master1 with the command="/usr/libexec/glusterfs/gsyncd"</span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px">,
Does it make any sense?</span></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">Missing entry is the problem as you observed. Delete /var/lib/glusterd/geo-replication/secret.pem.pub
file in master1 and then run gsec_create, georep create push pem again.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">--</span></span></span><br></div><div id="x_x_x_x_x_280554324Zm-_Id_-Sgn"><div>Aravinda<br></div><div>Kadalu Technologies<br></div></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; line-height : 0px; margin : 10px 0px; border-top : 1px solid rgb(204, 204, 204); min-height : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div id="x_x_x_x_x_280554324Zm-_Id_-Sgn1">---- On Thu, 01 Feb 2024 00:27:34 +0530 <b>Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWA9bcb887d-ecc0-68d2-b791-52853b0ee5bf" class="x_x_x_OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>></b> wrote ---<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Hi <a href="mailto:aravinda@kadalu.tech" id="x_255514570OWA4616a003-fc2a-05a8-d285-e202c6acbddb" class="x_x_x_x_x_1250338379tWKOu x_x_x_x_x_1250338379mention x_x_x_x_x_1250338379ms-bgc-nlr x_x_x_x_x_1250338379ms-fcl-b x_x_x_OWAAutoLink" target="_blank"> @Aravinda</a>,<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">As advised, I have removed all the master server-related entries from the
".ssh/authorized_keys" file on the secondary node. Then, I ran the "ssh-copy-id root@drtier1data" command on all the masters for passwordless SSH and verified that I can access the drtier1data server from all the master nodes.</span></span></span><br></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">After that, I ran the following commands.</span></span></span><br></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">gluster system:: execute gsec_create</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">gluster volume geo-replication tier1data drtier1data::drtier1data
create push-pem force</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">gluster volume geo-replication tier1data drtier1data::drtier1data
start</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">gluster volume geo-replication tier1data drtier1data::drtier1data
status</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[root@master3 ~]# gluster volume geo-replication
tier1data drtier1data::drtier1data status</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"> </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">MASTER NODE MASTER VOL MASTER
BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">master3 tier1data /opt/tier1data2019/brick
root drtier1data::drtier1data N/A Initializing... N/A N/A </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">master1 tier1data /opt/tier1data2019/brick
root drtier1data::drtier1data N/A Initializing... N/A N/A </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">master2 tier1data /opt/tier1data2019/brick
root drtier1data::drtier1data N/A Initializing... N/A N/A </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[root@master3 ~]# gluster volume geo-replication
tier1data drtier1data::drtier1data status</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"> </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">MASTER NODE MASTER VOL MASTER
BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">master3 tier1data /opt/tier1data2019/brick
root drtier1data::drtier1data Passive N/A N/A </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">master1 tier1data /opt/tier1data2019/brick
root drtier1data::drtier1data Active History Crawl N/A </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">master2 tier1data /opt/tier1data2019/brick
root drtier1data::drtier1data Passive N/A N/A </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[root@master3 ~]# gluster volume geo-replication
tier1data drtier1data::drtier1data status</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">MASTER NODE MASTER VOL MASTER
BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">master3 tier1data /opt/tier1data2019/brick
root drtier1data::drtier1data Passive N/A N/A </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">master1 tier1data /opt/tier1data2019/brick
root drtier1data::drtier1data N/A Faulty N/A N/A </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">master2 tier1data /opt/tier1data2019/brick
root drtier1data::drtier1data Passive N/A N/A </span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Still same thing happening, One thing I have noticed
that everytime master1 is trying to be primary node, how it's selected that which node will be primary node in the geo-replication?</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div id="x_x_x_x_x_1250338379appendonsend"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Second, I have noticed </span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">in </span></span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px">the ".ssh/authorized_keys" file on the secondry node, master2 and master3 </span></span></span></span><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">have 2 entries (command="/usr/libexec/glusterfs/gsyncd" AND command="tar ${SSH_ORIGINAL_COMMAND#* }", but
master1 node have only one entry with command="tar ${SSH_ORIGINAL_COMMAND#* }" , This indicates that there is a missing entry for master1 with the command="/usr/libexec/glusterfs/gsyncd"</span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px">,
Does it make any sense? </span></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px"><br></span></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px">Really appreciate
your help on this.</span></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px"><br></span></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px">Thanks,</span></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px">Anant</span></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px"><br></span></span></span></span></div><hr style="direction : ltr; margin-right : 0px; margin-left : 0px; display : inline-block; width : 98%;"><div id="x_x_x_x_x_1250338379divRplyFwdMsg" dir="ltr"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt"><b>From:</b> Aravinda <<a href="mailto:aravinda@kadalu.tech" id="x_255514570OWAde9bb5b5-18fb-21ff-2a72-7b34856c2e4f" class="x_x_x_OWAAutoLink" target="_blank">aravinda@kadalu.tech</a>><br> <b>Sent:</b> 31 January 2024 5:27 PM<br> <b>To:</b> Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWAa94e544c-4c0b-660d-41a7-070f6e2816a9" class="x_x_x_OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>><br> <b>Cc:</b> <a href="mailto:gluster-users@gluster.org" id="x_255514570OWA39ae0346-208e-f78e-b803-2cf4b24030e6" class="x_x_x_OWAAutoLink" target="_blank">gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" id="x_255514570OWA15cbd525-ecb3-704a-4023-2bd16d560dec" class="x_x_x_OWAAutoLink" target="_blank">gluster-users@gluster.org</a>>;
Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" id="x_255514570OWA54840956-cdf7-898b-4708-d493f53f38ab" class="x_x_x_OWAAutoLink" target="_blank">hunter86_bg@yahoo.com</a>><br> <b>Subject:</b> Re: [Gluster-users] Geo-replication status is getting Faulty after few seconds</span></span></span> <div> <br></div></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="highlight" style="background-color:rgb(250, 247, 190)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><b>EXTERNAL: Do not click links or open attachments if you do not recognize
the sender.</b></span></span></span></span><br></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">In one of the thread, I saw the SSH public key is manually copied to secondary node's authorized_keys
file. Check if that entry starts with "command=" or not. If not added, delete the entry in that file (or delete all Georep related entries in that file) and run Georep create push-pem force again.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">--</span></span></span><br></div><div id="x_x_x_x_x_Zm-_Id_-Sgn"><div>Aravinda<br></div></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">Kadalu Technologies</span></span></span><br></div><div style="direction : ltr; line-height : 0px; margin : 10px 0px; border-top : 1px solid rgb(204, 204, 204); min-height : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div id="x_x_x_x_x_Zm-_Id_-Sgn1">---- On Wed, 31 Jan 2024 17:19:45 +0530 <b>Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWA6f0459f8-33eb-7320-a081-7195391bfdf6" class="x_x_x_OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>></b> wrote
---<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Hi <a href="mailto:aravinda@kadalu.tech" id="x_255514570OWA916ebe74-025f-9e89-5e6c-166afce4df83" class="x_x_x_x_x_x_590519085tWKOu x_x_x_x_x_x_590519085mention x_x_x_x_x_x_590519085ms-bgc-nlr x_x_x_x_x_x_590519085ms-fcl-b x_x_x_x_OWAAutoLink" target="_blank"> @Aravinda</a>,<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">I used the exact same commands when I added master3 to the primary geo-replication.
However, the issue is that the master3 node is in a passive state, and master1 is stuck in a loop of Initializing -> Active -> Faulty. It never considers master2 or master3 as the primary master for geo-replication.</span></span></span><br></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">If master1 can connect to the secondary (drtier1data) server, and I see
the following message in the master1 logs which says "SSH connection between master and slave established.", do you still think it could be related to key issues? I am willing to rerun the commands from master1 if you advise.</span></span></span><br></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">2024-01-30 23:33:14.274611] I [resource(worker
/opt/tier1data2019/brick):1387:connect_remote] SSH: Initializing SSH connection between master and slave...</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-01-30 23:33:15.960004] I [resource(worker
/opt/tier1data2019/brick):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.6852}]</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-01-30 23:33:15.960300] I [resource(worker
/opt/tier1data2019/brick):1116:connect] GLUSTER: Mounting gluster volume locally...</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-01-30 23:33:16.995715] I [resource(worker
/opt/tier1data2019/brick):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.0353}]</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-01-30 23:33:16.995905] I [subcmds(worker
/opt/tier1data2019/brick):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-01-30 23:33:19.154376] I [master(worker
/opt/tier1data2019/brick):1662:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/tier1data_drtier1data_drtier1data/opt-tier1data2019-brick}]</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-01-30 23:33:19.154759] I [resource(worker
/opt/tier1data2019/brick):1292:service_loop] GLUSTER: Register time [{time=1706657599}]</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-01-30 23:33:19.191343] I [gsyncdstatus(worker
/opt/tier1data2019/brick):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-01-30 23:33:19.191940] I [gsyncdstatus(worker
/opt/tier1data2019/brick):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-01-30 23:33:19.192105] I [master(worker
/opt/tier1data2019/brick):1576:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1705935991, 0)}, {etime=1706657599}, {entry_stime=(1705935991, 0)}]</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-01-30 23:33:20.269529] I [master(worker
/opt/tier1data2019/brick):1605:crawl] _GMaster: slave's time [{stime=(1705935991, 0)}]</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-01-30 23:33:20.385018] E [syncdutils(worker
/opt/tier1data2019/brick):346:log_raise_exception] <top>: Gluster Mount process exited [{error=ENOTCONN}]</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-01-30 23:33:21.674] I [monitor(monitor):228:monitor]
Monitor: worker died in startup phase [{brick=/opt/tier1data2019/brick}]</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">[2024-01-30 23:33:21.11514] I [gsyncdstatus(monitor):248:set_worker_status]
GeorepStatus: Worker Status Change [{status=Faulty}]</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Many thanks,</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Anant</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><hr style="direction : ltr; margin-right : 0px; margin-left : 0px; display : inline-block; width : 98%;"><div dir="ltr" id="x_x_x_x_x_x_590519085divRplyFwdMsg"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt"><b>From:</b> Aravinda <<a href="mailto:aravinda@kadalu.tech" id="x_255514570OWA152b6899-79a2-9608-80e0-53e77426f26c" class="x_x_x_x_x_1250338379OWAAutoLink" target="_blank">aravinda@kadalu.tech</a>><br> <b>Sent:</b> 31 January 2024 11:14 AM<br> <b>To:</b> Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWA79c836ab-fe20-a157-8e45-8047ecb1c057" class="x_x_x_x_x_1250338379OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>><br> <b>Cc:</b> <a href="mailto:gluster-users@gluster.org" id="x_255514570OWA3390bec0-8540-9995-f3e9-14d49ea73bf1" class="x_x_x_x_x_1250338379OWAAutoLink" target="_blank">gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" id="x_255514570OWA83fdef4a-b971-2cbe-997c-99d56375fefd" class="x_x_x_x_x_1250338379OWAAutoLink" target="_blank">gluster-users@gluster.org</a>>;
Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" id="x_255514570OWA1aaf5f5c-808f-d435-b47c-6b395eaa73ce" class="x_x_x_x_x_1250338379OWAAutoLink" target="_blank">hunter86_bg@yahoo.com</a>><br> <b>Subject:</b> Re: [Gluster-users] Geo-replication status is getting Faulty after few seconds</span></span></span> <div> <br></div></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="highlight" style="background-color:rgb(250, 247, 190)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><b>EXTERNAL: Do not click links or open attachments if you do not recognize
the sender.</b></span></span></span></span><br></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">Hi Anant,</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">You need to run the gsec_create command when a new node is added to Primary or secondary</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">gluster system:: execute gsec_create</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">gluster volume geo-replication tier1data drtier1data::drtier1data create push-pem force</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">gluster volume geo-replication tier1data drtier1data::drtier1data start</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">gluster volume geo-replication tier1data drtier1data::drtier1data status</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">Or use the Geo-rep setup tool fix the key related issues and resetup (<a href="https://urldefense.com/v3/__https://github.com/aravindavk/gluster-georep-tools__;!!I_DbfM1H!F7q10L7Ll47-D3yZGbN2O2r5kpvRMP5-jO46OzkZu67d3vzF5AozNKd3umkbfJ7_29i2g8TKkhoGfcySbARDGIvyc-VOX7A$" id="x_255514570OWAdb8f84b7-692d-2d8a-719f-83fc525a7cd3" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">https://github.com/aravindavk/gluster-georep-tools</a>)</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">gluster-georep-setup tier1data drtier1data::drtier1data</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">--</span></span></span><br></div><div id="x_x_x_x_x_x_Zm-_Id_-Sgn"><div><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(36, 36, 36)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:14.6667px">gluster system:: execute gsec_create</span></span></span></span><br> <span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(36, 36, 36)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:14.6667px">gluster system:: execute gsec_create</span></span></span></span><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br> </span></span></span><br></div></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">Kadalu Technologies</span></span></span><br></div><div style="direction : ltr; line-height : 0px; margin : 10px 0px; border-top : 1px solid rgb(204, 204, 204); min-height : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div id="x_x_x_x_x_x_Zm-_Id_-Sgn1">---- On Wed, 31 Jan 2024 02:49:08 +0530 <b>Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWAcf251202-35dc-ab0f-dbf4-d52ca3c91c33" class="x_x_x_x_x_1250338379OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>></b> wrote
---<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Hi All,<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">As per the documentation, if we use `delete` only
it will start the replication from the time where it was left before deleting the session, So I tried that without any luck.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(12, 136, 42)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">gluster volume geo-replication tier1data drtier1data::drtier1data
delete</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(12, 136, 42)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">gluster volume geo-replication tier1data drtier1data::drtier1data
create push-pem force</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(12, 136, 42)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">gluster volume geo-replication tier1data drtier1data::drtier1data
start</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(12, 136, 42)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">gluster volume geo-replication tier1data drtier1data::drtier1data
status</span></span></span><br></div><div id="x_x_x_x_x_x_x_1293087509appendonsend"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">I have tried to check the drtier1data logs as
well, and all I can see is master1 connects to drtier1data and send disconnect after 5 seconds, please check the following logs from drtier1data.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 0)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[2024-01-30
21:04:03.016805 +0000] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-drtier1data-client-0: Connected, attached to remote volume [{conn-name=drtier1data-client-0}, {remote_subvol=/opt/tier1data2019/brick}] </i></span></span></span></span><br></div><div style="direction : ltr; text-align : left; margin : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 0)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[2024-01-30 21:04:03.020148
+0000] I [fuse-bridge.c:5296:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.33</i></span></span></span></span><br></div><div style="direction : ltr; text-align : left; margin : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 0)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[2024-01-30 21:04:03.020197
+0000] I [fuse-bridge.c:5924:fuse_graph_sync] 0-fuse: switched to graph 0</i></span></span></span></span><br></div><div style="direction : ltr; text-align : left; margin : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 0)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[2024-01-30 21:04:08.573873
+0000] I [fuse-bridge.c:6233:fuse_thread_proc] 0-fuse: initiating unmount of /tmp/gsyncd-aux-mount-c8c41k2k</i></span></span></span></span><br></div><div style="direction : ltr; text-align : left; margin : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 0)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[2024-01-30 21:04:08.575131
+0000] W [glusterfsd.c:1429:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x817a) [0x7fb907e2e17a] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x55f97b17dbfd] -->/usr/sbin/glusterfs(cleanup_and_exit+0x58) [0x55f97b17da48] ) 0-: received signum (15),
shutting down </i></span></span></span></span><br></div><div style="direction : ltr; text-align : left; margin : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 0)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[2024-01-30 21:04:08.575227
+0000] I [fuse-bridge.c:7063:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-c8c41k2k'.</i></span></span></span></span><br></div><div style="direction : ltr; text-align : left; margin : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 0)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[2024-01-30 21:04:08.575256
+0000] I [fuse-bridge.c:7068:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-c8c41k2k'.</i></span></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Can anyone suggest how can I find the reason of
getting these disconnect requests from master1 or what shall I check next?</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Many thanks,</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">A</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><hr style="direction : ltr; margin-right : 0px; margin-left : 0px; display : inline-block; width : 98%;"><div id="x_x_x_x_x_x_x_1293087509divRplyFwdMsg" dir="ltr"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt"><b>From:</b> Gluster-users <<a href="mailto:gluster-users-bounces@gluster.org" id="x_255514570OWAf943227d-6bce-a2d4-292f-1c414164d9a3" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users-bounces@gluster.org</a>>
on behalf of Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWA162d65fc-b6a7-937d-ba14-ad0f7ea0d952" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>><br> <b>Sent:</b> 30 January 2024 2:14 PM<br> <b>To:</b> <a href="mailto:gluster-users@gluster.org" id="x_255514570OWA31726160-ac00-43e7-2781-1e6e00124bfc" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" id="x_255514570OWAed4dbbc9-1d39-a70b-88da-8dda54190591" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users@gluster.org</a>>;
Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" id="x_255514570OWAd2c1f82e-41c8-e91c-c14c-527133666dc0" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">hunter86_bg@yahoo.com</a>><br> <b>Subject:</b> Re: [Gluster-users] Geo-replication status is getting Faulty after few seconds</span></span></span> <div> <br></div></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="highlight" style="background-color:rgb(250, 247, 190)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><b>EXTERNAL: Do not click links or open attachments if you do not recognize
the sender.</b></span></span></span></span><br></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Hello Everyone, <br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">I am looking for some help. Can anyone please suggest if it's possible to
promote a master node to be the primary in the geo-replication session?</span></span></span><br></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">We have three master nodes and one secondary node. We are facing issues
where geo-replication is consistently failing from the primary master node. We want to check if it works fine from another master node.</span></span></span><br></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Any guidance or assistance would be highly appreciated.<br> <br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Many thanks,<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Anant<br></div><div id="x_x_x_x_x_x_x_appendonsend"><br></div><hr style="direction : ltr; margin-right : 0px; margin-left : 0px; display : inline-block; width : 98%;"><div id="x_x_x_x_x_x_x_divRplyFwdMsg" dir="ltr"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt"><b>From:</b> Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWAaaed4a23-2998-cf42-78f3-619c3493555b" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>><br> <b>Sent:</b> 29 January 2024 3:55 PM<br> <b>To:</b> <a href="mailto:gluster-users@gluster.org" id="x_255514570OWA52427f48-c15a-ccb0-956e-b8a6768ec53d" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" id="x_255514570OWAef3b55e9-b597-dda9-f136-3186b1817c9d" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users@gluster.org</a>>;
Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" id="x_255514570OWAcd4a8f86-c904-00e3-b848-f0c855ca4967" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">hunter86_bg@yahoo.com</a>><br> <b>Subject:</b> Re: [Gluster-users] Geo-replication status is getting Faulty after few seconds</span></span></span> <div> <br></div></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Hi <a href="mailto:hunter86_bg@yahoo.com" id="x_255514570OWA494871b8-9759-98cd-f110-fef2066ca1d4" class="x_x_x_x_x_x_x_x_tWKOu x_x_x_x_x_x_x_x_mention x_x_x_x_x_x_x_x_ms-bgc-nlr x_x_x_x_x_x_x_x_ms-fcl-b x_x_x_x_x_OWAAutoLink" target="_blank"> @Strahil Nikolov</a>,<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">We have been running this geo-replication for
more than 5 years and it was working fine till last week, So I think it shouldn't be something which was missed in the initial setup, but I am unable to understand why it's not working now.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">I have enabled SSH Debug on the secondary node(drtier1data),
and I can see this in the logs.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:52 drtier1data sshd[1268110]:
debug1: server_input_channel_req: channel 0 request exec reply 1</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:52 drtier1data sshd[1268110]:
debug1: session_by_channel: session 0 channel 0</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:52 drtier1data sshd[1268110]:
debug1: session_input_channel_req: session 0 req exec</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:52 drtier1data sshd[1268110]:
Starting session: command for root from XX.236.28.58 port 53082 id 0</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:52 drtier1data sshd[1268095]:
debug1: session_new: session 0</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:58 drtier1data sshd[1268110]:
debug1: Received SIGCHLD.</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:58 drtier1data sshd[1268110]:
debug1: session_by_pid: pid 1268111</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:58 drtier1data sshd[1268110]:
debug1: session_exit_message: session 0 channel 0 pid 1268111</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:58 drtier1data sshd[1268110]:
debug1: session_exit_message: release channel 0</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:58 drtier1data sshd[1268110]:
Received disconnect from XX.236.28.58 port 53082:11: disconnected by user</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:58 drtier1data sshd[1268110]:
Disconnected from user root XX.236.28.58 port 53082</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:58 drtier1data sshd[1268110]:
debug1: do_cleanup</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:58 drtier1data sshd[1268095]:
debug1: do_cleanup</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:58 drtier1data sshd[1268095]:
debug1: PAM: cleanup</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:58 drtier1data sshd[1268095]:
debug1: PAM: closing session</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:58 drtier1data sshd[1268095]:
pam_unix(sshd:session): session closed for user root</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Jan 29 14:25:58 drtier1data sshd[1268095]:
debug1: PAM: deleting credentials</i></span></span></span><br></div><div id="x_x_x_x_x_x_x_x_appendonsend"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">As per the above logs, drtier1data node is getting </span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px"><i>SIGCHLD</i></span></span></span></span><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"> from
master1. (</span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px"><i>Received disconnect from XX.236.28.58 port 53082:11:
disconnected by user)</i></span></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Also, I have checked the gsyncd.log on master1,
which says "SSH: SSH connection between master and slave established. [{duration=1.7277}]", which means passwordless ssh is working fine.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">As per my understanding, Master1 can connect to
the drtier1data server, and then the geo-replication status changes to Active --> History Crawl and then something happens on the master1 which triggers the SSH disconnect.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">is it possible to change the master node in geo-replication
so that we can mark master2 as primary, instead of master1?</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">I am really struggling to fix this issue, Please
help, any pointer is appreciated !!!</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Many thanks,</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Anant</span></span></span><br></div><hr style="direction : ltr; margin-right : 0px; margin-left : 0px; display : inline-block; width : 98%;"><div dir="ltr" id="x_x_x_x_x_x_x_x_divRplyFwdMsg"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt"><b>From:</b> Gluster-users <<a href="mailto:gluster-users-bounces@gluster.org" id="x_255514570OWAbc31a34c-c537-3534-e38d-fd975f170cb6" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users-bounces@gluster.org</a>>
on behalf of Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWA5e1f7327-bddd-4f55-5574-45f37cd87d9f" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>><br> <b>Sent:</b> 29 January 2024 12:20 AM<br> <b>To:</b> <a href="mailto:gluster-users@gluster.org" id="x_255514570OWA9d17b2b1-eba4-c52e-1ab7-92f74ffdb791" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" id="x_255514570OWA9105c6e4-ac06-6fda-9459-b642c1783841" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users@gluster.org</a>>;
Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" id="x_255514570OWA54f12256-339f-3c65-26f4-166dd3a66083" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">hunter86_bg@yahoo.com</a>><br> <b>Subject:</b> Re: [Gluster-users] Geo-replication status is getting Faulty after few seconds</span></span></span> <div> <br></div></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="highlight" style="background-color:rgb(250, 247, 190)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><b>EXTERNAL: Do not click links or open attachments if you do not recognize
the sender.</b></span></span></span></span><br></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">HI </span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Strahil</span></span></span></span><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">,</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">As mentioned in my last email, I have copied the
gluster </span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt">public key from master3 to secondary server, and I can now ssh from all master nodes to secondary server,
but still getting the same error.</span></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i><br></i></span></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[root@master1 geo-replication]# ssh root@drtier1data
-i /var/lib/glusterd/geo-replication/secret.pem</i></span></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Last login: Mon Jan 29 00:14:32 2024 from </i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[root@drtier1data ~]# </i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i><br></i></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[root@master2 ~]# ssh -i /var/lib/glusterd/geo-replication/secret.pem root@drtier1data</i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Last login: Mon Jan 29 00:02:34 2024 from </i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[root@drtier1data ~]# </i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i><br></i></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[root@master3 ~]# ssh -i /var/lib/glusterd/geo-replication/secret.pem root@drtier1data </i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Last login: Mon Jan 29 00:14:41 2024 from </i></span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[root@drtier1data ~]#</i></span></span></span></span><br></div><div id="x_x_x_x_x_x_x_x_x_appendonsend"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Thanks,<br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Anant<br></div><hr style="direction : ltr; margin-right : 0px; margin-left : 0px; display : inline-block; width : 98%;"><div id="x_x_x_x_x_x_x_x_x_divRplyFwdMsg" dir="ltr"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt"><b>From:</b> Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" id="x_255514570OWA4e495c28-3836-dda0-fe8c-5ad9f4896c71" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">hunter86_bg@yahoo.com</a>><br> <b>Sent:</b> 28 January 2024 10:07 PM<br> <b>To:</b> Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWA3cd63301-feed-f324-91e8-fbaad7f67b86" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>>; <a href="mailto:gluster-users@gluster.org" id="x_255514570OWAdbbc7194-64c7-e5d3-b436-24a206770564" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank"> gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" id="x_255514570OWAfb7cbafc-3979-7ed7-7a04-179fe0d22354" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users@gluster.org</a>><br> <b>Subject:</b> Re: [Gluster-users] Geo-replication status is getting Faulty after few seconds</span></span></span> <div> <br></div></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="highlight" style="background-color:rgb(250, 247, 190)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><b>EXTERNAL: Do not click links or
open attachments if you do not recognize the sender.</b></span></span></span></span><br></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Gluster doesn't use the ssh key in /root/.ssh,
thus you need to exchange the public key that corresponds to /var/lib/glusterd/geo-replication/secret.pem . If you don't know the pub key, google how to obtain it from the private key.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Ensure that all hosts can ssh to the secondary
before proceeding with the troubleshooting.</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Best Regards,</span></span></span><br></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">Strahil Nikolov<br> <br></span></span></span></div><blockquote style="margin : 0px 0px 20px;"><div style="direction : ltr;"><span class="colour" style="color:rgb(109, 0, 246)"><span class="font" style="font-family:Roboto, sans-serif"><span class="size" style="font-size:12pt">On Sun, Jan 28, 2024 at 15:58, Anant Saraswat</span></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(109, 0, 246)"><span class="font" style="font-family:Roboto, sans-serif"><span class="size" style="font-size:12pt"><<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWAe7cdca02-06f6-09fd-bfd4-78969c7f99aa" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>>
wrote:</span></span></span><br></div><div style="margin : 10px 0px 0px; padding-top : 10px; padding-left : 20px; border-left : 1px solid rgb(109, 0, 246);"><div id="x_x_x_x_x_x_x_x_x_x_yiv9835342601"><div style="direction : ltr; font-family : Aptos, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Hi All,<br></div><div style="direction : ltr; font-family : Aptos, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">I have now copied </span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:16px"><i> /var/lib/glusterd/geo-replication/secret.pem.pub </i></span></span></span></span><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">(public key) from master3 to drtier1data /root/.ssh/authorized_keys, and now I can ssh from master node3 to drtier1data using the </span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">georep key (/var/lib/glusterd/geo-replication/secret.pem).</span></span></span></span><br></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></span></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">But I am still getting the same error, and geo-replication is getting faulty again and
again.</span></span></span></span><br></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i><br></i></span></span></span></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:38.897683] I [resource(worker /opt/tier1data2019/brick):1292:service_loop] GLUSTER: Register time [{time=1706449598}]</i></span></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:38.922491] I [gsyncdstatus(worker /opt/tier1data2019/brick):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:38.923127] I [gsyncdstatus(worker /opt/tier1data2019/brick):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:38.923313] I [master(worker /opt/tier1data2019/brick):1576:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1705935991, 0)}, {etime=1706449598},
{entry_stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:39.973584] I [master(worker /opt/tier1data2019/brick):1605:crawl] _GMaster: slave's time [{stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:40.98970] E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>: Gluster Mount process exited [{error=ENOTCONN}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:40.757691] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/opt/tier1data2019/brick}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:40.766860] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:50.793311] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:50.793469] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/opt/tier1data2019/brick}, {slave_node=drtier1data}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:50.874474] I [resource(worker /opt/tier1data2019/brick):1387:connect_remote] SSH: Initializing SSH connection between master and slave...</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:52.659114] I [resource(worker /opt/tier1data2019/brick):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.7844}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:52.659461] I [resource(worker /opt/tier1data2019/brick):1116:connect] GLUSTER: Mounting gluster volume locally...</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:53.698769] I [resource(worker /opt/tier1data2019/brick):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.0392}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:53.698984] I [subcmds(worker /opt/tier1data2019/brick):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:55.831999] I [master(worker /opt/tier1data2019/brick):1662:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/tier1data_drtier1data_drtier1data/opt-tier1data2019-brick}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:55.832354] I [resource(worker /opt/tier1data2019/brick):1292:service_loop] GLUSTER: Register time [{time=1706449615}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:55.854684] I [gsyncdstatus(worker /opt/tier1data2019/brick):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:55.855251] I [gsyncdstatus(worker /opt/tier1data2019/brick):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:55.855419] I [master(worker /opt/tier1data2019/brick):1576:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1705935991, 0)}, {etime=1706449615},
{entry_stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:56.905496] I [master(worker /opt/tier1data2019/brick):1605:crawl] _GMaster: slave's time [{stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:57.38262] E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>: Gluster Mount process exited [{error=ENOTCONN}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:57.704128] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/opt/tier1data2019/brick}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:46:57.706743] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:07.741438] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:07.741582] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/opt/tier1data2019/brick}, {slave_node=drtier1data}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:07.821284] I [resource(worker /opt/tier1data2019/brick):1387:connect_remote] SSH: Initializing SSH connection between master and slave...</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:09.573661] I [resource(worker /opt/tier1data2019/brick):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.7521}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:09.573955] I [resource(worker /opt/tier1data2019/brick):1116:connect] GLUSTER: Mounting gluster volume locally...</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:10.612173] I [resource(worker /opt/tier1data2019/brick):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.0381}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:10.612359] I [subcmds(worker /opt/tier1data2019/brick):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:12.751856] I [master(worker /opt/tier1data2019/brick):1662:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/tier1data_drtier1data_drtier1data/opt-tier1data2019-brick}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:12.752237] I [resource(worker /opt/tier1data2019/brick):1292:service_loop] GLUSTER: Register time [{time=1706449632}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:12.759138] I [gsyncdstatus(worker /opt/tier1data2019/brick):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:12.759690] I [gsyncdstatus(worker /opt/tier1data2019/brick):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:12.759868] I [master(worker /opt/tier1data2019/brick):1576:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1705935991, 0)}, {etime=1706449632},
{entry_stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:13.810321] I [master(worker /opt/tier1data2019/brick):1605:crawl] _GMaster: slave's time [{stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:13.924068] E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>: Gluster Mount process exited [{error=ENOTCONN}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:14.617663] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/opt/tier1data2019/brick}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:14.620035] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:24.646013] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:24.646157] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/opt/tier1data2019/brick}, {slave_node=drtier1data}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:24.725510] I [resource(worker /opt/tier1data2019/brick):1387:connect_remote] SSH: Initializing SSH connection between master and slave...</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:26.491939] I [resource(worker /opt/tier1data2019/brick):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.7662}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:26.492235] I [resource(worker /opt/tier1data2019/brick):1116:connect] GLUSTER: Mounting gluster volume locally...</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:27.530852] I [resource(worker /opt/tier1data2019/brick):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.0385}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:27.531036] I [subcmds(worker /opt/tier1data2019/brick):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:29.670099] I [master(worker /opt/tier1data2019/brick):1662:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/tier1data_drtier1data_drtier1data/opt-tier1data2019-brick}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:29.670640] I [resource(worker /opt/tier1data2019/brick):1292:service_loop] GLUSTER: Register time [{time=1706449649}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:29.696144] I [gsyncdstatus(worker /opt/tier1data2019/brick):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:29.696709] I [gsyncdstatus(worker /opt/tier1data2019/brick):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:29.696899] I [master(worker /opt/tier1data2019/brick):1576:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1705935991, 0)}, {etime=1706449649},
{entry_stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:30.751127] I [master(worker /opt/tier1data2019/brick):1605:crawl] _GMaster: slave's time [{stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:30.885824] E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>: Gluster Mount process exited [{error=ENOTCONN}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:31.535252] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/opt/tier1data2019/brick}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:31.538450] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:41.564276] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:41.564426] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/opt/tier1data2019/brick}, {slave_node=drtier1data}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:41.645110] I [resource(worker /opt/tier1data2019/brick):1387:connect_remote] SSH: Initializing SSH connection between master and slave...</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:43.435830] I [resource(worker /opt/tier1data2019/brick):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.7904}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:43.436285] I [resource(worker /opt/tier1data2019/brick):1116:connect] GLUSTER: Mounting gluster volume locally...</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:44.475671] I [resource(worker /opt/tier1data2019/brick):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.0393}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:44.475865] I [subcmds(worker /opt/tier1data2019/brick):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:46.630478] I [master(worker /opt/tier1data2019/brick):1662:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/tier1data_drtier1data_drtier1data/opt-tier1data2019-brick}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:46.630924] I [resource(worker /opt/tier1data2019/brick):1292:service_loop] GLUSTER: Register time [{time=1706449666}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:46.655069] I [gsyncdstatus(worker /opt/tier1data2019/brick):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:46.655752] I [gsyncdstatus(worker /opt/tier1data2019/brick):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:46.655926] I [master(worker /opt/tier1data2019/brick):1576:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1705935991, 0)}, {etime=1706449666},
{entry_stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:47.706875] I [master(worker /opt/tier1data2019/brick):1605:crawl] _GMaster: slave's time [{stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:47.834996] E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>: Gluster Mount process exited [{error=ENOTCONN}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:48.480822] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/opt/tier1data2019/brick}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:48.491306] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:58.518263] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:58.518412] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/opt/tier1data2019/brick}, {slave_node=drtier1data}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:47:58.601096] I [resource(worker /opt/tier1data2019/brick):1387:connect_remote] SSH: Initializing SSH connection between master and slave...</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:00.355000] I [resource(worker /opt/tier1data2019/brick):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.7537}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:00.355345] I [resource(worker /opt/tier1data2019/brick):1116:connect] GLUSTER: Mounting gluster volume locally...</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:01.395025] I [resource(worker /opt/tier1data2019/brick):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.0396}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:01.395212] I [subcmds(worker /opt/tier1data2019/brick):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:03.541059] I [master(worker /opt/tier1data2019/brick):1662:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/tier1data_drtier1data_drtier1data/opt-tier1data2019-brick}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:03.541481] I [resource(worker /opt/tier1data2019/brick):1292:service_loop] GLUSTER: Register time [{time=1706449683}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:03.567552] I [gsyncdstatus(worker /opt/tier1data2019/brick):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:03.568172] I [gsyncdstatus(worker /opt/tier1data2019/brick):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:03.568376] I [master(worker /opt/tier1data2019/brick):1576:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1705935991, 0)}, {etime=1706449683},
{entry_stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:04.621488] I [master(worker /opt/tier1data2019/brick):1605:crawl] _GMaster: slave's time [{stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:04.742268] E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>: Gluster Mount process exited [{error=ENOTCONN}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:04.919335] I [master(worker /opt/tier1data2019/brick):2013:syncjob] Syncer: Sync Time Taken [{job=3}, {num_files=10}, {return_code=3}, {duration=0.0180}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:04.919919] E [syncdutils(worker /opt/tier1data2019/brick):847:errlog] Popen: command returned error [{cmd=rsync -aR0 --inplace --files-from=- --super --stats
--numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args . -e ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-zo_ev6yu/75785990b3233f5dbbab9f43cc3ed895.sock
drtier1data:/proc/799165/cwd}, {error=3}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:05.399226] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/opt/tier1data2019/brick}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:05.403931] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:15.430175] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:15.430308] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/opt/tier1data2019/brick}, {slave_node=drtier1data}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:15.510770] I [resource(worker /opt/tier1data2019/brick):1387:connect_remote] SSH: Initializing SSH connection between master and slave...</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:17.240311] I [resource(worker /opt/tier1data2019/brick):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.7294}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:17.240509] I [resource(worker /opt/tier1data2019/brick):1116:connect] GLUSTER: Mounting gluster volume locally...</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:18.279007] I [resource(worker /opt/tier1data2019/brick):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.0384}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:18.279195] I [subcmds(worker /opt/tier1data2019/brick):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:20.455937] I [master(worker /opt/tier1data2019/brick):1662:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/tier1data_drtier1data_drtier1data/opt-tier1data2019-brick}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:20.456274] I [resource(worker /opt/tier1data2019/brick):1292:service_loop] GLUSTER: Register time [{time=1706449700}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:20.464288] I [gsyncdstatus(worker /opt/tier1data2019/brick):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:20.464807] I [gsyncdstatus(worker /opt/tier1data2019/brick):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:20.464970] I [master(worker /opt/tier1data2019/brick):1576:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1705935991, 0)}, {etime=1706449700},
{entry_stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:21.514201] I [master(worker /opt/tier1data2019/brick):1605:crawl] _GMaster: slave's time [{stime=(1705935991, 0)}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:21.644609] E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>: Gluster Mount process exited [{error=ENOTCONN}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:22.284920] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/opt/tier1data2019/brick}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:22.286189] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:32.312378] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:32.312526] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/opt/tier1data2019/brick}, {slave_node=drtier1data}]</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:32.393484] I [resource(worker /opt/tier1data2019/brick):1387:connect_remote] SSH: Initializing SSH connection between master and slave...</i></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:34.91825] I [resource(worker /opt/tier1data2019/brick):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.6981}]</i></span></span><br></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:10pt"><i>[2024-01-28 13:48:34.92130] I [resource(worker /opt/tier1data2019/brick):1116:connect] GLUSTER: Mounting gluster volume locally...</i></span></span></span><br></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:14.6667px"><br></span></span></span></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:14.6667px">Thanks,</span></span></span><br></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:14.6667px">Anant</span></span></span><br></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:14.6667px"><br></span></span></span></div><hr style="direction : ltr; display : inline-block; width : 98%;"><div id="x_x_x_x_x_x_x_x_x_x_yiv9835342601yqt01645" class="x_x_x_x_x_x_x_x_x_x_yiv9835342601yqt0981169724"><div dir="ltr" id="x_x_x_x_x_x_x_x_x_x_yiv9835342601divRplyFwdMsg"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt"><b>From:</b> Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWAfcce765f-ea77-0b51-aa99-d08208cd604e" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>><br> <b>Sent:</b> 28 January 2024 1:33 AM<br> <b>To:</b> Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" id="x_255514570OWAf03bbc01-a196-6bff-8f7d-7a67abf528dc" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">hunter86_bg@yahoo.com</a>>; <a href="mailto:gluster-users@gluster.org" id="x_255514570OWAcccbfe2f-d629-3205-370a-a5928a0d5ee4" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank"> gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" id="x_255514570OWA13cc4b7a-bec3-c902-454a-56861d617a32" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users@gluster.org</a>><br> <b>Subject:</b> Re: [Gluster-users] Geo-replication status is getting Faulty after few seconds</span></span></span> <div> <br></div></div><div style="direction : ltr; font-family : Aptos, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Hi <a href="mailto:hunter86_bg@yahoo.com" id="x_255514570OWAb89110e3-f4b4-308a-65e6-f06c6b28d51a" class="x_x_x_x_x_x_x_x_x_x_yiv9835342601x_tWKOu x_x_x_x_x_x_x_x_x_x_yiv9835342601x_mention x_x_x_x_x_x_x_x_x_x_yiv9835342601x_ms-bgc-nlr x_x_x_x_x_x_x_x_x_x_yiv9835342601x_ms-fcl-b x_x_x_x_x_x_x_x_x_OWAAutoLink" target="_blank"> @Strahil Nikolov</a>,<br></div><div style="direction : ltr; font-family : Aptos, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt">I have checked the ssh connection from all the master servers and I can ssh </span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:14.6667px">drtier1data</span></span></span><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"> from master1 and master2 server(old
master servers), but I am unable to ssh </span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:14.6667px">drtier1data</span></span></span><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"> from
master3 (new node).</span></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>[root@master3 ~]# ssh -i /var/lib/glusterd/geo-replication/secret.pem root@drtier1data</i></span></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Traceback (most recent call last):</i></span></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i> File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in <module></i></span></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i> main()</i></span></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i> File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 259, in main</i></span></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i> if args.subcmd in ("worker"):</i></span></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>TypeError: 'in <string>' requires string as left operand, not NoneType</i></span></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(237, 92, 87)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><i>Connection to drtier1data closed.</i></span></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><div style="direction : ltr;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><b>But</b> I am able to ssh </span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"> </span></span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:14.6667px">drtier1data</span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"> from
master3</span></span></span></span><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"> without using the </span></span></span><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:14.6667px">georep key.</span></span></span><br></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:14.6667px"><br></span></span></span></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(12, 136, 42)"><span class="size" style="font-size:14.6667px">[root@master3 ~]# ssh root@drtier1data</span></span></span><br></div><div style="direction : ltr;"><span class="colour" style="color:rgb(12, 136, 42)"><span class="size" style="font-size:14.6667px">Last login: Sun Jan 28 01:16:25 2024 from 87.246.74.32</span></span><br></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(12, 136, 42)"><span class="size" style="font-size:14.6667px">[root@drtier1data ~]# </span></span></span><br></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:14.6667px"><br></span></span></span></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:14.6667px">Also, today I restarted the gluster server on master1 as geo-replication is trying to be active from master1 server, and sometimes I
am getting the following error in gsyncd.log</span></span></span><br></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(0, 0, 0)"><span class="size" style="font-size:14.6667px"><br></span></span></span></div><div style="direction : ltr;"><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(200, 38, 19)"><span class="size" style="font-size:14.6667px"><i>[2024-01-28 01:27:24.722663] E [syncdutils(worker /opt/tier1data2019/brick):847:errlog] Popen: command returned error [{cmd=rsync
-aR0 --inplace --files-from=- --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args . -e ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto
-S /tmp/gsyncd-aux-ssh-0exuoeg7/75785990b3233f5dbbab9f43cc3ed895.sock drtier1data:/proc/553418/cwd}, {error=3}]</i></span></span></span><br></div><div id="x_x_x_x_x_x_x_x_x_x_yiv9835342601x_appendonsend"><br></div><div style="direction : ltr; font-family : Aptos, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);"><br></div><div style="direction : ltr; font-family : Aptos, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Many thanks,<br></div><div style="direction : ltr; font-family : Aptos, Calibri, Helvetica, sans-serif; font-size : 12pt; color : rgb(0, 0, 0);">Anant<br></div><hr style="direction : ltr; display : inline-block; width : 98%;"><div dir="ltr" id="x_x_x_x_x_x_x_x_x_x_yiv9835342601x_divRplyFwdMsg"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt"><b>From:</b> Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" id="x_255514570OWAcdeb4e56-4658-87c1-f3d0-abe4782f7352" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">hunter86_bg@yahoo.com</a>><br> <b>Sent:</b> 27 January 2024 5:25 AM<br> <b>To:</b> <a href="mailto:gluster-users@gluster.org" id="x_255514570OWA3014556e-1ab8-07fa-2df4-1077008278b5" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" id="x_255514570OWAa9c8d924-50a4-0f44-3412-555e8ae04586" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users@gluster.org</a>>;
Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWAef404747-cc1e-2e8c-fffd-7e4f28b2c639" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>><br> <b>Subject:</b> Re: [Gluster-users] Geo-replication status is getting Faulty after few seconds</span></span></span> <div> <br></div></div><div style="direction : ltr;"><span class="size" style="font-size:11pt">EXTERNAL: Do not click links or open attachments if you do not recognize the sender.<br> <br> Don't forget to test with the georep key. I think it was /var/lib/glusterd/geo-replication/secret.pem<br> <br> Best Regards,<br> Strahil Nikolov<br> <br> <br> В събота, 27 януари 2024 г. в 07:24:07 ч. Гринуич+2, Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" id="x_255514570OWAd2257af2-e3ed-e01a-1b82-59dc1a7fdf91" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">hunter86_bg@yahoo.com</a>> написа:<br> <br> <br> <br> <br> <br> Hi Anant,<br> <br> i would first start checking if you can do ssh from all masters to the slave node.If you haven't setup a dedicated user for the session, then gluster is using root.<br> <br> Best Regards,<br> Strahil Nikolov<br> <br> <br> <br> <br> <br> <br> В петък, 26 януари 2024 г. в 18:07:59 ч. Гринуич+2, Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWA97bb0bb3-87b5-e75b-35fc-a94afa3a9f84" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>>
написа:<br> <br> <br> <br> <br> <br> <br> <br> Hi All,<br> <br> <br> <br> <br> I have run the following commands on master3, and that has added master3 to geo-replication.<br> <br> <br> <br> <br> gluster system:: execute gsec_create<br> <br> gluster volume geo-replication tier1data drtier1data::drtier1data create push-pem force<br> <br> gluster volume geo-replication tier1data drtier1data::drtier1data stop<br> <br> gluster volume geo-replication tier1data drtier1data::drtier1data start<br> <br> <br> <br> Now I am able to start the geo-replication, but I am getting the same error.<br> <br> <br> <br> [2024-01-24 19:51:24.80892] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]<br> <br> [2024-01-24 19:51:24.81020] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/opt/tier1data2019/brick}, {slave_node=drtier1data}]<br> <br> [2024-01-24 19:51:24.158021] I [resource(worker /opt/tier1data2019/brick):1387:connect_remote] SSH: Initializing SSH connection between master and slave...<br> <br> [2024-01-24 19:51:25.951998] I [resource(worker /opt/tier1data2019/brick):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.7938}]<br> <br> [2024-01-24 19:51:25.952292] I [resource(worker /opt/tier1data2019/brick):1116:connect] GLUSTER: Mounting gluster volume locally...<br> <br> [2024-01-24 19:51:26.986974] I [resource(worker /opt/tier1data2019/brick):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.0346}]<br> <br> [2024-01-24 19:51:26.987137] I [subcmds(worker /opt/tier1data2019/brick):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor<br> <br> [2024-01-24 19:51:29.139131] I [master(worker /opt/tier1data2019/brick):1662:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/tier1data_drtier1data_drtier1data/opt-tier1data2019-brick}]<br> <br> [2024-01-24 19:51:29.139531] I [resource(worker /opt/tier1data2019/brick):1292:service_loop] GLUSTER: Register time [{time=1706125889}]<br> <br> [2024-01-24 19:51:29.173877] I [gsyncdstatus(worker /opt/tier1data2019/brick):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]<br> <br> [2024-01-24 19:51:29.174407] I [gsyncdstatus(worker /opt/tier1data2019/brick):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]<br> <br> [2024-01-24 19:51:29.174558] I [master(worker /opt/tier1data2019/brick):1576:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1705935991, 0)}, {etime=1706125889}, {entry_stime=(1705935991, 0)}]<br> <br> [2024-01-24 19:51:30.251965] I [master(worker /opt/tier1data2019/brick):1605:crawl] _GMaster: slave's time [{stime=(1705935991, 0)}]<br> <br> [2024-01-24 19:51:30.376715] E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>: Gluster Mount process exited [{error=ENOTCONN}]<br> <br> [2024-01-24 19:51:30.991856] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/opt/tier1data2019/brick}]<br> <br> [2024-01-24 19:51:30.993608] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]<br> <br> Any idea why it's stuck in this loop?<br> <br> <br> <br> Thanks,<br> <br> Anant<br> <br> <br> <br> <br> <br> ________________________________<br> From: Gluster-users <<a href="mailto:gluster-users-bounces@gluster.org" id="x_255514570OWAccb15f39-6c9f-f166-18eb-13a0fb99ff83" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users-bounces@gluster.org</a>> on behalf of Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="x_255514570OWA8e3daa51-82db-b1ef-bcc4-35ad70ca3d28" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">anant.saraswat@techblue.co.uk</a>><br> Sent: 22 January 2024 9:00 PM<br> To: <a href="mailto:gluster-users@gluster.org" id="x_255514570OWA7722b9a2-8a62-0b1c-6a6b-e702041a7563" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank"> gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" id="x_255514570OWA65407f9d-eb86-008c-6007-631300b1b775" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">gluster-users@gluster.org</a>><br> Subject: [Gluster-users] Geo-replication status is getting Faulty after few seconds<br> <br> <br> <br> EXTERNAL: Do not click links or open attachments if you do not recognize the sender.<br> <br> Hi There,<br> <br> <br> <br> <br> We have a Gluster setup with three master nodes in replicated mode and one slave node with geo-replication.<br> <br> <br> <br> <br> # gluster volume info<br> <br> Volume Name: tier1data<br> <br> Type: Replicate<br> <br> Volume ID: 93c45c14-f700-4d50-962b-7653be471e27<br> <br> Status: Started<br> <br> Snapshot Count: 0<br> <br> Number of Bricks: 1 x 3 = 3<br> <br> Transport-type: tcp<br> <br> Bricks:<br> <br> Brick1: master1:/opt/tier1data2019/brick<br> <br> Brick2: master2:/opt/tier1data2019/brick<br> <br> Brick3: master3:/opt/tier1data2019/brick<br> <br> <br> <br> <br> <br> master1 |master2 | ------------------------------geo-replication----------------------------- | drtier1datamaster3 |<br> <br> <br> <br> We added the master3 node a few months back, the initial setup consisted of 2 master nodes and one geo-replicated slave(drtier1data).<br> <br> <br> <br> Our geo-replication was functioning well with the initial two master nodes (master1 and master2), where master1 was active and master2 was in passive mode. However, today, we started experiencing issues where geo-replication suddenly stopped and became stuck
in a loop of Initializing..., Active.. Faulty on master1, while master2 remained in passive mode.<br> <br> <br> <br> Upon checking the gsyncd.log on the master1 node, we observed the following error (please refer to the attached logs for more details):<br> <br> E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception] <top>: Gluster Mount process exited [{error=ENOTCONN}]<br> <br> <br> <br> # gluster volume geo-replication tier1data status<br> <br> MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED<br> <br> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br> <br> master1 tier1data /opt/tier1data2019/brick root ssh://drtier1data::drtier1data N/A Faulty N/A N/A<br> <br> master2 tier1data /opt/tier1data2019/brick root ssh://drtier1data::drtier1data Passive N/A N/A<br> <br> <br> <br> Suspecting an issue on the drtier1data(slave), I attempted to restart Gluster on the slave node, also tried to restart drtier1data server without any luck.<br> <br> <br> <br> After that I tried the following command to get the Primary-log-file for geo-replication on master1, and got the following error.<br> <br> <br> <br> # gluster volume geo-replication tier1data drtier1data::drtier1data config log-file<br> <br> Staging failed on master3. Error: Geo-replication session between tier1data and drtier1data::drtier1data does not exist.<br> <br> geo-replication command failed<br> <br> <br> <br> <br> Master3 was the new node added a few months back, but geo-replication was working until today, and we never added this node under geo-replication.<br> <br> After that, I forcefully stopped the geo-replication, thinking that restarting geo-replication might fix the issue. However, now the geo-replication is not starting and is giving the same error.<br> <br> # gluster volume geo-replication tier1data drtier1data::drtier1data start force<br> <br> Staging failed on master3. Error: Geo-replication session between tier1data and drtier1data::drtier1data does not exist.<br> <br> geo-replication command failed<br> <br> Can anyone please suggest what I should do next to resolve this issue? As there is 5TB of data in this volume, I don't want to resync the entire data to drtier1data. Instead, I want to resume the sync from where it last stopped.<br> <br> <br> <br> Thanks in advance for any guidance/help.<br> <br> <br> <br> Kind regards,<br> <br> Anant<br> <br> <br> <br> DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential
information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this
email from your system.<br> <br> If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.<br> <br> <br> <br> DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential
information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this
email from your system.<br> <br> If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.<br> <br> ________<br> <br> <br> <br> Community Meeting Calendar:<br> <br> Schedule -<br> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br> Bridge: <a href="https://urldefense.com/v3/__https://meet.google.com/cpu-eiue-hvk__;!!I_DbfM1H!FIFMVBFvoomIXp1pMhjtLbD-1B_qztpAUPBHP5MST7a1hcf3FP8o6GkbQwzQUnS2nT_YIQ1MF7GV_PtM0CAQoOCo4VSgaw$" id="x_255514570OWA20f52945-12e3-6465-363e-1a1281751a90" class="x_x_x_x_x_x_x_x_x_x_yiv9835342601x_OWAAutoLink" target="_blank"> https://urldefense.com/v3/__https://meet.google.com/cpu-eiue-hvk__;!!I_DbfM1H!FIFMVBFvoomIXp1pMhjtLbD-1B_qztpAUPBHP5MST7a1hcf3FP8o6GkbQwzQUnS2nT_YIQ1MF7GV_PtM0CAQoOCo4VSgaw$</a><br> Gluster-users mailing list<br> <a href="mailto:Gluster-users@gluster.org" id="x_255514570OWA3869187c-5175-21bd-a207-88cd4685b341" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">Gluster-users@gluster.org</a><br> <a href="https://urldefense.com/v3/__https://lists.gluster.org/mailman/listinfo/gluster-users__;!!I_DbfM1H!FIFMVBFvoomIXp1pMhjtLbD-1B_qztpAUPBHP5MST7a1hcf3FP8o6GkbQwzQUnS2nT_YIQ1MF7GV_PtM0CAQoOBrmvmlMg$" id="x_255514570OWA10cf1611-1643-2b6d-7c1f-b63da7f8dd2e" class="x_x_x_x_x_x_x_x_x_x_yiv9835342601x_OWAAutoLink" target="_blank">https://urldefense.com/v3/__https://lists.gluster.org/mailman/listinfo/gluster-users__;!!I_DbfM1H!FIFMVBFvoomIXp1pMhjtLbD-1B_qztpAUPBHP5MST7a1hcf3FP8o6GkbQwzQUnS2nT_YIQ1MF7GV_PtM0CAQoOBrmvmlMg$</a></span></div><p style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:ARIAL"><span class="size" style="font-size:10pt">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email.
Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.<br> <br> If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.</span></span></p></div></div></div></blockquote><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:ARIAL"><span class="size" style="font-size:10pt">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy
this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.<br> <br> If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.</span></span></span></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:ARIAL"><span class="size" style="font-size:10pt">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy
this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.<br> <br> If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.</span></span></span></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:ARIAL"><span class="size" style="font-size:10pt">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy
this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.<br> <br> If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.</span></span></span></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">________<br> <br> <br> <br> Community Meeting Calendar:<br> <br> Schedule -<br> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br> Bridge: <a href="https://urldefense.com/v3/__https://meet.google.com/cpu-eiue-hvk__;!!I_DbfM1H!F7q10L7Ll47-D3yZGbN2O2r5kpvRMP5-jO46OzkZu67d3vzF5AozNKd3umkbfJ7_29i2g8TKkhoGfcySbARDGIvy66WGiu4$" id="x_255514570OWA7e46d96e-66f5-ce87-1b23-c7adf7870b43" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank"> https://meet.google.com/cpu-eiue-hvk</a><br> Gluster-users mailing list<br> <a href="mailto:Gluster-users@gluster.org" id="x_255514570OWA95dee224-ce1b-23ce-f17f-7f4a811e5a16" class="x_x_x_x_x_1250338379OWAAutoLink" target="_blank">Gluster-users@gluster.org</a><br> <a href="https://urldefense.com/v3/__https://lists.gluster.org/mailman/listinfo/gluster-users__;!!I_DbfM1H!F7q10L7Ll47-D3yZGbN2O2r5kpvRMP5-jO46OzkZu67d3vzF5AozNKd3umkbfJ7_29i2g8TKkhoGfcySbARDGIvyRXX72tM$" id="x_255514570OWAd6e9638b-abd7-b813-67de-e541d397cfb5" class="x_x_x_x_x_x_590519085OWAAutoLink" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:ARIAL"><span class="size" style="font-size:10pt">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy
this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.<br> <br> If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.</span></span></span></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:ARIAL"><span class="size" style="font-size:10pt">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy
this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.<br> <br> If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.</span></span></span></p><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">________<br> <br> <br> <br> Community Meeting Calendar:<br> <br> Schedule -<br> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br> Bridge: <a href="https://urldefense.com/v3/__https://meet.google.com/cpu-eiue-hvk__;!!I_DbfM1H!BhqOtEyWh2TghCsLjQAu6kNSYqtFoRKfU8nSfUnVX4ZZfmlMRVka_BTot4C1XsR1fBFnoBMgNlYUi-xIXH9J5jo8klnopO0$" id="x_255514570OWAafef304c-c4b7-6a2c-2e7d-aa9673109dc7" class="x_x_x_OWAAutoLink" target="_blank"> https://meet.google.com/cpu-eiue-hvk</a><br> Gluster-users mailing list<br> <a href="mailto:Gluster-users@gluster.org" id="x_255514570OWA23683d7c-828a-20f9-89fd-9d189831996d" class="x_255514570OWAAutoLink" target="_blank">Gluster-users@gluster.org</a><br> <a href="https://urldefense.com/v3/__https://lists.gluster.org/mailman/listinfo/gluster-users__;!!I_DbfM1H!BhqOtEyWh2TghCsLjQAu6kNSYqtFoRKfU8nSfUnVX4ZZfmlMRVka_BTot4C1XsR1fBFnoBMgNlYUi-xIXH9J5jo8xixa_Vs$" id="x_255514570OWA47e27610-c7e9-275b-059e-cff7b68eac49" class="x_x_x_OWAAutoLink" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></span></div><div style="direction : ltr; margin-right : 0px; margin-left : 0px;"><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif"><span class="size" style="font-size:12pt"><br></span></span></span></div><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="font" style="font-family:ARIAL"><span class="size" style="font-size:10pt">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you
have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please
notify the sender immediately by email if you have received this email by mistake and delete this email from your system.<br> <br> If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.</span></span></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="font" style="font-family:ARIAL"><span class="size" style="font-size:10pt">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you
have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please
notify the sender immediately by email if you have received this email by mistake and delete this email from your system.<br> <br> If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.</span></span></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></p><p style="margin-top: 0; margin-bottom: 0; direction: ltr; margin: 0px;"><span class="font" style="font-family:ARIAL"><span class="size" style="font-size:10pt">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you
have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please
notify the sender immediately by email if you have received this email by mistake and delete this email from your system.<br> <br> If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.</span></span></p><div style="margin-right : 0px; margin-left : 0px;"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt">________<br> <br> <br> <br> Community Meeting Calendar:<br> <br> Schedule -<br> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br> Bridge: <a href="https://urldefense.com/v3/__https://meet.google.com/cpu-eiue-hvk__;!!I_DbfM1H!AOBbofCF6CW4CcriOixqH6vEmi3V5M5q5SCixRgyT-Vf_R_7Ulo9sM79cpcZVzStDaufJeEq459m7-NuFgm8i89ChEJGLcM$" id="x_255514570OWA1e85501a-446c-eccd-f296-00637a0d81e8" class="x_255514570OWAAutoLink" target="_blank"> https://meet.google.com/cpu-eiue-hvk</a><br> Gluster-users mailing list<br> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br> <a href="https://urldefense.com/v3/__https://lists.gluster.org/mailman/listinfo/gluster-users__;!!I_DbfM1H!AOBbofCF6CW4CcriOixqH6vEmi3V5M5q5SCixRgyT-Vf_R_7Ulo9sM79cpcZVzStDaufJeEq459m7-NuFgm8i89CDLF-Mvs$" id="x_255514570OWA9c53d888-9739-522a-9452-b375a632bc90" class="x_255514570OWAAutoLink" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></span></span></div><div><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size:10pt"><br></span></span></div><br><p style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:ARIAL"><span class="size" style="font-size: 10pt; margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Calibri Light""></span></span></span><br></p><p style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:ARIAL"><span class="size" style="font-size: 10pt; margin-top: 0px; margin-bottom: 0px;">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please
notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if
you have received this email by mistake and delete this email from your system. <br> <br> If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.</span></span></p></div></blockquote></div></div><br></body></html>