<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Hi Everyone,</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">As I was getting "OSError: [Errno 107] Transport endpoint is not connected: '.gfid/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8'
" error in the primary master node gsyncd log, So I started searching this file details and I found this file in the brick, under the .glusterfs folder on master1 node.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Path on master1 - /opt/tier1data2019/brick/.glusterfs/d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[root@master1 ~]# ls -lrt /opt/tier1data2019/brick/.glusterfs/d5/3f/</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">-rw-r--r-- 2 root root 15996 Dec 14 10:10 d53feba6-dc8b-4645-a86c-befabd0e5069</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">-rw-r--r-- 2 root root 343111 Dec 18 10:55 d53fed32-b47a-48bf-889e-140c69b04479</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">-rw-r--r-- 2 root root 5060531 Dec 29 15:29 d53f184d-91e8-4bc1-b6e7-bb5f27ef8b41</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">-rw-r--r-- 2 root root 2149782 Jan 12 13:25 d53ffee5-fa66-4493-8bdf-f2093b3f6ce7</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">-rw-r--r-- 2 root root 1913460 Jan 18 10:40 d53f799b-0e87-4800-a3cd-fac9e1a30b54</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">-rw-r--r-- 2 root root 62940 Jan 22 09:35 d53fb9d4-8c64-4a83-b968-bbbfb9af4224</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">-rw-r--r-- 1 root root 174592 Jan 22 15:06 d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">-rw-r--r-- 2 root root 5633 Jan 26 08:36 d53f6bf6-9aac-476c-b8c5-0569fc8d5116</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">-rw-r--r-- 2 root root 801740 Feb 8 11:40 d53f71f8-e88b-4ece-b66e-228c2b08d6c8</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Now I have noticed two things:</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">First, this file is only present on the primary master node (master1) and doesn't exist on master2 and master3
nodes.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Second, this file has different file attributes than other files in the folder. If you check the second column
of the above output, every file has "2", but this file has "1".</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Now, can someone please guide me why this file has "1" and what I should do next? Is it safe to copy this file
to the remaining two master nodes, or should I delete it from master1?</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Many thanks,</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Anant</span></div>
<div id="appendonsend"></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div dir="ltr" id="divRplyFwdMsg"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Gluster-users <gluster-users-bounces@gluster.org> on behalf of Anant Saraswat <anant.saraswat@techblue.co.uk><br>
<b>Sent:</b> 08 February 2024 12:01 AM<br>
<b>To:</b> Aravinda <aravinda@kadalu.tech><br>
<b>Cc:</b> gluster-users@gluster.org <gluster-users@gluster.org><br>
<b>Subject:</b> Re: [Gluster-users] __Geo-replication status is getting Faulty after few seconds</span>
<div> </div>
</div>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="background-color: rgb(250, 247, 190);"><b>EXTERNAL: Do not click links or open attachments if you do not recognize the sender.</b></span></p>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Hi
<a href="mailto:aravinda@kadalu.tech" id="OWAAM737709" class="x_tWKOu x_mention x_ms-bgc-nlr x_ms-fcl-b" data-loopstyle="linkonly">
@Aravinda</a>,</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">I have checked the rsync version, and it's the same
on primary and secondary nodes. We have rsync version 3.1.3, protocol version 31, on all servers. It's very strange that we have not made any changes, that we are aware of, and this geo-replication was working fine for the last 5 years, and suddenly it has
stopped, and we are unable to understand the root cause of it.</span></p>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></p>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">I have checked the tcpdump and I can see that the
master node is sending RST to the secondary node when geo-replication connects, but we are not seeing any RST when we do the ssh using the root user from master to secondary node ourselves, which makes me think that geo-replication is able to connect to the
secondary node but after that, it's not liking something and tries to reset the connection, and this is repeating in a loop.</span></p>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></p>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">I have also enabled geo-replication debug logs and
I am getting this error in the master node gsyncd logs.</span></p>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></p>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.820978] D [repce(worker /opt/tier1data2019/brick):195:push]
RepceClient: call 2563661:140414778891136:1707345456.8209238 entry_ops([{'op': 'CREATE', 'skip_entry': False, 'gfid': '3d57e1e4-7bd2-44f6-a6d1-d628208b3697', 'entry': '.gfid/9a39167c-6c28-470a-b699-11eeaaff8edd/app_docmerge8795785720233840105.docx', 'mode':
33188, 'uid': 0, 'gid': 0}, {'op': 'UNLINK', 'skip_entry': False, 'gfid': '3d57e1e4-7bd2-44f6-a6d1-d628208b3697', 'entry': '.gfid/9a39167c-6c28-470a-b699-11eeaaff8edd/app_docmerge8795785720233840105.docx'}, {'op': 'CREATE', 'skip_entry': False, 'gfid': '7bd35f91-1408-476d-869a-9936f2d94afc',
'entry': '.gfid/9a39167c-6c28-470a-b699-11eeaaff8edd/0c3fb22f-0fbe-4445-845b-9d94d84a9888', 'mode': 33188, 'uid': 0, 'gid': 0}, {'op': 'CREATE', 'skip_entry': False, 'gfid': '3837018c-2f5e-43d4-ab58-0ed8b7456e73', 'entry': '.gfid/861afb81-386a-4b5b-af37-cef63a55a436/26fcd7e7-2c8c-4dcb-96f2-2c8a0d79f3d4',
'mode': 33188, 'uid': 0, 'gid': 0}, {'op': 'CREATE', 'skip_entry': False, 'gfid': 'db311b10-b1e2-4b84-adea-a6746214aeda', 'entry': '.gfid/861afb81-386a-4b5b-af37-cef63a55a436/0526d0da-1f36-4203-8563-7e23aacf6237', 'mode': 33188, 'uid': 0, 'gid': 0}, {'op':
'CREATE', 'skip_entry': False, 'gfid': '9bbb253a-226a-44b1-a968-7cfa76cf9463', 'entry': '.gfid/e861ff10-696a-4b03-9716-39d9e7dd08d7/app_docmergeLLRenewalLetterDocusign_1_22_15_1_18_153.doc', 'mode': 33188, 'uid': 0, 'gid': 0}, {'op': 'UNLINK', 'skip_entry':
False, 'gfid': '9bbb253a-226a-44b1-a968-7cfa76cf9463', 'entry': '.gfid/e861ff10-696a-4b03-9716-39d9e7dd08d7/app_docmergeLLRenewalLetterDocusign_1_22_15_1_18_153.doc'}, {'op': 'CREATE', 'skip_entry': False, 'gfid': 'f62d0c65-6ede-48ff-b9bf-c44a33e5e023', 'entry':
'.gfid/e861ff10-696a-4b03-9716-39d9e7dd08d7/85530794-c15f-44d4-8660-87a14c2c9c8c', 'mode': 33188, 'uid': 0, 'gid': 0}, {'op': 'CREATE', 'skip_entry': False, 'gfid': 'fd3d0af6-8ef5-4b76-bb47-0bc508df0ed0', 'entry': '.gfid/e861ff10-696a-4b03-9716-39d9e7dd08d7/app_docmergeMOA_1_22_15_1_20_501.doc',
'mode': 33188, 'uid': 0, 'gid': 0}, {'op': 'UNLINK', 'skip_entry': False, 'gfid': 'fd3d0af6-8ef5-4b76-bb47-0bc508df0ed0', 'entry': '.gfid/e861ff10-696a-4b03-9716-39d9e7dd08d7/app_docmergeMOA_1_22_15_1_20_501.doc'}, {'op': 'CREATE', 'skip_entry': False, 'gfid':
'e93c5771-9676-40d4-90cd-f0586ec05dd9', 'entry': '.gfid/e861ff10-696a-4b03-9716-39d9e7dd08d7/cc372667-3b77-468f-bac6-671d4eb069e9', 'mode': 33188, 'uid': 0, 'gid': 0}, {'op': 'CREATE', 'skip_entry': False, 'gfid': '02045f44-68ff-4a35-a843-08939afc46a4', 'entry':
'.gfid/e861ff10-696a-4b03-9716-39d9e7dd08d7/app_docmergeTTRenewalLetterASTNoFee-2022_1_22_15_1_19_530.doc', 'mode': 33188, 'uid': 0, 'gid': 0}, {'op': 'UNLINK', 'skip_entry': False, 'gfid': '02045f44-68ff-4a35-a843-08939afc46a4', 'entry': '.gfid/e861ff10-696a-4b03-9716-39d9e7dd08d7/app_docmergeTTRenewalLetterASTNoFee-2022_1_22_15_1_19_530.doc'},
{'op': 'CREATE', 'skip_entry': False, 'gfid': '6f5766c9-2dc3-4636-9041-9cf4ac64d26b', 'entry': '.gfid/e861ff10-696a-4b03-9716-39d9e7dd08d7/556a0e3c-510d-4396-8f32-335aafec1314', 'mode': 33188, 'uid': 0, 'gid': 0}, {'op': 'UNLINK', 'skip_entry': False, 'gfid':
'f78561f0-c9f2-4192-a82a-8368e0ad8b2b', 'entry': '.gfid/ec161c2e-bb32-4639-a7b2-9be961221d86/app_1705935977525.tmp'}, {'op': 'CREATE', 'skip_entry': False, 'gfid': 'd1e33edb-523e-41c1-a021-8bd3a5a2c7c0', 'entry': '.gfid/e861ff10-696a-4b03-9716-39d9e7dd08d7/c655e3e5-9d4c-43d7-9171-949f01612e6d',
'mode': 33188, 'uid': 0, 'gid': 0}, {'op': 'CREATE', 'skip_entry': False, 'gfid': 'b6f44b28-c2bf-4e70-b953-1c559ded7835', 'entry': '.gfid/9a39167c-6c28-470a-b699-11eeaaff8edd/app_docmerge7370453767656401681.docx', 'mode': 33188, 'uid': 0, 'gid': 0}, {'op':
'UNLINK', 'skip_entry': False, 'gfid': 'b6f44b28-c2bf-4e70-b953-1c559ded7835', 'entry': '.gfid/9a39167c-6c28-470a-b699-11eeaaff8edd/app_docmerge7370453767656401681.docx'}, {'op': 'CREATE', 'skip_entry': False, 'gfid': '2d845d9e-7a49-4200-a100-759fe831ba0e',
'entry': '.gfid/9a39167c-6c28-470a-b699-11eeaaff8edd/84d47d84-5749-4a19-8f73-293078d17c63', 'mode': 33188, 'uid': 0, 'gid': 0}, {'op': 'CREATE', 'skip_entry': False, 'gfid': '44554c17-21aa-427a-b796-7ecec6af2570', 'entry': '.gfid/9a39167c-6c28-470a-b699-11eeaaff8edd/app_docmerge8634804987715893755.docx',
'mode': 33188, 'uid': 0, 'gid': 0}, {'op': 'CREATE', 'skip_entry': False, 'gfid': '652bf5d7-3b7a-41d8-aa4f-e52296034821', 'entry': '.gfid/9a39167c-6c28-470a-b699-11eeaaff8edd/91a25682-69ea-4edc-9250-d6c7aac56853', 'mode': 33188, 'uid': 0, 'gid': 0}, {'op':
'UNLINK', 'skip_entry': False, 'gfid': '44554c17-21aa-427a-b796-7ecec6af2570', 'entry': '.gfid/9a39167c-6c28-470a-b699-11eeaaff8edd/app_docmerge8634804987715893755.docx'}, {'op': 'CREATE', 'skip_entry': False, 'gfid': '04720811-b90e-42b7-a5d1-656afd92e245',
'entry': '.gfid/9a39167c-6c28-470a-b699-11eeaaff8edd/a66cbc42-61dc-4896-bb69-c715f1a820db', 'mode': 33188, 'uid': 0, 'gid': 0}],) ...</span></p>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.909606] D [repce(worker /opt/tier1data2019/brick):215:__call__]
RepceClient: call 2563661:140414778891136:1707345456.8209238 entry_ops -> []</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.911032] D [master(worker /opt/tier1data2019/brick):317:a_syncdata]
_GMaster: files [{files={'.gfid/652bf5d7-3b7a-41d8-aa4f-e52296034821', '.gfid/2d845d9e-7a49-4200-a100-759fe831ba0e', '.gfid/3837018c-2f5e-43d4-ab58-0ed8b7456e73', '.gfid/e93c5771-9676-40d4-90cd-f0586ec05dd9', '.gfid/f62d0c65-6ede-48ff-b9bf-c44a33e5e023', '.gfid/7bd35f91-1408-476d-869a-9936f2d94afc',
'.gfid/04720811-b90e-42b7-a5d1-656afd92e245', '.gfid/6f5766c9-2dc3-4636-9041-9cf4ac64d26b', '.gfid/db311b10-b1e2-4b84-adea-a6746214aeda', '.gfid/d1e33edb-523e-41c1-a021-8bd3a5a2c7c0'}}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.911089] D [master(worker /opt/tier1data2019/brick):320:a_syncdata]
_GMaster: candidate for syncing [{file=.gfid/652bf5d7-3b7a-41d8-aa4f-e52296034821}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.911133] D [master(worker /opt/tier1data2019/brick):320:a_syncdata]
_GMaster: candidate for syncing [{file=.gfid/2d845d9e-7a49-4200-a100-759fe831ba0e}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.911169] D [master(worker /opt/tier1data2019/brick):320:a_syncdata]
_GMaster: candidate for syncing [{file=.gfid/3837018c-2f5e-43d4-ab58-0ed8b7456e73}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.911202] D [master(worker /opt/tier1data2019/brick):320:a_syncdata]
_GMaster: candidate for syncing [{file=.gfid/e93c5771-9676-40d4-90cd-f0586ec05dd9}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.911235] D [master(worker /opt/tier1data2019/brick):320:a_syncdata]
_GMaster: candidate for syncing [{file=.gfid/f62d0c65-6ede-48ff-b9bf-c44a33e5e023}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.911268] D [master(worker /opt/tier1data2019/brick):320:a_syncdata]
_GMaster: candidate for syncing [{file=.gfid/7bd35f91-1408-476d-869a-9936f2d94afc}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.911301] D [master(worker /opt/tier1data2019/brick):320:a_syncdata]
_GMaster: candidate for syncing [{file=.gfid/04720811-b90e-42b7-a5d1-656afd92e245}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.911333] D [master(worker /opt/tier1data2019/brick):320:a_syncdata]
_GMaster: candidate for syncing [{file=.gfid/6f5766c9-2dc3-4636-9041-9cf4ac64d26b}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.911366] D [master(worker /opt/tier1data2019/brick):320:a_syncdata]
_GMaster: candidate for syncing [{file=.gfid/db311b10-b1e2-4b84-adea-a6746214aeda}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.911398] D [master(worker /opt/tier1data2019/brick):320:a_syncdata]
_GMaster: candidate for syncing [{file=.gfid/d1e33edb-523e-41c1-a021-8bd3a5a2c7c0}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.911439] D [master(worker /opt/tier1data2019/brick):1344:process]
_GMaster: processing change [{changelog=/var/lib/misc/gluster/gsyncd/tier1data_drtier1data_drtier1data/opt-tier1data2019-brick/.history/.processing/CHANGELOG.1705936007}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.915193] E [syncdutils(worker /opt/tier1data2019/brick):346:log_raise_exception]
<top>: Gluster Mount process exited [{error=ENOTCONN}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:36.915252] E [syncdutils(worker /opt/tier1data2019/brick):363:log_raise_exception]
<top>: FULL EXCEPTION TRACE:</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Traceback (most recent call last):</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 317, in main</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> func(args)</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in subcmd_worker</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> local.service_loop(remote)</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1298, in service_loop</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> g3.crawlwrap(oneshot=True)</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 604, in crawlwrap</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> self.crawl()</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1614, in crawl</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> self.changelogs_batch_process(changes)</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1510, in changelogs_batch_process</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> self.process(batch)</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1345, in process</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> self.process_change(change, done, retry)</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1071, in process_change</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> st = lstat(pt)</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 589, in lstat</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> return errno_wrap(os.lstat, [e], [ENOENT], [ESTALE, EBUSY])</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 571, in errno_wrap</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> return call(*arg)</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">OSError: [Errno 107] Transport endpoint is not connected: '.gfid/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8'</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:37.344426] I [monitor(monitor):228:monitor] Monitor: worker died
in startup phase [{brick=/opt/tier1data2019/brick}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2024-02-07 22:37:37.346601] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus:
Worker Status Change [{status=Faulty}]</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div id="x_appendonsend"></div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Anant</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="direction: ltr; display: inline-block; width: 98%;">
<div id="x_divRplyFwdMsg" dir="ltr"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Aravinda <aravinda@kadalu.tech><br>
<b>Sent:</b> 07 February 2024 2:54 PM<br>
<b>To:</b> Anant Saraswat <anant.saraswat@techblue.co.uk><br>
<b>Cc:</b> Strahil Nikolov <hunter86_bg@yahoo.com>; gluster-users@gluster.org <gluster-users@gluster.org><br>
<b>Subject:</b> Re: [Gluster-users] __Geo-replication status is getting Faulty after few seconds</span>
<div> </div>
</div>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(250, 247, 190);"><b>EXTERNAL: Do
not click links or open attachments if you do not recognize the sender.</b></span></p>
<div style="direction: ltr;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">It will keep track of last sync time if you change to non-root user. But I don't think the issue is related to root vs non-root
user. </span></div>
<div style="direction: ltr;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Even in non-root user based Geo-rep, Primary volume is mounted using root user only. Only in the secondary node, it will use
Glusterd mountbroker to allow mounting the Secondary volume as non-priviliaged user.</span></div>
<div style="direction: ltr;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Check the rsync version in Primary and secondary nodes. Please fix the versions if not matching. </span></div>
<div style="direction: ltr;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">--</span></div>
<div id="x_x_Zm-_Id_-Sgn" data-sigid="3848334000000010003" data-zbluepencil-ignore="true">
<div>Aravinda</div>
<div>Kadalu Technologies</div>
</div>
<div style="direction: ltr;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr; line-height: 0px; margin-top: 10px; margin-bottom: 10px; border-top: 1px solid rgb(204, 204, 204); height: 0px;">
<span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div id="x_x_Zm-_Id_-Sgn1">---- On Wed, 07 Feb 2024 20:11:47 +0530 <b>Anant Saraswat <anant.saraswat@techblue.co.uk></b> wrote ---<br>
</div>
<div style="direction: ltr;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr; margin-right: 0px; margin-left: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">No, It was setup and running using the root user
only. </span></div>
<div style="direction: ltr; margin: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr; margin: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Do you think I should setup using
</span><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> a dedicated non-root user</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">?
will it keep the track of old files or will it consider it as a new geo-replication and copy all the files from the scratch?</span></div>
<div style="direction: ltr; margin: 0px;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<hr style="direction: ltr; margin-right: 0px; margin-left: 0px; display: inline-block; width: 98%;">
<div dir="ltr" id="x_x_x_1199982551divRplyFwdMsg"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" id="OWA4756b457-afd9-b67f-8230-3f7aa299dd52" class="x_OWAAutoLink" data-loopstyle="linkonly">hunter86_bg@yahoo.com</a>><br>
<b>Sent:</b> 07 February 2024 2:36 PM<br>
<b>To:</b> Anant Saraswat <<a href="mailto:anant.saraswat@techblue.co.uk" id="OWA49c32ff5-1f8e-6ae2-a28b-8ea002671071" class="x_OWAAutoLink" data-loopstyle="linkonly">anant.saraswat@techblue.co.uk</a>>; Aravinda <<a href="mailto:aravinda@kadalu.tech" id="OWAc39081d1-0b2e-e0ed-8cc0-a3e723191767" class="x_OWAAutoLink" data-loopstyle="linkonly">aravinda@kadalu.tech</a>><br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" id="OWAecefab6c-354a-76cc-385b-c8aa07eb54ba" class="x_OWAAutoLink" data-loopstyle="linkonly">gluster-users@gluster.org</a> <<a href="mailto:gluster-users@gluster.org" id="OWA3daff331-077d-27ed-9ce1-7ac3c58af7c5" class="x_OWAAutoLink" data-loopstyle="linkonly">gluster-users@gluster.org</a>><br>
<b>Subject:</b> Re: [Gluster-users] __Geo-replication status is getting Faulty after few seconds</span>
<div> </div>
</div>
<p style="direction: ltr; margin: 0px;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(250, 247, 190);"><b>EXTERNAL: Do not click links or open attachments if you do not recognize
the sender.</b></span></p>
<div style="direction: ltr; margin-right: 0px; margin-left: 0px;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Have you tried setting up gluster georep with a dedicated non-root user ?</span></div>
<div style="direction: ltr; margin-right: 0px; margin-left: 0px;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr; margin-right: 0px; margin-left: 0px;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Best Regards,</span></div>
<div style="direction: ltr; margin-right: 0px; margin-left: 0px;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Strahil Nikolov<br>
<br>
</span></div>
<blockquote style="margin: 0px 0px 20px;">
<div style="direction: ltr;"><span style="font-family: Roboto, sans-serif; font-size: 10pt; color: rgb(109, 0, 246);">On Tue, Feb 6, 2024 at 16:38, Anant Saraswat</span></div>
<div style="direction: ltr;"><span style="font-family: Roboto, sans-serif; font-size: 10pt; color: rgb(109, 0, 246);"><<a href="mailto:anant.saraswat@techblue.co.uk" id="OWA00253d86-b7a4-5520-7bef-b4cc63cf9bdd" class="x_OWAAutoLink" data-loopstyle="linkonly">anant.saraswat@techblue.co.uk</a>>
wrote:</span></div>
<div style="direction: ltr; margin: 10px 0px 0px; padding-top: 10px; padding-left: 20px; border-left: 1px solid rgb(109, 0, 246);">
<span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://urldefense.com/v3/__https://meet.google.com/cpu-eiue-hvk__;!!I_DbfM1H!Dm8_fHcUmz5wnOfTdrkMSb6PXqGdC_3VpklsIdfjPuKgee_Ds7JD__1KjwR4F62a67f5292of5PyQVk9y3-TRe_00eSiJw$" id="OWA329d8c5a-4869-7ebd-3082-4bb58e747808" class="x_x_x_1199982551OWAAutoLink" data-loopstyle="linkonly" data-auth="NotApplicable">
https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" id="OWA44a88427-98dd-1302-caab-8db590c7e511" class="x_x_x_1199982551OWAAutoLink" data-loopstyle="linkonly">Gluster-users@gluster.org</a><br>
<a href="https://urldefense.com/v3/__https://lists.gluster.org/mailman/listinfo/gluster-users__;!!I_DbfM1H!Dm8_fHcUmz5wnOfTdrkMSb6PXqGdC_3VpklsIdfjPuKgee_Ds7JD__1KjwR4F62a67f5292of5PyQVk9y3-TRe-GwoljEQ$" id="OWA0ef3fb17-e8d8-4524-0495-d1df8e72d974" class="x_x_x_1199982551OWAAutoLink" data-loopstyle="linkonly" data-auth="NotApplicable">https://lists.gluster.org/mailman/listinfo/gluster-users</a></span></div>
</blockquote>
<p style="direction: ltr; margin: 0px;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br>
</span></p>
<p style="direction: ltr; margin: 0px;"><span style="font-family: ARIAL; font-size: 10pt; color: rgb(0, 0, 0);">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy
this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.<br>
<br>
If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.</span></p>
<div style="direction: ltr;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<p style="direction: ltr; margin-top: 0px; margin-bottom: 0px; font-family: ARIAL; font-size: 10pt;">
DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential
information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this
email from your system.<br>
<br>
If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.</p>
<p style="FONT-SIZE: 10pt; FONT-FAMILY: ARIAL"><span style="FONT-FAMILY: Calibri Light"></p>
<p style="FONT-SIZE: 10pt; FONT-FAMILY: ARIAL">DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please
notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if
you have received this email by mistake and delete this email from your system. <br>
<br>
If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.
</span></p>
</body>
</html>