<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
As an update on this thread - I was able to work around the issue. <br>
<br>
I discovered that nearly all of the problematic files were coming
from one directory. I deleted that directory from the new servers,
and eventually geo-replication completed to the backup servers and
was staying in sync. <br>
<br>
Once I was in "Changelog Crawl" on all the bricks I was able to copy
the "bad" directory back and everything synced properly - no errors.
I can't really explain why that worked, other than that changlog
crawl was able to process the files in a different way than Hybrid
Crawl was. <br>
<br>
In any case - things are working now, the servers are staying in
sync with the geo-repo endpoints. Thank-you for your help everyone.
<br>
<br>
<tt>[root@storage01 ~]# gluster volume geo-replication status </tt><tt><br>
</tt><tt> </tt><tt><br>
</tt><tt>MASTER NODE MASTER VOL MASTER BRICK
SLAVE USER SLAVE SLAVE
NODE STATUS CRAWL STATUS LAST_SYNCED
</tt><tt><br>
</tt><tt>---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------</tt><tt><br>
</tt><tt>10.0.231.91 storage /data/storage_a/storage
geoaccount <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>
10.0.231.81 Active Changelog Crawl 2021-01-28
10:03:09 </tt><tt><br>
</tt><tt>10.0.231.91 storage /data/storage_c/storage
geoaccount <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>
10.0.231.82 Active Changelog Crawl 2021-01-28
10:03:13 </tt><tt><br>
</tt><tt>10.0.231.91 storage /data/storage_b/storage
geoaccount <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>
10.0.231.82 Active Changelog Crawl 2021-01-28
10:03:08 </tt><tt><br>
</tt><tt>10.0.231.93 storage /data/storage_c/storage
geoaccount <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>
10.0.231.81 Active Changelog Crawl 2021-01-28
10:03:13 </tt><tt><br>
</tt><tt>10.0.231.93 storage /data/storage_b/storage
geoaccount <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>
10.0.231.81 Active Changelog Crawl 2021-01-28
10:03:03 </tt><tt><br>
</tt><tt>10.0.231.93 storage /data/storage_a/storage
geoaccount <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>
10.0.231.81 Active Changelog Crawl 2021-01-28
10:03:11 </tt><tt><br>
</tt><tt>10.0.231.92 storage /data/storage_b/storage
geoaccount <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>
10.0.231.82 Active Changelog Crawl 2021-01-28
10:03:10 </tt><tt><br>
</tt><tt>10.0.231.92 storage /data/storage_a/storage
geoaccount <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>
10.0.231.82 Active Changelog Crawl 2021-01-28
10:03:04 </tt><tt><br>
</tt><tt>10.0.231.92 storage /data/storage_c/storage
geoaccount <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>
10.0.231.81 Active Changelog Crawl 2021-01-28 10:03:06</tt><br>
<br>
Thanks,<br>
-Matthew<br>
<div class="moz-signature"><font size="-1">
<p><br>
</p>
</font>
</div>
<div class="moz-cite-prefix">On 10/19/20 9:56 AM, Matthew Benstead
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:a25cb9da-fa64-38af-c26d-91476b1398c8@uvic.ca"> Thanks -
I found the file path from the GFID - but I don't see any weird
xattr's: <br>
<br>
<br>
<tt>[root@storage01 ~]# mkdir /mnt/storage2-gfid</tt><tt><br>
</tt><tt>[root@storage01 ~]# mount -t glusterfs -o aux-gfid-mount
10.0.231.91:/storage /mnt/storage2-gfid</tt><tt><br>
</tt><tt><br>
</tt><tt>[root@storage01 ~]# getfattr -n
trusted.glusterfs.pathinfo -e text
/mnt/storage2-gfid/.gfid/24bf0575-6ab0-4613-b42a-3b63b3c00165</tt><tt><br>
</tt><tt>getfattr: Removing leading '/' from absolute path names</tt><tt><br>
</tt><tt># file:
mnt/storage2-gfid/.gfid/24bf0575-6ab0-4613-b42a-3b63b3c00165</tt><tt><br>
</tt><tt>trusted.glusterfs.pathinfo="(<DISTRIBUTE:storage-dht>
(<REPLICATE:storage-replicate-1>
<POSIX(/data/storage_b/storage):storage03.pcic.uvic.ca:/data/storage_b/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h>
<POSIX(/data/storage_c/storage):storage01.pcic.uvic.ca:/data/storage_c/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h>
<POSIX(/data/storage_a/storage):storage02.pcic.uvic.ca:/data/storage_a/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h>))"</tt><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt><tt>[root@storage03 ~]# ls -lh
/data/storage_b/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt>-rw-rw-r--. 4 USER ftp 108K May 25 2018
/data/storage_b/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt><br>
</tt><tt>[root@storage03 ~]# getfattr -d -m. -e hex
/data/storage_b/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt>getfattr: Removing leading '/' from absolute path names</tt><tt><br>
</tt><tt># file:
data/storage_b/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt>security.selinux=0x73797374656d5f753a6f626a6563745f723a64656661756c745f743a733000</tt><tt><br>
</tt><tt>trusted.afr.dirty=0x000000000000000000000000</tt><tt><br>
</tt><tt>trusted.gfid=0x24bf05756ab04613b42a3b63b3c00165</tt><tt><br>
</tt><tt>trusted.gfid2path.12529fafe3b56cc3=0x33346136613135632d643031662d343230302d626164652d3738303136633634323638352f636169726f2e68</tt><tt><br>
</tt><tt>trusted.gfid2path.e445cee32fde338d=0x62376364343961322d303531392d346333342d626436342d3936616334346432353835302f636169726f2e68</tt><tt><br>
</tt><tt>trusted.gfid2path.fbe2d5aaa0195e94=0x62626138356435362d623330302d343566322d393863312d6233303233626635333762362f636169726f2e68</tt><tt><br>
</tt><tt>trusted.glusterfs.cf94a8f2-324b-40b3-bf72-c3766100ea99.xtime=0x5f8a063a0001f5c1</tt><tt><br>
</tt><tt>trusted.glusterfs.mdata=0x010000000000000000000000005ee25c8f000000002d2a4984000000005b08129b0000000000000000000000005ed26b83000000001f9b00c1</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.34a6a15c-d01f-4200-bade-78016c642685.contri.1=0x000000000001ae000000000000000001</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.b7cd49a2-0519-4c34-bd64-96ac44d25850.contri.1=0x000000000001ae000000000000000001</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.bba85d56-b300-45f2-98c1-b3023bf537b6.contri.1=0x000000000001ae000000000000000001</tt><tt><br>
</tt><tt>trusted.pgfid.34a6a15c-d01f-4200-bade-78016c642685=0x00000001</tt><tt><br>
</tt><tt>trusted.pgfid.b7cd49a2-0519-4c34-bd64-96ac44d25850=0x00000001</tt><tt><br>
</tt><tt>trusted.pgfid.bba85d56-b300-45f2-98c1-b3023bf537b6=0x00000001</tt><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt><tt>[root@storage02 ~]# ls -lh
/data/storage_a/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt>-rw-rw-r--. 4 USER ftp 108K May 25 2018
/data/storage_a/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt><br>
</tt><tt>[root@storage02 ~]# getfattr -d -m . -e hex
/data/storage_a/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt>getfattr: Removing leading '/' from absolute path names</tt><tt><br>
</tt><tt># file:
data/storage_a/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt>security.selinux=0x73797374656d5f753a6f626a6563745f723a64656661756c745f743a733000</tt><tt><br>
</tt><tt>trusted.afr.dirty=0x000000000000000000000000</tt><tt><br>
</tt><tt>trusted.gfid=0x24bf05756ab04613b42a3b63b3c00165</tt><tt><br>
</tt><tt>trusted.gfid2path.12529fafe3b56cc3=0x33346136613135632d643031662d343230302d626164652d3738303136633634323638352f636169726f2e68</tt><tt><br>
</tt><tt>trusted.gfid2path.e445cee32fde338d=0x62376364343961322d303531392d346333342d626436342d3936616334346432353835302f636169726f2e68</tt><tt><br>
</tt><tt>trusted.gfid2path.fbe2d5aaa0195e94=0x62626138356435362d623330302d343566322d393863312d6233303233626635333762362f636169726f2e68</tt><tt><br>
</tt><tt>trusted.glusterfs.cf94a8f2-324b-40b3-bf72-c3766100ea99.xtime=0x5f8a063a0001e6bc</tt><tt><br>
</tt><tt>trusted.glusterfs.mdata=0x010000000000000000000000005ee25c8f000000002d2a4984000000005b08129b0000000000000000000000005ed26b83000000001f9b00c1</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.34a6a15c-d01f-4200-bade-78016c642685.contri.1=0x000000000001ae000000000000000001</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.b7cd49a2-0519-4c34-bd64-96ac44d25850.contri.1=0x000000000001ae000000000000000001</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.bba85d56-b300-45f2-98c1-b3023bf537b6.contri.1=0x000000000001ae000000000000000001</tt><tt><br>
</tt><tt>trusted.pgfid.34a6a15c-d01f-4200-bade-78016c642685=0x00000001</tt><tt><br>
</tt><tt>trusted.pgfid.b7cd49a2-0519-4c34-bd64-96ac44d25850=0x00000001</tt><tt><br>
</tt><tt>trusted.pgfid.bba85d56-b300-45f2-98c1-b3023bf537b6=0x00000001</tt><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt><tt>[root@storage01 ~]# ls -lh
/data/storage_c/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt>-rw-rw-r--. 4 USER ftp 0 May 25 2018
/data/storage_c/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt><br>
</tt><tt>[root@storage01 ~]# getfattr -d -m. -e hex
/data/storage_c/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt>getfattr: Removing leading '/' from absolute path names</tt><tt><br>
</tt><tt># file:
data/storage_c/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt>security.selinux=0x73797374656d5f753a6f626a6563745f723a64656661756c745f743a733000</tt><tt><br>
</tt><tt>trusted.afr.dirty=0x000000000000000000000000</tt><tt><br>
</tt><tt>trusted.gfid=0x24bf05756ab04613b42a3b63b3c00165</tt><tt><br>
</tt><tt>trusted.gfid2path.12529fafe3b56cc3=0x33346136613135632d643031662d343230302d626164652d3738303136633634323638352f636169726f2e68</tt><tt><br>
</tt><tt>trusted.gfid2path.e445cee32fde338d=0x62376364343961322d303531392d346333342d626436342d3936616334346432353835302f636169726f2e68</tt><tt><br>
</tt><tt>trusted.gfid2path.fbe2d5aaa0195e94=0x62626138356435362d623330302d343566322d393863312d6233303233626635333762362f636169726f2e68</tt><tt><br>
</tt><tt>trusted.glusterfs.cf94a8f2-324b-40b3-bf72-c3766100ea99.xtime=0x5f8a063a00023a3a</tt><tt><br>
</tt><tt>trusted.glusterfs.mdata=0x010000000000000000000000005ee25c8f000000002d2a4984000000005b08129b0000000000000000000000005ed26b83000000001f9b00c1</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.34a6a15c-d01f-4200-bade-78016c642685.contri.1=0x00000000000000000000000000000001</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.b7cd49a2-0519-4c34-bd64-96ac44d25850.contri.1=0x00000000000000000000000000000001</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.bba85d56-b300-45f2-98c1-b3023bf537b6.contri.1=0x00000000000000000000000000000001</tt><tt><br>
</tt><tt>trusted.pgfid.34a6a15c-d01f-4200-bade-78016c642685=0x00000001</tt><tt><br>
</tt><tt>trusted.pgfid.b7cd49a2-0519-4c34-bd64-96ac44d25850=0x00000001</tt><tt><br>
</tt><tt>trusted.pgfid.bba85d56-b300-45f2-98c1-b3023bf537b6=0x00000001</tt><tt><br>
</tt><br>
And nothing weird with ACLs on the file in the mount: <br>
<br>
<tt>[root@storage01 ~]# getfacl
/storage2/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt>getfacl: Removing leading '/' from absolute path names</tt><tt><br>
</tt><tt># file:
storage2/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt># owner: USER</tt><tt><br>
</tt><tt># group: ftp</tt><tt><br>
</tt><tt>user::rw-</tt><tt><br>
</tt><tt>group::rw-</tt><tt><br>
</tt><tt>other::r--</tt><br>
<br>
<br>
<br>
The file doesn't exist on the geo-rep destination, but other files
in the same directory do, and don't seem to have any problems with
similar xattrs: <br>
<tt><br>
</tt><tt>[root@pcic-backup01 ~]# ls -lh
/pcic-backup/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h</tt><tt><br>
</tt><tt>ls: cannot access
/pcic-backup/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h:
No such file or directory</tt><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt><tt>[root@pcic-backup01 ~]# getfattr -d -m. -e hex
/data/brick/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo-svg.h
</tt><tt><br>
</tt><tt>getfattr: Removing leading '/' from absolute path names</tt><tt><br>
</tt><tt># file:
data/brick/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo-svg.h</tt><tt><br>
</tt><tt>security.selinux=0x73797374656d5f753a6f626a6563745f723a64656661756c745f743a733000</tt><tt><br>
</tt><tt>trusted.gfid=0x324a54f85b084a3091749c1f93ead51b</tt><tt><br>
</tt><tt>trusted.gfid2path.aabb9241b9daeeb1=0x33346136613135632d643031662d343230302d626164652d3738303136633634323638352f636169726f2d7376672e68</tt><tt><br>
</tt><tt>trusted.gfid2path.c8c216a2fda72d32=0x62626138356435362d623330302d343566322d393863312d6233303233626635333762362f636169726f2d7376672e68</tt><tt><br>
</tt><tt>trusted.gfid2path.e21e4ba36199a097=0x62376364343961322d303531392d346333342d626436342d3936616334346432353835302f636169726f2d7376672e68</tt><tt><br>
</tt><tt>trusted.glusterfs.mdata=0x010000000000000000000000005f7e60a9000000002ff17959000000005b08129b0000000000000000000000005f7e40260000000028b06e11</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.34a6a15c-d01f-4200-bade-78016c642685.contri.1=0x0000000000000c000000000000000001</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.bba85d56-b300-45f2-98c1-b3023bf537b6.contri.1=0x0000000000000c000000000000000001</tt><tt><br>
</tt><tt>trusted.pgfid.34a6a15c-d01f-4200-bade-78016c642685=0x00000001</tt><tt><br>
</tt><tt>trusted.pgfid.b7cd49a2-0519-4c34-bd64-96ac44d25850=0x00000001</tt><tt><br>
</tt><tt>trusted.pgfid.bba85d56-b300-45f2-98c1-b3023bf537b6=0x00000001</tt><br>
<br>
<br>
Anything else I can check? <br>
<br>
Thanks,<br>
-Matthew<br>
<div class="moz-signature">
<p>--<br>
Matthew Benstead<br>
System Administrator<br>
<a href="https://pacificclimate.org/" moz-do-not-send="true">Pacific
Climate Impacts Consortium</a><br>
University of Victoria, UH1<br>
PO Box 1800, STN CSC<br>
Victoria, BC, V8W 2Y2<br>
Phone: +1-250-721-8432<br>
Email: <a class="moz-txt-link-abbreviated"
href="mailto:matthewb@uvic.ca" moz-do-not-send="true">matthewb@uvic.ca</a></p>
</div>
<div class="moz-cite-prefix">On 10/18/20 9:01 PM, Strahil Nikolov
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:1154198174.625492.1603080077253@mail.yahoo.com">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[2020-10-16 20:30:25.039659] E [MSGID: 109009] [dht-helper.c:1384:dht_migration_complete_check_task] 0-pcic-backup-dht: 24bf0575-6ab0-4613-b42a-3b63b3c00165: gfid different on the target file on pcic-backup-readdir-ahead-0
[2020-10-16 20:30:25.039695] E [MSGID: 148002] [utime.c:146:gf_utime_set_mdata_setxattr_cbk] 0-pcic-backup-utime: dict set of key for set-ctime-mdata failed [Input/output error]
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">I would start by finding that gfid on the source and then identify the gfid of the file in the geo-rep volume.
Also, it seems that you got some acl/extended attributes issues on the geo-rep destination - so take a look.
Best Regards,
Strahil Nikolov
</pre>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>