[Gluster-users] GeoRep Faulty after Gluster 7 to 8 upgrade - gfchangelog: wrong result

Fri Mar 19 12:15:36 UTC 2021

Hi Shwetha,
Is that script mentioned in the documentation (release notes for example) ?
Do you think that the new glusterfs rpm can call (via rpm scriplets) the script safely ?

If yes, then it will be nice if the rpm itself runs the script instead of the user.

Best Regards,Strahil Nikolov

  On Fri, Mar 19, 2021 at 11:44, Shwetha Acharya<sacharya at redhat.com> wrote:   Hi Mathew,

The upgrade script https://github.com/gluster/glusterfs/commit/2857fe3fad4d2b30894847088a54b847b88a23b9 need to be run, to make sure that the changelogs and htimes files created in the older version are updated as per the new changes in the directory structure.

If not done, search algorithm during history crawl would give wrong result.

Regards,
Shwetha
On Fri, Mar 19, 2021 at 3:53 AM Strahil Nikolov <hunter86_bg at yahoo.com> wrote:

Sadly,
I'm out of ideas. It makes sense... if the changelog was changed - then it won't work after the upgrade.
I guess that once you delete the session, you can remove the extended attribute for the time.
Best Regards,Strahil Nikolov 

  On Wed, Mar 17, 2021 at 23:11, Matthew Benstead<matthewb at uvic.ca> wrote:    Yes, I've run through everything short of regenerating the keys and creating the session again with no errors. Everything looks ok. 

 But I did notice that the changelog format had changed, instead of them being dumped into one directory, they now seem to be separated in year/month/day directories... 

 Looks like this change in 8.0: https://github.com/gluster/glusterfs/issues/154

 [root at storage01 changelogs]# ls -lh | head
 total 16G
 drw-------. 3 root root   24 Mar  9 11:34 2021
 -rw-r--r--. 1 root root   51 Mar 17 13:19 CHANGELOG
 -rw-r--r--. 1 root root  13K Aug 13  2020 CHANGELOG.1597343197
 -rw-r--r--. 1 root root  51K Aug 13  2020 CHANGELOG.1597343212
 -rw-r--r--. 1 root root  86K Aug 13  2020 CHANGELOG.1597343227
 -rw-r--r--. 1 root root  99K Aug 13  2020 CHANGELOG.1597343242
 -rw-r--r--. 1 root root  69K Aug 13  2020 CHANGELOG.1597343257
 -rw-r--r--. 1 root root  69K Aug 13  2020 CHANGELOG.1597343272
 -rw-r--r--. 1 root root  72K Aug 13  2020 CHANGELOG.1597343287
 [root at storage01 changelogs]# ls -lh | tail
 -rw-r--r--. 1 root root   92 Mar  1 21:33 CHANGELOG.1614663193
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663731
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663760
 -rw-r--r--. 1 root root  511 Mar  1 21:47 CHANGELOG.1614664043
 -rw-r--r--. 1 root root  536 Mar  1 21:48 CHANGELOG.1614664101
 -rw-r--r--. 1 root root 2.8K Mar  1 21:48 CHANGELOG.1614664116
 -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666061
 -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666554
 drw-------. 2 root root   10 May  7  2020 csnap
 drw-------. 2 root root   38 Aug 13  2020 htime

 [root at storage01 changelogs]# ls -lh 2021/03/09/
 total 8.0K
 -rw-r--r--. 1 root root 51 Mar  9 11:26 CHANGELOG.1615318474
 -rw-r--r--. 1 root root 51 Mar  9 12:19 CHANGELOG.1615321197
 [root at storage01 changelogs]# ls -lh 2021/03/15/
 total 4.0K
 -rw-r--r--. 1 root root 51 Mar 15 13:38 CHANGELOG.1615842490
 [root at storage01 changelogs]# ls -lh 2021/03/16
 total 4.0K
 -rw-r--r--. 1 root root 331 Mar 16 12:04 CHANGELOG.1615921482

 But it looks like the htime file still records them...

 [root at storage01 changelogs]# ls -lh htime
 total 84M
 -rw-r--r--. 1 root root 84M Mar 17 13:31 HTIME.1597342860

 [root at storage01 changelogs]# head -c 256 htime/HTIME.1597342860 
/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342875/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342890/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342904/data/storage_a/storage/.glusterfs/changelogs/changelog[root at storage01 changelogs]# 

 [root at storage01 changelogs]# tail -c 256 htime/HTIME.1597342860 
/changelog.1616013484/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013499/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013514/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013529[root at storage01 changelogs]# 

 And there seems to be an xattr for time in the brick root - presumably for when changelogs were enabled: 

 [root at storage01 changelogs]# getfattr -d -m. -e hex /data/storage_a/storage 2>&1 | egrep xtime
trusted.glusterfs.cf94a8f2-324b-40b3-bf72-c3766100ea99.xtime=0x60510140000ef317

 Reading through the changes, it looks like there is a script to convert from one format to the other... I didn't see anything in the release notes for 8.0 about it... this seems like it could be the fix, and explain why gluster can't get through the changelogs....  Thoughts? 

   - https://github.com/gluster/glusterfs/issues/154#issuecomment-585701964
   - https://review.gluster.org/#/c/glusterfs/+/24121/   

 Thanks,
  -Matthew

--
 Matthew Benstead
 System Administrator
 Pacific Climate Impacts Consortium
 University of Victoria, UH1
 PO Box 1800, STN CSC
 Victoria, BC, V8W 2Y2
 Phone: +1-250-721-8432
 Email: matthewb at uvic.ca
  On 3/16/21 9:36 PM, Strahil Nikolov wrote:

  Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.  
 Have you verified all steps for creating the geo-replication ? 
  If yes , maybe using "reset-sync-time + delete + create" makes sense.Keep in mind that it will take a long time once the geo-rep is established again. 

  Best Regards, Strahil Nikolov 

  On Tue, Mar 16, 2021 at 22:34, Matthew Benstead <matthewb at uvic.ca> wrote:    Thanks Strahil, 

 I wanted to make sure the issue wasn't occurring because there were no new changes to sync from the master volume. So I created some files and restarted the sync, but it had no effect. 

 [root at storage01 ~]# cd /storage2/home/test/
 [root at storage01 test]# for nums in {1,2,3,4,5,6,7,8,9,0}; do touch $nums.txt; done

 [root at storage01 test]# gluster volume geo-replication storage  geoaccount at 10.0.231.81::pcic-backup start
 Starting geo-replication session between storage &  geoaccount at 10.0.231.81::pcic-backup has been successful
 [root at storage01 test]# gluster volume geo-replication status 

 MASTER NODE    MASTER VOL    MASTER BRICK               SLAVE USER    SLAVE                                        SLAVE NODE    STATUS             CRAWL STATUS    LAST_SYNCED          
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 10.0.231.91    storage       /data/storage_a/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.91    storage       /data/storage_c/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.91    storage       /data/storage_b/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.93    storage       /data/storage_c/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.93    storage       /data/storage_b/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.93    storage       /data/storage_a/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.92    storage       /data/storage_b/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.92    storage       /data/storage_a/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.92    storage       /data/storage_c/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 [root at storage01 test]# gluster volume geo-replication status 

 MASTER NODE    MASTER VOL    MASTER BRICK               SLAVE USER    SLAVE                                        SLAVE NODE    STATUS    CRAWL STATUS    LAST_SYNCED          
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
 10.0.231.91    storage       /data/storage_a/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.91    storage       /data/storage_c/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.91    storage       /data/storage_b/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.93    storage       /data/storage_c/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.93    storage       /data/storage_b/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.93    storage       /data/storage_a/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.92    storage       /data/storage_b/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.92    storage       /data/storage_a/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.92    storage       /data/storage_c/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 [root at storage01 test]# gluster volume geo-replication storage  geoaccount at 10.0.231.81::pcic-backup stop
 Stopping geo-replication session between storage &  geoaccount at 10.0.231.81::pcic-backup has been successful

 Still getting the same error about the history crawl failing: 

 [2021-03-16 19:05:05.227677] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666552}, {end=1615921505}] 
 [2021-03-16 19:05:05.227733] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1615921502}, {total_changelogs=1300114}] 
 [2021-03-16 19:05:05.408567] E [MSGID: 132009] [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong result [{for=end}, {start=1615921502}, {idx=1300113}] 

 [2021-03-16 19:05:05.228092] I [resource(worker /data/storage_c/storage):1292:service_loop] GLUSTER: Register time [{time=1615921505}]
 [2021-03-16 19:05:05.228626] D [repce(worker /data/storage_c/storage):195:push] RepceClient: call 124117:140500837320448:1615921505.23 keep_alive(None,) ...
 [2021-03-16 19:05:05.230076] D [repce(worker /data/storage_c/storage):215:__call__] RepceClient: call 124117:140500837320448:1615921505.23 keep_alive -> 1
 [2021-03-16 19:05:05.230693] D [master(worker /data/storage_c/storage):540:crawlwrap] _GMaster: primary master with volume id cf94a8f2-324b-40b3-bf72-c3766100ea99 ...
 [2021-03-16 19:05:05.237607] I [gsyncdstatus(worker /data/storage_c/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
 [2021-03-16 19:05:05.242046] I [gsyncdstatus(worker /data/storage_c/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
 [2021-03-16 19:05:05.242450] I [master(worker /data/storage_c/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664108, 0)}, {etime=1615921505}]
 [2021-03-16 19:05:05.244151] E [resource(worker /data/storage_c/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
 [2021-03-16 19:05:05.394129] E [resource(worker /data/storage_a/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
 [2021-03-16 19:05:05.408759] E [resource(worker /data/storage_b/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
 [2021-03-16 19:05:06.158694] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_a/storage}]
 [2021-03-16 19:05:06.163052] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
 [2021-03-16 19:05:06.204464] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_b/storage}]
 [2021-03-16 19:05:06.208961] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
 [2021-03-16 19:05:06.220495] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_c/storage}]
 [2021-03-16 19:05:06.223947] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]

 I confirmed NTP is working: 

 pcic-backup02 | CHANGED | rc=0 >>
      remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 +s216-232-132-95 68.69.221.61     2 u   29 1024  377   24.141    2.457   1.081
 *yyz-1.ip.0xt.ca 206.108.0.131    2 u  257 1024  377   57.119   -0.084   5.625
 +ip102.ip-198-27 192.168.10.254   2 u  189 1024  377   64.227   -3.012   8.867

 storage03 | CHANGED | rc=0 >>
      remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 *198.161.203.36  128.233.150.93   2 u   36 1024  377   16.055   -0.381   0.318
 +s206-75-147-25. 192.168.10.254   2 u  528 1024  377   23.648   -6.196   4.803
 +time.cloudflare 10.69.8.80       3 u  121 1024  377    2.408    0.507   0.791

 storage02 | CHANGED | rc=0 >>
      remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 *198.161.203.36  128.233.150.93   2 u  918 1024  377   15.952    0.226   0.197
 +linuxgeneration 16.164.40.197    2 u   88 1024  377   62.692   -1.160   2.007
 +dns3.switch.ca  206.108.0.131    2 u  857 1024  377   27.315    0.778   0.483

 storage01 | CHANGED | rc=0 >>
      remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 +198.161.203.36  128.233.150.93   2 u  121 1024  377   16.069    1.016   0.195
 +zero.gotroot.ca 30.114.5.31      2 u  543 1024  377    5.106   -2.462   4.923
 *ntp3.torix.ca   .PTP0.           1 u  300 1024  377   54.010    2.421  15.182

 pcic-backup01 | CHANGED | rc=0 >>
      remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 *dns3.switch.ca  206.108.0.131    2 u  983 1024  377   26.990    0.523   1.389
 +dns2.switch.ca  206.108.0.131    2 u  689 1024  377   26.975   -0.257   0.467
 +64.ip-54-39-23. 214.176.184.39   2 u  909 1024  377   64.262   -0.604   6.129

 And everything is working on the same version of gluster: 

 pcic-backup02 | CHANGED | rc=0 >>
 glusterfs 8.3
 pcic-backup01 | CHANGED | rc=0 >>
 glusterfs 8.3
 storage02 | CHANGED | rc=0 >>
 glusterfs 8.3
 storage01 | CHANGED | rc=0 >>
 glusterfs 8.3
 storage03 | CHANGED | rc=0 >>
 glusterfs 8.3

 SSH works, and the backup user/group is configured with mountbroker: 

 [root at storage01 ~]# ssh -i /root/.ssh/id_rsa  geoaccount at 10.0.231.81 uname -a
 Linux pcic-backup01 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
 [root at storage01 ~]# ssh -i /root/.ssh/id_rsa  geoaccount at 10.0.231.82 uname -a
 Linux pcic-backup02 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

 [root at pcic-backup01 ~]# grep geo /etc/passwd
 geoaccount:x:1000:1000::/home/geoaccount:/bin/bash
 [root at pcic-backup01 ~]# grep geo /etc/group
 geogroup:x:1000:geoaccount
 geoaccount:x:1001:geoaccount

 [root at pcic-backup01 ~]# gluster-mountbroker status
+-------------+-------------+---------------------------+--------------+--------------------------+
 |     NODE    | NODE STATUS |         MOUNT ROOT        |    GROUP     |          USERS           |
+-------------+-------------+---------------------------+--------------+--------------------------+
 | 10.0.231.82 |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
 |  localhost  |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
+-------------+-------------+---------------------------+--------------+--------------------------+

 So, then if I'm going to have to resync, what is the best way to do this? 

 With delete or delete reset-sync-time ?   https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-starting_geo-replication#Deleting_a_Geo-replication_Session

 Erasing the index? So I don't have to transfer the files again that are already on the backup? 

   - https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/sect-troubleshooting_geo-replication#Synchronization_Is_Not_Complete 
   - https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Geo%20Replication/#best-practices   

 Is it possible to use the special-sync-mode  option from here: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-disaster_recovery

 Thoughts? 

 Thanks,
  -Matthew
 --

  On 3/12/21 3:31 PM, Strahil Nikolov wrote:

 Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.  
 Usually, when I'm stuck - I just start over. For example, check the prerequisites: - Is ssh available (no firewall blocking) - Is time sync enabled (ntp/chrony) - Is DNS ok on all hosts (including PTR records) - Is the gluster version the same on all nodes (primary & secondary) 
  Then start over as if the geo rep was never existing. For example , stop it and start over with the secondary nodes's checks (mountbroker, user, group) . 
  Most probably somwthing will come up and you will fix it. 
  In worst case scenario, you will need to clean ip the geo-rep and start fresh. 

  Best Regards, Strahil Nikolov

  On Fri, Mar 12, 2021 at 20:01, Matthew Benstead <matthewb at uvic.ca> wrote:    Hi Strahil, 

 Yes, SELinux was put into permissive mode on the secondary nodes as well: 

 [root at pcic-backup01 ~]# sestatus | egrep -i  "^SELinux status|mode"
 SELinux status:                 enabled
 Current mode:                   permissive
 Mode from config file:          enforcing

 [root at pcic-backup02 ~]# sestatus | egrep -i  "^SELinux status|mode"
 SELinux status:                 enabled
 Current mode:                   permissive
 Mode from config file:          enforcing

 The secondary server logs didn't show anything interesting: 

 gsyncd.log: 

 [2021-03-11 19:15:28.81820] I [resource(slave 10.0.231.92/data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:28.101819] I [resource(slave 10.0.231.91/data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:28.107012] I [resource(slave 10.0.231.93/data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:28.124567] I [resource(slave 10.0.231.93/data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:28.128145] I [resource(slave 10.0.231.93/data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:29.425739] I [resource(slave 10.0.231.93/data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.3184}]
 [2021-03-11 19:15:29.427448] I [resource(slave 10.0.231.93/data/storage_c/storage):1166:service_loop] GLUSTER: slave listening
 [2021-03-11 19:15:29.433340] I [resource(slave 10.0.231.93/data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.3083}]
 [2021-03-11 19:15:29.434452] I [resource(slave 10.0.231.91/data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.3321}]
 [2021-03-11 19:15:29.434314] I [resource(slave 10.0.231.93/data/storage_b/storage):1166:service_loop] GLUSTER: slave listening
 [2021-03-11 19:15:29.435575] I [resource(slave 10.0.231.91/data/storage_a/storage):1166:service_loop] GLUSTER: slave listening
 [2021-03-11 19:15:29.439769] I [resource(slave 10.0.231.92/data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.3576}]
 [2021-03-11 19:15:29.440998] I [resource(slave 10.0.231.92/data/storage_c/storage):1166:service_loop] GLUSTER: slave listening
 [2021-03-11 19:15:29.454745] I [resource(slave 10.0.231.93/data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.3262}]
 [2021-03-11 19:15:29.456192] I [resource(slave 10.0.231.93/data/storage_a/storage):1166:service_loop] GLUSTER: slave listening
 [2021-03-11 19:15:32.594865] I [repce(slave 10.0.231.92/data/storage_c/storage):96:service_loop] RepceServer: terminating on reaching EOF.
 [2021-03-11 19:15:32.607815] I [repce(slave 10.0.231.93/data/storage_c/storage):96:service_loop] RepceServer: terminating on reaching EOF.
 [2021-03-11 19:15:32.647663] I [repce(slave 10.0.231.93/data/storage_b/storage):96:service_loop] RepceServer: terminating on reaching EOF.
 [2021-03-11 19:15:32.656280] I [repce(slave 10.0.231.91/data/storage_a/storage):96:service_loop] RepceServer: terminating on reaching EOF.
 [2021-03-11 19:15:32.668299] I [repce(slave 10.0.231.93/data/storage_a/storage):96:service_loop] RepceServer: terminating on reaching EOF.
 [2021-03-11 19:15:44.260689] I [resource(slave 10.0.231.92/data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:44.271457] I [resource(slave 10.0.231.93/data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:44.271883] I [resource(slave 10.0.231.93/data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:44.279670] I [resource(slave 10.0.231.91/data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:44.284261] I [resource(slave 10.0.231.93/data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:45.614280] I [resource(slave 10.0.231.93/data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.3419}]
 [2021-03-11 19:15:45.615622] I [resource(slave 10.0.231.93/data/storage_b/storage):1166:service_loop] GLUSTER: slave listening
 [2021-03-11 19:15:45.617986] I [resource(slave 10.0.231.93/data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.3461}]
 [2021-03-11 19:15:45.618180] I [resource(slave 10.0.231.91/data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.3380}]
 [2021-03-11 19:15:45.619539] I [resource(slave 10.0.231.91/data/storage_a/storage):1166:service_loop] GLUSTER: slave listening
 [2021-03-11 19:15:45.618999] I [resource(slave 10.0.231.93/data/storage_c/storage):1166:service_loop] GLUSTER: slave listening
 [2021-03-11 19:15:45.620843] I [resource(slave 10.0.231.93/data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.3361}]
 [2021-03-11 19:15:45.621347] I [resource(slave 10.0.231.92/data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.3604}]
 [2021-03-11 19:15:45.622179] I [resource(slave 10.0.231.93/data/storage_a/storage):1166:service_loop] GLUSTER: slave listening
 [2021-03-11 19:15:45.622541] I [resource(slave 10.0.231.92/data/storage_c/storage):1166:service_loop] GLUSTER: slave listening
 [2021-03-11 19:15:47.626054] I [repce(slave 10.0.231.91/data/storage_a/storage):96:service_loop] RepceServer: terminating on reaching EOF.
 [2021-03-11 19:15:48.778399] I [repce(slave 10.0.231.93/data/storage_c/storage):96:service_loop] RepceServer: terminating on reaching EOF.
 [2021-03-11 19:15:48.778491] I [repce(slave 10.0.231.92/data/storage_c/storage):96:service_loop] RepceServer: terminating on reaching EOF.
 [2021-03-11 19:15:48.796854] I [repce(slave 10.0.231.93/data/storage_a/storage):96:service_loop] RepceServer: terminating on reaching EOF.
 [2021-03-11 19:15:48.800697] I [repce(slave 10.0.231.93/data/storage_b/storage):96:service_loop] RepceServer: terminating on reaching EOF.

 The mnt geo-rep files were also uninteresting: 
 [2021-03-11 19:15:28.250150] I [MSGID: 100030] [glusterfsd.c:2689:main] 0-/usr/sbin/glusterfs: Started running version [{arg=/usr/sbin/glusterfs}, {version=8.3}, {cmdlinestr=/usr/sbin/glusterfs --user-map-root=g
 eoaccount --aux-gfid-mount --acl --log-level=INFO--log-file=/var/log/glusterfs/geo-replication-slaves/storage_10.0.231.81_pcic-backup/mnt-10.0.231.93-data-storage_b-storage.log --volfile-server=localhost --volf
 ile-id=pcic-backup --client-pid=-1 /var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI}] 
 [2021-03-11 19:15:28.253485] I [glusterfsd.c:2424:daemonize] 0-glusterfs: Pid of current running process is 157484
 [2021-03-11 19:15:28.267911] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
 [2021-03-11 19:15:28.267984] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
 [2021-03-11 19:15:28.268371] I [glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of available volfile servers: 10.0.231.82:24007 
 [2021-03-11 19:15:28.271729] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}] 
 [2021-03-11 19:15:28.271762] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}] 
 [2021-03-11 19:15:28.272223] I [MSGID: 114020] [client.c:2315:notify] 0-pcic-backup-client-0: parent translators are ready, attempting connect on transport [] 
 [2021-03-11 19:15:28.275883] I [MSGID: 114020] [client.c:2315:notify] 0-pcic-backup-client-1: parent translators are ready, attempting connect on transport [] 
 [2021-03-11 19:15:28.276154] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-pcic-backup-client-0: changing port to 49153 (from 0)
 [2021-03-11 19:15:28.276193] I [socket.c:849:__socket_shutdown] 0-pcic-backup-client-0: intentional socket shutdown(13)
 Final graph:
 ...
+------------------------------------------------------------------------------+
 [2021-03-11 19:15:28.282144] I [socket.c:849:__socket_shutdown] 0-pcic-backup-client-1: intentional socket shutdown(15)
 [2021-03-11 19:15:28.286536] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-pcic-backup-client-0: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}] 
 [2021-03-11 19:15:28.287208] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-pcic-backup-client-0: Connected, attached to remote volume [{conn-name=pcic-backup-client-0}, {remote_subvol=/data/brick}] 
 [2021-03-11 19:15:28.290162] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-pcic-backup-client-1: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}] 
 [2021-03-11 19:15:28.291122] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-pcic-backup-client-1: Connected, attached to remote volume [{conn-name=pcic-backup-client-1}, {remote_subvol=/data/brick}] 
 [2021-03-11 19:15:28.292703] I [fuse-bridge.c:5300:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.23
 [2021-03-11 19:15:28.292730] I [fuse-bridge.c:5926:fuse_graph_sync] 0-fuse: switched to graph 0
 [2021-03-11 19:15:32.809518] I [fuse-bridge.c:6242:fuse_thread_proc] 0-fuse: initiating unmount of /var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI
 [2021-03-11 19:15:32.810216] W [glusterfsd.c:1439:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7ea5) [0x7ff56b175ea5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55664e67db45] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55664e67d9ab] ) 0-: received signum (15), shutting down 
 [2021-03-11 19:15:32.810253] I [fuse-bridge.c:7074:fini] 0-fuse: Unmounting '/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI'.
 [2021-03-11 19:15:32.810268] I [fuse-bridge.c:7079:fini] 0-fuse: Closing fuse connection to '/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI'.

 I'm really at a loss for where to go from here, it seems like everything is set up correctly, and it has been working well through the 7.x minor versions, but the jump to 8 has broken something...

 There definitely are lots of changelogs on the servers that fit into the timeframe..... I haven't made any writes to the source volume.... do you think that's the problem? That it needs some new changelog info to sync? 
 I had been holding off making any writes in case I needed to go back to Gluster7.9 - not sure if that's really a good option or not. 

 [root at storage01 changelogs]# for dirs in {a,b,c}; do echo "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh /data/storage_$dirs/storage/.glusterfs/changelogs | head; echo ""; done
/data/storage_a/storage/.glusterfs/changelogs
 total 16G
 drw-------. 3 root root   24 Mar  9 11:34 2021
 -rw-r--r--. 1 root root   51 Mar 12 09:50 CHANGELOG
 -rw-r--r--. 1 root root  13K Aug 13  2020 CHANGELOG.1597343197
 -rw-r--r--. 1 root root  51K Aug 13  2020 CHANGELOG.1597343212
 -rw-r--r--. 1 root root  86K Aug 13  2020 CHANGELOG.1597343227
 -rw-r--r--. 1 root root  99K Aug 13  2020 CHANGELOG.1597343242
 -rw-r--r--. 1 root root  69K Aug 13  2020 CHANGELOG.1597343257
 -rw-r--r--. 1 root root  69K Aug 13  2020 CHANGELOG.1597343272
 -rw-r--r--. 1 root root  72K Aug 13  2020 CHANGELOG.1597343287

/data/storage_b/storage/.glusterfs/changelogs
 total 3.3G
 drw-------. 3 root root   24 Mar  9 11:34 2021
 -rw-r--r--. 1 root root   51 Mar 12 09:50 CHANGELOG
 -rw-r--r--. 1 root root  13K Aug 13  2020 CHANGELOG.1597343197
 -rw-r--r--. 1 root root  53K Aug 13  2020 CHANGELOG.1597343212
 -rw-r--r--. 1 root root  89K Aug 13  2020 CHANGELOG.1597343227
 -rw-r--r--. 1 root root  89K Aug 13  2020 CHANGELOG.1597343242
 -rw-r--r--. 1 root root  69K Aug 13  2020 CHANGELOG.1597343257
 -rw-r--r--. 1 root root  71K Aug 13  2020 CHANGELOG.1597343272
 -rw-r--r--. 1 root root  86K Aug 13  2020 CHANGELOG.1597343287

/data/storage_c/storage/.glusterfs/changelogs
 total 9.6G
 drw-------. 3 root root   16 Mar  9 11:34 2021
 -rw-r--r--. 1 root root   51 Mar 12 09:50 CHANGELOG
 -rw-r--r--. 1 root root  16K Aug 13  2020 CHANGELOG.1597343199
 -rw-r--r--. 1 root root  71K Aug 13  2020 CHANGELOG.1597343214
 -rw-r--r--. 1 root root 122K Aug 13  2020 CHANGELOG.1597343229
 -rw-r--r--. 1 root root  73K Aug 13  2020 CHANGELOG.1597343244
 -rw-r--r--. 1 root root 100K Aug 13  2020 CHANGELOG.1597343259
 -rw-r--r--. 1 root root  95K Aug 13  2020 CHANGELOG.1597343274
 -rw-r--r--. 1 root root  92K Aug 13  2020 CHANGELOG.1597343289

 [root at storage01 changelogs]# for dirs in {a,b,c}; do echo "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh /data/storage_$dirs/storage/.glusterfs/changelogs | tail; echo ""; done
/data/storage_a/storage/.glusterfs/changelogs
 -rw-r--r--. 1 root root   92 Mar  1 21:33 CHANGELOG.1614663193
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663731
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663760
 -rw-r--r--. 1 root root  511 Mar  1 21:47 CHANGELOG.1614664043
 -rw-r--r--. 1 root root  536 Mar  1 21:48 CHANGELOG.1614664101
 -rw-r--r--. 1 root root 2.8K Mar  1 21:48 CHANGELOG.1614664116
 -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666061
 -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666554
 drw-------. 2 root root   10 May  7  2020 csnap
 drw-------. 2 root root   38 Aug 13  2020 htime

/data/storage_b/storage/.glusterfs/changelogs
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663731
 -rw-r--r--. 1 root root  480 Mar  1 21:42 CHANGELOG.1614663745
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663760
 -rw-r--r--. 1 root root  524 Mar  1 21:47 CHANGELOG.1614664043
 -rw-r--r--. 1 root root  495 Mar  1 21:48 CHANGELOG.1614664100
 -rw-r--r--. 1 root root 1.6K Mar  1 21:48 CHANGELOG.1614664114
 -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666060
 -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666553
 drw-------. 2 root root   10 May  7  2020 csnap
 drw-------. 2 root root   38 Aug 13  2020 htime

/data/storage_c/storage/.glusterfs/changelogs
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663738
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663753
 -rw-r--r--. 1 root root  395 Mar  1 21:47 CHANGELOG.1614664051
 -rw-r--r--. 1 root root  316 Mar  1 21:48 CHANGELOG.1614664094
 -rw-r--r--. 1 root root 1.2K Mar  1 21:48 CHANGELOG.1614664109
 -rw-r--r--. 1 root root  174 Mar  1 21:48 CHANGELOG.1614664123
 -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666061
 -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666553
 drw-------. 2 root root    6 May  7  2020 csnap
 drw-------. 2 root root   30 Aug 13  2020 htime

 [root at storage02 ~]# for dirs in {a,b,c}; do echo "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh /data/storage_$dirs/storage/.glusterfs/changelogs | head; echo ""; done
/data/storage_a/storage/.glusterfs/changelogs
 total 9.6G
 drw-------. 3 root root   24 Mar  9 11:34 2021
 -rw-r--r--. 1 root root   51 Mar 12 09:50 CHANGELOG
 -rw-r--r--. 1 root root 4.2K Aug 13  2020 CHANGELOG.1597343193
 -rw-r--r--. 1 root root  32K Aug 13  2020 CHANGELOG.1597343208
 -rw-r--r--. 1 root root 107K Aug 13  2020 CHANGELOG.1597343223
 -rw-r--r--. 1 root root 120K Aug 13  2020 CHANGELOG.1597343238
 -rw-r--r--. 1 root root  72K Aug 13  2020 CHANGELOG.1597343253
 -rw-r--r--. 1 root root 111K Aug 13  2020 CHANGELOG.1597343268
 -rw-r--r--. 1 root root  91K Aug 13  2020 CHANGELOG.1597343283

/data/storage_b/storage/.glusterfs/changelogs
 total 16G
 drw-------. 3 root root   24 Mar  9 11:34 2021
 -rw-r--r--. 1 root root   51 Mar 12 09:50 CHANGELOG
 -rw-r--r--. 1 root root 3.9K Aug 13  2020 CHANGELOG.1597343193
 -rw-r--r--. 1 root root  35K Aug 13  2020 CHANGELOG.1597343208
 -rw-r--r--. 1 root root  85K Aug 13  2020 CHANGELOG.1597343223
 -rw-r--r--. 1 root root 103K Aug 13  2020 CHANGELOG.1597343238
 -rw-r--r--. 1 root root  70K Aug 13  2020 CHANGELOG.1597343253
 -rw-r--r--. 1 root root  72K Aug 13  2020 CHANGELOG.1597343268
 -rw-r--r--. 1 root root  73K Aug 13  2020 CHANGELOG.1597343283

/data/storage_c/storage/.glusterfs/changelogs
 total 3.3G
 drw-------. 3 root root   16 Mar  9 11:34 2021
 -rw-r--r--. 1 root root   51 Mar 12 09:51 CHANGELOG
 -rw-r--r--. 1 root root  21K Aug 13  2020 CHANGELOG.1597343202
 -rw-r--r--. 1 root root  75K Aug 13  2020 CHANGELOG.1597343217
 -rw-r--r--. 1 root root  92K Aug 13  2020 CHANGELOG.1597343232
 -rw-r--r--. 1 root root  77K Aug 13  2020 CHANGELOG.1597343247
 -rw-r--r--. 1 root root  66K Aug 13  2020 CHANGELOG.1597343262
 -rw-r--r--. 1 root root  84K Aug 13  2020 CHANGELOG.1597343277
 -rw-r--r--. 1 root root  81K Aug 13  2020 CHANGELOG.1597343292

 [root at storage02 ~]# for dirs in {a,b,c}; do echo "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh /data/storage_$dirs/storage/.glusterfs/changelogs | tail; echo ""; done
/data/storage_a/storage/.glusterfs/changelogs
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663734
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663749
 -rw-r--r--. 1 root root  395 Mar  1 21:47 CHANGELOG.1614664052
 -rw-r--r--. 1 root root  316 Mar  1 21:48 CHANGELOG.1614664096
 -rw-r--r--. 1 root root 1.2K Mar  1 21:48 CHANGELOG.1614664111
 -rw-r--r--. 1 root root  174 Mar  1 21:48 CHANGELOG.1614664126
 -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666056
 -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666560
 drw-------. 2 root root   10 May  7  2020 csnap
 drw-------. 2 root root   38 Aug 13  2020 htime

/data/storage_b/storage/.glusterfs/changelogs
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663735
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663749
 -rw-r--r--. 1 root root  511 Mar  1 21:47 CHANGELOG.1614664052
 -rw-r--r--. 1 root root  316 Mar  1 21:48 CHANGELOG.1614664096
 -rw-r--r--. 1 root root 1.8K Mar  1 21:48 CHANGELOG.1614664111
 -rw-r--r--. 1 root root 1.4K Mar  1 21:48 CHANGELOG.1614664126
 -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666060
 -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666556
 drw-------. 2 root root   10 May  7  2020 csnap
 drw-------. 2 root root   38 Aug 13  2020 htime

/data/storage_c/storage/.glusterfs/changelogs
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663738
 -rw-r--r--. 1 root root  521 Mar  1 21:42 CHANGELOG.1614663752
 -rw-r--r--. 1 root root  524 Mar  1 21:47 CHANGELOG.1614664042
 -rw-r--r--. 1 root root   92 Mar  1 21:47 CHANGELOG.1614664057
 -rw-r--r--. 1 root root  536 Mar  1 21:48 CHANGELOG.1614664102
 -rw-r--r--. 1 root root 1.6K Mar  1 21:48 CHANGELOG.1614664117
 -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666057
 -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666550
 drw-------. 2 root root    6 May  7  2020 csnap
 drw-------. 2 root root   30 Aug 13  2020 htime

 [root at storage03 ~]# for dirs in {a,b,c}; do echo "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh /data/storage_$dirs/storage/.glusterfs/changelogs | head; echo ""; done
/data/storage_a/storage/.glusterfs/changelogs
 total 3.4G
 drw-------. 3 root root   24 Mar  9 11:34 2021
 -rw-r--r--. 1 root root   51 Mar 12 09:50 CHANGELOG
 -rw-r--r--. 1 root root  19K Aug 13  2020 CHANGELOG.1597343201
 -rw-r--r--. 1 root root  66K Aug 13  2020 CHANGELOG.1597343215
 -rw-r--r--. 1 root root  91K Aug 13  2020 CHANGELOG.1597343230
 -rw-r--r--. 1 root root  82K Aug 13  2020 CHANGELOG.1597343245
 -rw-r--r--. 1 root root  64K Aug 13  2020 CHANGELOG.1597343259
 -rw-r--r--. 1 root root  75K Aug 13  2020 CHANGELOG.1597343274
 -rw-r--r--. 1 root root  81K Aug 13  2020 CHANGELOG.1597343289

/data/storage_b/storage/.glusterfs/changelogs
 total 9.6G
 drw-------. 3 root root   24 Mar  9 11:34 2021
 -rw-r--r--. 1 root root   51 Mar 12 09:51 CHANGELOG
 -rw-r--r--. 1 root root  19K Aug 13  2020 CHANGELOG.1597343201
 -rw-r--r--. 1 root root  80K Aug 13  2020 CHANGELOG.1597343215
 -rw-r--r--. 1 root root 119K Aug 13  2020 CHANGELOG.1597343230
 -rw-r--r--. 1 root root  65K Aug 13  2020 CHANGELOG.1597343244
 -rw-r--r--. 1 root root 100K Aug 13  2020 CHANGELOG.1597343259
 -rw-r--r--. 1 root root  95K Aug 13  2020 CHANGELOG.1597343274
 -rw-r--r--. 1 root root  92K Aug 13  2020 CHANGELOG.1597343289

/data/storage_c/storage/.glusterfs/changelogs
 total 16G
 drw-------. 3 root root   16 Mar  9 11:34 2021
 -rw-r--r--. 1 root root   51 Mar 12 09:51 CHANGELOG
 -rw-r--r--. 1 root root 3.9K Aug 13  2020 CHANGELOG.1597343193
 -rw-r--r--. 1 root root  35K Aug 13  2020 CHANGELOG.1597343208
 -rw-r--r--. 1 root root  85K Aug 13  2020 CHANGELOG.1597343223
 -rw-r--r--. 1 root root 103K Aug 13  2020 CHANGELOG.1597343238
 -rw-r--r--. 1 root root  70K Aug 13  2020 CHANGELOG.1597343253
 -rw-r--r--. 1 root root  71K Aug 13  2020 CHANGELOG.1597343268
 -rw-r--r--. 1 root root  73K Aug 13  2020 CHANGELOG.1597343283

 [root at storage03 ~]# for dirs in {a,b,c}; do echo "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh /data/storage_$dirs/storage/.glusterfs/changelogs | tail; echo ""; done
/data/storage_a/storage/.glusterfs/changelogs
 -rw-r--r--. 1 root root   92 Mar  1 21:33 CHANGELOG.1614663183
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663740
 -rw-r--r--. 1 root root  521 Mar  1 21:42 CHANGELOG.1614663755
 -rw-r--r--. 1 root root  524 Mar  1 21:47 CHANGELOG.1614664049
 -rw-r--r--. 1 root root 1.9K Mar  1 21:48 CHANGELOG.1614664106
 -rw-r--r--. 1 root root  174 Mar  1 21:48 CHANGELOG.1614664121
 -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666051
 -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666559
 drw-------. 2 root root   10 May  7  2020 csnap
 drw-------. 2 root root   38 Aug 13  2020 htime

/data/storage_b/storage/.glusterfs/changelogs
 -rw-r--r--. 1 root root  474 Mar  1 21:33 CHANGELOG.1614663182
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663739
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663753
 -rw-r--r--. 1 root root  395 Mar  1 21:47 CHANGELOG.1614664049
 -rw-r--r--. 1 root root 1.4K Mar  1 21:48 CHANGELOG.1614664106
 -rw-r--r--. 1 root root  174 Mar  1 21:48 CHANGELOG.1614664120
 -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666063
 -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666557
 drw-------. 2 root root   10 May  7  2020 csnap
 drw-------. 2 root root   38 Aug 13  2020 htime

/data/storage_c/storage/.glusterfs/changelogs
 -rw-r--r--. 1 root root  468 Mar  1 21:33 CHANGELOG.1614663183
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663740
 -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663754
 -rw-r--r--. 1 root root  511 Mar  1 21:47 CHANGELOG.1614664048
 -rw-r--r--. 1 root root 2.0K Mar  1 21:48 CHANGELOG.1614664105
 -rw-r--r--. 1 root root 1.4K Mar  1 21:48 CHANGELOG.1614664120
 -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666063
 -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666556
 drw-------. 2 root root    6 May  7  2020 csnap
 drw-------. 2 root root   30 Aug 13  2020 htime

 Thanks,
  -Matthew

--
 Matthew Benstead
 System Administrator
 Pacific Climate Impacts Consortium
 University of Victoria, UH1
 PO Box 1800, STN CSC
 Victoria, BC, V8W 2Y2
 Phone: +1-250-721-8432
 Email:  matthewb at uvic.ca
   On 3/11/21 11:37 PM, Strahil Nikolov wrote:

 Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.  
 Have you checked the secondary volume nodes' logs & SELINUX status ? 
  Best Regards, Strahil Nikolov

  On Thu, Mar 11, 2021 at 21:36, Matthew Benstead <matthewb at uvic.ca> wrote:    Hi Strahil, 

 It looks like perhaps the changelog_log_level and log_level options? I've set them to debug: 

 [root at storage01 ~]# gluster volume geo-replication storage geoaccount at 10.0.231.81::pcic-backup config | egrep -i "log_level"
changelog_log_level:INFO
 cli_log_level:INFO
gluster_log_level:INFO
 log_level:INFO
slave_gluster_log_level:INFO
 slave_log_level:INFO

 [root at storage01 ~]# gluster volume geo-replication storage geoaccount at 10.0.231.81::pcic-backup config changelog_log_level DEBUG
 geo-replication config updated successfully

 [root at storage01 ~]# gluster volume geo-replication storage geoaccount at 10.0.231.81::pcic-backup config log_level DEBUG
 geo-replication config updated successfully

 Then I restarted geo-replication: 

 [root at storage01 ~]# gluster volume geo-replication storage geoaccount at 10.0.231.81::pcic-backup start
 Starting geo-replication session between storage & geoaccount at 10.0.231.81::pcic-backup has been successful
 [root at storage01 ~]# gluster volume geo-replication status 

 MASTER NODE    MASTER VOL    MASTER BRICK               SLAVE USER    SLAVE                                        SLAVE NODE    STATUS             CRAWL STATUS    LAST_SYNCED          
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 10.0.231.91    storage       /data/storage_a/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.91    storage       /data/storage_c/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.91    storage       /data/storage_b/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.92    storage       /data/storage_b/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.92    storage       /data/storage_a/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.92    storage       /data/storage_c/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.93    storage       /data/storage_c/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.93    storage       /data/storage_b/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 10.0.231.93    storage       /data/storage_a/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Initializing...    N/A             N/A                  
 [root at storage01 ~]# gluster volume geo-replication status 

 MASTER NODE    MASTER VOL    MASTER BRICK               SLAVE USER    SLAVE                                        SLAVE NODE    STATUS    CRAWL STATUS    LAST_SYNCED          
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
 10.0.231.91    storage       /data/storage_a/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.91    storage       /data/storage_c/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.91    storage       /data/storage_b/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.92    storage       /data/storage_b/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.92    storage       /data/storage_a/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.92    storage       /data/storage_c/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.93    storage       /data/storage_c/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.93    storage       /data/storage_b/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A                  
 10.0.231.93    storage       /data/storage_a/storage    geoaccount    ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A  

 [root at storage01 ~]# gluster volume geo-replication storage geoaccount at 10.0.231.81::pcic-backup stop
 Stopping geo-replication session between storage & geoaccount at 10.0.231.81::pcic-backup has been successful

 The changelogs didn't really show anything new around changelog selection: 

 [root at storage01 storage_10.0.231.81_pcic-backup]# cat changes-data-storage_a-storage.log | egrep "2021-03-11"
 [2021-03-11 19:15:30.552889] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_a/storage}, {notify_filter=1}] 
 [2021-03-11 19:15:30.552893] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
 [2021-03-11 19:15:30.552894] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
 [2021-03-11 19:15:30.553633] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}] 
 [2021-03-11 19:15:30.553634] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}] 
 [2021-03-11 19:15:30.554236] D [rpcsvc.c:2831:rpcsvc_init] 0-rpc-service: RPC service inited.
 [2021-03-11 19:15:30.554403] D [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
 [2021-03-11 19:15:30.554420] D [rpc-transport.c:278:rpc_transport_load] 0-rpc-transport: attempt to load file/usr/lib64/glusterfs/8.3/rpc-transport/socket.so
 [2021-03-11 19:15:30.554933] D [socket.c:4485:socket_init]0-socket.gfchangelog: disabling nodelay
 [2021-03-11 19:15:30.554944] D [socket.c:4523:socket_init]0-socket.gfchangelog: Configured transport.tcp-user-timeout=42
 [2021-03-11 19:15:30.554949] D [socket.c:4543:socket_init]0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9
 [2021-03-11 19:15:30.555002] I [socket.c:929:__socket_server_bind]0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 23
 [2021-03-11 19:15:30.555324] D [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service: New program registered: LIBGFCHANGELOG REBORP, Num: 1886350951, Ver: 1, Port: 0
 [2021-03-11 19:15:30.555345] D [rpc-clnt.c:1020:rpc_clnt_connection_init] 0-gfchangelog: defaulting frame-timeout to 30mins
 [2021-03-11 19:15:30.555351] D [rpc-clnt.c:1032:rpc_clnt_connection_init] 0-gfchangelog: disable ping-timeout
 [2021-03-11 19:15:30.555358] D [rpc-transport.c:278:rpc_transport_load] 0-rpc-transport: attempt to load file/usr/lib64/glusterfs/8.3/rpc-transport/socket.so
 [2021-03-11 19:15:30.555399] D [socket.c:4485:socket_init] 0-gfchangelog: disabling nodelay
 [2021-03-11 19:15:30.555406] D [socket.c:4523:socket_init] 0-gfchangelog: Configured transport.tcp-user-timeout=42
 [2021-03-11 19:15:32.555711] D [rpc-clnt-ping.c:298:rpc_clnt_start_ping] 0-gfchangelog: ping timeout is 0, returning
 [2021-03-11 19:15:32.572157] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666553}, {end=1615490132}] 
 [2021-03-11 19:15:32.572436] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1615490121}, {total_changelogs=1256897}] 
 [2021-03-11 19:15:32.621244] E [MSGID: 132009] [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong result [{for=end}, {start=1615490121}, {idx=1256896}] 
 [2021-03-11 19:15:46.733182] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_a/storage}, {notify_filter=1}] 
 [2021-03-11 19:15:46.733316] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
 [2021-03-11 19:15:46.733348] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
 [2021-03-11 19:15:46.734031] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}] 
 [2021-03-11 19:15:46.734085] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}] 
 [2021-03-11 19:15:46.734591] D [rpcsvc.c:2831:rpcsvc_init] 0-rpc-service: RPC service inited.
 [2021-03-11 19:15:46.734755] D [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
 [2021-03-11 19:15:46.734772] D [rpc-transport.c:278:rpc_transport_load] 0-rpc-transport: attempt to load file/usr/lib64/glusterfs/8.3/rpc-transport/socket.so
 [2021-03-11 19:15:46.735256] D [socket.c:4485:socket_init]0-socket.gfchangelog: disabling nodelay
 [2021-03-11 19:15:46.735266] D [socket.c:4523:socket_init]0-socket.gfchangelog: Configured transport.tcp-user-timeout=42
 [2021-03-11 19:15:46.735271] D [socket.c:4543:socket_init]0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9
 [2021-03-11 19:15:46.735325] I [socket.c:929:__socket_server_bind]0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 21
 [2021-03-11 19:15:46.735704] D [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service: New program registered: LIBGFCHANGELOG REBORP, Num: 1886350951, Ver: 1, Port: 0
 [2021-03-11 19:15:46.735721] D [rpc-clnt.c:1020:rpc_clnt_connection_init] 0-gfchangelog: defaulting frame-timeout to 30mins
 [2021-03-11 19:15:46.735726] D [rpc-clnt.c:1032:rpc_clnt_connection_init] 0-gfchangelog: disable ping-timeout
 [2021-03-11 19:15:46.735733] D [rpc-transport.c:278:rpc_transport_load] 0-rpc-transport: attempt to load file/usr/lib64/glusterfs/8.3/rpc-transport/socket.so
 [2021-03-11 19:15:46.735771] D [socket.c:4485:socket_init] 0-gfchangelog: disabling nodelay
 [2021-03-11 19:15:46.735778] D [socket.c:4523:socket_init] 0-gfchangelog: Configured transport.tcp-user-timeout=42
 [2021-03-11 19:15:47.618464] D [rpc-clnt-ping.c:298:rpc_clnt_start_ping] 0-gfchangelog: ping timeout is 0, returning

 [root at storage01 storage_10.0.231.81_pcic-backup]# cat changes-data-storage_b-storage.log | egrep "2021-03-11"
 [2021-03-11 19:15:30.611457] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_b/storage}, {notify_filter=1}] 
 [2021-03-11 19:15:30.611574] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
 [2021-03-11 19:15:30.611641] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}] 
 [2021-03-11 19:15:30.611645] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}] 
 [2021-03-11 19:15:30.612325] D [rpcsvc.c:2831:rpcsvc_init] 0-rpc-service: RPC service inited.
 [2021-03-11 19:15:30.612488] D [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
 [2021-03-11 19:15:30.612507] D [rpc-transport.c:278:rpc_transport_load] 0-rpc-transport: attempt to load file/usr/lib64/glusterfs/8.3/rpc-transport/socket.so
 [2021-03-11 19:15:30.613005] D [socket.c:4485:socket_init]0-socket.gfchangelog: disabling nodelay
 [2021-03-11 19:15:30.613130] D [socket.c:4523:socket_init]0-socket.gfchangelog: Configured transport.tcp-user-timeout=42
 [2021-03-11 19:15:30.613142] D [socket.c:4543:socket_init]0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9
 [2021-03-11 19:15:30.613208] I [socket.c:929:__socket_server_bind]0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 22
 [2021-03-11 19:15:30.613545] D [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service: New program registered: LIBGFCHANGELOG REBORP, Num: 1886350951, Ver: 1, Port: 0
 [2021-03-11 19:15:30.613567] D [rpc-clnt.c:1020:rpc_clnt_connection_init] 0-gfchangelog: defaulting frame-timeout to 30mins
 [2021-03-11 19:15:30.613574] D [rpc-clnt.c:1032:rpc_clnt_connection_init] 0-gfchangelog: disable ping-timeout
 [2021-03-11 19:15:30.613582] D [rpc-transport.c:278:rpc_transport_load] 0-rpc-transport: attempt to load file/usr/lib64/glusterfs/8.3/rpc-transport/socket.so
 [2021-03-11 19:15:30.613637] D [socket.c:4485:socket_init] 0-gfchangelog: disabling nodelay
 [2021-03-11 19:15:30.613654] D [socket.c:4523:socket_init] 0-gfchangelog: Configured transport.tcp-user-timeout=42
 [2021-03-11 19:15:32.614273] D [rpc-clnt-ping.c:298:rpc_clnt_start_ping] 0-gfchangelog: ping timeout is 0, returning
 [2021-03-11 19:15:32.643628] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666552}, {end=1615490132}] 
 [2021-03-11 19:15:32.643716] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1615490123}, {total_changelogs=1264296}] 
 [2021-03-11 19:15:32.700397] E [MSGID: 132009] [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong result [{for=end}, {start=1615490123}, {idx=1264295}] 
 [2021-03-11 19:15:46.832322] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_b/storage}, {notify_filter=1}] 
 [2021-03-11 19:15:46.832394] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
 [2021-03-11 19:15:46.832465] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
 [2021-03-11 19:15:46.832531] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}] 
 [2021-03-11 19:15:46.833086] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}] 
 [2021-03-11 19:15:46.833648] D [rpcsvc.c:2831:rpcsvc_init] 0-rpc-service: RPC service inited.
 [2021-03-11 19:15:46.833817] D [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
 [2021-03-11 19:15:46.833835] D [rpc-transport.c:278:rpc_transport_load] 0-rpc-transport: attempt to load file/usr/lib64/glusterfs/8.3/rpc-transport/socket.so
 [2021-03-11 19:15:46.834368] D [socket.c:4485:socket_init]0-socket.gfchangelog: disabling nodelay
 [2021-03-11 19:15:46.834380] D [socket.c:4523:socket_init]0-socket.gfchangelog: Configured transport.tcp-user-timeout=42
 [2021-03-11 19:15:46.834386] D [socket.c:4543:socket_init]0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9
 [2021-03-11 19:15:46.834441] I [socket.c:929:__socket_server_bind]0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 23
 [2021-03-11 19:15:46.834768] D [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service: New program registered: LIBGFCHANGELOG REBORP, Num: 1886350951, Ver: 1, Port: 0
 [2021-03-11 19:15:46.834789] D [rpc-clnt.c:1020:rpc_clnt_connection_init] 0-gfchangelog: defaulting frame-timeout to 30mins
 [2021-03-11 19:15:46.834795] D [rpc-clnt.c:1032:rpc_clnt_connection_init] 0-gfchangelog: disable ping-timeout
 [2021-03-11 19:15:46.834802] D [rpc-transport.c:278:rpc_transport_load] 0-rpc-transport: attempt to load file/usr/lib64/glusterfs/8.3/rpc-transport/socket.so
 [2021-03-11 19:15:46.834845] D [socket.c:4485:socket_init] 0-gfchangelog: disabling nodelay
 [2021-03-11 19:15:46.834853] D [socket.c:4523:socket_init] 0-gfchangelog: Configured transport.tcp-user-timeout=42
 [2021-03-11 19:15:47.618476] D [rpc-clnt-ping.c:298:rpc_clnt_start_ping] 0-gfchangelog: ping timeout is 0, returning

 gsyncd logged a lot but I'm not sure if it's helpful: 

 [2021-03-11 19:15:00.41898] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:21.551302] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:21.631470] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:21.718386] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:21.804991] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:26.203999] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:26.284775] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:26.573355] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:26.653752] D [gsyncd(monitor):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:26.756994] D [monitor(monitor):304:distribute] <top>: master bricks: [{'host': '10.0.231.91', 'uuid': 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir': '/data/storage_a/storage'}, {'host': '10.0.2
 31.92', 'uuid': 'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d', 'dir': '/data/storage_b/storage'}, {'host': '10.0.231.93', 'uuid': '8b28b331-3780-46bc-9da3-fb27de4ab57b', 'dir': '/data/storage_c/storage'}, {'host': '10.
 0.231.92', 'uuid': 'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d', 'dir': '/data/storage_a/storage'}, {'host': '10.0.231.93', 'uuid': '8b28b331-3780-46bc-9da3-fb27de4ab57b', 'dir': '/data/storage_b/storage'}, {'host': '
 10.0.231.91', 'uuid': 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir': '/data/storage_c/storage'}, {'host': '10.0.231.93', 'uuid': '8b28b331-3780-46bc-9da3-fb27de4ab57b', 'dir': '/data/storage_a/storage'}, {'host'
 : '10.0.231.91', 'uuid': 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir': '/data/storage_b/storage'}, {'host': '10.0.231.92', 'uuid': 'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d', 'dir': '/data/storage_c/storage'}]
 [2021-03-11 19:15:26.757252] D [monitor(monitor):314:distribute] <top>: slave SSH gateway: geoaccount at 10.0.231.81
 [2021-03-11 19:15:27.416235] D [monitor(monitor):334:distribute] <top>: slave bricks: [{'host': '10.0.231.81', 'uuid': 'b88dea4f-31ec-416a-9110-3ccdc3910acd', 'dir': '/data/brick'}, {'host': '10.0.231.82', 'uuid
 ': 'be50a8de-3934-4fee-a80d-8e2e99017902', 'dir': '/data/brick'}]
 [2021-03-11 19:15:27.416825] D [syncdutils(monitor):932:is_hot] Volinfo: brickpath:'10.0.231.91:/data/storage_a/storage'
 [2021-03-11 19:15:27.417273] D [syncdutils(monitor):932:is_hot] Volinfo: brickpath:'10.0.231.91:/data/storage_c/storage'
 [2021-03-11 19:15:27.417515] D [syncdutils(monitor):932:is_hot] Volinfo: brickpath:'10.0.231.91:/data/storage_b/storage'
 [2021-03-11 19:15:27.417763] D [monitor(monitor):348:distribute] <top>: worker specs: [({'host': '10.0.231.91', 'uuid': 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir': '/data/storage_a/storage'}, ('geoaccount at 10.
 0.231.81', 'b88dea4f-31ec-416a-9110-3ccdc3910acd'), '1', False), ({'host': '10.0.231.91', 'uuid': 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir': '/data/storage_c/storage'}, ('geoaccount at 10.0.231.82', 'be50a8de-3
934-4fee-a80d-8e2e99017902'), '2', False), ({'host': '10.0.231.91', 'uuid': 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir': '/data/storage_b/storage'}, ('geoaccount at 10.0.231.82', 'be50a8de-3934-4fee-a80d-8e2e9901
 7902'), '3', False)]
 [2021-03-11 19:15:27.425009] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_c/storage},{slave_node=10.0.231.82}]
 [2021-03-11 19:15:27.426764] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_b/storage},{slave_node=10.0.231.82}]
 [2021-03-11 19:15:27.429208] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_a/storage},{slave_node=10.0.231.81}]
 [2021-03-11 19:15:27.432280] D [monitor(monitor):195:monitor] Monitor: Worker would mount volume privately
 [2021-03-11 19:15:27.434195] D [monitor(monitor):195:monitor] Monitor: Worker would mount volume privately
 [2021-03-11 19:15:27.436584] D [monitor(monitor):195:monitor] Monitor: Worker would mount volume privately
 [2021-03-11 19:15:27.478806] D [gsyncd(worker /data/storage_c/storage):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:27.478852] D [gsyncd(worker /data/storage_b/storage):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:27.480104] D [gsyncd(worker /data/storage_a/storage):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:27.500456] I [resource(worker /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
 [2021-03-11 19:15:27.501375] I [resource(worker /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
 [2021-03-11 19:15:27.502003] I [resource(worker /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
 [2021-03-11 19:15:27.525511] D [repce(worker /data/storage_a/storage):195:push] RepceClient: call 192117:140572692309824:1615490127.53 __repce_version__() ...
 [2021-03-11 19:15:27.525582] D [repce(worker /data/storage_b/storage):195:push] RepceClient: call 192115:139891296405312:1615490127.53 __repce_version__() ...
 [2021-03-11 19:15:27.526089] D [repce(worker /data/storage_c/storage):195:push] RepceClient: call 192114:140388828780352:1615490127.53 __repce_version__() ...
 [2021-03-11 19:15:29.435985] D [repce(worker /data/storage_a/storage):215:__call__] RepceClient: call 192117:140572692309824:1615490127.53 __repce_version__ -> 1.0
 [2021-03-11 19:15:29.436213] D [repce(worker /data/storage_a/storage):195:push] RepceClient: call 192117:140572692309824:1615490129.44 version() ...
 [2021-03-11 19:15:29.437136] D [repce(worker /data/storage_a/storage):215:__call__] RepceClient: call 192117:140572692309824:1615490129.44 version -> 1.0
 [2021-03-11 19:15:29.437268] D [repce(worker /data/storage_a/storage):195:push] RepceClient: call 192117:140572692309824:1615490129.44 pid() ...
 [2021-03-11 19:15:29.437915] D [repce(worker /data/storage_a/storage):215:__call__] RepceClient: call 192117:140572692309824:1615490129.44 pid -> 157321
 [2021-03-11 19:15:29.438004] I [resource(worker /data/storage_a/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9359}]
 [2021-03-11 19:15:29.438072] I [resource(worker /data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:29.494538] D [repce(worker /data/storage_b/storage):215:__call__] RepceClient: call 192115:139891296405312:1615490127.53 __repce_version__ -> 1.0
 [2021-03-11 19:15:29.494748] D [repce(worker /data/storage_b/storage):195:push] RepceClient: call 192115:139891296405312:1615490129.49 version() ...
 [2021-03-11 19:15:29.495290] D [repce(worker /data/storage_b/storage):215:__call__] RepceClient: call 192115:139891296405312:1615490129.49 version -> 1.0
 [2021-03-11 19:15:29.495400] D [repce(worker /data/storage_b/storage):195:push] RepceClient: call 192115:139891296405312:1615490129.5 pid() ...
 [2021-03-11 19:15:29.495872] D [repce(worker /data/storage_b/storage):215:__call__] RepceClient: call 192115:139891296405312:1615490129.5 pid -> 88110
 [2021-03-11 19:15:29.495960] I [resource(worker /data/storage_b/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9944}]
 [2021-03-11 19:15:29.496028] I [resource(worker /data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:29.501255] D [repce(worker /data/storage_c/storage):215:__call__] RepceClient: call 192114:140388828780352:1615490127.53 __repce_version__ -> 1.0
 [2021-03-11 19:15:29.501454] D [repce(worker /data/storage_c/storage):195:push] RepceClient: call 192114:140388828780352:1615490129.5 version() ...
 [2021-03-11 19:15:29.502258] D [repce(worker /data/storage_c/storage):215:__call__] RepceClient: call 192114:140388828780352:1615490129.5 version -> 1.0
 [2021-03-11 19:15:29.502444] D [repce(worker /data/storage_c/storage):195:push] RepceClient: call 192114:140388828780352:1615490129.5 pid() ...
 [2021-03-11 19:15:29.503140] D [repce(worker /data/storage_c/storage):215:__call__] RepceClient: call 192114:140388828780352:1615490129.5 pid -> 88111
 [2021-03-11 19:15:29.503232] I [resource(worker /data/storage_c/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=2.0026}]
 [2021-03-11 19:15:29.503302] I [resource(worker /data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:29.533899] D [resource(worker /data/storage_a/storage):880:inhibit] DirectMounter: auxiliary glusterfs mount in place
 [2021-03-11 19:15:29.595736] D [resource(worker /data/storage_b/storage):880:inhibit] DirectMounter: auxiliary glusterfs mount in place
 [2021-03-11 19:15:29.601110] D [resource(worker /data/storage_c/storage):880:inhibit] DirectMounter: auxiliary glusterfs mount in place
 [2021-03-11 19:15:30.541542] D [resource(worker /data/storage_a/storage):964:inhibit] DirectMounter: auxiliary glusterfs mount prepared
 [2021-03-11 19:15:30.541816] I [resource(worker /data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1037}]
 [2021-03-11 19:15:30.541887] I [subcmds(worker /data/storage_a/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
 [2021-03-11 19:15:30.542042] D [master(worker /data/storage_a/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=xsync}]
 [2021-03-11 19:15:30.542125] D [monitor(monitor):222:monitor] Monitor: worker(/data/storage_a/storage) connected
 [2021-03-11 19:15:30.543323] D [master(worker /data/storage_a/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=changelog}]
 [2021-03-11 19:15:30.544460] D [master(worker /data/storage_a/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=changeloghistory}]
 [2021-03-11 19:15:30.552103] D [master(worker /data/storage_a/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
 [2021-03-11 19:15:30.602937] D [resource(worker /data/storage_b/storage):964:inhibit] DirectMounter: auxiliary glusterfs mount prepared
 [2021-03-11 19:15:30.603117] I [resource(worker /data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1070}]
 [2021-03-11 19:15:30.603197] I [subcmds(worker /data/storage_b/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
 [2021-03-11 19:15:30.603353] D [master(worker /data/storage_b/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=xsync}]
 [2021-03-11 19:15:30.603338] D [monitor(monitor):222:monitor] Monitor: worker(/data/storage_b/storage) connected
 [2021-03-11 19:15:30.604620] D [master(worker /data/storage_b/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=changelog}]
 [2021-03-11 19:15:30.605600] D [master(worker /data/storage_b/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=changeloghistory}]
 [2021-03-11 19:15:30.608365] D [resource(worker /data/storage_c/storage):964:inhibit] DirectMounter: auxiliary glusterfs mount prepared
 [2021-03-11 19:15:30.608534] I [resource(worker /data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1052}]
 [2021-03-11 19:15:30.608612] I [subcmds(worker /data/storage_c/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
 [2021-03-11 19:15:30.608762] D [master(worker /data/storage_c/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=xsync}]
 [2021-03-11 19:15:30.608779] D [monitor(monitor):222:monitor] Monitor: worker(/data/storage_c/storage) connected
 [2021-03-11 19:15:30.610033] D [master(worker /data/storage_c/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=changelog}]
 [2021-03-11 19:15:30.610637] D [master(worker /data/storage_b/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
 [2021-03-11 19:15:30.610970] D [master(worker /data/storage_c/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=changeloghistory}]
 [2021-03-11 19:15:30.616197] D [master(worker /data/storage_c/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
 [2021-03-11 19:15:31.371265] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:31.451000] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:31.537257] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:31.623800] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:32.555840] D [master(worker /data/storage_a/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
 [2021-03-11 19:15:32.556051] D [master(worker /data/storage_a/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
 [2021-03-11 19:15:32.556122] D [master(worker /data/storage_a/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
 [2021-03-11 19:15:32.556179] I [master(worker /data/storage_a/storage):1645:register] _GMaster: Working dir[{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage}]
 [2021-03-11 19:15:32.556359] I [resource(worker /data/storage_a/storage):1292:service_loop] GLUSTER: Register time [{time=1615490132}]
 [2021-03-11 19:15:32.556823] D [repce(worker /data/storage_a/storage):195:push] RepceClient: call 192117:140570487928576:1615490132.56 keep_alive(None,) ...
 [2021-03-11 19:15:32.558429] D [repce(worker /data/storage_a/storage):215:__call__] RepceClient: call 192117:140570487928576:1615490132.56 keep_alive -> 1
 [2021-03-11 19:15:32.558974] D [master(worker /data/storage_a/storage):540:crawlwrap] _GMaster: primary master with volume id cf94a8f2-324b-40b3-bf72-c3766100ea99 ...
 [2021-03-11 19:15:32.567478] I [gsyncdstatus(worker/data/storage_a/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
 [2021-03-11 19:15:32.571824] I [gsyncdstatus(worker/data/storage_a/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
 [2021-03-11 19:15:32.572052] I [master(worker /data/storage_a/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666553, 0)}, {entry_stime=(1614664115, 0)}, {etime=1615490132}]
 [2021-03-11 19:15:32.614506] D [master(worker /data/storage_b/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
 [2021-03-11 19:15:32.614701] D [master(worker /data/storage_b/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
 [2021-03-11 19:15:32.614788] D [master(worker /data/storage_b/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
 [2021-03-11 19:15:32.614845] I [master(worker /data/storage_b/storage):1645:register] _GMaster: Working dir[{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage}]
 [2021-03-11 19:15:32.615000] I [resource(worker /data/storage_b/storage):1292:service_loop] GLUSTER: Register time [{time=1615490132}]
 [2021-03-11 19:15:32.615586] D [repce(worker /data/storage_b/storage):195:push] RepceClient: call 192115:139889215526656:1615490132.62 keep_alive(None,) ...
 [2021-03-11 19:15:32.617373] D [repce(worker /data/storage_b/storage):215:__call__] RepceClient: call 192115:139889215526656:1615490132.62 keep_alive -> 1
 [2021-03-11 19:15:32.618144] D [master(worker /data/storage_b/storage):540:crawlwrap] _GMaster: primary master with volume id cf94a8f2-324b-40b3-bf72-c3766100ea99 ...
 [2021-03-11 19:15:32.619323] D [master(worker /data/storage_c/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
 [2021-03-11 19:15:32.619491] D [master(worker /data/storage_c/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
 [2021-03-11 19:15:32.619739] D [master(worker /data/storage_c/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
 [2021-03-11 19:15:32.619863] I [master(worker /data/storage_c/storage):1645:register] _GMaster: Working dir[{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage}]
 [2021-03-11 19:15:32.620040] I [resource(worker /data/storage_c/storage):1292:service_loop] GLUSTER: Register time [{time=1615490132}]
 [2021-03-11 19:15:32.620599] D [repce(worker /data/storage_c/storage):195:push] RepceClient: call 192114:140386886469376:1615490132.62 keep_alive(None,) ...
 [2021-03-11 19:15:32.621397] E [resource(worker /data/storage_a/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
 [2021-03-11 19:15:32.622035] D [repce(worker /data/storage_c/storage):215:__call__] RepceClient: call 192114:140386886469376:1615490132.62 keep_alive -> 1
 [2021-03-11 19:15:32.622701] D [master(worker /data/storage_c/storage):540:crawlwrap] _GMaster: primary master with volume id cf94a8f2-324b-40b3-bf72-c3766100ea99 ...
 [2021-03-11 19:15:32.627031] I [gsyncdstatus(worker/data/storage_b/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
 [2021-03-11 19:15:32.643184] I [gsyncdstatus(worker/data/storage_b/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
 [2021-03-11 19:15:32.643528] I [master(worker /data/storage_b/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664113, 0)}, {etime=1615490132}]
 [2021-03-11 19:15:32.645148] I [gsyncdstatus(worker/data/storage_c/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
 [2021-03-11 19:15:32.649631] I [gsyncdstatus(worker/data/storage_c/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
 [2021-03-11 19:15:32.649882] I [master(worker /data/storage_c/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664108, 0)}, {etime=1615490132}]
 [2021-03-11 19:15:32.650907] E [resource(worker /data/storage_c/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
 [2021-03-11 19:15:32.700489] E [resource(worker /data/storage_b/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
 [2021-03-11 19:15:33.545886] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_a/storage}]
 [2021-03-11 19:15:33.550487] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
 [2021-03-11 19:15:33.606991] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_b/storage}]
 [2021-03-11 19:15:33.611573] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
 [2021-03-11 19:15:33.612337] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_c/storage}]
 [2021-03-11 19:15:33.615777] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
 [2021-03-11 19:15:34.684247] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:34.764971] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:34.851174] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:34.937166] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:36.994502] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:37.73805] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:37.159288] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:37.244153] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:38.916510] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:38.997649] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:39.84816] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:39.172045] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:40.896359] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:40.976135] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:41.62052] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:41.147902] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:42.791997] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:42.871239] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:42.956609] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:43.42473] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:43.566190] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
 [2021-03-11 19:15:43.566400] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_a/storage},{slave_node=10.0.231.81}]
 [2021-03-11 19:15:43.572240] D [monitor(monitor):195:monitor] Monitor: Worker would mount volume privately
 [2021-03-11 19:15:43.612744] D [gsyncd(worker /data/storage_a/storage):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:43.625689] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
 [2021-03-11 19:15:43.626060] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_b/storage},{slave_node=10.0.231.82}]
 [2021-03-11 19:15:43.632287] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
 [2021-03-11 19:15:43.632137] D [monitor(monitor):195:monitor] Monitor: Worker would mount volume privately
 [2021-03-11 19:15:43.632508] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_c/storage},{slave_node=10.0.231.82}]
 [2021-03-11 19:15:43.635565] I [resource(worker /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
 [2021-03-11 19:15:43.637835] D [monitor(monitor):195:monitor] Monitor: Worker would mount volume privately
 [2021-03-11 19:15:43.661304] D [repce(worker /data/storage_a/storage):195:push] RepceClient: call 192535:140367272073024:1615490143.66 __repce_version__() ...
 [2021-03-11 19:15:43.674499] D [gsyncd(worker /data/storage_b/storage):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:43.680706] D [gsyncd(worker /data/storage_c/storage):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:43.693773] I [resource(worker /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
 [2021-03-11 19:15:43.700957] I [resource(worker /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
 [2021-03-11 19:15:43.717686] D [repce(worker /data/storage_b/storage):195:push] RepceClient: call 192539:139907321804608:1615490143.72 __repce_version__() ...
 [2021-03-11 19:15:43.725369] D [repce(worker /data/storage_c/storage):195:push] RepceClient: call 192541:140653101852480:1615490143.73 __repce_version__() ...
 [2021-03-11 19:15:44.289117] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:44.375693] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:44.472251] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:44.558429] D [gsyncd(status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:45.619694] D [repce(worker /data/storage_a/storage):215:__call__] RepceClient: call 192535:140367272073024:1615490143.66 __repce_version__ -> 1.0
 [2021-03-11 19:15:45.619930] D [repce(worker /data/storage_a/storage):195:push] RepceClient: call 192535:140367272073024:1615490145.62 version() ...
 [2021-03-11 19:15:45.621191] D [repce(worker /data/storage_a/storage):215:__call__] RepceClient: call 192535:140367272073024:1615490145.62 version -> 1.0
 [2021-03-11 19:15:45.621332] D [repce(worker /data/storage_a/storage):195:push] RepceClient: call 192535:140367272073024:1615490145.62 pid() ...
 [2021-03-11 19:15:45.621859] D [repce(worker /data/storage_a/storage):215:__call__] RepceClient: call 192535:140367272073024:1615490145.62 pid -> 158229
 [2021-03-11 19:15:45.621939] I [resource(worker /data/storage_a/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9862}]
 [2021-03-11 19:15:45.622000] I [resource(worker /data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:45.714468] D [resource(worker /data/storage_a/storage):880:inhibit] DirectMounter: auxiliary glusterfs mount in place
 [2021-03-11 19:15:45.718441] D [repce(worker /data/storage_c/storage):215:__call__] RepceClient: call 192541:140653101852480:1615490143.73 __repce_version__ -> 1.0
 [2021-03-11 19:15:45.718643] D [repce(worker /data/storage_c/storage):195:push] RepceClient: call 192541:140653101852480:1615490145.72 version() ...
 [2021-03-11 19:15:45.719492] D [repce(worker /data/storage_c/storage):215:__call__] RepceClient: call 192541:140653101852480:1615490145.72 version -> 1.0
 [2021-03-11 19:15:45.719772] D [repce(worker /data/storage_c/storage):195:push] RepceClient: call 192541:140653101852480:1615490145.72 pid() ...
 [2021-03-11 19:15:45.720202] D [repce(worker /data/storage_b/storage):215:__call__] RepceClient: call 192539:139907321804608:1615490143.72 __repce_version__ -> 1.0
 [2021-03-11 19:15:45.720381] D [repce(worker /data/storage_b/storage):195:push] RepceClient: call 192539:139907321804608:1615490145.72 version() ...
 [2021-03-11 19:15:45.720463] D [repce(worker /data/storage_c/storage):215:__call__] RepceClient: call 192541:140653101852480:1615490145.72 pid -> 88921
 [2021-03-11 19:15:45.720694] I [resource(worker /data/storage_c/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=2.0196}]
 [2021-03-11 19:15:45.720882] I [resource(worker /data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:45.721146] D [repce(worker /data/storage_b/storage):215:__call__] RepceClient: call 192539:139907321804608:1615490145.72 version -> 1.0
 [2021-03-11 19:15:45.721271] D [repce(worker /data/storage_b/storage):195:push] RepceClient: call 192539:139907321804608:1615490145.72 pid() ...
 [2021-03-11 19:15:45.721795] D [repce(worker /data/storage_b/storage):215:__call__] RepceClient: call 192539:139907321804608:1615490145.72 pid -> 88924
 [2021-03-11 19:15:45.721911] I [resource(worker /data/storage_b/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=2.0280}]
 [2021-03-11 19:15:45.721993] I [resource(worker /data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
 [2021-03-11 19:15:45.816891] D [resource(worker /data/storage_b/storage):880:inhibit] DirectMounter: auxiliary glusterfs mount in place
 [2021-03-11 19:15:45.816960] D [resource(worker /data/storage_c/storage):880:inhibit] DirectMounter: auxiliary glusterfs mount in place
 [2021-03-11 19:15:46.721534] D [resource(worker /data/storage_a/storage):964:inhibit] DirectMounter: auxiliary glusterfs mount prepared
 [2021-03-11 19:15:46.721726] I [resource(worker /data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.0997}]
 [2021-03-11 19:15:46.721796] I [subcmds(worker /data/storage_a/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
 [2021-03-11 19:15:46.721971] D [master(worker /data/storage_a/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=xsync}]
 [2021-03-11 19:15:46.722122] D [monitor(monitor):222:monitor] Monitor: worker(/data/storage_a/storage) connected
 [2021-03-11 19:15:46.723871] D [master(worker /data/storage_a/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=changelog}]
 [2021-03-11 19:15:46.725100] D [master(worker /data/storage_a/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=changeloghistory}]
 [2021-03-11 19:15:46.732400] D [master(worker /data/storage_a/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
 [2021-03-11 19:15:46.823477] D [resource(worker /data/storage_c/storage):964:inhibit] DirectMounter: auxiliary glusterfs mount prepared
 [2021-03-11 19:15:46.823645] I [resource(worker /data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1027}]
 [2021-03-11 19:15:46.823754] I [subcmds(worker /data/storage_c/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
 [2021-03-11 19:15:46.823932] D [master(worker /data/storage_c/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=xsync}]
 [2021-03-11 19:15:46.823904] D [resource(worker /data/storage_b/storage):964:inhibit] DirectMounter: auxiliary glusterfs mount prepared
 [2021-03-11 19:15:46.823930] D [monitor(monitor):222:monitor] Monitor: worker(/data/storage_c/storage) connected
 [2021-03-11 19:15:46.824103] I [resource(worker /data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1020}]
 [2021-03-11 19:15:46.824184] I [subcmds(worker /data/storage_b/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
 [2021-03-11 19:15:46.824340] D [master(worker /data/storage_b/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=xsync}]
 [2021-03-11 19:15:46.824321] D [monitor(monitor):222:monitor] Monitor: worker(/data/storage_b/storage) connected
 [2021-03-11 19:15:46.825100] D [master(worker /data/storage_c/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=changelog}]
 [2021-03-11 19:15:46.825414] D [master(worker /data/storage_b/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=changelog}]
 [2021-03-11 19:15:46.826375] D [master(worker /data/storage_b/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=changeloghistory}]
 [2021-03-11 19:15:46.826574] D [master(worker /data/storage_c/storage):105:gmaster_builder] <top>: setting up change detection mode [{mode=changeloghistory}]
 [2021-03-11 19:15:46.831506] D [master(worker /data/storage_b/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
 [2021-03-11 19:15:46.833168] D [master(worker /data/storage_c/storage):778:setup_working_dir] _GMaster: changelog working dir/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
 [2021-03-11 19:15:47.275141] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:47.320247] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:47.570877] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:47.615571] D [gsyncd(config-get):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:47.620893] E [syncdutils(worker /data/storage_a/storage):325:log_raise_exception] <top>: connection to peer is broken
 [2021-03-11 19:15:47.620939] E [syncdutils(worker /data/storage_c/storage):325:log_raise_exception] <top>: connection to peer is broken
 [2021-03-11 19:15:47.621668] E [syncdutils(worker /data/storage_a/storage):847:errlog] Popen: command returned error [{cmd=ssh -oPasswordAuthentication=no-oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-_AyCOc/79fa3dc75e30f532b4a40bc08c2b10a1.sock geoaccount at 10.0.231.81 /nonexistent/gsyncd slave storage geoaccount at 10.0.231.81::pcic-backup --master-node 10.0.231.91 --master-node-id afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick /data/storage_a/storage --local-node 10.0.231.81 --local-node-id b88dea4f-31ec-416a-9110-3ccdc3910acd --slave-timeout 120 --slave-log-level INFO --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin --master-dist-count 3}, {error=255}]
 [2021-03-11 19:15:47.621685] E [syncdutils(worker /data/storage_c/storage):847:errlog] Popen: command returned error [{cmd=ssh -oPasswordAuthentication=no-oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-WOgOEu/e15fc58bb13552de0710eaf018209548.sock geoaccount at 10.0.231.82 /nonexistent/gsyncd slave storage geoaccount at 10.0.231.81::pcic-backup --master-node 10.0.231.91 --master-node-id afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick /data/storage_c/storage --local-node 10.0.231.82 --local-node-id be50a8de-3934-4fee-a80d-8e2e99017902 --slave-timeout 120 --slave-log-level INFO --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin --master-dist-count 3}, {error=255}]
 [2021-03-11 19:15:47.621776] E [syncdutils(worker /data/storage_a/storage):851:logerr] Popen: ssh> Killed by signal 15.
 [2021-03-11 19:15:47.621819] E [syncdutils(worker /data/storage_c/storage):851:logerr] Popen: ssh> Killed by signal 15.
 [2021-03-11 19:15:47.621850] E [syncdutils(worker /data/storage_b/storage):325:log_raise_exception] <top>: connection to peer is broken
 [2021-03-11 19:15:47.622437] E [syncdutils(worker /data/storage_b/storage):847:errlog] Popen: command returned error [{cmd=ssh -oPasswordAuthentication=no-oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-Vy935W/e15fc58bb13552de0710eaf018209548.sock geoaccount at 10.0.231.82 /nonexistent/gsyncd slave storage geoaccount at 10.0.231.81::pcic-backup --master-node 10.0.231.91 --master-node-id afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick /data/storage_b/storage --local-node 10.0.231.82 --local-node-id be50a8de-3934-4fee-a80d-8e2e99017902 --slave-timeout 120 --slave-log-level INFO --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin --master-dist-count 3}, {error=255}]
 [2021-03-11 19:15:47.622556] E [syncdutils(worker /data/storage_b/storage):851:logerr] Popen: ssh> Killed by signal 15.
 [2021-03-11 19:15:47.723756] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_a/storage}]
 [2021-03-11 19:15:47.731405] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
 [2021-03-11 19:15:47.825223] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_c/storage}]
 [2021-03-11 19:15:47.825685] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_b/storage}]
 [2021-03-11 19:15:47.829011] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
 [2021-03-11 19:15:47.830965] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
 [2021-03-11 19:15:48.669634] D [gsyncd(monitor-status):303:main] <top>: Using session config file[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
 [2021-03-11 19:15:48.683784] I [subcmds(monitor-status):29:subcmd_monitor_status] <top>: Monitor Status Change [{status=Stopped}]

 Thanks,
  -Matthew

  On 3/11/21 9:37 AM, Strahil Nikolov wrote:

 Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.

I think you have to increase the debug logs for geo-rep session.
I will try to find the command necessary to increase it.

Best Regards,
Strahil Nikolov

В четвъртък, 11 март 2021 г., 00:38:41 ч. Гринуич+2, Matthew Benstead <matthewb at uvic.ca> написа:

Thanks Strahil,

Right - I had come across your message in early January that v8 from the CentOS Sig was missing the SELinux rules, and had put SELinux into permissive mode after the upgrade when I saw denied messages in the audit logs.

[root at storage01 ~]# sestatus | egrep "^SELinux status|[mM]ode"
SELinux status:                 enabled
Current mode:                   permissive
Mode from config file:          enforcing

Yes - I am using an unprivileged user for georep:

[root at pcic-backup01 ~]# gluster-mountbroker status
+-------------+-------------+---------------------------+--------------+--------------------------+
|     NODE    | NODE STATUS |         MOUNT ROOT        |    GROUP     |          USERS           |
+-------------+-------------+---------------------------+--------------+--------------------------+
| 10.0.231.82 |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
|  localhost  |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
+-------------+-------------+---------------------------+--------------+--------------------------+

[root at pcic-backup02 ~]# gluster-mountbroker status
+-------------+-------------+---------------------------+--------------+--------------------------+
|     NODE    | NODE STATUS |         MOUNT ROOT        |    GROUP     |          USERS           |
+-------------+-------------+---------------------------+--------------+--------------------------+
| 10.0.231.81 |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
|  localhost  |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
+-------------+-------------+---------------------------+--------------+--------------------------+

Thanks,
 -Matthew

--
Matthew Benstead
System AdministratorPacific Climate Impacts ConsortiumUniversity of Victoria, UH1PO Box 1800, STN CSCVictoria, BC, V8W 2Y2Phone: +1-250-721-8432Email: matthewb at uvic.ca

On 3/10/21 2:11 PM, Strahil Nikolov wrote:

  Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.

I have tested georep on v8.3 and it was running quite well untill you involve SELINUX.

Are you using SELINUX ?

Are you using unprivileged user for the georep ?

Also, you can check https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-troubleshooting_geo-replication .

Best Regards,

Strahil Nikolov

On Thu, Mar 11, 2021 at 0:03, Matthew Benstead

<matthewb at uvic.ca> wrote:

Hello,

I recently upgraded my Distributed-Replicate cluster from Gluster 7.9 to 8.3 on CentOS7 using the CentOS Storage SIG packages. I had geo-replication syncing properly before the upgrade, but not it is not working after.

After I had upgraded both master and slave clusters I attempted to start geo-replication again, but it goes to faulty quickly:

[root at storage01 ~]# gluster volume geo-replication storage  geoaccount at 10.0.231.81::pcic-backup start
Starting geo-replication session between storage &  geoaccount at 10.0.231.81::pcic-backup has been successful\

[root at storage01 ~]# gluster volume geo-replication status

MASTER NODE    MASTER VOL    MASTER BRICK               SLAVE USER    SLAVE                                        SLAVE NODE    STATUS    CRAWL STATUS    LAST_SYNCED
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.0.231.91    storage       /data/storage_a/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
10.0.231.91    storage       /data/storage_c/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
10.0.231.91    storage       /data/storage_b/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
10.0.231.92    storage       /data/storage_b/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
10.0.231.92    storage       /data/storage_a/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
10.0.231.92    storage       /data/storage_c/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
10.0.231.93    storage       /data/storage_c/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
10.0.231.93    storage       /data/storage_b/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
10.0.231.93    storage       /data/storage_a/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A

[root at storage01 ~]# gluster volume geo-replication storage  geoaccount at 10.0.231.81::pcic-backup stop
Stopping geo-replication session between storage &  geoaccount at 10.0.231.81::pcic-backup has been successful

I went through the gsyncd logs and see it attempts to go back through the changlogs - which would make sense - but fails:

[2021-03-10 19:18:42.165807] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:42.166136] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]
[2021-03-10 19:18:42.167829] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]
[2021-03-10 19:18:42.172343] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:42.172580] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]
[2021-03-10 19:18:42.235574] I [resource(worker /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:42.236613] I [resource(worker /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:42.238614] I [resource(worker /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:44.144856] I [resource(worker /data/storage_b/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9059}]
[2021-03-10 19:18:44.145065] I [resource(worker /data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
[2021-03-10 19:18:44.162873] I [resource(worker /data/storage_a/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9259}]
[2021-03-10 19:18:44.163412] I [resource(worker /data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
[2021-03-10 19:18:44.167506] I [resource(worker /data/storage_c/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9316}]
[2021-03-10 19:18:44.167746] I [resource(worker /data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
[2021-03-10 19:18:45.251372] I [resource(worker /data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1062}]
[2021-03-10 19:18:45.251583] I [subcmds(worker /data/storage_b/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2021-03-10 19:18:45.271950] I [resource(worker /data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1041}]
[2021-03-10 19:18:45.272118] I [subcmds(worker /data/storage_c/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2021-03-10 19:18:45.275180] I [resource(worker /data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1116}]
[2021-03-10 19:18:45.275361] I [subcmds(worker /data/storage_a/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2021-03-10 19:18:47.265618] I [master(worker /data/storage_b/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage}]
[2021-03-10 19:18:47.265954] I [resource(worker /data/storage_b/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
[2021-03-10 19:18:47.276746] I [gsyncdstatus(worker /data/storage_b/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
[2021-03-10 19:18:47.281194] I [gsyncdstatus(worker /data/storage_b/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
[2021-03-10 19:18:47.281404] I [master(worker /data/storage_b/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664113, 0)}, {etime=1615403927}]
[2021-03-10 19:18:47.285340] I [master(worker /data/storage_c/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage}]
[2021-03-10 19:18:47.285579] I [resource(worker /data/storage_c/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
[2021-03-10 19:18:47.287383] I [master(worker /data/storage_a/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage}]
[2021-03-10 19:18:47.287697] I [resource(worker /data/storage_a/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
[2021-03-10 19:18:47.298415] I [gsyncdstatus(worker /data/storage_c/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
[2021-03-10 19:18:47.301342] I [gsyncdstatus(worker /data/storage_a/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
[2021-03-10 19:18:47.304183] I [gsyncdstatus(worker /data/storage_c/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
[2021-03-10 19:18:47.304418] I [master(worker /data/storage_c/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664108, 0)}, {etime=1615403927}]
[2021-03-10 19:18:47.305294] E [resource(worker /data/storage_c/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
[2021-03-10 19:18:47.308124] I [gsyncdstatus(worker /data/storage_a/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
[2021-03-10 19:18:47.308509] I [master(worker /data/storage_a/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666553, 0)}, {entry_stime=(1614664115, 0)}, {etime=1615403927}]
[2021-03-10 19:18:47.357470] E [resource(worker /data/storage_b/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
[2021-03-10 19:18:47.383949] E [resource(worker /data/storage_a/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
[2021-03-10 19:18:48.255340] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_b/storage}]
[2021-03-10 19:18:48.260052] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
[2021-03-10 19:18:48.275651] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_c/storage}]
[2021-03-10 19:18:48.278064] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_a/storage}]
[2021-03-10 19:18:48.280453] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
[2021-03-10 19:18:48.282274] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
[2021-03-10 19:18:58.275702] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:58.276041] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]
[2021-03-10 19:18:58.296252] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:58.296506] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]
[2021-03-10 19:18:58.301290] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:58.301521] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]
[2021-03-10 19:18:58.345817] I [resource(worker /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:58.361268] I [resource(worker /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:58.367985] I [resource(worker /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:59.115143] I [subcmds(monitor-status):29:subcmd_monitor_status] <top>: Monitor Status Change [{status=Stopped}]

It seems like there is an issue selecting the changelogs - perhaps similar to this issue?  https://github.com/gluster/glusterfs/issues/1766

[root at storage01 storage_10.0.231.81_pcic-backup]# cat changes-data-storage_a-storage.log
[2021-03-10 19:18:45.284764] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_a/storage}, {notify_filter=1}]
[2021-03-10 19:18:45.285275] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}]
[2021-03-10 19:18:45.285269] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}]
[2021-03-10 19:18:45.286615] I [socket.c:929:__socket_server_bind] 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 21
[2021-03-10 19:18:47.308607] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666553}, {end=1615403927}]
[2021-03-10 19:18:47.308659] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1615403927}, {total_changelogs=1250878}]
[2021-03-10 19:18:47.383774] E [MSGID: 132009] [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong result [{for=end}, {start=1615403927}, {idx=1250877}]

[root at storage01 storage_10.0.231.81_pcic-backup]# tail -7 changes-data-storage_b-storage.log
[2021-03-10 19:18:45.263211] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}]
[2021-03-10 19:18:45.263151] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_b/storage}, {notify_filter=1}]
[2021-03-10 19:18:45.263294] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}]
[2021-03-10 19:18:45.264598] I [socket.c:929:__socket_server_bind] 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 23
[2021-03-10 19:18:47.281499] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666552}, {end=1615403927}]
[2021-03-10 19:18:47.281551] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1615403927}, {total_changelogs=1258258}]
[2021-03-10 19:18:47.357244] E [MSGID: 132009] [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong result [{for=end}, {start=1615403927}, {idx=1258257}]

Any ideas on where to debug this? I'd prefer not to have to remove and re-sync everything as there is about 240TB on the cluster...

Thanks,
 -Matthew

________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210319/fe071cc6/attachment.html>