[Gluster-users] GeoRep Faulty after Gluster 7 to 8 upgrade - gfchangelog: wrong result
Matthew Benstead
matthewb at uvic.ca
Wed Mar 17 21:11:42 UTC 2021
Yes, I've run through everything short of regenerating the keys and
creating the session again with no errors. Everything looks ok.
But I did notice that the changelog format had changed, instead of them
being dumped into one directory, they now seem to be separated in
year/month/day directories...
Looks like this change in 8.0:
https://github.com/gluster/glusterfs/issues/154
[root at storage01 changelogs]# ls -lh | head
total 16G
drw-------. 3 root root 24 Mar 9 11:34 2021
-rw-r--r--. 1 root root 51 Mar 17 13:19 CHANGELOG
-rw-r--r--. 1 root root 13K Aug 13 2020 CHANGELOG.1597343197
-rw-r--r--. 1 root root 51K Aug 13 2020 CHANGELOG.1597343212
-rw-r--r--. 1 root root 86K Aug 13 2020 CHANGELOG.1597343227
-rw-r--r--. 1 root root 99K Aug 13 2020 CHANGELOG.1597343242
-rw-r--r--. 1 root root 69K Aug 13 2020 CHANGELOG.1597343257
-rw-r--r--. 1 root root 69K Aug 13 2020 CHANGELOG.1597343272
-rw-r--r--. 1 root root 72K Aug 13 2020 CHANGELOG.1597343287
[root at storage01 changelogs]# ls -lh | tail
-rw-r--r--. 1 root root 92 Mar 1 21:33 CHANGELOG.1614663193
-rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663731
-rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663760
-rw-r--r--. 1 root root 511 Mar 1 21:47 CHANGELOG.1614664043
-rw-r--r--. 1 root root 536 Mar 1 21:48 CHANGELOG.1614664101
-rw-r--r--. 1 root root 2.8K Mar 1 21:48 CHANGELOG.1614664116
-rw-r--r--. 1 root root 92 Mar 1 22:20 CHANGELOG.1614666061
-rw-r--r--. 1 root root 92 Mar 1 22:29 CHANGELOG.1614666554
drw-------. 2 root root 10 May 7 2020 csnap
drw-------. 2 root root 38 Aug 13 2020 htime
[root at storage01 changelogs]# ls -lh 2021/03/09/
total 8.0K
-rw-r--r--. 1 root root 51 Mar 9 11:26 CHANGELOG.1615318474
-rw-r--r--. 1 root root 51 Mar 9 12:19 CHANGELOG.1615321197
[root at storage01 changelogs]# ls -lh 2021/03/15/
total 4.0K
-rw-r--r--. 1 root root 51 Mar 15 13:38 CHANGELOG.1615842490
[root at storage01 changelogs]# ls -lh 2021/03/16
total 4.0K
-rw-r--r--. 1 root root 331 Mar 16 12:04 CHANGELOG.1615921482
But it looks like the htime file still records them...
[root at storage01 changelogs]# ls -lh htime
total 84M
-rw-r--r--. 1 root root 84M Mar 17 13:31 HTIME.1597342860
[root at storage01 changelogs]# head -c 256 htime/HTIME.1597342860
/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342875/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342890/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342904/data/storage_a/storage/.glusterfs/changelogs/changelog[root at storage01
changelogs]#
[root at storage01 changelogs]# tail -c 256 htime/HTIME.1597342860
/changelog.1616013484/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013499/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013514/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013529[root at storage01
changelogs]#
And there seems to be an xattr for time in the brick root - presumably
for when changelogs were enabled:
[root at storage01 changelogs]# getfattr -d -m. -e hex
/data/storage_a/storage 2>&1 | egrep xtime
trusted.glusterfs.cf94a8f2-324b-40b3-bf72-c3766100ea99.xtime=0x60510140000ef317
Reading through the changes, it looks like there is a script to convert
from one format to the other... I didn't see anything in the release
notes for 8.0 about it... this seems like it could be the fix, and
explain why gluster can't get through the changelogs.... Thoughts?
* https://github.com/gluster/glusterfs/issues/154#issuecomment-585701964
* https://review.gluster.org/#/c/glusterfs/+/24121/
Thanks,
-Matthew
--
Matthew Benstead
System Administrator
Pacific Climate Impacts Consortium <https://pacificclimate.org/>
University of Victoria, UH1
PO Box 1800, STN CSC
Victoria, BC, V8W 2Y2
Phone: +1-250-721-8432
Email: matthewb at uvic.ca
On 3/16/21 9:36 PM, Strahil Nikolov wrote:
> Notice: This message was sent from outside the University of Victoria
> email system. Please be cautious with links and sensitive information.
>
> Have you verified all steps for creating the geo-replication ?
>
> If yes , maybe using "reset-sync-time + delete + create" makes
> sense.Keep in mind that it will take a long time once the geo-rep is
> established again.
>
>
> Best Regards,
> Strahil Nikolov
>
> On Tue, Mar 16, 2021 at 22:34, Matthew Benstead
> <matthewb at uvic.ca> wrote:
> Thanks Strahil,
>
> I wanted to make sure the issue wasn't occurring because there
> were no new changes to sync from the master volume. So I created
> some files and restarted the sync, but it had no effect.
>
> [root at storage01 ~]# cd /storage2/home/test/
> [root at storage01 test]# for nums in {1,2,3,4,5,6,7,8,9,0}; do touch
> $nums.txt; done
>
> [root at storage01 test]# gluster volume geo-replication storage
> geoaccount at 10.0.231.81::pcic-backup
> <mailto:geoaccount at 10.0.231.81::pcic-backup> start
> Starting geo-replication session between storage &
> geoaccount at 10.0.231.81::pcic-backup
> <mailto:geoaccount at 10.0.231.81::pcic-backup> has been successful
> [root at storage01 test]# gluster volume geo-replication status
>
> MASTER NODE MASTER VOL MASTER BRICK SLAVE
> USER SLAVE SLAVE NODE
> STATUS CRAWL STATUS LAST_SYNCED
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 10.0.231.91 storage /data/storage_a/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Initializing... N/A N/A
> 10.0.231.91 storage /data/storage_c/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Initializing... N/A N/A
> 10.0.231.91 storage /data/storage_b/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Initializing... N/A N/A
> 10.0.231.93 storage /data/storage_c/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Initializing... N/A N/A
> 10.0.231.93 storage /data/storage_b/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Initializing... N/A N/A
> 10.0.231.93 storage /data/storage_a/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Initializing... N/A N/A
> 10.0.231.92 storage /data/storage_b/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Initializing... N/A N/A
> 10.0.231.92 storage /data/storage_a/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Initializing... N/A N/A
> 10.0.231.92 storage /data/storage_c/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Initializing... N/A N/A
> [root at storage01 test]# gluster volume geo-replication status
>
> MASTER NODE MASTER VOL MASTER BRICK SLAVE
> USER SLAVE SLAVE NODE
> STATUS CRAWL STATUS LAST_SYNCED
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 10.0.231.91 storage /data/storage_a/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Faulty N/A N/A
> 10.0.231.91 storage /data/storage_c/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Faulty N/A N/A
> 10.0.231.91 storage /data/storage_b/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Faulty N/A N/A
> 10.0.231.93 storage /data/storage_c/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Faulty N/A N/A
> 10.0.231.93 storage /data/storage_b/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Faulty N/A N/A
> 10.0.231.93 storage /data/storage_a/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Faulty N/A N/A
> 10.0.231.92 storage /data/storage_b/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Faulty N/A N/A
> 10.0.231.92 storage /data/storage_a/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Faulty N/A N/A
> 10.0.231.92 storage /data/storage_c/storage
> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
> N/A Faulty N/A N/A
> [root at storage01 test]# gluster volume geo-replication storage
> geoaccount at 10.0.231.81::pcic-backup
> <mailto:geoaccount at 10.0.231.81::pcic-backup> stop
> Stopping geo-replication session between storage &
> geoaccount at 10.0.231.81::pcic-backup
> <mailto:geoaccount at 10.0.231.81::pcic-backup> has been successful
>
> Still getting the same error about the history crawl failing:
>
> [2021-03-16 19:05:05.227677] I [MSGID: 132035]
> [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog:
> Requesting historical changelogs [{start=1614666552},
> {end=1615921505}]
> [2021-03-16 19:05:05.227733] I [MSGID: 132019]
> [gf-history-changelog.c:755:gf_changelog_extract_min_max]
> 0-gfchangelog: changelogs min max [{min=1597342860},
> {max=1615921502}, {total_changelogs=1300114}]
> [2021-03-16 19:05:05.408567] E [MSGID: 132009]
> [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog:
> wrong result [{for=end}, {start=1615921502}, {idx=1300113}]
>
>
> [2021-03-16 19:05:05.228092] I [resource(worker
> /data/storage_c/storage):1292:service_loop] GLUSTER: Register time
> [{time=1615921505}]
> [2021-03-16 19:05:05.228626] D [repce(worker
> /data/storage_c/storage):195:push] RepceClient: call
> 124117:140500837320448:1615921505.23 keep_alive(None,) ...
> [2021-03-16 19:05:05.230076] D [repce(worker
> /data/storage_c/storage):215:__call__] RepceClient: call
> 124117:140500837320448:1615921505.23 keep_alive -> 1
> [2021-03-16 19:05:05.230693] D [master(worker
> /data/storage_c/storage):540:crawlwrap] _GMaster: primary master
> with volume id cf94a8f2-324b-40b3-bf72-c3766100ea99 ...
> [2021-03-16 19:05:05.237607] I [gsyncdstatus(worker
> /data/storage_c/storage):281:set_active] GeorepStatus: Worker
> Status Change [{status=Active}]
> [2021-03-16 19:05:05.242046] I [gsyncdstatus(worker
> /data/storage_c/storage):253:set_worker_crawl_status]
> GeorepStatus: Crawl Status Change [{status=History Crawl}]
> [2021-03-16 19:05:05.242450] I [master(worker
> /data/storage_c/storage):1559:crawl] _GMaster: starting history
> crawl [{turns=1}, {stime=(1614666552, 0)},
> {entry_stime=(1614664108, 0)}, {etime=1615921505}]
> [2021-03-16 19:05:05.244151] E [resource(worker
> /data/storage_c/storage):1312:service_loop] GLUSTER: Changelog
> History Crawl failed [{error=[Errno 0] Success}]
> [2021-03-16 19:05:05.394129] E [resource(worker
> /data/storage_a/storage):1312:service_loop] GLUSTER: Changelog
> History Crawl failed [{error=[Errno 0] Success}]
> [2021-03-16 19:05:05.408759] E [resource(worker
> /data/storage_b/storage):1312:service_loop] GLUSTER: Changelog
> History Crawl failed [{error=[Errno 0] Success}]
> [2021-03-16 19:05:06.158694] I [monitor(monitor):228:monitor]
> Monitor: worker died in startup phase
> [{brick=/data/storage_a/storage}]
> [2021-03-16 19:05:06.163052] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker
> Status Change [{status=Faulty}]
> [2021-03-16 19:05:06.204464] I [monitor(monitor):228:monitor]
> Monitor: worker died in startup phase
> [{brick=/data/storage_b/storage}]
> [2021-03-16 19:05:06.208961] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker
> Status Change [{status=Faulty}]
> [2021-03-16 19:05:06.220495] I [monitor(monitor):228:monitor]
> Monitor: worker died in startup phase
> [{brick=/data/storage_c/storage}]
> [2021-03-16 19:05:06.223947] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker
> Status Change [{status=Faulty}]
>
> I confirmed NTP is working:
>
>
> pcic-backup02 | CHANGED | rc=0 >>
> remote refid st t when poll reach delay
> offset jitter
> ==============================================================================
> +s216-232-132-95 68.69.221.61 2 u 29 1024 377 24.141
> 2.457 1.081
> *yyz-1.ip.0xt.ca 206.108.0.131 2 u 257 1024 377 57.119
> -0.084 5.625
> +ip102.ip-198-27 192.168.10.254 2 u 189 1024 377 64.227
> -3.012 8.867
>
> storage03 | CHANGED | rc=0 >>
> remote refid st t when poll reach delay
> offset jitter
> ==============================================================================
> *198.161.203.36 128.233.150.93 2 u 36 1024 377 16.055
> -0.381 0.318
> +s206-75-147-25. 192.168.10.254 2 u 528 1024 377 23.648
> -6.196 4.803
> +time.cloudflare 10.69.8.80 3 u 121 1024 377 2.408
> 0.507 0.791
>
> storage02 | CHANGED | rc=0 >>
> remote refid st t when poll reach delay
> offset jitter
> ==============================================================================
> *198.161.203.36 128.233.150.93 2 u 918 1024 377 15.952
> 0.226 0.197
> +linuxgeneration 16.164.40.197 2 u 88 1024 377 62.692
> -1.160 2.007
> +dns3.switch.ca 206.108.0.131 2 u 857 1024 377 27.315
> 0.778 0.483
>
> storage01 | CHANGED | rc=0 >>
> remote refid st t when poll reach delay
> offset jitter
> ==============================================================================
> +198.161.203.36 128.233.150.93 2 u 121 1024 377 16.069
> 1.016 0.195
> +zero.gotroot.ca 30.114.5.31 2 u 543 1024 377 5.106
> -2.462 4.923
> *ntp3.torix.ca .PTP0. 1 u 300 1024 377 54.010
> 2.421 15.182
>
> pcic-backup01 | CHANGED | rc=0 >>
> remote refid st t when poll reach delay
> offset jitter
> ==============================================================================
> *dns3.switch.ca 206.108.0.131 2 u 983 1024 377 26.990
> 0.523 1.389
> +dns2.switch.ca 206.108.0.131 2 u 689 1024 377 26.975
> -0.257 0.467
> +64.ip-54-39-23. 214.176.184.39 2 u 909 1024 377 64.262
> -0.604 6.129
>
> And everything is working on the same version of gluster:
>
> pcic-backup02 | CHANGED | rc=0 >>
> glusterfs 8.3
> pcic-backup01 | CHANGED | rc=0 >>
> glusterfs 8.3
> storage02 | CHANGED | rc=0 >>
> glusterfs 8.3
> storage01 | CHANGED | rc=0 >>
> glusterfs 8.3
> storage03 | CHANGED | rc=0 >>
> glusterfs 8.3
>
> SSH works, and the backup user/group is configured with mountbroker:
>
> [root at storage01 ~]# ssh -i /root/.ssh/id_rsa
> geoaccount at 10.0.231.81 <mailto:geoaccount at 10.0.231.81> uname -a
> Linux pcic-backup01 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3
> 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> [root at storage01 ~]# ssh -i /root/.ssh/id_rsa
> geoaccount at 10.0.231.82 <mailto:geoaccount at 10.0.231.82> uname -a
> Linux pcic-backup02 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3
> 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>
>
> [root at pcic-backup01 ~]# grep geo /etc/passwd
> geoaccount:x:1000:1000::/home/geoaccount:/bin/bash
> [root at pcic-backup01 ~]# grep geo /etc/group
> geogroup:x:1000:geoaccount
> geoaccount:x:1001:geoaccount
>
> [root at pcic-backup01 ~]# gluster-mountbroker status
> +-------------+-------------+---------------------------+--------------+--------------------------+
> | NODE | NODE STATUS | MOUNT ROOT |
> GROUP | USERS |
> +-------------+-------------+---------------------------+--------------+--------------------------+
> | 10.0.231.82 | UP | /var/mountbroker-root(OK) |
> geogroup(OK) | geoaccount(pcic-backup) |
> | localhost | UP | /var/mountbroker-root(OK) |
> geogroup(OK) | geoaccount(pcic-backup) |
> +-------------+-------------+---------------------------+--------------+--------------------------+
>
>
>
>
> So, then if I'm going to have to resync, what is the best way to
> do this?
>
> With delete or delete reset-sync-time ?
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-starting_geo-replication#Deleting_a_Geo-replication_Session
> <https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-starting_geo-replication#Deleting_a_Geo-replication_Session>
>
>
>
> Erasing the index? So I don't have to transfer the files again
> that are already on the backup?
>
> * https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/sect-troubleshooting_geo-replication#Synchronization_Is_Not_Complete
> <https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/sect-troubleshooting_geo-replication#Synchronization_Is_Not_Complete>
>
> * https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Geo%20Replication/#best-practices
> <https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Geo%20Replication/#best-practices>
>
>
>
>
> Is it possible to use the special-sync-mode option from here:
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-disaster_recovery
> <https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-disaster_recovery>
>
>
>
> Thoughts?
>
> Thanks,
> -Matthew
> --
>
> On 3/12/21 3:31 PM, Strahil Nikolov wrote:
>> Notice: This message was sent from outside the University of
>> Victoria email system. Please be cautious with links and
>> sensitive information.
>>
>> Usually, when I'm stuck - I just start over.
>> For example, check the prerequisites:
>> - Is ssh available (no firewall blocking)
>> - Is time sync enabled (ntp/chrony)
>> - Is DNS ok on all hosts (including PTR records)
>> - Is the gluster version the same on all nodes (primary & secondary)
>>
>> Then start over as if the geo rep was never existing. For example
>> , stop it and start over with the secondary nodes's checks
>> (mountbroker, user, group) .
>>
>> Most probably somwthing will come up and you will fix it.
>>
>> In worst case scenario, you will need to clean ip the geo-rep and
>> start fresh.
>>
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> On Fri, Mar 12, 2021 at 20:01, Matthew Benstead
>> <matthewb at uvic.ca> <mailto:matthewb at uvic.ca> wrote:
>> Hi Strahil,
>>
>> Yes, SELinux was put into permissive mode on the secondary
>> nodes as well:
>>
>> [root at pcic-backup01 ~]# sestatus | egrep -i "^SELinux
>> status|mode"
>> SELinux status: enabled
>> Current mode: permissive
>> Mode from config file: enforcing
>>
>> [root at pcic-backup02 ~]# sestatus | egrep -i "^SELinux
>> status|mode"
>> SELinux status: enabled
>> Current mode: permissive
>> Mode from config file: enforcing
>>
>> The secondary server logs didn't show anything interesting:
>>
>> gsyncd.log:
>>
>> [2021-03-11 19:15:28.81820] I [resource(slave
>> 10.0.231.92/data/storage_c/storage):1116:connect] GLUSTER:
>> Mounting gluster volume locally...
>> [2021-03-11 19:15:28.101819] I [resource(slave
>> 10.0.231.91/data/storage_a/storage):1116:connect] GLUSTER:
>> Mounting gluster volume locally...
>> [2021-03-11 19:15:28.107012] I [resource(slave
>> 10.0.231.93/data/storage_c/storage):1116:connect] GLUSTER:
>> Mounting gluster volume locally...
>> [2021-03-11 19:15:28.124567] I [resource(slave
>> 10.0.231.93/data/storage_b/storage):1116:connect] GLUSTER:
>> Mounting gluster volume locally...
>> [2021-03-11 19:15:28.128145] I [resource(slave
>> 10.0.231.93/data/storage_a/storage):1116:connect] GLUSTER:
>> Mounting gluster volume locally...
>> [2021-03-11 19:15:29.425739] I [resource(slave
>> 10.0.231.93/data/storage_c/storage):1139:connect] GLUSTER:
>> Mounted gluster volume [{duration=1.3184}]
>> [2021-03-11 19:15:29.427448] I [resource(slave
>> 10.0.231.93/data/storage_c/storage):1166:service_loop]
>> GLUSTER: slave listening
>> [2021-03-11 19:15:29.433340] I [resource(slave
>> 10.0.231.93/data/storage_b/storage):1139:connect] GLUSTER:
>> Mounted gluster volume [{duration=1.3083}]
>> [2021-03-11 19:15:29.434452] I [resource(slave
>> 10.0.231.91/data/storage_a/storage):1139:connect] GLUSTER:
>> Mounted gluster volume [{duration=1.3321}]
>> [2021-03-11 19:15:29.434314] I [resource(slave
>> 10.0.231.93/data/storage_b/storage):1166:service_loop]
>> GLUSTER: slave listening
>> [2021-03-11 19:15:29.435575] I [resource(slave
>> 10.0.231.91/data/storage_a/storage):1166:service_loop]
>> GLUSTER: slave listening
>> [2021-03-11 19:15:29.439769] I [resource(slave
>> 10.0.231.92/data/storage_c/storage):1139:connect] GLUSTER:
>> Mounted gluster volume [{duration=1.3576}]
>> [2021-03-11 19:15:29.440998] I [resource(slave
>> 10.0.231.92/data/storage_c/storage):1166:service_loop]
>> GLUSTER: slave listening
>> [2021-03-11 19:15:29.454745] I [resource(slave
>> 10.0.231.93/data/storage_a/storage):1139:connect] GLUSTER:
>> Mounted gluster volume [{duration=1.3262}]
>> [2021-03-11 19:15:29.456192] I [resource(slave
>> 10.0.231.93/data/storage_a/storage):1166:service_loop]
>> GLUSTER: slave listening
>> [2021-03-11 19:15:32.594865] I [repce(slave
>> 10.0.231.92/data/storage_c/storage):96:service_loop]
>> RepceServer: terminating on reaching EOF.
>> [2021-03-11 19:15:32.607815] I [repce(slave
>> 10.0.231.93/data/storage_c/storage):96:service_loop]
>> RepceServer: terminating on reaching EOF.
>> [2021-03-11 19:15:32.647663] I [repce(slave
>> 10.0.231.93/data/storage_b/storage):96:service_loop]
>> RepceServer: terminating on reaching EOF.
>> [2021-03-11 19:15:32.656280] I [repce(slave
>> 10.0.231.91/data/storage_a/storage):96:service_loop]
>> RepceServer: terminating on reaching EOF.
>> [2021-03-11 19:15:32.668299] I [repce(slave
>> 10.0.231.93/data/storage_a/storage):96:service_loop]
>> RepceServer: terminating on reaching EOF.
>> [2021-03-11 19:15:44.260689] I [resource(slave
>> 10.0.231.92/data/storage_c/storage):1116:connect] GLUSTER:
>> Mounting gluster volume locally...
>> [2021-03-11 19:15:44.271457] I [resource(slave
>> 10.0.231.93/data/storage_c/storage):1116:connect] GLUSTER:
>> Mounting gluster volume locally...
>> [2021-03-11 19:15:44.271883] I [resource(slave
>> 10.0.231.93/data/storage_b/storage):1116:connect] GLUSTER:
>> Mounting gluster volume locally...
>> [2021-03-11 19:15:44.279670] I [resource(slave
>> 10.0.231.91/data/storage_a/storage):1116:connect] GLUSTER:
>> Mounting gluster volume locally...
>> [2021-03-11 19:15:44.284261] I [resource(slave
>> 10.0.231.93/data/storage_a/storage):1116:connect] GLUSTER:
>> Mounting gluster volume locally...
>> [2021-03-11 19:15:45.614280] I [resource(slave
>> 10.0.231.93/data/storage_b/storage):1139:connect] GLUSTER:
>> Mounted gluster volume [{duration=1.3419}]
>> [2021-03-11 19:15:45.615622] I [resource(slave
>> 10.0.231.93/data/storage_b/storage):1166:service_loop]
>> GLUSTER: slave listening
>> [2021-03-11 19:15:45.617986] I [resource(slave
>> 10.0.231.93/data/storage_c/storage):1139:connect] GLUSTER:
>> Mounted gluster volume [{duration=1.3461}]
>> [2021-03-11 19:15:45.618180] I [resource(slave
>> 10.0.231.91/data/storage_a/storage):1139:connect] GLUSTER:
>> Mounted gluster volume [{duration=1.3380}]
>> [2021-03-11 19:15:45.619539] I [resource(slave
>> 10.0.231.91/data/storage_a/storage):1166:service_loop]
>> GLUSTER: slave listening
>> [2021-03-11 19:15:45.618999] I [resource(slave
>> 10.0.231.93/data/storage_c/storage):1166:service_loop]
>> GLUSTER: slave listening
>> [2021-03-11 19:15:45.620843] I [resource(slave
>> 10.0.231.93/data/storage_a/storage):1139:connect] GLUSTER:
>> Mounted gluster volume [{duration=1.3361}]
>> [2021-03-11 19:15:45.621347] I [resource(slave
>> 10.0.231.92/data/storage_c/storage):1139:connect] GLUSTER:
>> Mounted gluster volume [{duration=1.3604}]
>> [2021-03-11 19:15:45.622179] I [resource(slave
>> 10.0.231.93/data/storage_a/storage):1166:service_loop]
>> GLUSTER: slave listening
>> [2021-03-11 19:15:45.622541] I [resource(slave
>> 10.0.231.92/data/storage_c/storage):1166:service_loop]
>> GLUSTER: slave listening
>> [2021-03-11 19:15:47.626054] I [repce(slave
>> 10.0.231.91/data/storage_a/storage):96:service_loop]
>> RepceServer: terminating on reaching EOF.
>> [2021-03-11 19:15:48.778399] I [repce(slave
>> 10.0.231.93/data/storage_c/storage):96:service_loop]
>> RepceServer: terminating on reaching EOF.
>> [2021-03-11 19:15:48.778491] I [repce(slave
>> 10.0.231.92/data/storage_c/storage):96:service_loop]
>> RepceServer: terminating on reaching EOF.
>> [2021-03-11 19:15:48.796854] I [repce(slave
>> 10.0.231.93/data/storage_a/storage):96:service_loop]
>> RepceServer: terminating on reaching EOF.
>> [2021-03-11 19:15:48.800697] I [repce(slave
>> 10.0.231.93/data/storage_b/storage):96:service_loop]
>> RepceServer: terminating on reaching EOF.
>>
>> The mnt geo-rep files were also uninteresting:
>> [2021-03-11 19:15:28.250150] I [MSGID: 100030]
>> [glusterfsd.c:2689:main] 0-/usr/sbin/glusterfs: Started
>> running version [{arg=/usr/sbin/glusterfs}, {version=8.3},
>> {cmdlinestr=/usr/sbin/glusterfs --user-map-root=g
>> eoaccount --aux-gfid-mount --acl --log-level=INFO
>> --log-file=/var/log/glusterfs/geo-replication-slaves/storage_10.0.231.81_pcic-backup/mnt-10.0.231.93-data-storage_b-storage.log
>> --volfile-server=localhost --volf
>> ile-id=pcic-backup --client-pid=-1
>> /var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI}]
>> [2021-03-11 19:15:28.253485] I [glusterfsd.c:2424:daemonize]
>> 0-glusterfs: Pid of current running process is 157484
>> [2021-03-11 19:15:28.267911] I [MSGID: 101190]
>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>> Started thread with index [{index=0}]
>> [2021-03-11 19:15:28.267984] I [MSGID: 101190]
>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>> Started thread with index [{index=1}]
>> [2021-03-11 19:15:28.268371] I
>> [glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs:
>> Received list of available volfile servers: 10.0.231.82:24007
>> [2021-03-11 19:15:28.271729] I [MSGID: 101190]
>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>> Started thread with index [{index=2}]
>> [2021-03-11 19:15:28.271762] I [MSGID: 101190]
>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>> Started thread with index [{index=3}]
>> [2021-03-11 19:15:28.272223] I [MSGID: 114020]
>> [client.c:2315:notify] 0-pcic-backup-client-0: parent
>> translators are ready, attempting connect on transport []
>> [2021-03-11 19:15:28.275883] I [MSGID: 114020]
>> [client.c:2315:notify] 0-pcic-backup-client-1: parent
>> translators are ready, attempting connect on transport []
>> [2021-03-11 19:15:28.276154] I
>> [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-pcic-backup-client-0:
>> changing port to 49153 (from 0)
>> [2021-03-11 19:15:28.276193] I
>> [socket.c:849:__socket_shutdown] 0-pcic-backup-client-0:
>> intentional socket shutdown(13)
>> Final graph:
>> ...
>> +------------------------------------------------------------------------------+
>> [2021-03-11 19:15:28.282144] I
>> [socket.c:849:__socket_shutdown] 0-pcic-backup-client-1:
>> intentional socket shutdown(15)
>> [2021-03-11 19:15:28.286536] I [MSGID: 114057]
>> [client-handshake.c:1128:select_server_supported_programs]
>> 0-pcic-backup-client-0: Using Program
>> [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
>> [2021-03-11 19:15:28.287208] I [MSGID: 114046]
>> [client-handshake.c:857:client_setvolume_cbk]
>> 0-pcic-backup-client-0: Connected, attached to remote volume
>> [{conn-name=pcic-backup-client-0}, {remote_subvol=/data/brick}]
>> [2021-03-11 19:15:28.290162] I [MSGID: 114057]
>> [client-handshake.c:1128:select_server_supported_programs]
>> 0-pcic-backup-client-1: Using Program
>> [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
>> [2021-03-11 19:15:28.291122] I [MSGID: 114046]
>> [client-handshake.c:857:client_setvolume_cbk]
>> 0-pcic-backup-client-1: Connected, attached to remote volume
>> [{conn-name=pcic-backup-client-1}, {remote_subvol=/data/brick}]
>> [2021-03-11 19:15:28.292703] I [fuse-bridge.c:5300:fuse_init]
>> 0-glusterfs-fuse: FUSE inited with protocol versions:
>> glusterfs 7.24 kernel 7.23
>> [2021-03-11 19:15:28.292730] I
>> [fuse-bridge.c:5926:fuse_graph_sync] 0-fuse: switched to graph 0
>> [2021-03-11 19:15:32.809518] I
>> [fuse-bridge.c:6242:fuse_thread_proc] 0-fuse: initiating
>> unmount of /var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI
>> [2021-03-11 19:15:32.810216] W
>> [glusterfsd.c:1439:cleanup_and_exit]
>> (-->/lib64/libpthread.so.0(+0x7ea5) [0x7ff56b175ea5]
>> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5)
>> [0x55664e67db45]
>> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b)
>> [0x55664e67d9ab] ) 0-: received signum (15), shutting down
>> [2021-03-11 19:15:32.810253] I [fuse-bridge.c:7074:fini]
>> 0-fuse: Unmounting
>> '/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI'.
>> [2021-03-11 19:15:32.810268] I [fuse-bridge.c:7079:fini]
>> 0-fuse: Closing fuse connection to
>> '/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI'.
>>
>>
>> I'm really at a loss for where to go from here, it seems like
>> everything is set up correctly, and it has been working well
>> through the 7.x minor versions, but the jump to 8 has broken
>> something...
>>
>> There definitely are lots of changelogs on the servers that
>> fit into the timeframe..... I haven't made any writes to the
>> source volume.... do you think that's the problem? That it
>> needs some new changelog info to sync?
>> I had been holding off making any writes in case I needed to
>> go back to Gluster7.9 - not sure if that's really a good
>> option or not.
>>
>> [root at storage01 changelogs]# for dirs in {a,b,c}; do echo
>> "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh
>> /data/storage_$dirs/storage/.glusterfs/changelogs | head;
>> echo ""; done
>> /data/storage_a/storage/.glusterfs/changelogs
>> total 16G
>> drw-------. 3 root root 24 Mar 9 11:34 2021
>> -rw-r--r--. 1 root root 51 Mar 12 09:50 CHANGELOG
>> -rw-r--r--. 1 root root 13K Aug 13 2020 CHANGELOG.1597343197
>> -rw-r--r--. 1 root root 51K Aug 13 2020 CHANGELOG.1597343212
>> -rw-r--r--. 1 root root 86K Aug 13 2020 CHANGELOG.1597343227
>> -rw-r--r--. 1 root root 99K Aug 13 2020 CHANGELOG.1597343242
>> -rw-r--r--. 1 root root 69K Aug 13 2020 CHANGELOG.1597343257
>> -rw-r--r--. 1 root root 69K Aug 13 2020 CHANGELOG.1597343272
>> -rw-r--r--. 1 root root 72K Aug 13 2020 CHANGELOG.1597343287
>>
>> /data/storage_b/storage/.glusterfs/changelogs
>> total 3.3G
>> drw-------. 3 root root 24 Mar 9 11:34 2021
>> -rw-r--r--. 1 root root 51 Mar 12 09:50 CHANGELOG
>> -rw-r--r--. 1 root root 13K Aug 13 2020 CHANGELOG.1597343197
>> -rw-r--r--. 1 root root 53K Aug 13 2020 CHANGELOG.1597343212
>> -rw-r--r--. 1 root root 89K Aug 13 2020 CHANGELOG.1597343227
>> -rw-r--r--. 1 root root 89K Aug 13 2020 CHANGELOG.1597343242
>> -rw-r--r--. 1 root root 69K Aug 13 2020 CHANGELOG.1597343257
>> -rw-r--r--. 1 root root 71K Aug 13 2020 CHANGELOG.1597343272
>> -rw-r--r--. 1 root root 86K Aug 13 2020 CHANGELOG.1597343287
>>
>> /data/storage_c/storage/.glusterfs/changelogs
>> total 9.6G
>> drw-------. 3 root root 16 Mar 9 11:34 2021
>> -rw-r--r--. 1 root root 51 Mar 12 09:50 CHANGELOG
>> -rw-r--r--. 1 root root 16K Aug 13 2020 CHANGELOG.1597343199
>> -rw-r--r--. 1 root root 71K Aug 13 2020 CHANGELOG.1597343214
>> -rw-r--r--. 1 root root 122K Aug 13 2020 CHANGELOG.1597343229
>> -rw-r--r--. 1 root root 73K Aug 13 2020 CHANGELOG.1597343244
>> -rw-r--r--. 1 root root 100K Aug 13 2020 CHANGELOG.1597343259
>> -rw-r--r--. 1 root root 95K Aug 13 2020 CHANGELOG.1597343274
>> -rw-r--r--. 1 root root 92K Aug 13 2020 CHANGELOG.1597343289
>>
>> [root at storage01 changelogs]# for dirs in {a,b,c}; do echo
>> "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh
>> /data/storage_$dirs/storage/.glusterfs/changelogs | tail;
>> echo ""; done
>> /data/storage_a/storage/.glusterfs/changelogs
>> -rw-r--r--. 1 root root 92 Mar 1 21:33 CHANGELOG.1614663193
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663731
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663760
>> -rw-r--r--. 1 root root 511 Mar 1 21:47 CHANGELOG.1614664043
>> -rw-r--r--. 1 root root 536 Mar 1 21:48 CHANGELOG.1614664101
>> -rw-r--r--. 1 root root 2.8K Mar 1 21:48 CHANGELOG.1614664116
>> -rw-r--r--. 1 root root 92 Mar 1 22:20 CHANGELOG.1614666061
>> -rw-r--r--. 1 root root 92 Mar 1 22:29 CHANGELOG.1614666554
>> drw-------. 2 root root 10 May 7 2020 csnap
>> drw-------. 2 root root 38 Aug 13 2020 htime
>>
>> /data/storage_b/storage/.glusterfs/changelogs
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663731
>> -rw-r--r--. 1 root root 480 Mar 1 21:42 CHANGELOG.1614663745
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663760
>> -rw-r--r--. 1 root root 524 Mar 1 21:47 CHANGELOG.1614664043
>> -rw-r--r--. 1 root root 495 Mar 1 21:48 CHANGELOG.1614664100
>> -rw-r--r--. 1 root root 1.6K Mar 1 21:48 CHANGELOG.1614664114
>> -rw-r--r--. 1 root root 92 Mar 1 22:20 CHANGELOG.1614666060
>> -rw-r--r--. 1 root root 92 Mar 1 22:29 CHANGELOG.1614666553
>> drw-------. 2 root root 10 May 7 2020 csnap
>> drw-------. 2 root root 38 Aug 13 2020 htime
>>
>> /data/storage_c/storage/.glusterfs/changelogs
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663738
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663753
>> -rw-r--r--. 1 root root 395 Mar 1 21:47 CHANGELOG.1614664051
>> -rw-r--r--. 1 root root 316 Mar 1 21:48 CHANGELOG.1614664094
>> -rw-r--r--. 1 root root 1.2K Mar 1 21:48 CHANGELOG.1614664109
>> -rw-r--r--. 1 root root 174 Mar 1 21:48 CHANGELOG.1614664123
>> -rw-r--r--. 1 root root 92 Mar 1 22:20 CHANGELOG.1614666061
>> -rw-r--r--. 1 root root 92 Mar 1 22:29 CHANGELOG.1614666553
>> drw-------. 2 root root 6 May 7 2020 csnap
>> drw-------. 2 root root 30 Aug 13 2020 htime
>>
>> [root at storage02 ~]# for dirs in {a,b,c}; do echo
>> "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh
>> /data/storage_$dirs/storage/.glusterfs/changelogs | head;
>> echo ""; done
>> /data/storage_a/storage/.glusterfs/changelogs
>> total 9.6G
>> drw-------. 3 root root 24 Mar 9 11:34 2021
>> -rw-r--r--. 1 root root 51 Mar 12 09:50 CHANGELOG
>> -rw-r--r--. 1 root root 4.2K Aug 13 2020 CHANGELOG.1597343193
>> -rw-r--r--. 1 root root 32K Aug 13 2020 CHANGELOG.1597343208
>> -rw-r--r--. 1 root root 107K Aug 13 2020 CHANGELOG.1597343223
>> -rw-r--r--. 1 root root 120K Aug 13 2020 CHANGELOG.1597343238
>> -rw-r--r--. 1 root root 72K Aug 13 2020 CHANGELOG.1597343253
>> -rw-r--r--. 1 root root 111K Aug 13 2020 CHANGELOG.1597343268
>> -rw-r--r--. 1 root root 91K Aug 13 2020 CHANGELOG.1597343283
>>
>> /data/storage_b/storage/.glusterfs/changelogs
>> total 16G
>> drw-------. 3 root root 24 Mar 9 11:34 2021
>> -rw-r--r--. 1 root root 51 Mar 12 09:50 CHANGELOG
>> -rw-r--r--. 1 root root 3.9K Aug 13 2020 CHANGELOG.1597343193
>> -rw-r--r--. 1 root root 35K Aug 13 2020 CHANGELOG.1597343208
>> -rw-r--r--. 1 root root 85K Aug 13 2020 CHANGELOG.1597343223
>> -rw-r--r--. 1 root root 103K Aug 13 2020 CHANGELOG.1597343238
>> -rw-r--r--. 1 root root 70K Aug 13 2020 CHANGELOG.1597343253
>> -rw-r--r--. 1 root root 72K Aug 13 2020 CHANGELOG.1597343268
>> -rw-r--r--. 1 root root 73K Aug 13 2020 CHANGELOG.1597343283
>>
>> /data/storage_c/storage/.glusterfs/changelogs
>> total 3.3G
>> drw-------. 3 root root 16 Mar 9 11:34 2021
>> -rw-r--r--. 1 root root 51 Mar 12 09:51 CHANGELOG
>> -rw-r--r--. 1 root root 21K Aug 13 2020 CHANGELOG.1597343202
>> -rw-r--r--. 1 root root 75K Aug 13 2020 CHANGELOG.1597343217
>> -rw-r--r--. 1 root root 92K Aug 13 2020 CHANGELOG.1597343232
>> -rw-r--r--. 1 root root 77K Aug 13 2020 CHANGELOG.1597343247
>> -rw-r--r--. 1 root root 66K Aug 13 2020 CHANGELOG.1597343262
>> -rw-r--r--. 1 root root 84K Aug 13 2020 CHANGELOG.1597343277
>> -rw-r--r--. 1 root root 81K Aug 13 2020 CHANGELOG.1597343292
>>
>> [root at storage02 ~]# for dirs in {a,b,c}; do echo
>> "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh
>> /data/storage_$dirs/storage/.glusterfs/changelogs | tail;
>> echo ""; done
>> /data/storage_a/storage/.glusterfs/changelogs
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663734
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663749
>> -rw-r--r--. 1 root root 395 Mar 1 21:47 CHANGELOG.1614664052
>> -rw-r--r--. 1 root root 316 Mar 1 21:48 CHANGELOG.1614664096
>> -rw-r--r--. 1 root root 1.2K Mar 1 21:48 CHANGELOG.1614664111
>> -rw-r--r--. 1 root root 174 Mar 1 21:48 CHANGELOG.1614664126
>> -rw-r--r--. 1 root root 92 Mar 1 22:20 CHANGELOG.1614666056
>> -rw-r--r--. 1 root root 92 Mar 1 22:29 CHANGELOG.1614666560
>> drw-------. 2 root root 10 May 7 2020 csnap
>> drw-------. 2 root root 38 Aug 13 2020 htime
>>
>> /data/storage_b/storage/.glusterfs/changelogs
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663735
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663749
>> -rw-r--r--. 1 root root 511 Mar 1 21:47 CHANGELOG.1614664052
>> -rw-r--r--. 1 root root 316 Mar 1 21:48 CHANGELOG.1614664096
>> -rw-r--r--. 1 root root 1.8K Mar 1 21:48 CHANGELOG.1614664111
>> -rw-r--r--. 1 root root 1.4K Mar 1 21:48 CHANGELOG.1614664126
>> -rw-r--r--. 1 root root 92 Mar 1 22:20 CHANGELOG.1614666060
>> -rw-r--r--. 1 root root 92 Mar 1 22:29 CHANGELOG.1614666556
>> drw-------. 2 root root 10 May 7 2020 csnap
>> drw-------. 2 root root 38 Aug 13 2020 htime
>>
>> /data/storage_c/storage/.glusterfs/changelogs
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663738
>> -rw-r--r--. 1 root root 521 Mar 1 21:42 CHANGELOG.1614663752
>> -rw-r--r--. 1 root root 524 Mar 1 21:47 CHANGELOG.1614664042
>> -rw-r--r--. 1 root root 92 Mar 1 21:47 CHANGELOG.1614664057
>> -rw-r--r--. 1 root root 536 Mar 1 21:48 CHANGELOG.1614664102
>> -rw-r--r--. 1 root root 1.6K Mar 1 21:48 CHANGELOG.1614664117
>> -rw-r--r--. 1 root root 92 Mar 1 22:20 CHANGELOG.1614666057
>> -rw-r--r--. 1 root root 92 Mar 1 22:29 CHANGELOG.1614666550
>> drw-------. 2 root root 6 May 7 2020 csnap
>> drw-------. 2 root root 30 Aug 13 2020 htime
>>
>>
>> [root at storage03 ~]# for dirs in {a,b,c}; do echo
>> "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh
>> /data/storage_$dirs/storage/.glusterfs/changelogs | head;
>> echo ""; done
>> /data/storage_a/storage/.glusterfs/changelogs
>> total 3.4G
>> drw-------. 3 root root 24 Mar 9 11:34 2021
>> -rw-r--r--. 1 root root 51 Mar 12 09:50 CHANGELOG
>> -rw-r--r--. 1 root root 19K Aug 13 2020 CHANGELOG.1597343201
>> -rw-r--r--. 1 root root 66K Aug 13 2020 CHANGELOG.1597343215
>> -rw-r--r--. 1 root root 91K Aug 13 2020 CHANGELOG.1597343230
>> -rw-r--r--. 1 root root 82K Aug 13 2020 CHANGELOG.1597343245
>> -rw-r--r--. 1 root root 64K Aug 13 2020 CHANGELOG.1597343259
>> -rw-r--r--. 1 root root 75K Aug 13 2020 CHANGELOG.1597343274
>> -rw-r--r--. 1 root root 81K Aug 13 2020 CHANGELOG.1597343289
>>
>> /data/storage_b/storage/.glusterfs/changelogs
>> total 9.6G
>> drw-------. 3 root root 24 Mar 9 11:34 2021
>> -rw-r--r--. 1 root root 51 Mar 12 09:51 CHANGELOG
>> -rw-r--r--. 1 root root 19K Aug 13 2020 CHANGELOG.1597343201
>> -rw-r--r--. 1 root root 80K Aug 13 2020 CHANGELOG.1597343215
>> -rw-r--r--. 1 root root 119K Aug 13 2020 CHANGELOG.1597343230
>> -rw-r--r--. 1 root root 65K Aug 13 2020 CHANGELOG.1597343244
>> -rw-r--r--. 1 root root 100K Aug 13 2020 CHANGELOG.1597343259
>> -rw-r--r--. 1 root root 95K Aug 13 2020 CHANGELOG.1597343274
>> -rw-r--r--. 1 root root 92K Aug 13 2020 CHANGELOG.1597343289
>>
>> /data/storage_c/storage/.glusterfs/changelogs
>> total 16G
>> drw-------. 3 root root 16 Mar 9 11:34 2021
>> -rw-r--r--. 1 root root 51 Mar 12 09:51 CHANGELOG
>> -rw-r--r--. 1 root root 3.9K Aug 13 2020 CHANGELOG.1597343193
>> -rw-r--r--. 1 root root 35K Aug 13 2020 CHANGELOG.1597343208
>> -rw-r--r--. 1 root root 85K Aug 13 2020 CHANGELOG.1597343223
>> -rw-r--r--. 1 root root 103K Aug 13 2020 CHANGELOG.1597343238
>> -rw-r--r--. 1 root root 70K Aug 13 2020 CHANGELOG.1597343253
>> -rw-r--r--. 1 root root 71K Aug 13 2020 CHANGELOG.1597343268
>> -rw-r--r--. 1 root root 73K Aug 13 2020 CHANGELOG.1597343283
>>
>> [root at storage03 ~]# for dirs in {a,b,c}; do echo
>> "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh
>> /data/storage_$dirs/storage/.glusterfs/changelogs | tail;
>> echo ""; done
>> /data/storage_a/storage/.glusterfs/changelogs
>> -rw-r--r--. 1 root root 92 Mar 1 21:33 CHANGELOG.1614663183
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663740
>> -rw-r--r--. 1 root root 521 Mar 1 21:42 CHANGELOG.1614663755
>> -rw-r--r--. 1 root root 524 Mar 1 21:47 CHANGELOG.1614664049
>> -rw-r--r--. 1 root root 1.9K Mar 1 21:48 CHANGELOG.1614664106
>> -rw-r--r--. 1 root root 174 Mar 1 21:48 CHANGELOG.1614664121
>> -rw-r--r--. 1 root root 92 Mar 1 22:20 CHANGELOG.1614666051
>> -rw-r--r--. 1 root root 92 Mar 1 22:29 CHANGELOG.1614666559
>> drw-------. 2 root root 10 May 7 2020 csnap
>> drw-------. 2 root root 38 Aug 13 2020 htime
>>
>> /data/storage_b/storage/.glusterfs/changelogs
>> -rw-r--r--. 1 root root 474 Mar 1 21:33 CHANGELOG.1614663182
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663739
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663753
>> -rw-r--r--. 1 root root 395 Mar 1 21:47 CHANGELOG.1614664049
>> -rw-r--r--. 1 root root 1.4K Mar 1 21:48 CHANGELOG.1614664106
>> -rw-r--r--. 1 root root 174 Mar 1 21:48 CHANGELOG.1614664120
>> -rw-r--r--. 1 root root 92 Mar 1 22:20 CHANGELOG.1614666063
>> -rw-r--r--. 1 root root 92 Mar 1 22:29 CHANGELOG.1614666557
>> drw-------. 2 root root 10 May 7 2020 csnap
>> drw-------. 2 root root 38 Aug 13 2020 htime
>>
>> /data/storage_c/storage/.glusterfs/changelogs
>> -rw-r--r--. 1 root root 468 Mar 1 21:33 CHANGELOG.1614663183
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663740
>> -rw-r--r--. 1 root root 92 Mar 1 21:42 CHANGELOG.1614663754
>> -rw-r--r--. 1 root root 511 Mar 1 21:47 CHANGELOG.1614664048
>> -rw-r--r--. 1 root root 2.0K Mar 1 21:48 CHANGELOG.1614664105
>> -rw-r--r--. 1 root root 1.4K Mar 1 21:48 CHANGELOG.1614664120
>> -rw-r--r--. 1 root root 92 Mar 1 22:20 CHANGELOG.1614666063
>> -rw-r--r--. 1 root root 92 Mar 1 22:29 CHANGELOG.1614666556
>> drw-------. 2 root root 6 May 7 2020 csnap
>> drw-------. 2 root root 30 Aug 13 2020 htime
>>
>> Thanks,
>> -Matthew
>>
>> --
>> Matthew Benstead
>> System Administrator
>> Pacific Climate Impacts Consortium <https://pacificclimate.org/>
>> University of Victoria, UH1
>> PO Box 1800, STN CSC
>> Victoria, BC, V8W 2Y2
>> Phone: +1-250-721-8432
>> Email: matthewb at uvic.ca <mailto:matthewb at uvic.ca>
>>
>> On 3/11/21 11:37 PM, Strahil Nikolov wrote:
>>> Notice: This message was sent from outside the University of
>>> Victoria email system. Please be cautious with links and
>>> sensitive information.
>>>
>>> Have you checked the secondary volume nodes' logs & SELINUX
>>> status ?
>>>
>>> Best Regards,
>>> Strahil Nikolov
>>>
>>> On Thu, Mar 11, 2021 at 21:36, Matthew Benstead
>>> <matthewb at uvic.ca> <mailto:matthewb at uvic.ca> wrote:
>>> Hi Strahil,
>>>
>>> It looks like perhaps the changelog_log_level and
>>> log_level options? I've set them to debug:
>>>
>>> [root at storage01 ~]# gluster volume geo-replication
>>> storage geoaccount at 10.0.231.81::pcic-backup
>>> <mailto:geoaccount at 10.0.231.81::pcic-backup> config |
>>> egrep -i "log_level"
>>> changelog_log_level:INFO
>>> cli_log_level:INFO
>>> gluster_log_level:INFO
>>> log_level:INFO
>>> slave_gluster_log_level:INFO
>>> slave_log_level:INFO
>>>
>>> [root at storage01 ~]# gluster volume geo-replication
>>> storage geoaccount at 10.0.231.81::pcic-backup
>>> <mailto:geoaccount at 10.0.231.81::pcic-backup> config
>>> changelog_log_level DEBUG
>>> geo-replication config updated successfully
>>>
>>> [root at storage01 ~]# gluster volume geo-replication
>>> storage geoaccount at 10.0.231.81::pcic-backup
>>> <mailto:geoaccount at 10.0.231.81::pcic-backup> config
>>> log_level DEBUG
>>> geo-replication config updated successfully
>>>
>>>
>>> Then I restarted geo-replication:
>>>
>>> [root at storage01 ~]# gluster volume geo-replication
>>> storage geoaccount at 10.0.231.81::pcic-backup
>>> <mailto:geoaccount at 10.0.231.81::pcic-backup> start
>>> Starting geo-replication session between storage &
>>> geoaccount at 10.0.231.81::pcic-backup
>>> <mailto:geoaccount at 10.0.231.81::pcic-backup> has been
>>> successful
>>> [root at storage01 ~]# gluster volume geo-replication status
>>>
>>> MASTER NODE MASTER VOL MASTER BRICK
>>> SLAVE USER
>>> SLAVE SLAVE
>>> NODE STATUS CRAWL STATUS
>>> LAST_SYNCED
>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>> 10.0.231.91 storage /data/storage_a/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Initializing... N/A
>>> N/A
>>> 10.0.231.91 storage /data/storage_c/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Initializing... N/A
>>> N/A
>>> 10.0.231.91 storage /data/storage_b/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Initializing... N/A
>>> N/A
>>> 10.0.231.92 storage /data/storage_b/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Initializing... N/A
>>> N/A
>>> 10.0.231.92 storage /data/storage_a/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Initializing... N/A
>>> N/A
>>> 10.0.231.92 storage /data/storage_c/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Initializing... N/A
>>> N/A
>>> 10.0.231.93 storage /data/storage_c/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Initializing... N/A
>>> N/A
>>> 10.0.231.93 storage /data/storage_b/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Initializing... N/A
>>> N/A
>>> 10.0.231.93 storage /data/storage_a/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Initializing... N/A
>>> N/A
>>> [root at storage01 ~]# gluster volume geo-replication status
>>>
>>> MASTER NODE MASTER VOL MASTER BRICK
>>> SLAVE USER
>>> SLAVE SLAVE
>>> NODE STATUS CRAWL STATUS LAST_SYNCED
>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>> 10.0.231.91 storage /data/storage_a/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Faulty N/A
>>> N/A
>>> 10.0.231.91 storage /data/storage_c/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Faulty N/A
>>> N/A
>>> 10.0.231.91 storage /data/storage_b/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Faulty N/A
>>> N/A
>>> 10.0.231.92 storage /data/storage_b/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Faulty N/A
>>> N/A
>>> 10.0.231.92 storage /data/storage_a/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Faulty N/A
>>> N/A
>>> 10.0.231.92 storage /data/storage_c/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Faulty N/A
>>> N/A
>>> 10.0.231.93 storage /data/storage_c/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Faulty N/A
>>> N/A
>>> 10.0.231.93 storage /data/storage_b/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Faulty N/A
>>> N/A
>>> 10.0.231.93 storage /data/storage_a/storage
>>> geoaccount ssh://geoaccount@10.0.231.81::pcic-backup
>>> <mailto:ssh://geoaccount@10.0.231.81::pcic-backup>
>>> N/A Faulty N/A N/A
>>>
>>> [root at storage01 ~]# gluster volume geo-replication
>>> storage geoaccount at 10.0.231.81::pcic-backup
>>> <mailto:geoaccount at 10.0.231.81::pcic-backup> stop
>>> Stopping geo-replication session between storage &
>>> geoaccount at 10.0.231.81::pcic-backup
>>> <mailto:geoaccount at 10.0.231.81::pcic-backup> has been
>>> successful
>>>
>>>
>>> The changelogs didn't really show anything new around
>>> changelog selection:
>>>
>>> [root at storage01 storage_10.0.231.81_pcic-backup]# cat
>>> changes-data-storage_a-storage.log | egrep "2021-03-11"
>>> [2021-03-11 19:15:30.552889] I [MSGID: 132028]
>>> [gf-changelog.c:577:gf_changelog_register_generic]
>>> 0-gfchangelog: Registering brick
>>> [{brick=/data/storage_a/storage}, {notify_filter=1}]
>>> [2021-03-11 19:15:30.552893] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=0}]
>>> [2021-03-11 19:15:30.552894] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=1}]
>>> [2021-03-11 19:15:30.553633] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=3}]
>>> [2021-03-11 19:15:30.553634] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=2}]
>>> [2021-03-11 19:15:30.554236] D
>>> [rpcsvc.c:2831:rpcsvc_init] 0-rpc-service: RPC service
>>> inited.
>>> [2021-03-11 19:15:30.554403] D
>>> [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service:
>>> New program registered: GF-DUMP, Num: 123451501, Ver: 1,
>>> Port: 0
>>> [2021-03-11 19:15:30.554420] D
>>> [rpc-transport.c:278:rpc_transport_load]
>>> 0-rpc-transport: attempt to load file
>>> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
>>> [2021-03-11 19:15:30.554933] D
>>> [socket.c:4485:socket_init] 0-socket.gfchangelog:
>>> disabling nodelay
>>> [2021-03-11 19:15:30.554944] D
>>> [socket.c:4523:socket_init] 0-socket.gfchangelog:
>>> Configured transport.tcp-user-timeout=42
>>> [2021-03-11 19:15:30.554949] D
>>> [socket.c:4543:socket_init] 0-socket.gfchangelog:
>>> Reconfigured transport.keepalivecnt=9
>>> [2021-03-11 19:15:30.555002] I
>>> [socket.c:929:__socket_server_bind]
>>> 0-socket.gfchangelog: closing (AF_UNIX) reuse check
>>> socket 23
>>> [2021-03-11 19:15:30.555324] D
>>> [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service:
>>> New program registered: LIBGFCHANGELOG REBORP, Num:
>>> 1886350951, Ver: 1, Port: 0
>>> [2021-03-11 19:15:30.555345] D
>>> [rpc-clnt.c:1020:rpc_clnt_connection_init]
>>> 0-gfchangelog: defaulting frame-timeout to 30mins
>>> [2021-03-11 19:15:30.555351] D
>>> [rpc-clnt.c:1032:rpc_clnt_connection_init]
>>> 0-gfchangelog: disable ping-timeout
>>> [2021-03-11 19:15:30.555358] D
>>> [rpc-transport.c:278:rpc_transport_load]
>>> 0-rpc-transport: attempt to load file
>>> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
>>> [2021-03-11 19:15:30.555399] D
>>> [socket.c:4485:socket_init] 0-gfchangelog: disabling nodelay
>>> [2021-03-11 19:15:30.555406] D
>>> [socket.c:4523:socket_init] 0-gfchangelog: Configured
>>> transport.tcp-user-timeout=42
>>> [2021-03-11 19:15:32.555711] D
>>> [rpc-clnt-ping.c:298:rpc_clnt_start_ping] 0-gfchangelog:
>>> ping timeout is 0, returning
>>> [2021-03-11 19:15:32.572157] I [MSGID: 132035]
>>> [gf-history-changelog.c:837:gf_history_changelog]
>>> 0-gfchangelog: Requesting historical changelogs
>>> [{start=1614666553}, {end=1615490132}]
>>> [2021-03-11 19:15:32.572436] I [MSGID: 132019]
>>> [gf-history-changelog.c:755:gf_changelog_extract_min_max]
>>> 0-gfchangelog: changelogs min max [{min=1597342860},
>>> {max=1615490121}, {total_changelogs=1256897}]
>>> [2021-03-11 19:15:32.621244] E [MSGID: 132009]
>>> [gf-history-changelog.c:941:gf_history_changelog]
>>> 0-gfchangelog: wrong result [{for=end},
>>> {start=1615490121}, {idx=1256896}]
>>> [2021-03-11 19:15:46.733182] I [MSGID: 132028]
>>> [gf-changelog.c:577:gf_changelog_register_generic]
>>> 0-gfchangelog: Registering brick
>>> [{brick=/data/storage_a/storage}, {notify_filter=1}]
>>> [2021-03-11 19:15:46.733316] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=0}]
>>> [2021-03-11 19:15:46.733348] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=1}]
>>> [2021-03-11 19:15:46.734031] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=2}]
>>> [2021-03-11 19:15:46.734085] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=3}]
>>> [2021-03-11 19:15:46.734591] D
>>> [rpcsvc.c:2831:rpcsvc_init] 0-rpc-service: RPC service
>>> inited.
>>> [2021-03-11 19:15:46.734755] D
>>> [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service:
>>> New program registered: GF-DUMP, Num: 123451501, Ver: 1,
>>> Port: 0
>>> [2021-03-11 19:15:46.734772] D
>>> [rpc-transport.c:278:rpc_transport_load]
>>> 0-rpc-transport: attempt to load file
>>> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
>>> [2021-03-11 19:15:46.735256] D
>>> [socket.c:4485:socket_init] 0-socket.gfchangelog:
>>> disabling nodelay
>>> [2021-03-11 19:15:46.735266] D
>>> [socket.c:4523:socket_init] 0-socket.gfchangelog:
>>> Configured transport.tcp-user-timeout=42
>>> [2021-03-11 19:15:46.735271] D
>>> [socket.c:4543:socket_init] 0-socket.gfchangelog:
>>> Reconfigured transport.keepalivecnt=9
>>> [2021-03-11 19:15:46.735325] I
>>> [socket.c:929:__socket_server_bind]
>>> 0-socket.gfchangelog: closing (AF_UNIX) reuse check
>>> socket 21
>>> [2021-03-11 19:15:46.735704] D
>>> [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service:
>>> New program registered: LIBGFCHANGELOG REBORP, Num:
>>> 1886350951, Ver: 1, Port: 0
>>> [2021-03-11 19:15:46.735721] D
>>> [rpc-clnt.c:1020:rpc_clnt_connection_init]
>>> 0-gfchangelog: defaulting frame-timeout to 30mins
>>> [2021-03-11 19:15:46.735726] D
>>> [rpc-clnt.c:1032:rpc_clnt_connection_init]
>>> 0-gfchangelog: disable ping-timeout
>>> [2021-03-11 19:15:46.735733] D
>>> [rpc-transport.c:278:rpc_transport_load]
>>> 0-rpc-transport: attempt to load file
>>> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
>>> [2021-03-11 19:15:46.735771] D
>>> [socket.c:4485:socket_init] 0-gfchangelog: disabling nodelay
>>> [2021-03-11 19:15:46.735778] D
>>> [socket.c:4523:socket_init] 0-gfchangelog: Configured
>>> transport.tcp-user-timeout=42
>>> [2021-03-11 19:15:47.618464] D
>>> [rpc-clnt-ping.c:298:rpc_clnt_start_ping] 0-gfchangelog:
>>> ping timeout is 0, returning
>>>
>>>
>>> [root at storage01 storage_10.0.231.81_pcic-backup]# cat
>>> changes-data-storage_b-storage.log | egrep "2021-03-11"
>>> [2021-03-11 19:15:30.611457] I [MSGID: 132028]
>>> [gf-changelog.c:577:gf_changelog_register_generic]
>>> 0-gfchangelog: Registering brick
>>> [{brick=/data/storage_b/storage}, {notify_filter=1}]
>>> [2021-03-11 19:15:30.611574] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=1}]
>>> [2021-03-11 19:15:30.611641] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=3}]
>>> [2021-03-11 19:15:30.611645] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=2}]
>>> [2021-03-11 19:15:30.612325] D
>>> [rpcsvc.c:2831:rpcsvc_init] 0-rpc-service: RPC service
>>> inited.
>>> [2021-03-11 19:15:30.612488] D
>>> [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service:
>>> New program registered: GF-DUMP, Num: 123451501, Ver: 1,
>>> Port: 0
>>> [2021-03-11 19:15:30.612507] D
>>> [rpc-transport.c:278:rpc_transport_load]
>>> 0-rpc-transport: attempt to load file
>>> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
>>> [2021-03-11 19:15:30.613005] D
>>> [socket.c:4485:socket_init] 0-socket.gfchangelog:
>>> disabling nodelay
>>> [2021-03-11 19:15:30.613130] D
>>> [socket.c:4523:socket_init] 0-socket.gfchangelog:
>>> Configured transport.tcp-user-timeout=42
>>> [2021-03-11 19:15:30.613142] D
>>> [socket.c:4543:socket_init] 0-socket.gfchangelog:
>>> Reconfigured transport.keepalivecnt=9
>>> [2021-03-11 19:15:30.613208] I
>>> [socket.c:929:__socket_server_bind]
>>> 0-socket.gfchangelog: closing (AF_UNIX) reuse check
>>> socket 22
>>> [2021-03-11 19:15:30.613545] D
>>> [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service:
>>> New program registered: LIBGFCHANGELOG REBORP, Num:
>>> 1886350951, Ver: 1, Port: 0
>>> [2021-03-11 19:15:30.613567] D
>>> [rpc-clnt.c:1020:rpc_clnt_connection_init]
>>> 0-gfchangelog: defaulting frame-timeout to 30mins
>>> [2021-03-11 19:15:30.613574] D
>>> [rpc-clnt.c:1032:rpc_clnt_connection_init]
>>> 0-gfchangelog: disable ping-timeout
>>> [2021-03-11 19:15:30.613582] D
>>> [rpc-transport.c:278:rpc_transport_load]
>>> 0-rpc-transport: attempt to load file
>>> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
>>> [2021-03-11 19:15:30.613637] D
>>> [socket.c:4485:socket_init] 0-gfchangelog: disabling nodelay
>>> [2021-03-11 19:15:30.613654] D
>>> [socket.c:4523:socket_init] 0-gfchangelog: Configured
>>> transport.tcp-user-timeout=42
>>> [2021-03-11 19:15:32.614273] D
>>> [rpc-clnt-ping.c:298:rpc_clnt_start_ping] 0-gfchangelog:
>>> ping timeout is 0, returning
>>> [2021-03-11 19:15:32.643628] I [MSGID: 132035]
>>> [gf-history-changelog.c:837:gf_history_changelog]
>>> 0-gfchangelog: Requesting historical changelogs
>>> [{start=1614666552}, {end=1615490132}]
>>> [2021-03-11 19:15:32.643716] I [MSGID: 132019]
>>> [gf-history-changelog.c:755:gf_changelog_extract_min_max]
>>> 0-gfchangelog: changelogs min max [{min=1597342860},
>>> {max=1615490123}, {total_changelogs=1264296}]
>>> [2021-03-11 19:15:32.700397] E [MSGID: 132009]
>>> [gf-history-changelog.c:941:gf_history_changelog]
>>> 0-gfchangelog: wrong result [{for=end},
>>> {start=1615490123}, {idx=1264295}]
>>> [2021-03-11 19:15:46.832322] I [MSGID: 132028]
>>> [gf-changelog.c:577:gf_changelog_register_generic]
>>> 0-gfchangelog: Registering brick
>>> [{brick=/data/storage_b/storage}, {notify_filter=1}]
>>> [2021-03-11 19:15:46.832394] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=0}]
>>> [2021-03-11 19:15:46.832465] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=1}]
>>> [2021-03-11 19:15:46.832531] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=2}]
>>> [2021-03-11 19:15:46.833086] I [MSGID: 101190]
>>> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll:
>>> Started thread with index [{index=3}]
>>> [2021-03-11 19:15:46.833648] D
>>> [rpcsvc.c:2831:rpcsvc_init] 0-rpc-service: RPC service
>>> inited.
>>> [2021-03-11 19:15:46.833817] D
>>> [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service:
>>> New program registered: GF-DUMP, Num: 123451501, Ver: 1,
>>> Port: 0
>>> [2021-03-11 19:15:46.833835] D
>>> [rpc-transport.c:278:rpc_transport_load]
>>> 0-rpc-transport: attempt to load file
>>> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
>>> [2021-03-11 19:15:46.834368] D
>>> [socket.c:4485:socket_init] 0-socket.gfchangelog:
>>> disabling nodelay
>>> [2021-03-11 19:15:46.834380] D
>>> [socket.c:4523:socket_init] 0-socket.gfchangelog:
>>> Configured transport.tcp-user-timeout=42
>>> [2021-03-11 19:15:46.834386] D
>>> [socket.c:4543:socket_init] 0-socket.gfchangelog:
>>> Reconfigured transport.keepalivecnt=9
>>> [2021-03-11 19:15:46.834441] I
>>> [socket.c:929:__socket_server_bind]
>>> 0-socket.gfchangelog: closing (AF_UNIX) reuse check
>>> socket 23
>>> [2021-03-11 19:15:46.834768] D
>>> [rpcsvc.c:2342:rpcsvc_program_register] 0-rpc-service:
>>> New program registered: LIBGFCHANGELOG REBORP, Num:
>>> 1886350951, Ver: 1, Port: 0
>>> [2021-03-11 19:15:46.834789] D
>>> [rpc-clnt.c:1020:rpc_clnt_connection_init]
>>> 0-gfchangelog: defaulting frame-timeout to 30mins
>>> [2021-03-11 19:15:46.834795] D
>>> [rpc-clnt.c:1032:rpc_clnt_connection_init]
>>> 0-gfchangelog: disable ping-timeout
>>> [2021-03-11 19:15:46.834802] D
>>> [rpc-transport.c:278:rpc_transport_load]
>>> 0-rpc-transport: attempt to load file
>>> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
>>> [2021-03-11 19:15:46.834845] D
>>> [socket.c:4485:socket_init] 0-gfchangelog: disabling nodelay
>>> [2021-03-11 19:15:46.834853] D
>>> [socket.c:4523:socket_init] 0-gfchangelog: Configured
>>> transport.tcp-user-timeout=42
>>> [2021-03-11 19:15:47.618476] D
>>> [rpc-clnt-ping.c:298:rpc_clnt_start_ping] 0-gfchangelog:
>>> ping timeout is 0, returning
>>>
>>>
>>> gsyncd logged a lot but I'm not sure if it's helpful:
>>>
>>> [2021-03-11 19:15:00.41898] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:21.551302] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:21.631470] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:21.718386] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:21.804991] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:26.203999] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:26.284775] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:26.573355] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:26.653752] D
>>> [gsyncd(monitor):303:main] <top>: Using session config
>>> file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:26.756994] D
>>> [monitor(monitor):304:distribute] <top>: master bricks:
>>> [{'host': '10.0.231.91', 'uuid':
>>> 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir':
>>> '/data/storage_a/storage'}, {'host': '10.0.2
>>> 31.92', 'uuid': 'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d',
>>> 'dir': '/data/storage_b/storage'}, {'host':
>>> '10.0.231.93', 'uuid':
>>> '8b28b331-3780-46bc-9da3-fb27de4ab57b', 'dir':
>>> '/data/storage_c/storage'}, {'host': '10.
>>> 0.231.92', 'uuid':
>>> 'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d', 'dir':
>>> '/data/storage_a/storage'}, {'host': '10.0.231.93',
>>> 'uuid': '8b28b331-3780-46bc-9da3-fb27de4ab57b', 'dir':
>>> '/data/storage_b/storage'}, {'host': '
>>> 10.0.231.91', 'uuid':
>>> 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir':
>>> '/data/storage_c/storage'}, {'host': '10.0.231.93',
>>> 'uuid': '8b28b331-3780-46bc-9da3-fb27de4ab57b', 'dir':
>>> '/data/storage_a/storage'}, {'host'
>>> : '10.0.231.91', 'uuid':
>>> 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir':
>>> '/data/storage_b/storage'}, {'host': '10.0.231.92',
>>> 'uuid': 'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d', 'dir':
>>> '/data/storage_c/storage'}]
>>> [2021-03-11 19:15:26.757252] D
>>> [monitor(monitor):314:distribute] <top>: slave SSH
>>> gateway: geoaccount at 10.0.231.81
>>> <mailto:geoaccount at 10.0.231.81>
>>> [2021-03-11 19:15:27.416235] D
>>> [monitor(monitor):334:distribute] <top>: slave bricks:
>>> [{'host': '10.0.231.81', 'uuid':
>>> 'b88dea4f-31ec-416a-9110-3ccdc3910acd', 'dir':
>>> '/data/brick'}, {'host': '10.0.231.82', 'uuid
>>> ': 'be50a8de-3934-4fee-a80d-8e2e99017902', 'dir':
>>> '/data/brick'}]
>>> [2021-03-11 19:15:27.416825] D
>>> [syncdutils(monitor):932:is_hot] Volinfo: brickpath:
>>> '10.0.231.91:/data/storage_a/storage'
>>> [2021-03-11 19:15:27.417273] D
>>> [syncdutils(monitor):932:is_hot] Volinfo: brickpath:
>>> '10.0.231.91:/data/storage_c/storage'
>>> [2021-03-11 19:15:27.417515] D
>>> [syncdutils(monitor):932:is_hot] Volinfo: brickpath:
>>> '10.0.231.91:/data/storage_b/storage'
>>> [2021-03-11 19:15:27.417763] D
>>> [monitor(monitor):348:distribute] <top>: worker specs:
>>> [({'host': '10.0.231.91', 'uuid':
>>> 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir':
>>> '/data/storage_a/storage'}, ('geoaccount at 10.
>>> 0.231.81', 'b88dea4f-31ec-416a-9110-3ccdc3910acd'), '1',
>>> False), ({'host': '10.0.231.91', 'uuid':
>>> 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir':
>>> '/data/storage_c/storage'}, ('geoaccount at 10.0.231.82
>>> <mailto:geoaccount at 10.0.231.82>', 'be50a8de-3
>>> 934-4fee-a80d-8e2e99017902'), '2', False), ({'host':
>>> '10.0.231.91', 'uuid':
>>> 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir':
>>> '/data/storage_b/storage'}, ('geoaccount at 10.0.231.82
>>> <mailto:geoaccount at 10.0.231.82>',
>>> 'be50a8de-3934-4fee-a80d-8e2e9901
>>> 7902'), '3', False)]
>>> [2021-03-11 19:15:27.425009] I
>>> [monitor(monitor):160:monitor] Monitor: starting gsyncd
>>> worker [{brick=/data/storage_c/storage},
>>> {slave_node=10.0.231.82}]
>>> [2021-03-11 19:15:27.426764] I
>>> [monitor(monitor):160:monitor] Monitor: starting gsyncd
>>> worker [{brick=/data/storage_b/storage},
>>> {slave_node=10.0.231.82}]
>>> [2021-03-11 19:15:27.429208] I
>>> [monitor(monitor):160:monitor] Monitor: starting gsyncd
>>> worker [{brick=/data/storage_a/storage},
>>> {slave_node=10.0.231.81}]
>>> [2021-03-11 19:15:27.432280] D
>>> [monitor(monitor):195:monitor] Monitor: Worker would
>>> mount volume privately
>>> [2021-03-11 19:15:27.434195] D
>>> [monitor(monitor):195:monitor] Monitor: Worker would
>>> mount volume privately
>>> [2021-03-11 19:15:27.436584] D
>>> [monitor(monitor):195:monitor] Monitor: Worker would
>>> mount volume privately
>>> [2021-03-11 19:15:27.478806] D [gsyncd(worker
>>> /data/storage_c/storage):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:27.478852] D [gsyncd(worker
>>> /data/storage_b/storage):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:27.480104] D [gsyncd(worker
>>> /data/storage_a/storage):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:27.500456] I [resource(worker
>>> /data/storage_c/storage):1387:connect_remote] SSH:
>>> Initializing SSH connection between master and slave...
>>> [2021-03-11 19:15:27.501375] I [resource(worker
>>> /data/storage_b/storage):1387:connect_remote] SSH:
>>> Initializing SSH connection between master and slave...
>>> [2021-03-11 19:15:27.502003] I [resource(worker
>>> /data/storage_a/storage):1387:connect_remote] SSH:
>>> Initializing SSH connection between master and slave...
>>> [2021-03-11 19:15:27.525511] D [repce(worker
>>> /data/storage_a/storage):195:push] RepceClient: call
>>> 192117:140572692309824:1615490127.53 __repce_version__() ...
>>> [2021-03-11 19:15:27.525582] D [repce(worker
>>> /data/storage_b/storage):195:push] RepceClient: call
>>> 192115:139891296405312:1615490127.53 __repce_version__() ...
>>> [2021-03-11 19:15:27.526089] D [repce(worker
>>> /data/storage_c/storage):195:push] RepceClient: call
>>> 192114:140388828780352:1615490127.53 __repce_version__() ...
>>> [2021-03-11 19:15:29.435985] D [repce(worker
>>> /data/storage_a/storage):215:__call__] RepceClient: call
>>> 192117:140572692309824:1615490127.53 __repce_version__
>>> -> 1.0
>>> [2021-03-11 19:15:29.436213] D [repce(worker
>>> /data/storage_a/storage):195:push] RepceClient: call
>>> 192117:140572692309824:1615490129.44 version() ...
>>> [2021-03-11 19:15:29.437136] D [repce(worker
>>> /data/storage_a/storage):215:__call__] RepceClient: call
>>> 192117:140572692309824:1615490129.44 version -> 1.0
>>> [2021-03-11 19:15:29.437268] D [repce(worker
>>> /data/storage_a/storage):195:push] RepceClient: call
>>> 192117:140572692309824:1615490129.44 pid() ...
>>> [2021-03-11 19:15:29.437915] D [repce(worker
>>> /data/storage_a/storage):215:__call__] RepceClient: call
>>> 192117:140572692309824:1615490129.44 pid -> 157321
>>> [2021-03-11 19:15:29.438004] I [resource(worker
>>> /data/storage_a/storage):1436:connect_remote] SSH: SSH
>>> connection between master and slave established.
>>> [{duration=1.9359}]
>>> [2021-03-11 19:15:29.438072] I [resource(worker
>>> /data/storage_a/storage):1116:connect] GLUSTER: Mounting
>>> gluster volume locally...
>>> [2021-03-11 19:15:29.494538] D [repce(worker
>>> /data/storage_b/storage):215:__call__] RepceClient: call
>>> 192115:139891296405312:1615490127.53 __repce_version__
>>> -> 1.0
>>> [2021-03-11 19:15:29.494748] D [repce(worker
>>> /data/storage_b/storage):195:push] RepceClient: call
>>> 192115:139891296405312:1615490129.49 version() ...
>>> [2021-03-11 19:15:29.495290] D [repce(worker
>>> /data/storage_b/storage):215:__call__] RepceClient: call
>>> 192115:139891296405312:1615490129.49 version -> 1.0
>>> [2021-03-11 19:15:29.495400] D [repce(worker
>>> /data/storage_b/storage):195:push] RepceClient: call
>>> 192115:139891296405312:1615490129.5 pid() ...
>>> [2021-03-11 19:15:29.495872] D [repce(worker
>>> /data/storage_b/storage):215:__call__] RepceClient: call
>>> 192115:139891296405312:1615490129.5 pid -> 88110
>>> [2021-03-11 19:15:29.495960] I [resource(worker
>>> /data/storage_b/storage):1436:connect_remote] SSH: SSH
>>> connection between master and slave established.
>>> [{duration=1.9944}]
>>> [2021-03-11 19:15:29.496028] I [resource(worker
>>> /data/storage_b/storage):1116:connect] GLUSTER: Mounting
>>> gluster volume locally...
>>> [2021-03-11 19:15:29.501255] D [repce(worker
>>> /data/storage_c/storage):215:__call__] RepceClient: call
>>> 192114:140388828780352:1615490127.53 __repce_version__
>>> -> 1.0
>>> [2021-03-11 19:15:29.501454] D [repce(worker
>>> /data/storage_c/storage):195:push] RepceClient: call
>>> 192114:140388828780352:1615490129.5 version() ...
>>> [2021-03-11 19:15:29.502258] D [repce(worker
>>> /data/storage_c/storage):215:__call__] RepceClient: call
>>> 192114:140388828780352:1615490129.5 version -> 1.0
>>> [2021-03-11 19:15:29.502444] D [repce(worker
>>> /data/storage_c/storage):195:push] RepceClient: call
>>> 192114:140388828780352:1615490129.5 pid() ...
>>> [2021-03-11 19:15:29.503140] D [repce(worker
>>> /data/storage_c/storage):215:__call__] RepceClient: call
>>> 192114:140388828780352:1615490129.5 pid -> 88111
>>> [2021-03-11 19:15:29.503232] I [resource(worker
>>> /data/storage_c/storage):1436:connect_remote] SSH: SSH
>>> connection between master and slave established.
>>> [{duration=2.0026}]
>>> [2021-03-11 19:15:29.503302] I [resource(worker
>>> /data/storage_c/storage):1116:connect] GLUSTER: Mounting
>>> gluster volume locally...
>>> [2021-03-11 19:15:29.533899] D [resource(worker
>>> /data/storage_a/storage):880:inhibit] DirectMounter:
>>> auxiliary glusterfs mount in place
>>> [2021-03-11 19:15:29.595736] D [resource(worker
>>> /data/storage_b/storage):880:inhibit] DirectMounter:
>>> auxiliary glusterfs mount in place
>>> [2021-03-11 19:15:29.601110] D [resource(worker
>>> /data/storage_c/storage):880:inhibit] DirectMounter:
>>> auxiliary glusterfs mount in place
>>> [2021-03-11 19:15:30.541542] D [resource(worker
>>> /data/storage_a/storage):964:inhibit] DirectMounter:
>>> auxiliary glusterfs mount prepared
>>> [2021-03-11 19:15:30.541816] I [resource(worker
>>> /data/storage_a/storage):1139:connect] GLUSTER: Mounted
>>> gluster volume [{duration=1.1037}]
>>> [2021-03-11 19:15:30.541887] I [subcmds(worker
>>> /data/storage_a/storage):84:subcmd_worker] <top>: Worker
>>> spawn successful. Acknowledging back to monitor
>>> [2021-03-11 19:15:30.542042] D [master(worker
>>> /data/storage_a/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=xsync}]
>>> [2021-03-11 19:15:30.542125] D
>>> [monitor(monitor):222:monitor] Monitor:
>>> worker(/data/storage_a/storage) connected
>>> [2021-03-11 19:15:30.543323] D [master(worker
>>> /data/storage_a/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=changelog}]
>>> [2021-03-11 19:15:30.544460] D [master(worker
>>> /data/storage_a/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=changeloghistory}]
>>> [2021-03-11 19:15:30.552103] D [master(worker
>>> /data/storage_a/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
>>> [2021-03-11 19:15:30.602937] D [resource(worker
>>> /data/storage_b/storage):964:inhibit] DirectMounter:
>>> auxiliary glusterfs mount prepared
>>> [2021-03-11 19:15:30.603117] I [resource(worker
>>> /data/storage_b/storage):1139:connect] GLUSTER: Mounted
>>> gluster volume [{duration=1.1070}]
>>> [2021-03-11 19:15:30.603197] I [subcmds(worker
>>> /data/storage_b/storage):84:subcmd_worker] <top>: Worker
>>> spawn successful. Acknowledging back to monitor
>>> [2021-03-11 19:15:30.603353] D [master(worker
>>> /data/storage_b/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=xsync}]
>>> [2021-03-11 19:15:30.603338] D
>>> [monitor(monitor):222:monitor] Monitor:
>>> worker(/data/storage_b/storage) connected
>>> [2021-03-11 19:15:30.604620] D [master(worker
>>> /data/storage_b/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=changelog}]
>>> [2021-03-11 19:15:30.605600] D [master(worker
>>> /data/storage_b/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=changeloghistory}]
>>> [2021-03-11 19:15:30.608365] D [resource(worker
>>> /data/storage_c/storage):964:inhibit] DirectMounter:
>>> auxiliary glusterfs mount prepared
>>> [2021-03-11 19:15:30.608534] I [resource(worker
>>> /data/storage_c/storage):1139:connect] GLUSTER: Mounted
>>> gluster volume [{duration=1.1052}]
>>> [2021-03-11 19:15:30.608612] I [subcmds(worker
>>> /data/storage_c/storage):84:subcmd_worker] <top>: Worker
>>> spawn successful. Acknowledging back to monitor
>>> [2021-03-11 19:15:30.608762] D [master(worker
>>> /data/storage_c/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=xsync}]
>>> [2021-03-11 19:15:30.608779] D
>>> [monitor(monitor):222:monitor] Monitor:
>>> worker(/data/storage_c/storage) connected
>>> [2021-03-11 19:15:30.610033] D [master(worker
>>> /data/storage_c/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=changelog}]
>>> [2021-03-11 19:15:30.610637] D [master(worker
>>> /data/storage_b/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
>>> [2021-03-11 19:15:30.610970] D [master(worker
>>> /data/storage_c/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=changeloghistory}]
>>> [2021-03-11 19:15:30.616197] D [master(worker
>>> /data/storage_c/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
>>> [2021-03-11 19:15:31.371265] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:31.451000] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:31.537257] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:31.623800] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:32.555840] D [master(worker
>>> /data/storage_a/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
>>> [2021-03-11 19:15:32.556051] D [master(worker
>>> /data/storage_a/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
>>> [2021-03-11 19:15:32.556122] D [master(worker
>>> /data/storage_a/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
>>> [2021-03-11 19:15:32.556179] I [master(worker
>>> /data/storage_a/storage):1645:register] _GMaster:
>>> Working dir
>>> [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage}]
>>> [2021-03-11 19:15:32.556359] I [resource(worker
>>> /data/storage_a/storage):1292:service_loop] GLUSTER:
>>> Register time [{time=1615490132}]
>>> [2021-03-11 19:15:32.556823] D [repce(worker
>>> /data/storage_a/storage):195:push] RepceClient: call
>>> 192117:140570487928576:1615490132.56 keep_alive(None,) ...
>>> [2021-03-11 19:15:32.558429] D [repce(worker
>>> /data/storage_a/storage):215:__call__] RepceClient: call
>>> 192117:140570487928576:1615490132.56 keep_alive -> 1
>>> [2021-03-11 19:15:32.558974] D [master(worker
>>> /data/storage_a/storage):540:crawlwrap] _GMaster:
>>> primary master with volume id
>>> cf94a8f2-324b-40b3-bf72-c3766100ea99 ...
>>> [2021-03-11 19:15:32.567478] I [gsyncdstatus(worker
>>> /data/storage_a/storage):281:set_active] GeorepStatus:
>>> Worker Status Change [{status=Active}]
>>> [2021-03-11 19:15:32.571824] I [gsyncdstatus(worker
>>> /data/storage_a/storage):253:set_worker_crawl_status]
>>> GeorepStatus: Crawl Status Change [{status=History Crawl}]
>>> [2021-03-11 19:15:32.572052] I [master(worker
>>> /data/storage_a/storage):1559:crawl] _GMaster: starting
>>> history crawl [{turns=1}, {stime=(1614666553, 0)},
>>> {entry_stime=(1614664115, 0)}, {etime=1615490132}]
>>> [2021-03-11 19:15:32.614506] D [master(worker
>>> /data/storage_b/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
>>> [2021-03-11 19:15:32.614701] D [master(worker
>>> /data/storage_b/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
>>> [2021-03-11 19:15:32.614788] D [master(worker
>>> /data/storage_b/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
>>> [2021-03-11 19:15:32.614845] I [master(worker
>>> /data/storage_b/storage):1645:register] _GMaster:
>>> Working dir
>>> [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage}]
>>> [2021-03-11 19:15:32.615000] I [resource(worker
>>> /data/storage_b/storage):1292:service_loop] GLUSTER:
>>> Register time [{time=1615490132}]
>>> [2021-03-11 19:15:32.615586] D [repce(worker
>>> /data/storage_b/storage):195:push] RepceClient: call
>>> 192115:139889215526656:1615490132.62 keep_alive(None,) ...
>>> [2021-03-11 19:15:32.617373] D [repce(worker
>>> /data/storage_b/storage):215:__call__] RepceClient: call
>>> 192115:139889215526656:1615490132.62 keep_alive -> 1
>>> [2021-03-11 19:15:32.618144] D [master(worker
>>> /data/storage_b/storage):540:crawlwrap] _GMaster:
>>> primary master with volume id
>>> cf94a8f2-324b-40b3-bf72-c3766100ea99 ...
>>> [2021-03-11 19:15:32.619323] D [master(worker
>>> /data/storage_c/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
>>> [2021-03-11 19:15:32.619491] D [master(worker
>>> /data/storage_c/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
>>> [2021-03-11 19:15:32.619739] D [master(worker
>>> /data/storage_c/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
>>> [2021-03-11 19:15:32.619863] I [master(worker
>>> /data/storage_c/storage):1645:register] _GMaster:
>>> Working dir
>>> [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage}]
>>> [2021-03-11 19:15:32.620040] I [resource(worker
>>> /data/storage_c/storage):1292:service_loop] GLUSTER:
>>> Register time [{time=1615490132}]
>>> [2021-03-11 19:15:32.620599] D [repce(worker
>>> /data/storage_c/storage):195:push] RepceClient: call
>>> 192114:140386886469376:1615490132.62 keep_alive(None,) ...
>>> [2021-03-11 19:15:32.621397] E [resource(worker
>>> /data/storage_a/storage):1312:service_loop] GLUSTER:
>>> Changelog History Crawl failed [{error=[Errno 0] Success}]
>>> [2021-03-11 19:15:32.622035] D [repce(worker
>>> /data/storage_c/storage):215:__call__] RepceClient: call
>>> 192114:140386886469376:1615490132.62 keep_alive -> 1
>>> [2021-03-11 19:15:32.622701] D [master(worker
>>> /data/storage_c/storage):540:crawlwrap] _GMaster:
>>> primary master with volume id
>>> cf94a8f2-324b-40b3-bf72-c3766100ea99 ...
>>> [2021-03-11 19:15:32.627031] I [gsyncdstatus(worker
>>> /data/storage_b/storage):281:set_active] GeorepStatus:
>>> Worker Status Change [{status=Active}]
>>> [2021-03-11 19:15:32.643184] I [gsyncdstatus(worker
>>> /data/storage_b/storage):253:set_worker_crawl_status]
>>> GeorepStatus: Crawl Status Change [{status=History Crawl}]
>>> [2021-03-11 19:15:32.643528] I [master(worker
>>> /data/storage_b/storage):1559:crawl] _GMaster: starting
>>> history crawl [{turns=1}, {stime=(1614666552, 0)},
>>> {entry_stime=(1614664113, 0)}, {etime=1615490132}]
>>> [2021-03-11 19:15:32.645148] I [gsyncdstatus(worker
>>> /data/storage_c/storage):281:set_active] GeorepStatus:
>>> Worker Status Change [{status=Active}]
>>> [2021-03-11 19:15:32.649631] I [gsyncdstatus(worker
>>> /data/storage_c/storage):253:set_worker_crawl_status]
>>> GeorepStatus: Crawl Status Change [{status=History Crawl}]
>>> [2021-03-11 19:15:32.649882] I [master(worker
>>> /data/storage_c/storage):1559:crawl] _GMaster: starting
>>> history crawl [{turns=1}, {stime=(1614666552, 0)},
>>> {entry_stime=(1614664108, 0)}, {etime=1615490132}]
>>> [2021-03-11 19:15:32.650907] E [resource(worker
>>> /data/storage_c/storage):1312:service_loop] GLUSTER:
>>> Changelog History Crawl failed [{error=[Errno 0] Success}]
>>> [2021-03-11 19:15:32.700489] E [resource(worker
>>> /data/storage_b/storage):1312:service_loop] GLUSTER:
>>> Changelog History Crawl failed [{error=[Errno 0] Success}]
>>> [2021-03-11 19:15:33.545886] I
>>> [monitor(monitor):228:monitor] Monitor: worker died in
>>> startup phase [{brick=/data/storage_a/storage}]
>>> [2021-03-11 19:15:33.550487] I
>>> [gsyncdstatus(monitor):248:set_worker_status]
>>> GeorepStatus: Worker Status Change [{status=Faulty}]
>>> [2021-03-11 19:15:33.606991] I
>>> [monitor(monitor):228:monitor] Monitor: worker died in
>>> startup phase [{brick=/data/storage_b/storage}]
>>> [2021-03-11 19:15:33.611573] I
>>> [gsyncdstatus(monitor):248:set_worker_status]
>>> GeorepStatus: Worker Status Change [{status=Faulty}]
>>> [2021-03-11 19:15:33.612337] I
>>> [monitor(monitor):228:monitor] Monitor: worker died in
>>> startup phase [{brick=/data/storage_c/storage}]
>>> [2021-03-11 19:15:33.615777] I
>>> [gsyncdstatus(monitor):248:set_worker_status]
>>> GeorepStatus: Worker Status Change [{status=Faulty}]
>>> [2021-03-11 19:15:34.684247] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:34.764971] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:34.851174] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:34.937166] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:36.994502] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:37.73805] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:37.159288] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:37.244153] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:38.916510] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:38.997649] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:39.84816] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:39.172045] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:40.896359] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:40.976135] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:41.62052] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:41.147902] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:42.791997] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:42.871239] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:42.956609] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:43.42473] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:43.566190] I
>>> [gsyncdstatus(monitor):248:set_worker_status]
>>> GeorepStatus: Worker Status Change
>>> [{status=Initializing...}]
>>> [2021-03-11 19:15:43.566400] I
>>> [monitor(monitor):160:monitor] Monitor: starting gsyncd
>>> worker [{brick=/data/storage_a/storage},
>>> {slave_node=10.0.231.81}]
>>> [2021-03-11 19:15:43.572240] D
>>> [monitor(monitor):195:monitor] Monitor: Worker would
>>> mount volume privately
>>> [2021-03-11 19:15:43.612744] D [gsyncd(worker
>>> /data/storage_a/storage):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:43.625689] I
>>> [gsyncdstatus(monitor):248:set_worker_status]
>>> GeorepStatus: Worker Status Change
>>> [{status=Initializing...}]
>>> [2021-03-11 19:15:43.626060] I
>>> [monitor(monitor):160:monitor] Monitor: starting gsyncd
>>> worker [{brick=/data/storage_b/storage},
>>> {slave_node=10.0.231.82}]
>>> [2021-03-11 19:15:43.632287] I
>>> [gsyncdstatus(monitor):248:set_worker_status]
>>> GeorepStatus: Worker Status Change
>>> [{status=Initializing...}]
>>> [2021-03-11 19:15:43.632137] D
>>> [monitor(monitor):195:monitor] Monitor: Worker would
>>> mount volume privately
>>> [2021-03-11 19:15:43.632508] I
>>> [monitor(monitor):160:monitor] Monitor: starting gsyncd
>>> worker [{brick=/data/storage_c/storage},
>>> {slave_node=10.0.231.82}]
>>> [2021-03-11 19:15:43.635565] I [resource(worker
>>> /data/storage_a/storage):1387:connect_remote] SSH:
>>> Initializing SSH connection between master and slave...
>>> [2021-03-11 19:15:43.637835] D
>>> [monitor(monitor):195:monitor] Monitor: Worker would
>>> mount volume privately
>>> [2021-03-11 19:15:43.661304] D [repce(worker
>>> /data/storage_a/storage):195:push] RepceClient: call
>>> 192535:140367272073024:1615490143.66 __repce_version__() ...
>>> [2021-03-11 19:15:43.674499] D [gsyncd(worker
>>> /data/storage_b/storage):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:43.680706] D [gsyncd(worker
>>> /data/storage_c/storage):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:43.693773] I [resource(worker
>>> /data/storage_b/storage):1387:connect_remote] SSH:
>>> Initializing SSH connection between master and slave...
>>> [2021-03-11 19:15:43.700957] I [resource(worker
>>> /data/storage_c/storage):1387:connect_remote] SSH:
>>> Initializing SSH connection between master and slave...
>>> [2021-03-11 19:15:43.717686] D [repce(worker
>>> /data/storage_b/storage):195:push] RepceClient: call
>>> 192539:139907321804608:1615490143.72 __repce_version__() ...
>>> [2021-03-11 19:15:43.725369] D [repce(worker
>>> /data/storage_c/storage):195:push] RepceClient: call
>>> 192541:140653101852480:1615490143.73 __repce_version__() ...
>>> [2021-03-11 19:15:44.289117] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:44.375693] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:44.472251] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:44.558429] D [gsyncd(status):303:main]
>>> <top>: Using session config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:45.619694] D [repce(worker
>>> /data/storage_a/storage):215:__call__] RepceClient: call
>>> 192535:140367272073024:1615490143.66 __repce_version__
>>> -> 1.0
>>> [2021-03-11 19:15:45.619930] D [repce(worker
>>> /data/storage_a/storage):195:push] RepceClient: call
>>> 192535:140367272073024:1615490145.62 version() ...
>>> [2021-03-11 19:15:45.621191] D [repce(worker
>>> /data/storage_a/storage):215:__call__] RepceClient: call
>>> 192535:140367272073024:1615490145.62 version -> 1.0
>>> [2021-03-11 19:15:45.621332] D [repce(worker
>>> /data/storage_a/storage):195:push] RepceClient: call
>>> 192535:140367272073024:1615490145.62 pid() ...
>>> [2021-03-11 19:15:45.621859] D [repce(worker
>>> /data/storage_a/storage):215:__call__] RepceClient: call
>>> 192535:140367272073024:1615490145.62 pid -> 158229
>>> [2021-03-11 19:15:45.621939] I [resource(worker
>>> /data/storage_a/storage):1436:connect_remote] SSH: SSH
>>> connection between master and slave established.
>>> [{duration=1.9862}]
>>> [2021-03-11 19:15:45.622000] I [resource(worker
>>> /data/storage_a/storage):1116:connect] GLUSTER: Mounting
>>> gluster volume locally...
>>> [2021-03-11 19:15:45.714468] D [resource(worker
>>> /data/storage_a/storage):880:inhibit] DirectMounter:
>>> auxiliary glusterfs mount in place
>>> [2021-03-11 19:15:45.718441] D [repce(worker
>>> /data/storage_c/storage):215:__call__] RepceClient: call
>>> 192541:140653101852480:1615490143.73 __repce_version__
>>> -> 1.0
>>> [2021-03-11 19:15:45.718643] D [repce(worker
>>> /data/storage_c/storage):195:push] RepceClient: call
>>> 192541:140653101852480:1615490145.72 version() ...
>>> [2021-03-11 19:15:45.719492] D [repce(worker
>>> /data/storage_c/storage):215:__call__] RepceClient: call
>>> 192541:140653101852480:1615490145.72 version -> 1.0
>>> [2021-03-11 19:15:45.719772] D [repce(worker
>>> /data/storage_c/storage):195:push] RepceClient: call
>>> 192541:140653101852480:1615490145.72 pid() ...
>>> [2021-03-11 19:15:45.720202] D [repce(worker
>>> /data/storage_b/storage):215:__call__] RepceClient: call
>>> 192539:139907321804608:1615490143.72 __repce_version__
>>> -> 1.0
>>> [2021-03-11 19:15:45.720381] D [repce(worker
>>> /data/storage_b/storage):195:push] RepceClient: call
>>> 192539:139907321804608:1615490145.72 version() ...
>>> [2021-03-11 19:15:45.720463] D [repce(worker
>>> /data/storage_c/storage):215:__call__] RepceClient: call
>>> 192541:140653101852480:1615490145.72 pid -> 88921
>>> [2021-03-11 19:15:45.720694] I [resource(worker
>>> /data/storage_c/storage):1436:connect_remote] SSH: SSH
>>> connection between master and slave established.
>>> [{duration=2.0196}]
>>> [2021-03-11 19:15:45.720882] I [resource(worker
>>> /data/storage_c/storage):1116:connect] GLUSTER: Mounting
>>> gluster volume locally...
>>> [2021-03-11 19:15:45.721146] D [repce(worker
>>> /data/storage_b/storage):215:__call__] RepceClient: call
>>> 192539:139907321804608:1615490145.72 version -> 1.0
>>> [2021-03-11 19:15:45.721271] D [repce(worker
>>> /data/storage_b/storage):195:push] RepceClient: call
>>> 192539:139907321804608:1615490145.72 pid() ...
>>> [2021-03-11 19:15:45.721795] D [repce(worker
>>> /data/storage_b/storage):215:__call__] RepceClient: call
>>> 192539:139907321804608:1615490145.72 pid -> 88924
>>> [2021-03-11 19:15:45.721911] I [resource(worker
>>> /data/storage_b/storage):1436:connect_remote] SSH: SSH
>>> connection between master and slave established.
>>> [{duration=2.0280}]
>>> [2021-03-11 19:15:45.721993] I [resource(worker
>>> /data/storage_b/storage):1116:connect] GLUSTER: Mounting
>>> gluster volume locally...
>>> [2021-03-11 19:15:45.816891] D [resource(worker
>>> /data/storage_b/storage):880:inhibit] DirectMounter:
>>> auxiliary glusterfs mount in place
>>> [2021-03-11 19:15:45.816960] D [resource(worker
>>> /data/storage_c/storage):880:inhibit] DirectMounter:
>>> auxiliary glusterfs mount in place
>>> [2021-03-11 19:15:46.721534] D [resource(worker
>>> /data/storage_a/storage):964:inhibit] DirectMounter:
>>> auxiliary glusterfs mount prepared
>>> [2021-03-11 19:15:46.721726] I [resource(worker
>>> /data/storage_a/storage):1139:connect] GLUSTER: Mounted
>>> gluster volume [{duration=1.0997}]
>>> [2021-03-11 19:15:46.721796] I [subcmds(worker
>>> /data/storage_a/storage):84:subcmd_worker] <top>: Worker
>>> spawn successful. Acknowledging back to monitor
>>> [2021-03-11 19:15:46.721971] D [master(worker
>>> /data/storage_a/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=xsync}]
>>> [2021-03-11 19:15:46.722122] D
>>> [monitor(monitor):222:monitor] Monitor:
>>> worker(/data/storage_a/storage) connected
>>> [2021-03-11 19:15:46.723871] D [master(worker
>>> /data/storage_a/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=changelog}]
>>> [2021-03-11 19:15:46.725100] D [master(worker
>>> /data/storage_a/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=changeloghistory}]
>>> [2021-03-11 19:15:46.732400] D [master(worker
>>> /data/storage_a/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
>>> [2021-03-11 19:15:46.823477] D [resource(worker
>>> /data/storage_c/storage):964:inhibit] DirectMounter:
>>> auxiliary glusterfs mount prepared
>>> [2021-03-11 19:15:46.823645] I [resource(worker
>>> /data/storage_c/storage):1139:connect] GLUSTER: Mounted
>>> gluster volume [{duration=1.1027}]
>>> [2021-03-11 19:15:46.823754] I [subcmds(worker
>>> /data/storage_c/storage):84:subcmd_worker] <top>: Worker
>>> spawn successful. Acknowledging back to monitor
>>> [2021-03-11 19:15:46.823932] D [master(worker
>>> /data/storage_c/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=xsync}]
>>> [2021-03-11 19:15:46.823904] D [resource(worker
>>> /data/storage_b/storage):964:inhibit] DirectMounter:
>>> auxiliary glusterfs mount prepared
>>> [2021-03-11 19:15:46.823930] D
>>> [monitor(monitor):222:monitor] Monitor:
>>> worker(/data/storage_c/storage) connected
>>> [2021-03-11 19:15:46.824103] I [resource(worker
>>> /data/storage_b/storage):1139:connect] GLUSTER: Mounted
>>> gluster volume [{duration=1.1020}]
>>> [2021-03-11 19:15:46.824184] I [subcmds(worker
>>> /data/storage_b/storage):84:subcmd_worker] <top>: Worker
>>> spawn successful. Acknowledging back to monitor
>>> [2021-03-11 19:15:46.824340] D [master(worker
>>> /data/storage_b/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=xsync}]
>>> [2021-03-11 19:15:46.824321] D
>>> [monitor(monitor):222:monitor] Monitor:
>>> worker(/data/storage_b/storage) connected
>>> [2021-03-11 19:15:46.825100] D [master(worker
>>> /data/storage_c/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=changelog}]
>>> [2021-03-11 19:15:46.825414] D [master(worker
>>> /data/storage_b/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=changelog}]
>>> [2021-03-11 19:15:46.826375] D [master(worker
>>> /data/storage_b/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=changeloghistory}]
>>> [2021-03-11 19:15:46.826574] D [master(worker
>>> /data/storage_c/storage):105:gmaster_builder] <top>:
>>> setting up change detection mode [{mode=changeloghistory}]
>>> [2021-03-11 19:15:46.831506] D [master(worker
>>> /data/storage_b/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
>>> [2021-03-11 19:15:46.833168] D [master(worker
>>> /data/storage_c/storage):778:setup_working_dir]
>>> _GMaster: changelog working dir
>>> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
>>> [2021-03-11 19:15:47.275141] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:47.320247] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:47.570877] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:47.615571] D
>>> [gsyncd(config-get):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:47.620893] E [syncdutils(worker
>>> /data/storage_a/storage):325:log_raise_exception] <top>:
>>> connection to peer is broken
>>> [2021-03-11 19:15:47.620939] E [syncdutils(worker
>>> /data/storage_c/storage):325:log_raise_exception] <top>:
>>> connection to peer is broken
>>> [2021-03-11 19:15:47.621668] E [syncdutils(worker
>>> /data/storage_a/storage):847:errlog] Popen: command
>>> returned error [{cmd=ssh -oPasswordAuthentication=no
>>> -oStrictHostKeyChecking=no -i
>>> /var/lib/glusterd/geo-replication/secret.pem -p 22
>>> -oControlMaster=auto -S
>>> /tmp/gsyncd-aux-ssh-_AyCOc/79fa3dc75e30f532b4a40bc08c2b10a1.sock
>>> geoaccount at 10.0.231.81 <mailto:geoaccount at 10.0.231.81>
>>> /nonexistent/gsyncd slave storage
>>> geoaccount at 10.0.231.81::pcic-backup
>>> <mailto:geoaccount at 10.0.231.81::pcic-backup>
>>> --master-node 10.0.231.91 --master-node-id
>>> afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick
>>> /data/storage_a/storage --local-node 10.0.231.81
>>> --local-node-id b88dea4f-31ec-416a-9110-3ccdc3910acd
>>> --slave-timeout 120 --slave-log-level INFO
>>> --slave-gluster-log-level INFO
>>> --slave-gluster-command-dir /usr/sbin
>>> --master-dist-count 3}, {error=255}]
>>> [2021-03-11 19:15:47.621685] E [syncdutils(worker
>>> /data/storage_c/storage):847:errlog] Popen: command
>>> returned error [{cmd=ssh -oPasswordAuthentication=no
>>> -oStrictHostKeyChecking=no -i
>>> /var/lib/glusterd/geo-replication/secret.pem -p 22
>>> -oControlMaster=auto -S
>>> /tmp/gsyncd-aux-ssh-WOgOEu/e15fc58bb13552de0710eaf018209548.sock
>>> geoaccount at 10.0.231.82 <mailto:geoaccount at 10.0.231.82>
>>> /nonexistent/gsyncd slave storage
>>> geoaccount at 10.0.231.81::pcic-backup
>>> <mailto:geoaccount at 10.0.231.81::pcic-backup>
>>> --master-node 10.0.231.91 --master-node-id
>>> afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick
>>> /data/storage_c/storage --local-node 10.0.231.82
>>> --local-node-id be50a8de-3934-4fee-a80d-8e2e99017902
>>> --slave-timeout 120 --slave-log-level INFO
>>> --slave-gluster-log-level INFO
>>> --slave-gluster-command-dir /usr/sbin
>>> --master-dist-count 3}, {error=255}]
>>> [2021-03-11 19:15:47.621776] E [syncdutils(worker
>>> /data/storage_a/storage):851:logerr] Popen: ssh> Killed
>>> by signal 15.
>>> [2021-03-11 19:15:47.621819] E [syncdutils(worker
>>> /data/storage_c/storage):851:logerr] Popen: ssh> Killed
>>> by signal 15.
>>> [2021-03-11 19:15:47.621850] E [syncdutils(worker
>>> /data/storage_b/storage):325:log_raise_exception] <top>:
>>> connection to peer is broken
>>> [2021-03-11 19:15:47.622437] E [syncdutils(worker
>>> /data/storage_b/storage):847:errlog] Popen: command
>>> returned error [{cmd=ssh -oPasswordAuthentication=no
>>> -oStrictHostKeyChecking=no -i
>>> /var/lib/glusterd/geo-replication/secret.pem -p 22
>>> -oControlMaster=auto -S
>>> /tmp/gsyncd-aux-ssh-Vy935W/e15fc58bb13552de0710eaf018209548.sock
>>> geoaccount at 10.0.231.82 <mailto:geoaccount at 10.0.231.82>
>>> /nonexistent/gsyncd slave storage
>>> geoaccount at 10.0.231.81::pcic-backup
>>> <mailto:geoaccount at 10.0.231.81::pcic-backup>
>>> --master-node 10.0.231.91 --master-node-id
>>> afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick
>>> /data/storage_b/storage --local-node 10.0.231.82
>>> --local-node-id be50a8de-3934-4fee-a80d-8e2e99017902
>>> --slave-timeout 120 --slave-log-level INFO
>>> --slave-gluster-log-level INFO
>>> --slave-gluster-command-dir /usr/sbin
>>> --master-dist-count 3}, {error=255}]
>>> [2021-03-11 19:15:47.622556] E [syncdutils(worker
>>> /data/storage_b/storage):851:logerr] Popen: ssh> Killed
>>> by signal 15.
>>> [2021-03-11 19:15:47.723756] I
>>> [monitor(monitor):228:monitor] Monitor: worker died in
>>> startup phase [{brick=/data/storage_a/storage}]
>>> [2021-03-11 19:15:47.731405] I
>>> [gsyncdstatus(monitor):248:set_worker_status]
>>> GeorepStatus: Worker Status Change [{status=Faulty}]
>>> [2021-03-11 19:15:47.825223] I
>>> [monitor(monitor):228:monitor] Monitor: worker died in
>>> startup phase [{brick=/data/storage_c/storage}]
>>> [2021-03-11 19:15:47.825685] I
>>> [monitor(monitor):228:monitor] Monitor: worker died in
>>> startup phase [{brick=/data/storage_b/storage}]
>>> [2021-03-11 19:15:47.829011] I
>>> [gsyncdstatus(monitor):248:set_worker_status]
>>> GeorepStatus: Worker Status Change [{status=Faulty}]
>>> [2021-03-11 19:15:47.830965] I
>>> [gsyncdstatus(monitor):248:set_worker_status]
>>> GeorepStatus: Worker Status Change [{status=Faulty}]
>>> [2021-03-11 19:15:48.669634] D
>>> [gsyncd(monitor-status):303:main] <top>: Using session
>>> config file
>>> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
>>> [2021-03-11 19:15:48.683784] I
>>> [subcmds(monitor-status):29:subcmd_monitor_status]
>>> <top>: Monitor Status Change [{status=Stopped}]
>>>
>>>
>>> Thanks,
>>> -Matthew
>>>
>>>
>>> On 3/11/21 9:37 AM, Strahil Nikolov wrote:
>>>> Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.
>>>>
>>>>
>>>> I think you have to increase the debug logs for geo-rep session.
>>>> I will try to find the command necessary to increase it.
>>>>
>>>>
>>>> Best Regards,
>>>> Strahil Nikolov
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> В четвъртък, 11 март 2021 г., 00:38:41 ч. Гринуич+2, Matthew Benstead <matthewb at uvic.ca> <mailto:matthewb at uvic.ca> написа:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks Strahil,
>>>>
>>>> Right - I had come across your message in early January that v8 from the CentOS Sig was missing the SELinux rules, and had put SELinux into permissive mode after the upgrade when I saw denied messages in the audit logs.
>>>>
>>>> [root at storage01 ~]# sestatus | egrep "^SELinux status|[mM]ode"
>>>> SELinux status: enabled
>>>> Current mode: permissive
>>>> Mode from config file: enforcing
>>>>
>>>> Yes - I am using an unprivileged user for georep:
>>>>
>>>> [root at pcic-backup01 ~]# gluster-mountbroker status
>>>> +-------------+-------------+---------------------------+--------------+--------------------------+
>>>> | NODE | NODE STATUS | MOUNT ROOT | GROUP | USERS |
>>>> +-------------+-------------+---------------------------+--------------+--------------------------+
>>>> | 10.0.231.82 | UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup) |
>>>> | localhost | UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup) |
>>>> +-------------+-------------+---------------------------+--------------+--------------------------+
>>>>
>>>> [root at pcic-backup02 ~]# gluster-mountbroker status
>>>> +-------------+-------------+---------------------------+--------------+--------------------------+
>>>> | NODE | NODE STATUS | MOUNT ROOT | GROUP | USERS |
>>>> +-------------+-------------+---------------------------+--------------+--------------------------+
>>>> | 10.0.231.81 | UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup) |
>>>> | localhost | UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup) |
>>>> +-------------+-------------+---------------------------+--------------+--------------------------+
>>>>
>>>> Thanks,
>>>> -Matthew
>>>>
>>>>
>>>> --
>>>> Matthew Benstead
>>>> System AdministratorPacific Climate Impacts ConsortiumUniversity of Victoria, UH1PO Box 1800, STN CSCVictoria, BC, V8W 2Y2Phone: +1-250-721-8432Email: matthewb at uvic.ca <mailto:matthewb at uvic.ca>
>>>>
>>>>
>>>> On 3/10/21 2:11 PM, Strahil Nikolov wrote:
>>>>
>>>>
>>>>>
>>>>> Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.
>>>>>
>>>>>
>>>>> I have tested georep on v8.3 and it was running quite well untill you involve SELINUX.
>>>>>
>>>>>
>>>>>
>>>>> Are you using SELINUX ?
>>>>>
>>>>> Are you using unprivileged user for the georep ?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Also, you can check https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-troubleshooting_geo-replication <https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-troubleshooting_geo-replication> .
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Strahil Nikolov
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 11, 2021 at 0:03, Matthew Benstead
>>>>>>
>>>>>> <matthewb at uvic.ca> <mailto:matthewb at uvic.ca> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I recently upgraded my Distributed-Replicate cluster from Gluster 7.9 to 8.3 on CentOS7 using the CentOS Storage SIG packages. I had geo-replication syncing properly before the upgrade, but not it is not working after.
>>>>>>
>>>>>> After I had upgraded both master and slave clusters I attempted to start geo-replication again, but it goes to faulty quickly:
>>>>>>
>>>>>> [root at storage01 ~]# gluster volume geo-replication storage geoaccount at 10.0.231.81::pcic-backup <mailto:geoaccount at 10.0.231.81::pcic-backup> start
>>>>>> Starting geo-replication session between storage & geoaccount at 10.0.231.81::pcic-backup <mailto:geoaccount at 10.0.231.81::pcic-backup> has been successful\
>>>>>>
>>>>>> [root at storage01 ~]# gluster volume geo-replication status
>>>>>>
>>>>>> MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED
>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>>> 10.0.231.91 storage /data/storage_a/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup <mailto:ssh://geoaccount@10.0.231.81::pcic-backup> N/A Faulty N/A N/A
>>>>>> 10.0.231.91 storage /data/storage_c/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup <mailto:ssh://geoaccount@10.0.231.81::pcic-backup> N/A Faulty N/A N/A
>>>>>> 10.0.231.91 storage /data/storage_b/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup <mailto:ssh://geoaccount@10.0.231.81::pcic-backup> N/A Faulty N/A N/A
>>>>>> 10.0.231.92 storage /data/storage_b/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup <mailto:ssh://geoaccount@10.0.231.81::pcic-backup> N/A Faulty N/A N/A
>>>>>> 10.0.231.92 storage /data/storage_a/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup <mailto:ssh://geoaccount@10.0.231.81::pcic-backup> N/A Faulty N/A N/A
>>>>>> 10.0.231.92 storage /data/storage_c/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup <mailto:ssh://geoaccount@10.0.231.81::pcic-backup> N/A Faulty N/A N/A
>>>>>> 10.0.231.93 storage /data/storage_c/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup <mailto:ssh://geoaccount@10.0.231.81::pcic-backup> N/A Faulty N/A N/A
>>>>>> 10.0.231.93 storage /data/storage_b/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup <mailto:ssh://geoaccount@10.0.231.81::pcic-backup> N/A Faulty N/A N/A
>>>>>> 10.0.231.93 storage /data/storage_a/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup <mailto:ssh://geoaccount@10.0.231.81::pcic-backup> N/A Faulty N/A N/A
>>>>>>
>>>>>> [root at storage01 ~]# gluster volume geo-replication storage geoaccount at 10.0.231.81::pcic-backup <mailto:geoaccount at 10.0.231.81::pcic-backup> stop
>>>>>> Stopping geo-replication session between storage & geoaccount at 10.0.231.81::pcic-backup <mailto:geoaccount at 10.0.231.81::pcic-backup> has been successful
>>>>>>
>>>>>>
>>>>>> I went through the gsyncd logs and see it attempts to go back through the changlogs - which would make sense - but fails:
>>>>>>
>>>>>> [2021-03-10 19:18:42.165807] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
>>>>>> [2021-03-10 19:18:42.166136] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]
>>>>>> [2021-03-10 19:18:42.167829] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]
>>>>>> [2021-03-10 19:18:42.172343] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
>>>>>> [2021-03-10 19:18:42.172580] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]
>>>>>> [2021-03-10 19:18:42.235574] I [resource(worker /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
>>>>>> [2021-03-10 19:18:42.236613] I [resource(worker /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
>>>>>> [2021-03-10 19:18:42.238614] I [resource(worker /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
>>>>>> [2021-03-10 19:18:44.144856] I [resource(worker /data/storage_b/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9059}]
>>>>>> [2021-03-10 19:18:44.145065] I [resource(worker /data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
>>>>>> [2021-03-10 19:18:44.162873] I [resource(worker /data/storage_a/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9259}]
>>>>>> [2021-03-10 19:18:44.163412] I [resource(worker /data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
>>>>>> [2021-03-10 19:18:44.167506] I [resource(worker /data/storage_c/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9316}]
>>>>>> [2021-03-10 19:18:44.167746] I [resource(worker /data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
>>>>>> [2021-03-10 19:18:45.251372] I [resource(worker /data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1062}]
>>>>>> [2021-03-10 19:18:45.251583] I [subcmds(worker /data/storage_b/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
>>>>>> [2021-03-10 19:18:45.271950] I [resource(worker /data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1041}]
>>>>>> [2021-03-10 19:18:45.272118] I [subcmds(worker /data/storage_c/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
>>>>>> [2021-03-10 19:18:45.275180] I [resource(worker /data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1116}]
>>>>>> [2021-03-10 19:18:45.275361] I [subcmds(worker /data/storage_a/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
>>>>>> [2021-03-10 19:18:47.265618] I [master(worker /data/storage_b/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage}]
>>>>>> [2021-03-10 19:18:47.265954] I [resource(worker /data/storage_b/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
>>>>>> [2021-03-10 19:18:47.276746] I [gsyncdstatus(worker /data/storage_b/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
>>>>>> [2021-03-10 19:18:47.281194] I [gsyncdstatus(worker /data/storage_b/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
>>>>>> [2021-03-10 19:18:47.281404] I [master(worker /data/storage_b/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664113, 0)}, {etime=1615403927}]
>>>>>> [2021-03-10 19:18:47.285340] I [master(worker /data/storage_c/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage}]
>>>>>> [2021-03-10 19:18:47.285579] I [resource(worker /data/storage_c/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
>>>>>> [2021-03-10 19:18:47.287383] I [master(worker /data/storage_a/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage}]
>>>>>> [2021-03-10 19:18:47.287697] I [resource(worker /data/storage_a/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
>>>>>> [2021-03-10 19:18:47.298415] I [gsyncdstatus(worker /data/storage_c/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
>>>>>> [2021-03-10 19:18:47.301342] I [gsyncdstatus(worker /data/storage_a/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
>>>>>> [2021-03-10 19:18:47.304183] I [gsyncdstatus(worker /data/storage_c/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
>>>>>> [2021-03-10 19:18:47.304418] I [master(worker /data/storage_c/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664108, 0)}, {etime=1615403927}]
>>>>>> [2021-03-10 19:18:47.305294] E [resource(worker /data/storage_c/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
>>>>>> [2021-03-10 19:18:47.308124] I [gsyncdstatus(worker /data/storage_a/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
>>>>>> [2021-03-10 19:18:47.308509] I [master(worker /data/storage_a/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666553, 0)}, {entry_stime=(1614664115, 0)}, {etime=1615403927}]
>>>>>> [2021-03-10 19:18:47.357470] E [resource(worker /data/storage_b/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
>>>>>> [2021-03-10 19:18:47.383949] E [resource(worker /data/storage_a/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
>>>>>> [2021-03-10 19:18:48.255340] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_b/storage}]
>>>>>> [2021-03-10 19:18:48.260052] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
>>>>>> [2021-03-10 19:18:48.275651] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_c/storage}]
>>>>>> [2021-03-10 19:18:48.278064] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_a/storage}]
>>>>>> [2021-03-10 19:18:48.280453] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
>>>>>> [2021-03-10 19:18:48.282274] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
>>>>>> [2021-03-10 19:18:58.275702] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
>>>>>> [2021-03-10 19:18:58.276041] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]
>>>>>> [2021-03-10 19:18:58.296252] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
>>>>>> [2021-03-10 19:18:58.296506] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]
>>>>>> [2021-03-10 19:18:58.301290] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
>>>>>> [2021-03-10 19:18:58.301521] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]
>>>>>> [2021-03-10 19:18:58.345817] I [resource(worker /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
>>>>>> [2021-03-10 19:18:58.361268] I [resource(worker /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
>>>>>> [2021-03-10 19:18:58.367985] I [resource(worker /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
>>>>>> [2021-03-10 19:18:59.115143] I [subcmds(monitor-status):29:subcmd_monitor_status] <top>: Monitor Status Change [{status=Stopped}]
>>>>>>
>>>>>> It seems like there is an issue selecting the changelogs - perhaps similar to this issue? https://github.com/gluster/glusterfs/issues/1766 <https://github.com/gluster/glusterfs/issues/1766>
>>>>>>
>>>>>> [root at storage01 storage_10.0.231.81_pcic-backup]# cat changes-data-storage_a-storage.log
>>>>>> [2021-03-10 19:18:45.284764] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_a/storage}, {notify_filter=1}]
>>>>>> [2021-03-10 19:18:45.285275] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}]
>>>>>> [2021-03-10 19:18:45.285269] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}]
>>>>>> [2021-03-10 19:18:45.286615] I [socket.c:929:__socket_server_bind] 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 21
>>>>>> [2021-03-10 19:18:47.308607] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666553}, {end=1615403927}]
>>>>>> [2021-03-10 19:18:47.308659] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1615403927}, {total_changelogs=1250878}]
>>>>>> [2021-03-10 19:18:47.383774] E [MSGID: 132009] [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong result [{for=end}, {start=1615403927}, {idx=1250877}]
>>>>>>
>>>>>> [root at storage01 storage_10.0.231.81_pcic-backup]# tail -7 changes-data-storage_b-storage.log
>>>>>> [2021-03-10 19:18:45.263211] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}]
>>>>>> [2021-03-10 19:18:45.263151] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_b/storage}, {notify_filter=1}]
>>>>>> [2021-03-10 19:18:45.263294] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}]
>>>>>> [2021-03-10 19:18:45.264598] I [socket.c:929:__socket_server_bind] 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 23
>>>>>> [2021-03-10 19:18:47.281499] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666552}, {end=1615403927}]
>>>>>> [2021-03-10 19:18:47.281551] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1615403927}, {total_changelogs=1258258}]
>>>>>> [2021-03-10 19:18:47.357244] E [MSGID: 132009] [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong result [{for=end}, {start=1615403927}, {idx=1258257}]
>>>>>>
>>>>>> Any ideas on where to debug this? I'd prefer not to have to remove and re-sync everything as there is about 240TB on the cluster...
>>>>>>
>>>>>> Thanks,
>>>>>> -Matthew
>>>>>>
>>>>>>
>>>>>> ________
>>>>>>
>>>>>>
>>>>>>
>>>>>> Community Meeting Calendar:
>>>>>>
>>>>>> Schedule -
>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>>> Bridge: https://meet.google.com/cpu-eiue-hvk <https://meet.google.com/cpu-eiue-hvk>
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users <https://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>>>
>>>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210317/ce6bc65e/attachment.html>
More information about the Gluster-users
mailing list