[Gluster-users] GeoRep Faulty after Gluster 7 to 8 upgrade - gfchangelog: wrong result

Shwetha Acharya sacharya at redhat.com
Fri Mar 19 09:43:36 UTC 2021


Hi Mathew,

The upgrade script
https://github.com/gluster/glusterfs/commit/2857fe3fad4d2b30894847088a54b847b88a23b9
need to be run, to make sure that the changelogs and htimes files
created in the older version are updated as per the new changes in the
directory structure.

If not done, search algorithm during history crawl would give wrong result.

Regards,
Shwetha

On Fri, Mar 19, 2021 at 3:53 AM Strahil Nikolov <hunter86_bg at yahoo.com>
wrote:

> Sadly,
>
> I'm out of ideas. It makes sense... if the changelog was changed - then it
> won't work after the upgrade.
>
> I guess that once you delete the session, you can remove the extended
> attribute for the time.
>
> Best Regards,
> Strahil Nikolov
>
> On Wed, Mar 17, 2021 at 23:11, Matthew Benstead
> <matthewb at uvic.ca> wrote:
> Yes, I've run through everything short of regenerating the keys and
> creating the session again with no errors. Everything looks ok.
>
> But I did notice that the changelog format had changed, instead of them
> being dumped into one directory, they now seem to be separated in
> year/month/day directories...
>
> Looks like this change in 8.0:
> https://github.com/gluster/glusterfs/issues/154
>
> [root at storage01 changelogs]# ls -lh | head
> total 16G
> drw-------. 3 root root   24 Mar  9 11:34 2021
> -rw-r--r--. 1 root root   51 Mar 17 13:19 CHANGELOG
> -rw-r--r--. 1 root root  13K Aug 13  2020 CHANGELOG.1597343197
> -rw-r--r--. 1 root root  51K Aug 13  2020 CHANGELOG.1597343212
> -rw-r--r--. 1 root root  86K Aug 13  2020 CHANGELOG.1597343227
> -rw-r--r--. 1 root root  99K Aug 13  2020 CHANGELOG.1597343242
> -rw-r--r--. 1 root root  69K Aug 13  2020 CHANGELOG.1597343257
> -rw-r--r--. 1 root root  69K Aug 13  2020 CHANGELOG.1597343272
> -rw-r--r--. 1 root root  72K Aug 13  2020 CHANGELOG.1597343287
> [root at storage01 changelogs]# ls -lh | tail
> -rw-r--r--. 1 root root   92 Mar  1 21:33 CHANGELOG.1614663193
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663731
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663760
> -rw-r--r--. 1 root root  511 Mar  1 21:47 CHANGELOG.1614664043
> -rw-r--r--. 1 root root  536 Mar  1 21:48 CHANGELOG.1614664101
> -rw-r--r--. 1 root root 2.8K Mar  1 21:48 CHANGELOG.1614664116
> -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666061
> -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666554
> drw-------. 2 root root   10 May  7  2020 csnap
> drw-------. 2 root root   38 Aug 13  2020 htime
>
>
> [root at storage01 changelogs]# ls -lh 2021/03/09/
> total 8.0K
> -rw-r--r--. 1 root root 51 Mar  9 11:26 CHANGELOG.1615318474
> -rw-r--r--. 1 root root 51 Mar  9 12:19 CHANGELOG.1615321197
> [root at storage01 changelogs]# ls -lh 2021/03/15/
> total 4.0K
> -rw-r--r--. 1 root root 51 Mar 15 13:38 CHANGELOG.1615842490
> [root at storage01 changelogs]# ls -lh 2021/03/16
> total 4.0K
> -rw-r--r--. 1 root root 331 Mar 16 12:04 CHANGELOG.1615921482
>
> But it looks like the htime file still records them...
>
> [root at storage01 changelogs]# ls -lh htime
> total 84M
> -rw-r--r--. 1 root root 84M Mar 17 13:31 HTIME.1597342860
>
> [root at storage01 changelogs]# head -c 256 htime/HTIME.1597342860
>
> /data/storage_a/storage/.glusterfs/changelogs/changelog.1597342875/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342890/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342904/data/storage_a/storage/.glusterfs/changelogs/changelog[root at storage01
> changelogs]#
>
> [root at storage01 changelogs]# tail -c 256 htime/HTIME.1597342860
>
> /changelog.1616013484/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013499/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013514/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013529[root at storage01
> changelogs]#
>
>
> And there seems to be an xattr for time in the brick root - presumably for
> when changelogs were enabled:
>
> [root at storage01 changelogs]# getfattr -d -m. -e hex
> /data/storage_a/storage 2>&1 | egrep xtime
>
> trusted.glusterfs.cf94a8f2-324b-40b3-bf72-c3766100ea99.xtime=0x60510140000ef317
>
>
> Reading through the changes, it looks like there is a script to convert
> from one format to the other... I didn't see anything in the release notes
> for 8.0 about it... this seems like it could be the fix, and explain why
> gluster can't get through the changelogs....  Thoughts?
>
>    -
>    https://github.com/gluster/glusterfs/issues/154#issuecomment-585701964
>    - https://review.gluster.org/#/c/glusterfs/+/24121/
>
>
> Thanks,
>  -Matthew
>
> --
> Matthew Benstead
> System Administrator
> Pacific Climate Impacts Consortium <https://pacificclimate.org/>
> University of Victoria, UH1
> PO Box 1800, STN CSC
> Victoria, BC, V8W 2Y2
> Phone: +1-250-721-8432
> Email: matthewb at uvic.ca
> On 3/16/21 9:36 PM, Strahil Nikolov wrote:
>
> Notice: This message was sent from outside the University of Victoria
> email system. Please be cautious with links and sensitive information.
>
> Have you verified all steps for creating the geo-replication ?
>
> If yes , maybe using "reset-sync-time + delete + create" makes sense.Keep
> in mind that it will take a long time once the geo-rep is established again.
>
>
> Best Regards,
> Strahil Nikolov
>
> On Tue, Mar 16, 2021 at 22:34, Matthew Benstead
> <matthewb at uvic.ca> <matthewb at uvic.ca> wrote:
> Thanks Strahil,
>
> I wanted to make sure the issue wasn't occurring because there were no new
> changes to sync from the master volume. So I created some files and
> restarted the sync, but it had no effect.
>
> [root at storage01 ~]# cd /storage2/home/test/
> [root at storage01 test]# for nums in {1,2,3,4,5,6,7,8,9,0}; do touch
> $nums.txt; done
>
> [root at storage01 test]# gluster volume geo-replication storage
> geoaccount at 10.0.231.81::pcic-backup start
> Starting geo-replication session between storage &
> geoaccount at 10.0.231.81::pcic-backup has been successful
> [root at storage01 test]# gluster volume geo-replication status
>
> MASTER NODE    MASTER VOL    MASTER BRICK               SLAVE USER
> SLAVE                                        SLAVE NODE
> STATUS             CRAWL STATUS    LAST_SYNCED
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 10.0.231.91    storage       /data/storage_a/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.91    storage       /data/storage_c/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.91    storage       /data/storage_b/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.93    storage       /data/storage_c/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.93    storage       /data/storage_b/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.93    storage       /data/storage_a/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.92    storage       /data/storage_b/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.92    storage       /data/storage_a/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.92    storage       /data/storage_c/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> [root at storage01 test]# gluster volume geo-replication status
>
> MASTER NODE    MASTER VOL    MASTER BRICK               SLAVE USER
> SLAVE                                        SLAVE NODE    STATUS    CRAWL
> STATUS    LAST_SYNCED
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 10.0.231.91    storage       /data/storage_a/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.91    storage       /data/storage_c/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.91    storage       /data/storage_b/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.93    storage       /data/storage_c/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.93    storage       /data/storage_b/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.93    storage       /data/storage_a/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.92    storage       /data/storage_b/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.92    storage       /data/storage_a/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.92    storage       /data/storage_c/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> [root at storage01 test]# gluster volume geo-replication storage
> geoaccount at 10.0.231.81::pcic-backup stop
> Stopping geo-replication session between storage &
> geoaccount at 10.0.231.81::pcic-backup has been successful
>
> Still getting the same error about the history crawl failing:
>
> [2021-03-16 19:05:05.227677] I [MSGID: 132035]
> [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting
> historical changelogs [{start=1614666552}, {end=1615921505}]
> [2021-03-16 19:05:05.227733] I [MSGID: 132019]
> [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog:
> changelogs min max [{min=1597342860}, {max=1615921502},
> {total_changelogs=1300114}]
> [2021-03-16 19:05:05.408567] E [MSGID: 132009]
> [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong
> result [{for=end}, {start=1615921502}, {idx=1300113}]
>
>
> [2021-03-16 19:05:05.228092] I [resource(worker
> /data/storage_c/storage):1292:service_loop] GLUSTER: Register time
> [{time=1615921505}]
> [2021-03-16 19:05:05.228626] D [repce(worker
> /data/storage_c/storage):195:push] RepceClient: call
> 124117:140500837320448:1615921505.23 keep_alive(None,) ...
> [2021-03-16 19:05:05.230076] D [repce(worker
> /data/storage_c/storage):215:__call__] RepceClient: call
> 124117:140500837320448:1615921505.23 keep_alive -> 1
> [2021-03-16 19:05:05.230693] D [master(worker
> /data/storage_c/storage):540:crawlwrap] _GMaster: primary master with
> volume id cf94a8f2-324b-40b3-bf72-c3766100ea99 ...
> [2021-03-16 19:05:05.237607] I [gsyncdstatus(worker
> /data/storage_c/storage):281:set_active] GeorepStatus: Worker Status Change
> [{status=Active}]
> [2021-03-16 19:05:05.242046] I [gsyncdstatus(worker
> /data/storage_c/storage):253:set_worker_crawl_status] GeorepStatus: Crawl
> Status Change [{status=History Crawl}]
> [2021-03-16 19:05:05.242450] I [master(worker
> /data/storage_c/storage):1559:crawl] _GMaster: starting history crawl
> [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664108, 0)},
> {etime=1615921505}]
> [2021-03-16 19:05:05.244151] E [resource(worker
> /data/storage_c/storage):1312:service_loop] GLUSTER: Changelog History
> Crawl failed [{error=[Errno 0] Success}]
> [2021-03-16 19:05:05.394129] E [resource(worker
> /data/storage_a/storage):1312:service_loop] GLUSTER: Changelog History
> Crawl failed [{error=[Errno 0] Success}]
> [2021-03-16 19:05:05.408759] E [resource(worker
> /data/storage_b/storage):1312:service_loop] GLUSTER: Changelog History
> Crawl failed [{error=[Errno 0] Success}]
> [2021-03-16 19:05:06.158694] I [monitor(monitor):228:monitor] Monitor:
> worker died in startup phase [{brick=/data/storage_a/storage}]
> [2021-03-16 19:05:06.163052] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change [{status=Faulty}]
> [2021-03-16 19:05:06.204464] I [monitor(monitor):228:monitor] Monitor:
> worker died in startup phase [{brick=/data/storage_b/storage}]
> [2021-03-16 19:05:06.208961] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change [{status=Faulty}]
> [2021-03-16 19:05:06.220495] I [monitor(monitor):228:monitor] Monitor:
> worker died in startup phase [{brick=/data/storage_c/storage}]
> [2021-03-16 19:05:06.223947] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change [{status=Faulty}]
>
> I confirmed NTP is working:
>
>
> pcic-backup02 | CHANGED | rc=0 >>
>      remote           refid      st t when poll reach   delay   offset
> jitter
>
> ==============================================================================
> +s216-232-132-95 68.69.221.61     2 u   29 1024  377   24.141    2.457
> 1.081
> *yyz-1.ip.0xt.ca 206.108.0.131    2 u  257 1024  377   57.119   -0.084
> 5.625
> +ip102.ip-198-27 192.168.10.254   2 u  189 1024  377   64.227   -3.012
> 8.867
>
> storage03 | CHANGED | rc=0 >>
>      remote           refid      st t when poll reach   delay   offset
> jitter
>
> ==============================================================================
> *198.161.203.36  128.233.150.93   2 u   36 1024  377   16.055   -0.381
> 0.318
> +s206-75-147-25. 192.168.10.254   2 u  528 1024  377   23.648   -6.196
> 4.803
> +time.cloudflare 10.69.8.80       3 u  121 1024  377    2.408    0.507
> 0.791
>
> storage02 | CHANGED | rc=0 >>
>      remote           refid      st t when poll reach   delay   offset
> jitter
>
> ==============================================================================
> *198.161.203.36  128.233.150.93   2 u  918 1024  377   15.952    0.226
> 0.197
> +linuxgeneration 16.164.40.197    2 u   88 1024  377   62.692   -1.160
> 2.007
> +dns3.switch.ca  206.108.0.131    2 u  857 1024  377   27.315    0.778
> 0.483
>
> storage01 | CHANGED | rc=0 >>
>      remote           refid      st t when poll reach   delay   offset
> jitter
>
> ==============================================================================
> +198.161.203.36  128.233.150.93   2 u  121 1024  377   16.069    1.016
> 0.195
> +zero.gotroot.ca 30.114.5.31      2 u  543 1024  377    5.106   -2.462
> 4.923
> *ntp3.torix.ca   .PTP0.           1 u  300 1024  377   54.010    2.421
> 15.182
>
> pcic-backup01 | CHANGED | rc=0 >>
>      remote           refid      st t when poll reach   delay   offset
> jitter
>
> ==============================================================================
> *dns3.switch.ca  206.108.0.131    2 u  983 1024  377   26.990    0.523
> 1.389
> +dns2.switch.ca  206.108.0.131    2 u  689 1024  377   26.975   -0.257
> 0.467
> +64.ip-54-39-23. 214.176.184.39   2 u  909 1024  377   64.262   -0.604
> 6.129
>
> And everything is working on the same version of gluster:
>
> pcic-backup02 | CHANGED | rc=0 >>
> glusterfs 8.3
> pcic-backup01 | CHANGED | rc=0 >>
> glusterfs 8.3
> storage02 | CHANGED | rc=0 >>
> glusterfs 8.3
> storage01 | CHANGED | rc=0 >>
> glusterfs 8.3
> storage03 | CHANGED | rc=0 >>
> glusterfs 8.3
>
> SSH works, and the backup user/group is configured with mountbroker:
>
> [root at storage01 ~]# ssh -i /root/.ssh/id_rsa geoaccount at 10.0.231.81 uname
> -a
> Linux pcic-backup01 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> [root at storage01 ~]# ssh -i /root/.ssh/id_rsa geoaccount at 10.0.231.82 uname
> -a
> Linux pcic-backup02 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>
>
> [root at pcic-backup01 ~]# grep geo /etc/passwd
> geoaccount:x:1000:1000::/home/geoaccount:/bin/bash
> [root at pcic-backup01 ~]# grep geo /etc/group
> geogroup:x:1000:geoaccount
> geoaccount:x:1001:geoaccount
>
> [root at pcic-backup01 ~]# gluster-mountbroker status
>
> +-------------+-------------+---------------------------+--------------+--------------------------+
> |     NODE    | NODE STATUS |         MOUNT ROOT        |    GROUP
> |          USERS           |
>
> +-------------+-------------+---------------------------+--------------+--------------------------+
> | 10.0.231.82 |          UP | /var/mountbroker-root(OK) | geogroup(OK) |
> geoaccount(pcic-backup)  |
> |  localhost  |          UP | /var/mountbroker-root(OK) | geogroup(OK) |
> geoaccount(pcic-backup)  |
>
> +-------------+-------------+---------------------------+--------------+--------------------------+
>
>
>
>
> So, then if I'm going to have to resync, what is the best way to do this?
>
> With delete or delete reset-sync-time ?
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-starting_geo-replication#Deleting_a_Geo-replication_Session
>
>
>
> Erasing the index? So I don't have to transfer the files again that are
> already on the backup?
>
>    -
>    https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/sect-troubleshooting_geo-replication#Synchronization_Is_Not_Complete
>    -
>    https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Geo%20Replication/#best-practices
>
>
>
>
> Is it possible to use the special-sync-mode  option from here:
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-disaster_recovery
>
>
>
> Thoughts?
>
> Thanks,
>  -Matthew
> --
>
> On 3/12/21 3:31 PM, Strahil Nikolov wrote:
>
> Notice: This message was sent from outside the University of Victoria
> email system. Please be cautious with links and sensitive information.
>
> Usually, when I'm stuck - I just start over.
> For example, check the prerequisites:
> - Is ssh available (no firewall blocking)
> - Is time sync enabled (ntp/chrony)
> - Is DNS ok on all hosts (including PTR records)
> - Is the gluster version the same on all nodes (primary & secondary)
>
> Then start over as if the geo rep was never existing. For example , stop
> it and start over with the secondary nodes's checks (mountbroker, user,
> group) .
>
> Most probably somwthing will come up and you will fix it.
>
> In worst case scenario, you will need to clean ip the geo-rep and start
> fresh.
>
>
> Best Regards,
> Strahil Nikolov
>
> On Fri, Mar 12, 2021 at 20:01, Matthew Benstead
> <matthewb at uvic.ca> <matthewb at uvic.ca> wrote:
> Hi Strahil,
>
> Yes, SELinux was put into permissive mode on the secondary nodes as well:
>
> [root at pcic-backup01 ~]# sestatus | egrep -i  "^SELinux status|mode"
> SELinux status:                 enabled
> Current mode:                   permissive
> Mode from config file:          enforcing
>
> [root at pcic-backup02 ~]# sestatus | egrep -i  "^SELinux status|mode"
> SELinux status:                 enabled
> Current mode:                   permissive
> Mode from config file:          enforcing
>
> The secondary server logs didn't show anything interesting:
>
> gsyncd.log:
>
> [2021-03-11 19:15:28.81820] I [resource(slave
> 10.0.231.92/data/storage_c/storage):1116:connect] GLUSTER: Mounting
> gluster volume locally...
> [2021-03-11 19:15:28.101819] I [resource(slave
> 10.0.231.91/data/storage_a/storage):1116:connect] GLUSTER: Mounting
> gluster volume locally...
> [2021-03-11 19:15:28.107012] I [resource(slave
> 10.0.231.93/data/storage_c/storage):1116:connect] GLUSTER: Mounting
> gluster volume locally...
> [2021-03-11 19:15:28.124567] I [resource(slave
> 10.0.231.93/data/storage_b/storage):1116:connect] GLUSTER: Mounting
> gluster volume locally...
> [2021-03-11 19:15:28.128145] I [resource(slave
> 10.0.231.93/data/storage_a/storage):1116:connect] GLUSTER: Mounting
> gluster volume locally...
> [2021-03-11 19:15:29.425739] I [resource(slave
> 10.0.231.93/data/storage_c/storage):1139:connect] GLUSTER: Mounted
> gluster volume [{duration=1.3184}]
> [2021-03-11 19:15:29.427448] I [resource(slave
> 10.0.231.93/data/storage_c/storage):1166:service_loop] GLUSTER: slave
> listening
> [2021-03-11 19:15:29.433340] I [resource(slave
> 10.0.231.93/data/storage_b/storage):1139:connect] GLUSTER: Mounted
> gluster volume [{duration=1.3083}]
> [2021-03-11 19:15:29.434452] I [resource(slave
> 10.0.231.91/data/storage_a/storage):1139:connect] GLUSTER: Mounted
> gluster volume [{duration=1.3321}]
> [2021-03-11 19:15:29.434314] I [resource(slave
> 10.0.231.93/data/storage_b/storage):1166:service_loop] GLUSTER: slave
> listening
> [2021-03-11 19:15:29.435575] I [resource(slave
> 10.0.231.91/data/storage_a/storage):1166:service_loop] GLUSTER: slave
> listening
> [2021-03-11 19:15:29.439769] I [resource(slave
> 10.0.231.92/data/storage_c/storage):1139:connect] GLUSTER: Mounted
> gluster volume [{duration=1.3576}]
> [2021-03-11 19:15:29.440998] I [resource(slave
> 10.0.231.92/data/storage_c/storage):1166:service_loop] GLUSTER: slave
> listening
> [2021-03-11 19:15:29.454745] I [resource(slave
> 10.0.231.93/data/storage_a/storage):1139:connect] GLUSTER: Mounted
> gluster volume [{duration=1.3262}]
> [2021-03-11 19:15:29.456192] I [resource(slave
> 10.0.231.93/data/storage_a/storage):1166:service_loop] GLUSTER: slave
> listening
> [2021-03-11 19:15:32.594865] I [repce(slave
> 10.0.231.92/data/storage_c/storage):96:service_loop] RepceServer:
> terminating on reaching EOF.
> [2021-03-11 19:15:32.607815] I [repce(slave
> 10.0.231.93/data/storage_c/storage):96:service_loop] RepceServer:
> terminating on reaching EOF.
> [2021-03-11 19:15:32.647663] I [repce(slave
> 10.0.231.93/data/storage_b/storage):96:service_loop] RepceServer:
> terminating on reaching EOF.
> [2021-03-11 19:15:32.656280] I [repce(slave
> 10.0.231.91/data/storage_a/storage):96:service_loop] RepceServer:
> terminating on reaching EOF.
> [2021-03-11 19:15:32.668299] I [repce(slave
> 10.0.231.93/data/storage_a/storage):96:service_loop] RepceServer:
> terminating on reaching EOF.
> [2021-03-11 19:15:44.260689] I [resource(slave
> 10.0.231.92/data/storage_c/storage):1116:connect] GLUSTER: Mounting
> gluster volume locally...
> [2021-03-11 19:15:44.271457] I [resource(slave
> 10.0.231.93/data/storage_c/storage):1116:connect] GLUSTER: Mounting
> gluster volume locally...
> [2021-03-11 19:15:44.271883] I [resource(slave
> 10.0.231.93/data/storage_b/storage):1116:connect] GLUSTER: Mounting
> gluster volume locally...
> [2021-03-11 19:15:44.279670] I [resource(slave
> 10.0.231.91/data/storage_a/storage):1116:connect] GLUSTER: Mounting
> gluster volume locally...
> [2021-03-11 19:15:44.284261] I [resource(slave
> 10.0.231.93/data/storage_a/storage):1116:connect] GLUSTER: Mounting
> gluster volume locally...
> [2021-03-11 19:15:45.614280] I [resource(slave
> 10.0.231.93/data/storage_b/storage):1139:connect] GLUSTER: Mounted
> gluster volume [{duration=1.3419}]
> [2021-03-11 19:15:45.615622] I [resource(slave
> 10.0.231.93/data/storage_b/storage):1166:service_loop] GLUSTER: slave
> listening
> [2021-03-11 19:15:45.617986] I [resource(slave
> 10.0.231.93/data/storage_c/storage):1139:connect] GLUSTER: Mounted
> gluster volume [{duration=1.3461}]
> [2021-03-11 19:15:45.618180] I [resource(slave
> 10.0.231.91/data/storage_a/storage):1139:connect] GLUSTER: Mounted
> gluster volume [{duration=1.3380}]
> [2021-03-11 19:15:45.619539] I [resource(slave
> 10.0.231.91/data/storage_a/storage):1166:service_loop] GLUSTER: slave
> listening
> [2021-03-11 19:15:45.618999] I [resource(slave
> 10.0.231.93/data/storage_c/storage):1166:service_loop] GLUSTER: slave
> listening
> [2021-03-11 19:15:45.620843] I [resource(slave
> 10.0.231.93/data/storage_a/storage):1139:connect] GLUSTER: Mounted
> gluster volume [{duration=1.3361}]
> [2021-03-11 19:15:45.621347] I [resource(slave
> 10.0.231.92/data/storage_c/storage):1139:connect] GLUSTER: Mounted
> gluster volume [{duration=1.3604}]
> [2021-03-11 19:15:45.622179] I [resource(slave
> 10.0.231.93/data/storage_a/storage):1166:service_loop] GLUSTER: slave
> listening
> [2021-03-11 19:15:45.622541] I [resource(slave
> 10.0.231.92/data/storage_c/storage):1166:service_loop] GLUSTER: slave
> listening
> [2021-03-11 19:15:47.626054] I [repce(slave
> 10.0.231.91/data/storage_a/storage):96:service_loop] RepceServer:
> terminating on reaching EOF.
> [2021-03-11 19:15:48.778399] I [repce(slave
> 10.0.231.93/data/storage_c/storage):96:service_loop] RepceServer:
> terminating on reaching EOF.
> [2021-03-11 19:15:48.778491] I [repce(slave
> 10.0.231.92/data/storage_c/storage):96:service_loop] RepceServer:
> terminating on reaching EOF.
> [2021-03-11 19:15:48.796854] I [repce(slave
> 10.0.231.93/data/storage_a/storage):96:service_loop] RepceServer:
> terminating on reaching EOF.
> [2021-03-11 19:15:48.800697] I [repce(slave
> 10.0.231.93/data/storage_b/storage):96:service_loop] RepceServer:
> terminating on reaching EOF.
>
> The mnt geo-rep files were also uninteresting:
> [2021-03-11 19:15:28.250150] I [MSGID: 100030] [glusterfsd.c:2689:main]
> 0-/usr/sbin/glusterfs: Started running version [{arg=/usr/sbin/glusterfs},
> {version=8.3}, {cmdlinestr=/usr/sbin/glusterfs --user-map-root=g
> eoaccount --aux-gfid-mount --acl --log-level=INFO
> --log-file=/var/log/glusterfs/geo-replication-slaves/storage_10.0.231.81_pcic-backup/mnt-10.0.231.93-data-storage_b-storage.log
> --volfile-server=localhost --volf
> ile-id=pcic-backup --client-pid=-1
> /var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI}]
> [2021-03-11 19:15:28.253485] I [glusterfsd.c:2424:daemonize] 0-glusterfs:
> Pid of current running process is 157484
> [2021-03-11 19:15:28.267911] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=0}]
> [2021-03-11 19:15:28.267984] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=1}]
> [2021-03-11 19:15:28.268371] I [glusterfsd-mgmt.c:2170:mgmt_getspec_cbk]
> 0-glusterfs: Received list of available volfile servers: 10.0.231.82:24007
> [2021-03-11 19:15:28.271729] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=2}]
> [2021-03-11 19:15:28.271762] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=3}]
> [2021-03-11 19:15:28.272223] I [MSGID: 114020] [client.c:2315:notify]
> 0-pcic-backup-client-0: parent translators are ready, attempting connect on
> transport []
> [2021-03-11 19:15:28.275883] I [MSGID: 114020] [client.c:2315:notify]
> 0-pcic-backup-client-1: parent translators are ready, attempting connect on
> transport []
> [2021-03-11 19:15:28.276154] I [rpc-clnt.c:1975:rpc_clnt_reconfig]
> 0-pcic-backup-client-0: changing port to 49153 (from 0)
> [2021-03-11 19:15:28.276193] I [socket.c:849:__socket_shutdown]
> 0-pcic-backup-client-0: intentional socket shutdown(13)
> Final graph:
> ...
>
> +------------------------------------------------------------------------------+
> [2021-03-11 19:15:28.282144] I [socket.c:849:__socket_shutdown]
> 0-pcic-backup-client-1: intentional socket shutdown(15)
> [2021-03-11 19:15:28.286536] I [MSGID: 114057]
> [client-handshake.c:1128:select_server_supported_programs]
> 0-pcic-backup-client-0: Using Program [{Program-name=GlusterFS 4.x v1},
> {Num=1298437}, {Version=400}]
> [2021-03-11 19:15:28.287208] I [MSGID: 114046]
> [client-handshake.c:857:client_setvolume_cbk] 0-pcic-backup-client-0:
> Connected, attached to remote volume [{conn-name=pcic-backup-client-0},
> {remote_subvol=/data/brick}]
> [2021-03-11 19:15:28.290162] I [MSGID: 114057]
> [client-handshake.c:1128:select_server_supported_programs]
> 0-pcic-backup-client-1: Using Program [{Program-name=GlusterFS 4.x v1},
> {Num=1298437}, {Version=400}]
> [2021-03-11 19:15:28.291122] I [MSGID: 114046]
> [client-handshake.c:857:client_setvolume_cbk] 0-pcic-backup-client-1:
> Connected, attached to remote volume [{conn-name=pcic-backup-client-1},
> {remote_subvol=/data/brick}]
> [2021-03-11 19:15:28.292703] I [fuse-bridge.c:5300:fuse_init]
> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel
> 7.23
> [2021-03-11 19:15:28.292730] I [fuse-bridge.c:5926:fuse_graph_sync]
> 0-fuse: switched to graph 0
> [2021-03-11 19:15:32.809518] I [fuse-bridge.c:6242:fuse_thread_proc]
> 0-fuse: initiating unmount of
> /var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI
> [2021-03-11 19:15:32.810216] W [glusterfsd.c:1439:cleanup_and_exit]
> (-->/lib64/libpthread.so.0(+0x7ea5) [0x7ff56b175ea5]
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55664e67db45]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55664e67d9ab] ) 0-:
> received signum (15), shutting down
> [2021-03-11 19:15:32.810253] I [fuse-bridge.c:7074:fini] 0-fuse:
> Unmounting '/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI'.
> [2021-03-11 19:15:32.810268] I [fuse-bridge.c:7079:fini] 0-fuse: Closing
> fuse connection to '/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI'.
>
>
> I'm really at a loss for where to go from here, it seems like everything
> is set up correctly, and it has been working well through the 7.x minor
> versions, but the jump to 8 has broken something...
>
> There definitely are lots of changelogs on the servers that fit into the
> timeframe..... I haven't made any writes to the source volume.... do you
> think that's the problem? That it needs some new changelog info to sync?
> I had been holding off making any writes in case I needed to go back to
> Gluster7.9 - not sure if that's really a good option or not.
>
> [root at storage01 changelogs]# for dirs in {a,b,c}; do echo
> "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh
> /data/storage_$dirs/storage/.glusterfs/changelogs | head; echo ""; done
> /data/storage_a/storage/.glusterfs/changelogs
> total 16G
> drw-------. 3 root root   24 Mar  9 11:34 2021
> -rw-r--r--. 1 root root   51 Mar 12 09:50 CHANGELOG
> -rw-r--r--. 1 root root  13K Aug 13  2020 CHANGELOG.1597343197
> -rw-r--r--. 1 root root  51K Aug 13  2020 CHANGELOG.1597343212
> -rw-r--r--. 1 root root  86K Aug 13  2020 CHANGELOG.1597343227
> -rw-r--r--. 1 root root  99K Aug 13  2020 CHANGELOG.1597343242
> -rw-r--r--. 1 root root  69K Aug 13  2020 CHANGELOG.1597343257
> -rw-r--r--. 1 root root  69K Aug 13  2020 CHANGELOG.1597343272
> -rw-r--r--. 1 root root  72K Aug 13  2020 CHANGELOG.1597343287
>
> /data/storage_b/storage/.glusterfs/changelogs
> total 3.3G
> drw-------. 3 root root   24 Mar  9 11:34 2021
> -rw-r--r--. 1 root root   51 Mar 12 09:50 CHANGELOG
> -rw-r--r--. 1 root root  13K Aug 13  2020 CHANGELOG.1597343197
> -rw-r--r--. 1 root root  53K Aug 13  2020 CHANGELOG.1597343212
> -rw-r--r--. 1 root root  89K Aug 13  2020 CHANGELOG.1597343227
> -rw-r--r--. 1 root root  89K Aug 13  2020 CHANGELOG.1597343242
> -rw-r--r--. 1 root root  69K Aug 13  2020 CHANGELOG.1597343257
> -rw-r--r--. 1 root root  71K Aug 13  2020 CHANGELOG.1597343272
> -rw-r--r--. 1 root root  86K Aug 13  2020 CHANGELOG.1597343287
>
> /data/storage_c/storage/.glusterfs/changelogs
> total 9.6G
> drw-------. 3 root root   16 Mar  9 11:34 2021
> -rw-r--r--. 1 root root   51 Mar 12 09:50 CHANGELOG
> -rw-r--r--. 1 root root  16K Aug 13  2020 CHANGELOG.1597343199
> -rw-r--r--. 1 root root  71K Aug 13  2020 CHANGELOG.1597343214
> -rw-r--r--. 1 root root 122K Aug 13  2020 CHANGELOG.1597343229
> -rw-r--r--. 1 root root  73K Aug 13  2020 CHANGELOG.1597343244
> -rw-r--r--. 1 root root 100K Aug 13  2020 CHANGELOG.1597343259
> -rw-r--r--. 1 root root  95K Aug 13  2020 CHANGELOG.1597343274
> -rw-r--r--. 1 root root  92K Aug 13  2020 CHANGELOG.1597343289
>
> [root at storage01 changelogs]# for dirs in {a,b,c}; do echo
> "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh
> /data/storage_$dirs/storage/.glusterfs/changelogs | tail; echo ""; done
> /data/storage_a/storage/.glusterfs/changelogs
> -rw-r--r--. 1 root root   92 Mar  1 21:33 CHANGELOG.1614663193
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663731
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663760
> -rw-r--r--. 1 root root  511 Mar  1 21:47 CHANGELOG.1614664043
> -rw-r--r--. 1 root root  536 Mar  1 21:48 CHANGELOG.1614664101
> -rw-r--r--. 1 root root 2.8K Mar  1 21:48 CHANGELOG.1614664116
> -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666061
> -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666554
> drw-------. 2 root root   10 May  7  2020 csnap
> drw-------. 2 root root   38 Aug 13  2020 htime
>
> /data/storage_b/storage/.glusterfs/changelogs
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663731
> -rw-r--r--. 1 root root  480 Mar  1 21:42 CHANGELOG.1614663745
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663760
> -rw-r--r--. 1 root root  524 Mar  1 21:47 CHANGELOG.1614664043
> -rw-r--r--. 1 root root  495 Mar  1 21:48 CHANGELOG.1614664100
> -rw-r--r--. 1 root root 1.6K Mar  1 21:48 CHANGELOG.1614664114
> -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666060
> -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666553
> drw-------. 2 root root   10 May  7  2020 csnap
> drw-------. 2 root root   38 Aug 13  2020 htime
>
> /data/storage_c/storage/.glusterfs/changelogs
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663738
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663753
> -rw-r--r--. 1 root root  395 Mar  1 21:47 CHANGELOG.1614664051
> -rw-r--r--. 1 root root  316 Mar  1 21:48 CHANGELOG.1614664094
> -rw-r--r--. 1 root root 1.2K Mar  1 21:48 CHANGELOG.1614664109
> -rw-r--r--. 1 root root  174 Mar  1 21:48 CHANGELOG.1614664123
> -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666061
> -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666553
> drw-------. 2 root root    6 May  7  2020 csnap
> drw-------. 2 root root   30 Aug 13  2020 htime
>
> [root at storage02 ~]# for dirs in {a,b,c}; do echo
> "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh
> /data/storage_$dirs/storage/.glusterfs/changelogs | head; echo ""; done
> /data/storage_a/storage/.glusterfs/changelogs
> total 9.6G
> drw-------. 3 root root   24 Mar  9 11:34 2021
> -rw-r--r--. 1 root root   51 Mar 12 09:50 CHANGELOG
> -rw-r--r--. 1 root root 4.2K Aug 13  2020 CHANGELOG.1597343193
> -rw-r--r--. 1 root root  32K Aug 13  2020 CHANGELOG.1597343208
> -rw-r--r--. 1 root root 107K Aug 13  2020 CHANGELOG.1597343223
> -rw-r--r--. 1 root root 120K Aug 13  2020 CHANGELOG.1597343238
> -rw-r--r--. 1 root root  72K Aug 13  2020 CHANGELOG.1597343253
> -rw-r--r--. 1 root root 111K Aug 13  2020 CHANGELOG.1597343268
> -rw-r--r--. 1 root root  91K Aug 13  2020 CHANGELOG.1597343283
>
> /data/storage_b/storage/.glusterfs/changelogs
> total 16G
> drw-------. 3 root root   24 Mar  9 11:34 2021
> -rw-r--r--. 1 root root   51 Mar 12 09:50 CHANGELOG
> -rw-r--r--. 1 root root 3.9K Aug 13  2020 CHANGELOG.1597343193
> -rw-r--r--. 1 root root  35K Aug 13  2020 CHANGELOG.1597343208
> -rw-r--r--. 1 root root  85K Aug 13  2020 CHANGELOG.1597343223
> -rw-r--r--. 1 root root 103K Aug 13  2020 CHANGELOG.1597343238
> -rw-r--r--. 1 root root  70K Aug 13  2020 CHANGELOG.1597343253
> -rw-r--r--. 1 root root  72K Aug 13  2020 CHANGELOG.1597343268
> -rw-r--r--. 1 root root  73K Aug 13  2020 CHANGELOG.1597343283
>
> /data/storage_c/storage/.glusterfs/changelogs
> total 3.3G
> drw-------. 3 root root   16 Mar  9 11:34 2021
> -rw-r--r--. 1 root root   51 Mar 12 09:51 CHANGELOG
> -rw-r--r--. 1 root root  21K Aug 13  2020 CHANGELOG.1597343202
> -rw-r--r--. 1 root root  75K Aug 13  2020 CHANGELOG.1597343217
> -rw-r--r--. 1 root root  92K Aug 13  2020 CHANGELOG.1597343232
> -rw-r--r--. 1 root root  77K Aug 13  2020 CHANGELOG.1597343247
> -rw-r--r--. 1 root root  66K Aug 13  2020 CHANGELOG.1597343262
> -rw-r--r--. 1 root root  84K Aug 13  2020 CHANGELOG.1597343277
> -rw-r--r--. 1 root root  81K Aug 13  2020 CHANGELOG.1597343292
>
> [root at storage02 ~]# for dirs in {a,b,c}; do echo
> "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh
> /data/storage_$dirs/storage/.glusterfs/changelogs | tail; echo ""; done
> /data/storage_a/storage/.glusterfs/changelogs
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663734
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663749
> -rw-r--r--. 1 root root  395 Mar  1 21:47 CHANGELOG.1614664052
> -rw-r--r--. 1 root root  316 Mar  1 21:48 CHANGELOG.1614664096
> -rw-r--r--. 1 root root 1.2K Mar  1 21:48 CHANGELOG.1614664111
> -rw-r--r--. 1 root root  174 Mar  1 21:48 CHANGELOG.1614664126
> -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666056
> -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666560
> drw-------. 2 root root   10 May  7  2020 csnap
> drw-------. 2 root root   38 Aug 13  2020 htime
>
> /data/storage_b/storage/.glusterfs/changelogs
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663735
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663749
> -rw-r--r--. 1 root root  511 Mar  1 21:47 CHANGELOG.1614664052
> -rw-r--r--. 1 root root  316 Mar  1 21:48 CHANGELOG.1614664096
> -rw-r--r--. 1 root root 1.8K Mar  1 21:48 CHANGELOG.1614664111
> -rw-r--r--. 1 root root 1.4K Mar  1 21:48 CHANGELOG.1614664126
> -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666060
> -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666556
> drw-------. 2 root root   10 May  7  2020 csnap
> drw-------. 2 root root   38 Aug 13  2020 htime
>
> /data/storage_c/storage/.glusterfs/changelogs
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663738
> -rw-r--r--. 1 root root  521 Mar  1 21:42 CHANGELOG.1614663752
> -rw-r--r--. 1 root root  524 Mar  1 21:47 CHANGELOG.1614664042
> -rw-r--r--. 1 root root   92 Mar  1 21:47 CHANGELOG.1614664057
> -rw-r--r--. 1 root root  536 Mar  1 21:48 CHANGELOG.1614664102
> -rw-r--r--. 1 root root 1.6K Mar  1 21:48 CHANGELOG.1614664117
> -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666057
> -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666550
> drw-------. 2 root root    6 May  7  2020 csnap
> drw-------. 2 root root   30 Aug 13  2020 htime
>
>
> [root at storage03 ~]# for dirs in {a,b,c}; do echo
> "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh
> /data/storage_$dirs/storage/.glusterfs/changelogs | head; echo ""; done
> /data/storage_a/storage/.glusterfs/changelogs
> total 3.4G
> drw-------. 3 root root   24 Mar  9 11:34 2021
> -rw-r--r--. 1 root root   51 Mar 12 09:50 CHANGELOG
> -rw-r--r--. 1 root root  19K Aug 13  2020 CHANGELOG.1597343201
> -rw-r--r--. 1 root root  66K Aug 13  2020 CHANGELOG.1597343215
> -rw-r--r--. 1 root root  91K Aug 13  2020 CHANGELOG.1597343230
> -rw-r--r--. 1 root root  82K Aug 13  2020 CHANGELOG.1597343245
> -rw-r--r--. 1 root root  64K Aug 13  2020 CHANGELOG.1597343259
> -rw-r--r--. 1 root root  75K Aug 13  2020 CHANGELOG.1597343274
> -rw-r--r--. 1 root root  81K Aug 13  2020 CHANGELOG.1597343289
>
> /data/storage_b/storage/.glusterfs/changelogs
> total 9.6G
> drw-------. 3 root root   24 Mar  9 11:34 2021
> -rw-r--r--. 1 root root   51 Mar 12 09:51 CHANGELOG
> -rw-r--r--. 1 root root  19K Aug 13  2020 CHANGELOG.1597343201
> -rw-r--r--. 1 root root  80K Aug 13  2020 CHANGELOG.1597343215
> -rw-r--r--. 1 root root 119K Aug 13  2020 CHANGELOG.1597343230
> -rw-r--r--. 1 root root  65K Aug 13  2020 CHANGELOG.1597343244
> -rw-r--r--. 1 root root 100K Aug 13  2020 CHANGELOG.1597343259
> -rw-r--r--. 1 root root  95K Aug 13  2020 CHANGELOG.1597343274
> -rw-r--r--. 1 root root  92K Aug 13  2020 CHANGELOG.1597343289
>
> /data/storage_c/storage/.glusterfs/changelogs
> total 16G
> drw-------. 3 root root   16 Mar  9 11:34 2021
> -rw-r--r--. 1 root root   51 Mar 12 09:51 CHANGELOG
> -rw-r--r--. 1 root root 3.9K Aug 13  2020 CHANGELOG.1597343193
> -rw-r--r--. 1 root root  35K Aug 13  2020 CHANGELOG.1597343208
> -rw-r--r--. 1 root root  85K Aug 13  2020 CHANGELOG.1597343223
> -rw-r--r--. 1 root root 103K Aug 13  2020 CHANGELOG.1597343238
> -rw-r--r--. 1 root root  70K Aug 13  2020 CHANGELOG.1597343253
> -rw-r--r--. 1 root root  71K Aug 13  2020 CHANGELOG.1597343268
> -rw-r--r--. 1 root root  73K Aug 13  2020 CHANGELOG.1597343283
>
> [root at storage03 ~]# for dirs in {a,b,c}; do echo
> "/data/storage_$dirs/storage/.glusterfs/changelogs"; ls -lh
> /data/storage_$dirs/storage/.glusterfs/changelogs | tail; echo ""; done
> /data/storage_a/storage/.glusterfs/changelogs
> -rw-r--r--. 1 root root   92 Mar  1 21:33 CHANGELOG.1614663183
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663740
> -rw-r--r--. 1 root root  521 Mar  1 21:42 CHANGELOG.1614663755
> -rw-r--r--. 1 root root  524 Mar  1 21:47 CHANGELOG.1614664049
> -rw-r--r--. 1 root root 1.9K Mar  1 21:48 CHANGELOG.1614664106
> -rw-r--r--. 1 root root  174 Mar  1 21:48 CHANGELOG.1614664121
> -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666051
> -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666559
> drw-------. 2 root root   10 May  7  2020 csnap
> drw-------. 2 root root   38 Aug 13  2020 htime
>
> /data/storage_b/storage/.glusterfs/changelogs
> -rw-r--r--. 1 root root  474 Mar  1 21:33 CHANGELOG.1614663182
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663739
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663753
> -rw-r--r--. 1 root root  395 Mar  1 21:47 CHANGELOG.1614664049
> -rw-r--r--. 1 root root 1.4K Mar  1 21:48 CHANGELOG.1614664106
> -rw-r--r--. 1 root root  174 Mar  1 21:48 CHANGELOG.1614664120
> -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666063
> -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666557
> drw-------. 2 root root   10 May  7  2020 csnap
> drw-------. 2 root root   38 Aug 13  2020 htime
>
> /data/storage_c/storage/.glusterfs/changelogs
> -rw-r--r--. 1 root root  468 Mar  1 21:33 CHANGELOG.1614663183
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663740
> -rw-r--r--. 1 root root   92 Mar  1 21:42 CHANGELOG.1614663754
> -rw-r--r--. 1 root root  511 Mar  1 21:47 CHANGELOG.1614664048
> -rw-r--r--. 1 root root 2.0K Mar  1 21:48 CHANGELOG.1614664105
> -rw-r--r--. 1 root root 1.4K Mar  1 21:48 CHANGELOG.1614664120
> -rw-r--r--. 1 root root   92 Mar  1 22:20 CHANGELOG.1614666063
> -rw-r--r--. 1 root root   92 Mar  1 22:29 CHANGELOG.1614666556
> drw-------. 2 root root    6 May  7  2020 csnap
> drw-------. 2 root root   30 Aug 13  2020 htime
>
> Thanks,
>  -Matthew
>
> --
> Matthew Benstead
> System Administrator
> Pacific Climate Impacts Consortium <https://pacificclimate.org/>
> University of Victoria, UH1
> PO Box 1800, STN CSC
> Victoria, BC, V8W 2Y2
> Phone: +1-250-721-8432
> Email: matthewb at uvic.ca
> On 3/11/21 11:37 PM, Strahil Nikolov wrote:
>
> Notice: This message was sent from outside the University of Victoria
> email system. Please be cautious with links and sensitive information.
>
> Have you checked the secondary volume nodes' logs & SELINUX status ?
>
> Best Regards,
> Strahil Nikolov
>
> On Thu, Mar 11, 2021 at 21:36, Matthew Benstead
> <matthewb at uvic.ca> <matthewb at uvic.ca> wrote:
> Hi Strahil,
>
> It looks like perhaps the changelog_log_level and log_level options? I've
> set them to debug:
>
> [root at storage01 ~]# gluster volume geo-replication storage
> geoaccount at 10.0.231.81::pcic-backup config | egrep -i "log_level"
> changelog_log_level:INFO
> cli_log_level:INFO
> gluster_log_level:INFO
> log_level:INFO
> slave_gluster_log_level:INFO
> slave_log_level:INFO
>
> [root at storage01 ~]# gluster volume geo-replication storage
> geoaccount at 10.0.231.81::pcic-backup config changelog_log_level DEBUG
> geo-replication config updated successfully
>
> [root at storage01 ~]# gluster volume geo-replication storage
> geoaccount at 10.0.231.81::pcic-backup config log_level DEBUG
> geo-replication config updated successfully
>
>
> Then I restarted geo-replication:
>
> [root at storage01 ~]# gluster volume geo-replication storage
> geoaccount at 10.0.231.81::pcic-backup start
> Starting geo-replication session between storage &
> geoaccount at 10.0.231.81::pcic-backup has been successful
> [root at storage01 ~]# gluster volume geo-replication status
>
> MASTER NODE    MASTER VOL    MASTER BRICK               SLAVE USER
> SLAVE                                        SLAVE NODE
> STATUS             CRAWL STATUS    LAST_SYNCED
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 10.0.231.91    storage       /data/storage_a/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.91    storage       /data/storage_c/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.91    storage       /data/storage_b/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.92    storage       /data/storage_b/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.92    storage       /data/storage_a/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.92    storage       /data/storage_c/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.93    storage       /data/storage_c/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.93    storage       /data/storage_b/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> 10.0.231.93    storage       /data/storage_a/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A
> Initializing...    N/A             N/A
> [root at storage01 ~]# gluster volume geo-replication status
>
> MASTER NODE    MASTER VOL    MASTER BRICK               SLAVE USER
> SLAVE                                        SLAVE NODE    STATUS    CRAWL
> STATUS    LAST_SYNCED
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 10.0.231.91    storage       /data/storage_a/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.91    storage       /data/storage_c/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.91    storage       /data/storage_b/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.92    storage       /data/storage_b/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.92    storage       /data/storage_a/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.92    storage       /data/storage_c/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.93    storage       /data/storage_c/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.93    storage       /data/storage_b/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
> 10.0.231.93    storage       /data/storage_a/storage    geoaccount
> ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty
> N/A             N/A
>
> [root at storage01 ~]# gluster volume geo-replication storage
> geoaccount at 10.0.231.81::pcic-backup stop
> Stopping geo-replication session between storage &
> geoaccount at 10.0.231.81::pcic-backup has been successful
>
>
> The changelogs didn't really show anything new around changelog selection:
>
> [root at storage01 storage_10.0.231.81_pcic-backup]# cat
> changes-data-storage_a-storage.log | egrep "2021-03-11"
> [2021-03-11 19:15:30.552889] I [MSGID: 132028]
> [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog:
> Registering brick [{brick=/data/storage_a/storage}, {notify_filter=1}]
> [2021-03-11 19:15:30.552893] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=0}]
> [2021-03-11 19:15:30.552894] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=1}]
> [2021-03-11 19:15:30.553633] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=3}]
> [2021-03-11 19:15:30.553634] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=2}]
> [2021-03-11 19:15:30.554236] D [rpcsvc.c:2831:rpcsvc_init] 0-rpc-service:
> RPC service inited.
> [2021-03-11 19:15:30.554403] D [rpcsvc.c:2342:rpcsvc_program_register]
> 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1,
> Port: 0
> [2021-03-11 19:15:30.554420] D [rpc-transport.c:278:rpc_transport_load]
> 0-rpc-transport: attempt to load file
> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
> [2021-03-11 19:15:30.554933] D [socket.c:4485:socket_init]
> 0-socket.gfchangelog: disabling nodelay
> [2021-03-11 19:15:30.554944] D [socket.c:4523:socket_init]
> 0-socket.gfchangelog: Configured transport.tcp-user-timeout=42
> [2021-03-11 19:15:30.554949] D [socket.c:4543:socket_init]
> 0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9
> [2021-03-11 19:15:30.555002] I [socket.c:929:__socket_server_bind]
> 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 23
> [2021-03-11 19:15:30.555324] D [rpcsvc.c:2342:rpcsvc_program_register]
> 0-rpc-service: New program registered: LIBGFCHANGELOG REBORP, Num:
> 1886350951, Ver: 1, Port: 0
> [2021-03-11 19:15:30.555345] D [rpc-clnt.c:1020:rpc_clnt_connection_init]
> 0-gfchangelog: defaulting frame-timeout to 30mins
> [2021-03-11 19:15:30.555351] D [rpc-clnt.c:1032:rpc_clnt_connection_init]
> 0-gfchangelog: disable ping-timeout
> [2021-03-11 19:15:30.555358] D [rpc-transport.c:278:rpc_transport_load]
> 0-rpc-transport: attempt to load file
> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
> [2021-03-11 19:15:30.555399] D [socket.c:4485:socket_init] 0-gfchangelog:
> disabling nodelay
> [2021-03-11 19:15:30.555406] D [socket.c:4523:socket_init] 0-gfchangelog:
> Configured transport.tcp-user-timeout=42
> [2021-03-11 19:15:32.555711] D [rpc-clnt-ping.c:298:rpc_clnt_start_ping]
> 0-gfchangelog: ping timeout is 0, returning
> [2021-03-11 19:15:32.572157] I [MSGID: 132035]
> [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting
> historical changelogs [{start=1614666553}, {end=1615490132}]
> [2021-03-11 19:15:32.572436] I [MSGID: 132019]
> [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog:
> changelogs min max [{min=1597342860}, {max=1615490121},
> {total_changelogs=1256897}]
> [2021-03-11 19:15:32.621244] E [MSGID: 132009]
> [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong
> result [{for=end}, {start=1615490121}, {idx=1256896}]
> [2021-03-11 19:15:46.733182] I [MSGID: 132028]
> [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog:
> Registering brick [{brick=/data/storage_a/storage}, {notify_filter=1}]
> [2021-03-11 19:15:46.733316] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=0}]
> [2021-03-11 19:15:46.733348] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=1}]
> [2021-03-11 19:15:46.734031] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=2}]
> [2021-03-11 19:15:46.734085] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=3}]
> [2021-03-11 19:15:46.734591] D [rpcsvc.c:2831:rpcsvc_init] 0-rpc-service:
> RPC service inited.
> [2021-03-11 19:15:46.734755] D [rpcsvc.c:2342:rpcsvc_program_register]
> 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1,
> Port: 0
> [2021-03-11 19:15:46.734772] D [rpc-transport.c:278:rpc_transport_load]
> 0-rpc-transport: attempt to load file
> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
> [2021-03-11 19:15:46.735256] D [socket.c:4485:socket_init]
> 0-socket.gfchangelog: disabling nodelay
> [2021-03-11 19:15:46.735266] D [socket.c:4523:socket_init]
> 0-socket.gfchangelog: Configured transport.tcp-user-timeout=42
> [2021-03-11 19:15:46.735271] D [socket.c:4543:socket_init]
> 0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9
> [2021-03-11 19:15:46.735325] I [socket.c:929:__socket_server_bind]
> 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 21
> [2021-03-11 19:15:46.735704] D [rpcsvc.c:2342:rpcsvc_program_register]
> 0-rpc-service: New program registered: LIBGFCHANGELOG REBORP, Num:
> 1886350951, Ver: 1, Port: 0
> [2021-03-11 19:15:46.735721] D [rpc-clnt.c:1020:rpc_clnt_connection_init]
> 0-gfchangelog: defaulting frame-timeout to 30mins
> [2021-03-11 19:15:46.735726] D [rpc-clnt.c:1032:rpc_clnt_connection_init]
> 0-gfchangelog: disable ping-timeout
> [2021-03-11 19:15:46.735733] D [rpc-transport.c:278:rpc_transport_load]
> 0-rpc-transport: attempt to load file
> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
> [2021-03-11 19:15:46.735771] D [socket.c:4485:socket_init] 0-gfchangelog:
> disabling nodelay
> [2021-03-11 19:15:46.735778] D [socket.c:4523:socket_init] 0-gfchangelog:
> Configured transport.tcp-user-timeout=42
> [2021-03-11 19:15:47.618464] D [rpc-clnt-ping.c:298:rpc_clnt_start_ping]
> 0-gfchangelog: ping timeout is 0, returning
>
>
> [root at storage01 storage_10.0.231.81_pcic-backup]# cat
> changes-data-storage_b-storage.log | egrep "2021-03-11"
> [2021-03-11 19:15:30.611457] I [MSGID: 132028]
> [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog:
> Registering brick [{brick=/data/storage_b/storage}, {notify_filter=1}]
> [2021-03-11 19:15:30.611574] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=1}]
> [2021-03-11 19:15:30.611641] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=3}]
> [2021-03-11 19:15:30.611645] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=2}]
> [2021-03-11 19:15:30.612325] D [rpcsvc.c:2831:rpcsvc_init] 0-rpc-service:
> RPC service inited.
> [2021-03-11 19:15:30.612488] D [rpcsvc.c:2342:rpcsvc_program_register]
> 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1,
> Port: 0
> [2021-03-11 19:15:30.612507] D [rpc-transport.c:278:rpc_transport_load]
> 0-rpc-transport: attempt to load file
> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
> [2021-03-11 19:15:30.613005] D [socket.c:4485:socket_init]
> 0-socket.gfchangelog: disabling nodelay
> [2021-03-11 19:15:30.613130] D [socket.c:4523:socket_init]
> 0-socket.gfchangelog: Configured transport.tcp-user-timeout=42
> [2021-03-11 19:15:30.613142] D [socket.c:4543:socket_init]
> 0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9
> [2021-03-11 19:15:30.613208] I [socket.c:929:__socket_server_bind]
> 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 22
> [2021-03-11 19:15:30.613545] D [rpcsvc.c:2342:rpcsvc_program_register]
> 0-rpc-service: New program registered: LIBGFCHANGELOG REBORP, Num:
> 1886350951, Ver: 1, Port: 0
> [2021-03-11 19:15:30.613567] D [rpc-clnt.c:1020:rpc_clnt_connection_init]
> 0-gfchangelog: defaulting frame-timeout to 30mins
> [2021-03-11 19:15:30.613574] D [rpc-clnt.c:1032:rpc_clnt_connection_init]
> 0-gfchangelog: disable ping-timeout
> [2021-03-11 19:15:30.613582] D [rpc-transport.c:278:rpc_transport_load]
> 0-rpc-transport: attempt to load file
> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
> [2021-03-11 19:15:30.613637] D [socket.c:4485:socket_init] 0-gfchangelog:
> disabling nodelay
> [2021-03-11 19:15:30.613654] D [socket.c:4523:socket_init] 0-gfchangelog:
> Configured transport.tcp-user-timeout=42
> [2021-03-11 19:15:32.614273] D [rpc-clnt-ping.c:298:rpc_clnt_start_ping]
> 0-gfchangelog: ping timeout is 0, returning
> [2021-03-11 19:15:32.643628] I [MSGID: 132035]
> [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting
> historical changelogs [{start=1614666552}, {end=1615490132}]
> [2021-03-11 19:15:32.643716] I [MSGID: 132019]
> [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog:
> changelogs min max [{min=1597342860}, {max=1615490123},
> {total_changelogs=1264296}]
> [2021-03-11 19:15:32.700397] E [MSGID: 132009]
> [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong
> result [{for=end}, {start=1615490123}, {idx=1264295}]
> [2021-03-11 19:15:46.832322] I [MSGID: 132028]
> [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog:
> Registering brick [{brick=/data/storage_b/storage}, {notify_filter=1}]
> [2021-03-11 19:15:46.832394] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=0}]
> [2021-03-11 19:15:46.832465] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=1}]
> [2021-03-11 19:15:46.832531] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=2}]
> [2021-03-11 19:15:46.833086] I [MSGID: 101190]
> [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index [{index=3}]
> [2021-03-11 19:15:46.833648] D [rpcsvc.c:2831:rpcsvc_init] 0-rpc-service:
> RPC service inited.
> [2021-03-11 19:15:46.833817] D [rpcsvc.c:2342:rpcsvc_program_register]
> 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1,
> Port: 0
> [2021-03-11 19:15:46.833835] D [rpc-transport.c:278:rpc_transport_load]
> 0-rpc-transport: attempt to load file
> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
> [2021-03-11 19:15:46.834368] D [socket.c:4485:socket_init]
> 0-socket.gfchangelog: disabling nodelay
> [2021-03-11 19:15:46.834380] D [socket.c:4523:socket_init]
> 0-socket.gfchangelog: Configured transport.tcp-user-timeout=42
> [2021-03-11 19:15:46.834386] D [socket.c:4543:socket_init]
> 0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9
> [2021-03-11 19:15:46.834441] I [socket.c:929:__socket_server_bind]
> 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 23
> [2021-03-11 19:15:46.834768] D [rpcsvc.c:2342:rpcsvc_program_register]
> 0-rpc-service: New program registered: LIBGFCHANGELOG REBORP, Num:
> 1886350951, Ver: 1, Port: 0
> [2021-03-11 19:15:46.834789] D [rpc-clnt.c:1020:rpc_clnt_connection_init]
> 0-gfchangelog: defaulting frame-timeout to 30mins
> [2021-03-11 19:15:46.834795] D [rpc-clnt.c:1032:rpc_clnt_connection_init]
> 0-gfchangelog: disable ping-timeout
> [2021-03-11 19:15:46.834802] D [rpc-transport.c:278:rpc_transport_load]
> 0-rpc-transport: attempt to load file
> /usr/lib64/glusterfs/8.3/rpc-transport/socket.so
> [2021-03-11 19:15:46.834845] D [socket.c:4485:socket_init] 0-gfchangelog:
> disabling nodelay
> [2021-03-11 19:15:46.834853] D [socket.c:4523:socket_init] 0-gfchangelog:
> Configured transport.tcp-user-timeout=42
> [2021-03-11 19:15:47.618476] D [rpc-clnt-ping.c:298:rpc_clnt_start_ping]
> 0-gfchangelog: ping timeout is 0, returning
>
>
> gsyncd logged a lot but I'm not sure if it's helpful:
>
> [2021-03-11 19:15:00.41898] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:21.551302] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:21.631470] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:21.718386] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:21.804991] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:26.203999] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:26.284775] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:26.573355] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:26.653752] D [gsyncd(monitor):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:26.756994] D [monitor(monitor):304:distribute] <top>:
> master bricks: [{'host': '10.0.231.91', 'uuid':
> 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir': '/data/storage_a/storage'},
> {'host': '10.0.2
> 31.92', 'uuid': 'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d', 'dir':
> '/data/storage_b/storage'}, {'host': '10.0.231.93', 'uuid':
> '8b28b331-3780-46bc-9da3-fb27de4ab57b', 'dir': '/data/storage_c/storage'},
> {'host': '10.
> 0.231.92', 'uuid': 'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d', 'dir':
> '/data/storage_a/storage'}, {'host': '10.0.231.93', 'uuid':
> '8b28b331-3780-46bc-9da3-fb27de4ab57b', 'dir': '/data/storage_b/storage'},
> {'host': '
> 10.0.231.91', 'uuid': 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir':
> '/data/storage_c/storage'}, {'host': '10.0.231.93', 'uuid':
> '8b28b331-3780-46bc-9da3-fb27de4ab57b', 'dir': '/data/storage_a/storage'},
> {'host'
> : '10.0.231.91', 'uuid': 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir':
> '/data/storage_b/storage'}, {'host': '10.0.231.92', 'uuid':
> 'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d', 'dir': '/data/storage_c/storage'}]
> [2021-03-11 19:15:26.757252] D [monitor(monitor):314:distribute] <top>:
> slave SSH gateway: geoaccount at 10.0.231.81
> [2021-03-11 19:15:27.416235] D [monitor(monitor):334:distribute] <top>:
> slave bricks: [{'host': '10.0.231.81', 'uuid':
> 'b88dea4f-31ec-416a-9110-3ccdc3910acd', 'dir': '/data/brick'}, {'host':
> '10.0.231.82', 'uuid
> ': 'be50a8de-3934-4fee-a80d-8e2e99017902', 'dir': '/data/brick'}]
> [2021-03-11 19:15:27.416825] D [syncdutils(monitor):932:is_hot] Volinfo:
> brickpath: '10.0.231.91:/data/storage_a/storage'
> [2021-03-11 19:15:27.417273] D [syncdutils(monitor):932:is_hot] Volinfo:
> brickpath: '10.0.231.91:/data/storage_c/storage'
> [2021-03-11 19:15:27.417515] D [syncdutils(monitor):932:is_hot] Volinfo:
> brickpath: '10.0.231.91:/data/storage_b/storage'
> [2021-03-11 19:15:27.417763] D [monitor(monitor):348:distribute] <top>:
> worker specs: [({'host': '10.0.231.91', 'uuid':
> 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir': '/data/storage_a/storage'},
> ('geoaccount at 10.
> 0.231.81', 'b88dea4f-31ec-416a-9110-3ccdc3910acd'), '1', False), ({'host':
> '10.0.231.91', 'uuid': 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir':
> '/data/storage_c/storage'}, ('geoaccount at 10.0.231.82', 'be50a8de-3
> 934-4fee-a80d-8e2e99017902'), '2', False), ({'host': '10.0.231.91',
> 'uuid': 'afc24654-2887-41f6-a9c2-8e835de243b6', 'dir':
> '/data/storage_b/storage'}, ('geoaccount at 10.0.231.82',
> 'be50a8de-3934-4fee-a80d-8e2e9901
> 7902'), '3', False)]
> [2021-03-11 19:15:27.425009] I [monitor(monitor):160:monitor] Monitor:
> starting gsyncd worker [{brick=/data/storage_c/storage},
> {slave_node=10.0.231.82}]
> [2021-03-11 19:15:27.426764] I [monitor(monitor):160:monitor] Monitor:
> starting gsyncd worker [{brick=/data/storage_b/storage},
> {slave_node=10.0.231.82}]
> [2021-03-11 19:15:27.429208] I [monitor(monitor):160:monitor] Monitor:
> starting gsyncd worker [{brick=/data/storage_a/storage},
> {slave_node=10.0.231.81}]
> [2021-03-11 19:15:27.432280] D [monitor(monitor):195:monitor] Monitor:
> Worker would mount volume privately
> [2021-03-11 19:15:27.434195] D [monitor(monitor):195:monitor] Monitor:
> Worker would mount volume privately
> [2021-03-11 19:15:27.436584] D [monitor(monitor):195:monitor] Monitor:
> Worker would mount volume privately
> [2021-03-11 19:15:27.478806] D [gsyncd(worker
> /data/storage_c/storage):303:main] <top>: Using session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:27.478852] D [gsyncd(worker
> /data/storage_b/storage):303:main] <top>: Using session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:27.480104] D [gsyncd(worker
> /data/storage_a/storage):303:main] <top>: Using session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:27.500456] I [resource(worker
> /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH
> connection between master and slave...
> [2021-03-11 19:15:27.501375] I [resource(worker
> /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH
> connection between master and slave...
> [2021-03-11 19:15:27.502003] I [resource(worker
> /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH
> connection between master and slave...
> [2021-03-11 19:15:27.525511] D [repce(worker
> /data/storage_a/storage):195:push] RepceClient: call
> 192117:140572692309824:1615490127.53 __repce_version__() ...
> [2021-03-11 19:15:27.525582] D [repce(worker
> /data/storage_b/storage):195:push] RepceClient: call
> 192115:139891296405312:1615490127.53 __repce_version__() ...
> [2021-03-11 19:15:27.526089] D [repce(worker
> /data/storage_c/storage):195:push] RepceClient: call
> 192114:140388828780352:1615490127.53 __repce_version__() ...
> [2021-03-11 19:15:29.435985] D [repce(worker
> /data/storage_a/storage):215:__call__] RepceClient: call
> 192117:140572692309824:1615490127.53 __repce_version__ -> 1.0
> [2021-03-11 19:15:29.436213] D [repce(worker
> /data/storage_a/storage):195:push] RepceClient: call
> 192117:140572692309824:1615490129.44 version() ...
> [2021-03-11 19:15:29.437136] D [repce(worker
> /data/storage_a/storage):215:__call__] RepceClient: call
> 192117:140572692309824:1615490129.44 version -> 1.0
> [2021-03-11 19:15:29.437268] D [repce(worker
> /data/storage_a/storage):195:push] RepceClient: call
> 192117:140572692309824:1615490129.44 pid() ...
> [2021-03-11 19:15:29.437915] D [repce(worker
> /data/storage_a/storage):215:__call__] RepceClient: call
> 192117:140572692309824:1615490129.44 pid -> 157321
> [2021-03-11 19:15:29.438004] I [resource(worker
> /data/storage_a/storage):1436:connect_remote] SSH: SSH connection between
> master and slave established. [{duration=1.9359}]
> [2021-03-11 19:15:29.438072] I [resource(worker
> /data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume
> locally...
> [2021-03-11 19:15:29.494538] D [repce(worker
> /data/storage_b/storage):215:__call__] RepceClient: call
> 192115:139891296405312:1615490127.53 __repce_version__ -> 1.0
> [2021-03-11 19:15:29.494748] D [repce(worker
> /data/storage_b/storage):195:push] RepceClient: call
> 192115:139891296405312:1615490129.49 version() ...
> [2021-03-11 19:15:29.495290] D [repce(worker
> /data/storage_b/storage):215:__call__] RepceClient: call
> 192115:139891296405312:1615490129.49 version -> 1.0
> [2021-03-11 19:15:29.495400] D [repce(worker
> /data/storage_b/storage):195:push] RepceClient: call
> 192115:139891296405312:1615490129.5 pid() ...
> [2021-03-11 19:15:29.495872] D [repce(worker
> /data/storage_b/storage):215:__call__] RepceClient: call
> 192115:139891296405312:1615490129.5 pid -> 88110
> [2021-03-11 19:15:29.495960] I [resource(worker
> /data/storage_b/storage):1436:connect_remote] SSH: SSH connection between
> master and slave established. [{duration=1.9944}]
> [2021-03-11 19:15:29.496028] I [resource(worker
> /data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume
> locally...
> [2021-03-11 19:15:29.501255] D [repce(worker
> /data/storage_c/storage):215:__call__] RepceClient: call
> 192114:140388828780352:1615490127.53 __repce_version__ -> 1.0
> [2021-03-11 19:15:29.501454] D [repce(worker
> /data/storage_c/storage):195:push] RepceClient: call
> 192114:140388828780352:1615490129.5 version() ...
> [2021-03-11 19:15:29.502258] D [repce(worker
> /data/storage_c/storage):215:__call__] RepceClient: call
> 192114:140388828780352:1615490129.5 version -> 1.0
> [2021-03-11 19:15:29.502444] D [repce(worker
> /data/storage_c/storage):195:push] RepceClient: call
> 192114:140388828780352:1615490129.5 pid() ...
> [2021-03-11 19:15:29.503140] D [repce(worker
> /data/storage_c/storage):215:__call__] RepceClient: call
> 192114:140388828780352:1615490129.5 pid -> 88111
> [2021-03-11 19:15:29.503232] I [resource(worker
> /data/storage_c/storage):1436:connect_remote] SSH: SSH connection between
> master and slave established. [{duration=2.0026}]
> [2021-03-11 19:15:29.503302] I [resource(worker
> /data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume
> locally...
> [2021-03-11 19:15:29.533899] D [resource(worker
> /data/storage_a/storage):880:inhibit] DirectMounter: auxiliary glusterfs
> mount in place
> [2021-03-11 19:15:29.595736] D [resource(worker
> /data/storage_b/storage):880:inhibit] DirectMounter: auxiliary glusterfs
> mount in place
> [2021-03-11 19:15:29.601110] D [resource(worker
> /data/storage_c/storage):880:inhibit] DirectMounter: auxiliary glusterfs
> mount in place
> [2021-03-11 19:15:30.541542] D [resource(worker
> /data/storage_a/storage):964:inhibit] DirectMounter: auxiliary glusterfs
> mount prepared
> [2021-03-11 19:15:30.541816] I [resource(worker
> /data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume
> [{duration=1.1037}]
> [2021-03-11 19:15:30.541887] I [subcmds(worker
> /data/storage_a/storage):84:subcmd_worker] <top>: Worker spawn successful.
> Acknowledging back to monitor
> [2021-03-11 19:15:30.542042] D [master(worker
> /data/storage_a/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=xsync}]
> [2021-03-11 19:15:30.542125] D [monitor(monitor):222:monitor] Monitor:
> worker(/data/storage_a/storage) connected
> [2021-03-11 19:15:30.543323] D [master(worker
> /data/storage_a/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=changelog}]
> [2021-03-11 19:15:30.544460] D [master(worker
> /data/storage_a/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=changeloghistory}]
> [2021-03-11 19:15:30.552103] D [master(worker
> /data/storage_a/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
> [2021-03-11 19:15:30.602937] D [resource(worker
> /data/storage_b/storage):964:inhibit] DirectMounter: auxiliary glusterfs
> mount prepared
> [2021-03-11 19:15:30.603117] I [resource(worker
> /data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume
> [{duration=1.1070}]
> [2021-03-11 19:15:30.603197] I [subcmds(worker
> /data/storage_b/storage):84:subcmd_worker] <top>: Worker spawn successful.
> Acknowledging back to monitor
> [2021-03-11 19:15:30.603353] D [master(worker
> /data/storage_b/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=xsync}]
> [2021-03-11 19:15:30.603338] D [monitor(monitor):222:monitor] Monitor:
> worker(/data/storage_b/storage) connected
> [2021-03-11 19:15:30.604620] D [master(worker
> /data/storage_b/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=changelog}]
> [2021-03-11 19:15:30.605600] D [master(worker
> /data/storage_b/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=changeloghistory}]
> [2021-03-11 19:15:30.608365] D [resource(worker
> /data/storage_c/storage):964:inhibit] DirectMounter: auxiliary glusterfs
> mount prepared
> [2021-03-11 19:15:30.608534] I [resource(worker
> /data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume
> [{duration=1.1052}]
> [2021-03-11 19:15:30.608612] I [subcmds(worker
> /data/storage_c/storage):84:subcmd_worker] <top>: Worker spawn successful.
> Acknowledging back to monitor
> [2021-03-11 19:15:30.608762] D [master(worker
> /data/storage_c/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=xsync}]
> [2021-03-11 19:15:30.608779] D [monitor(monitor):222:monitor] Monitor:
> worker(/data/storage_c/storage) connected
> [2021-03-11 19:15:30.610033] D [master(worker
> /data/storage_c/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=changelog}]
> [2021-03-11 19:15:30.610637] D [master(worker
> /data/storage_b/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
> [2021-03-11 19:15:30.610970] D [master(worker
> /data/storage_c/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=changeloghistory}]
> [2021-03-11 19:15:30.616197] D [master(worker
> /data/storage_c/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
> [2021-03-11 19:15:31.371265] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:31.451000] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:31.537257] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:31.623800] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:32.555840] D [master(worker
> /data/storage_a/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
> [2021-03-11 19:15:32.556051] D [master(worker
> /data/storage_a/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
> [2021-03-11 19:15:32.556122] D [master(worker
> /data/storage_a/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
> [2021-03-11 19:15:32.556179] I [master(worker
> /data/storage_a/storage):1645:register] _GMaster: Working dir
> [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage}]
> [2021-03-11 19:15:32.556359] I [resource(worker
> /data/storage_a/storage):1292:service_loop] GLUSTER: Register time
> [{time=1615490132}]
> [2021-03-11 19:15:32.556823] D [repce(worker
> /data/storage_a/storage):195:push] RepceClient: call
> 192117:140570487928576:1615490132.56 keep_alive(None,) ...
> [2021-03-11 19:15:32.558429] D [repce(worker
> /data/storage_a/storage):215:__call__] RepceClient: call
> 192117:140570487928576:1615490132.56 keep_alive -> 1
> [2021-03-11 19:15:32.558974] D [master(worker
> /data/storage_a/storage):540:crawlwrap] _GMaster: primary master with
> volume id cf94a8f2-324b-40b3-bf72-c3766100ea99 ...
> [2021-03-11 19:15:32.567478] I [gsyncdstatus(worker
> /data/storage_a/storage):281:set_active] GeorepStatus: Worker Status Change
> [{status=Active}]
> [2021-03-11 19:15:32.571824] I [gsyncdstatus(worker
> /data/storage_a/storage):253:set_worker_crawl_status] GeorepStatus: Crawl
> Status Change [{status=History Crawl}]
> [2021-03-11 19:15:32.572052] I [master(worker
> /data/storage_a/storage):1559:crawl] _GMaster: starting history crawl
> [{turns=1}, {stime=(1614666553, 0)}, {entry_stime=(1614664115, 0)},
> {etime=1615490132}]
> [2021-03-11 19:15:32.614506] D [master(worker
> /data/storage_b/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
> [2021-03-11 19:15:32.614701] D [master(worker
> /data/storage_b/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
> [2021-03-11 19:15:32.614788] D [master(worker
> /data/storage_b/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
> [2021-03-11 19:15:32.614845] I [master(worker
> /data/storage_b/storage):1645:register] _GMaster: Working dir
> [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage}]
> [2021-03-11 19:15:32.615000] I [resource(worker
> /data/storage_b/storage):1292:service_loop] GLUSTER: Register time
> [{time=1615490132}]
> [2021-03-11 19:15:32.615586] D [repce(worker
> /data/storage_b/storage):195:push] RepceClient: call
> 192115:139889215526656:1615490132.62 keep_alive(None,) ...
> [2021-03-11 19:15:32.617373] D [repce(worker
> /data/storage_b/storage):215:__call__] RepceClient: call
> 192115:139889215526656:1615490132.62 keep_alive -> 1
> [2021-03-11 19:15:32.618144] D [master(worker
> /data/storage_b/storage):540:crawlwrap] _GMaster: primary master with
> volume id cf94a8f2-324b-40b3-bf72-c3766100ea99 ...
> [2021-03-11 19:15:32.619323] D [master(worker
> /data/storage_c/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
> [2021-03-11 19:15:32.619491] D [master(worker
> /data/storage_c/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
> [2021-03-11 19:15:32.619739] D [master(worker
> /data/storage_c/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
> [2021-03-11 19:15:32.619863] I [master(worker
> /data/storage_c/storage):1645:register] _GMaster: Working dir
> [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage}]
> [2021-03-11 19:15:32.620040] I [resource(worker
> /data/storage_c/storage):1292:service_loop] GLUSTER: Register time
> [{time=1615490132}]
> [2021-03-11 19:15:32.620599] D [repce(worker
> /data/storage_c/storage):195:push] RepceClient: call
> 192114:140386886469376:1615490132.62 keep_alive(None,) ...
> [2021-03-11 19:15:32.621397] E [resource(worker
> /data/storage_a/storage):1312:service_loop] GLUSTER: Changelog History
> Crawl failed [{error=[Errno 0] Success}]
> [2021-03-11 19:15:32.622035] D [repce(worker
> /data/storage_c/storage):215:__call__] RepceClient: call
> 192114:140386886469376:1615490132.62 keep_alive -> 1
> [2021-03-11 19:15:32.622701] D [master(worker
> /data/storage_c/storage):540:crawlwrap] _GMaster: primary master with
> volume id cf94a8f2-324b-40b3-bf72-c3766100ea99 ...
> [2021-03-11 19:15:32.627031] I [gsyncdstatus(worker
> /data/storage_b/storage):281:set_active] GeorepStatus: Worker Status Change
> [{status=Active}]
> [2021-03-11 19:15:32.643184] I [gsyncdstatus(worker
> /data/storage_b/storage):253:set_worker_crawl_status] GeorepStatus: Crawl
> Status Change [{status=History Crawl}]
> [2021-03-11 19:15:32.643528] I [master(worker
> /data/storage_b/storage):1559:crawl] _GMaster: starting history crawl
> [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664113, 0)},
> {etime=1615490132}]
> [2021-03-11 19:15:32.645148] I [gsyncdstatus(worker
> /data/storage_c/storage):281:set_active] GeorepStatus: Worker Status Change
> [{status=Active}]
> [2021-03-11 19:15:32.649631] I [gsyncdstatus(worker
> /data/storage_c/storage):253:set_worker_crawl_status] GeorepStatus: Crawl
> Status Change [{status=History Crawl}]
> [2021-03-11 19:15:32.649882] I [master(worker
> /data/storage_c/storage):1559:crawl] _GMaster: starting history crawl
> [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664108, 0)},
> {etime=1615490132}]
> [2021-03-11 19:15:32.650907] E [resource(worker
> /data/storage_c/storage):1312:service_loop] GLUSTER: Changelog History
> Crawl failed [{error=[Errno 0] Success}]
> [2021-03-11 19:15:32.700489] E [resource(worker
> /data/storage_b/storage):1312:service_loop] GLUSTER: Changelog History
> Crawl failed [{error=[Errno 0] Success}]
> [2021-03-11 19:15:33.545886] I [monitor(monitor):228:monitor] Monitor:
> worker died in startup phase [{brick=/data/storage_a/storage}]
> [2021-03-11 19:15:33.550487] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change [{status=Faulty}]
> [2021-03-11 19:15:33.606991] I [monitor(monitor):228:monitor] Monitor:
> worker died in startup phase [{brick=/data/storage_b/storage}]
> [2021-03-11 19:15:33.611573] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change [{status=Faulty}]
> [2021-03-11 19:15:33.612337] I [monitor(monitor):228:monitor] Monitor:
> worker died in startup phase [{brick=/data/storage_c/storage}]
> [2021-03-11 19:15:33.615777] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change [{status=Faulty}]
> [2021-03-11 19:15:34.684247] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:34.764971] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:34.851174] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:34.937166] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:36.994502] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:37.73805] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:37.159288] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:37.244153] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:38.916510] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:38.997649] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:39.84816] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:39.172045] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:40.896359] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:40.976135] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:41.62052] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:41.147902] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:42.791997] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:42.871239] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:42.956609] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:43.42473] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:43.566190] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change [{status=Initializing...}]
> [2021-03-11 19:15:43.566400] I [monitor(monitor):160:monitor] Monitor:
> starting gsyncd worker [{brick=/data/storage_a/storage},
> {slave_node=10.0.231.81}]
> [2021-03-11 19:15:43.572240] D [monitor(monitor):195:monitor] Monitor:
> Worker would mount volume privately
> [2021-03-11 19:15:43.612744] D [gsyncd(worker
> /data/storage_a/storage):303:main] <top>: Using session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:43.625689] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change [{status=Initializing...}]
> [2021-03-11 19:15:43.626060] I [monitor(monitor):160:monitor] Monitor:
> starting gsyncd worker [{brick=/data/storage_b/storage},
> {slave_node=10.0.231.82}]
> [2021-03-11 19:15:43.632287] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change [{status=Initializing...}]
> [2021-03-11 19:15:43.632137] D [monitor(monitor):195:monitor] Monitor:
> Worker would mount volume privately
> [2021-03-11 19:15:43.632508] I [monitor(monitor):160:monitor] Monitor:
> starting gsyncd worker [{brick=/data/storage_c/storage},
> {slave_node=10.0.231.82}]
> [2021-03-11 19:15:43.635565] I [resource(worker
> /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH
> connection between master and slave...
> [2021-03-11 19:15:43.637835] D [monitor(monitor):195:monitor] Monitor:
> Worker would mount volume privately
> [2021-03-11 19:15:43.661304] D [repce(worker
> /data/storage_a/storage):195:push] RepceClient: call
> 192535:140367272073024:1615490143.66 __repce_version__() ...
> [2021-03-11 19:15:43.674499] D [gsyncd(worker
> /data/storage_b/storage):303:main] <top>: Using session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:43.680706] D [gsyncd(worker
> /data/storage_c/storage):303:main] <top>: Using session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:43.693773] I [resource(worker
> /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH
> connection between master and slave...
> [2021-03-11 19:15:43.700957] I [resource(worker
> /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH
> connection between master and slave...
> [2021-03-11 19:15:43.717686] D [repce(worker
> /data/storage_b/storage):195:push] RepceClient: call
> 192539:139907321804608:1615490143.72 __repce_version__() ...
> [2021-03-11 19:15:43.725369] D [repce(worker
> /data/storage_c/storage):195:push] RepceClient: call
> 192541:140653101852480:1615490143.73 __repce_version__() ...
> [2021-03-11 19:15:44.289117] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:44.375693] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:44.472251] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:44.558429] D [gsyncd(status):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:45.619694] D [repce(worker
> /data/storage_a/storage):215:__call__] RepceClient: call
> 192535:140367272073024:1615490143.66 __repce_version__ -> 1.0
> [2021-03-11 19:15:45.619930] D [repce(worker
> /data/storage_a/storage):195:push] RepceClient: call
> 192535:140367272073024:1615490145.62 version() ...
> [2021-03-11 19:15:45.621191] D [repce(worker
> /data/storage_a/storage):215:__call__] RepceClient: call
> 192535:140367272073024:1615490145.62 version -> 1.0
> [2021-03-11 19:15:45.621332] D [repce(worker
> /data/storage_a/storage):195:push] RepceClient: call
> 192535:140367272073024:1615490145.62 pid() ...
> [2021-03-11 19:15:45.621859] D [repce(worker
> /data/storage_a/storage):215:__call__] RepceClient: call
> 192535:140367272073024:1615490145.62 pid -> 158229
> [2021-03-11 19:15:45.621939] I [resource(worker
> /data/storage_a/storage):1436:connect_remote] SSH: SSH connection between
> master and slave established. [{duration=1.9862}]
> [2021-03-11 19:15:45.622000] I [resource(worker
> /data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume
> locally...
> [2021-03-11 19:15:45.714468] D [resource(worker
> /data/storage_a/storage):880:inhibit] DirectMounter: auxiliary glusterfs
> mount in place
> [2021-03-11 19:15:45.718441] D [repce(worker
> /data/storage_c/storage):215:__call__] RepceClient: call
> 192541:140653101852480:1615490143.73 __repce_version__ -> 1.0
> [2021-03-11 19:15:45.718643] D [repce(worker
> /data/storage_c/storage):195:push] RepceClient: call
> 192541:140653101852480:1615490145.72 version() ...
> [2021-03-11 19:15:45.719492] D [repce(worker
> /data/storage_c/storage):215:__call__] RepceClient: call
> 192541:140653101852480:1615490145.72 version -> 1.0
> [2021-03-11 19:15:45.719772] D [repce(worker
> /data/storage_c/storage):195:push] RepceClient: call
> 192541:140653101852480:1615490145.72 pid() ...
> [2021-03-11 19:15:45.720202] D [repce(worker
> /data/storage_b/storage):215:__call__] RepceClient: call
> 192539:139907321804608:1615490143.72 __repce_version__ -> 1.0
> [2021-03-11 19:15:45.720381] D [repce(worker
> /data/storage_b/storage):195:push] RepceClient: call
> 192539:139907321804608:1615490145.72 version() ...
> [2021-03-11 19:15:45.720463] D [repce(worker
> /data/storage_c/storage):215:__call__] RepceClient: call
> 192541:140653101852480:1615490145.72 pid -> 88921
> [2021-03-11 19:15:45.720694] I [resource(worker
> /data/storage_c/storage):1436:connect_remote] SSH: SSH connection between
> master and slave established. [{duration=2.0196}]
> [2021-03-11 19:15:45.720882] I [resource(worker
> /data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume
> locally...
> [2021-03-11 19:15:45.721146] D [repce(worker
> /data/storage_b/storage):215:__call__] RepceClient: call
> 192539:139907321804608:1615490145.72 version -> 1.0
> [2021-03-11 19:15:45.721271] D [repce(worker
> /data/storage_b/storage):195:push] RepceClient: call
> 192539:139907321804608:1615490145.72 pid() ...
> [2021-03-11 19:15:45.721795] D [repce(worker
> /data/storage_b/storage):215:__call__] RepceClient: call
> 192539:139907321804608:1615490145.72 pid -> 88924
> [2021-03-11 19:15:45.721911] I [resource(worker
> /data/storage_b/storage):1436:connect_remote] SSH: SSH connection between
> master and slave established. [{duration=2.0280}]
> [2021-03-11 19:15:45.721993] I [resource(worker
> /data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume
> locally...
> [2021-03-11 19:15:45.816891] D [resource(worker
> /data/storage_b/storage):880:inhibit] DirectMounter: auxiliary glusterfs
> mount in place
> [2021-03-11 19:15:45.816960] D [resource(worker
> /data/storage_c/storage):880:inhibit] DirectMounter: auxiliary glusterfs
> mount in place
> [2021-03-11 19:15:46.721534] D [resource(worker
> /data/storage_a/storage):964:inhibit] DirectMounter: auxiliary glusterfs
> mount prepared
> [2021-03-11 19:15:46.721726] I [resource(worker
> /data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume
> [{duration=1.0997}]
> [2021-03-11 19:15:46.721796] I [subcmds(worker
> /data/storage_a/storage):84:subcmd_worker] <top>: Worker spawn successful.
> Acknowledging back to monitor
> [2021-03-11 19:15:46.721971] D [master(worker
> /data/storage_a/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=xsync}]
> [2021-03-11 19:15:46.722122] D [monitor(monitor):222:monitor] Monitor:
> worker(/data/storage_a/storage) connected
> [2021-03-11 19:15:46.723871] D [master(worker
> /data/storage_a/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=changelog}]
> [2021-03-11 19:15:46.725100] D [master(worker
> /data/storage_a/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=changeloghistory}]
> [2021-03-11 19:15:46.732400] D [master(worker
> /data/storage_a/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage
> [2021-03-11 19:15:46.823477] D [resource(worker
> /data/storage_c/storage):964:inhibit] DirectMounter: auxiliary glusterfs
> mount prepared
> [2021-03-11 19:15:46.823645] I [resource(worker
> /data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume
> [{duration=1.1027}]
> [2021-03-11 19:15:46.823754] I [subcmds(worker
> /data/storage_c/storage):84:subcmd_worker] <top>: Worker spawn successful.
> Acknowledging back to monitor
> [2021-03-11 19:15:46.823932] D [master(worker
> /data/storage_c/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=xsync}]
> [2021-03-11 19:15:46.823904] D [resource(worker
> /data/storage_b/storage):964:inhibit] DirectMounter: auxiliary glusterfs
> mount prepared
> [2021-03-11 19:15:46.823930] D [monitor(monitor):222:monitor] Monitor:
> worker(/data/storage_c/storage) connected
> [2021-03-11 19:15:46.824103] I [resource(worker
> /data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume
> [{duration=1.1020}]
> [2021-03-11 19:15:46.824184] I [subcmds(worker
> /data/storage_b/storage):84:subcmd_worker] <top>: Worker spawn successful.
> Acknowledging back to monitor
> [2021-03-11 19:15:46.824340] D [master(worker
> /data/storage_b/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=xsync}]
> [2021-03-11 19:15:46.824321] D [monitor(monitor):222:monitor] Monitor:
> worker(/data/storage_b/storage) connected
> [2021-03-11 19:15:46.825100] D [master(worker
> /data/storage_c/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=changelog}]
> [2021-03-11 19:15:46.825414] D [master(worker
> /data/storage_b/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=changelog}]
> [2021-03-11 19:15:46.826375] D [master(worker
> /data/storage_b/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=changeloghistory}]
> [2021-03-11 19:15:46.826574] D [master(worker
> /data/storage_c/storage):105:gmaster_builder] <top>: setting up change
> detection mode [{mode=changeloghistory}]
> [2021-03-11 19:15:46.831506] D [master(worker
> /data/storage_b/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage
> [2021-03-11 19:15:46.833168] D [master(worker
> /data/storage_c/storage):778:setup_working_dir] _GMaster: changelog working
> dir
> /var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage
> [2021-03-11 19:15:47.275141] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:47.320247] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:47.570877] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:47.615571] D [gsyncd(config-get):303:main] <top>: Using
> session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:47.620893] E [syncdutils(worker
> /data/storage_a/storage):325:log_raise_exception] <top>: connection to peer
> is broken
> [2021-03-11 19:15:47.620939] E [syncdutils(worker
> /data/storage_c/storage):325:log_raise_exception] <top>: connection to peer
> is broken
> [2021-03-11 19:15:47.621668] E [syncdutils(worker
> /data/storage_a/storage):847:errlog] Popen: command returned error
> [{cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
> /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
> /tmp/gsyncd-aux-ssh-_AyCOc/79fa3dc75e30f532b4a40bc08c2b10a1.sock
> geoaccount at 10.0.231.81 /nonexistent/gsyncd slave storage
> geoaccount at 10.0.231.81::pcic-backup --master-node 10.0.231.91
> --master-node-id afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick
> /data/storage_a/storage --local-node 10.0.231.81 --local-node-id
> b88dea4f-31ec-416a-9110-3ccdc3910acd --slave-timeout 120 --slave-log-level
> INFO --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin
> --master-dist-count 3}, {error=255}]
> [2021-03-11 19:15:47.621685] E [syncdutils(worker
> /data/storage_c/storage):847:errlog] Popen: command returned error
> [{cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
> /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
> /tmp/gsyncd-aux-ssh-WOgOEu/e15fc58bb13552de0710eaf018209548.sock
> geoaccount at 10.0.231.82 /nonexistent/gsyncd slave storage
> geoaccount at 10.0.231.81::pcic-backup --master-node 10.0.231.91
> --master-node-id afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick
> /data/storage_c/storage --local-node 10.0.231.82 --local-node-id
> be50a8de-3934-4fee-a80d-8e2e99017902 --slave-timeout 120 --slave-log-level
> INFO --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin
> --master-dist-count 3}, {error=255}]
> [2021-03-11 19:15:47.621776] E [syncdutils(worker
> /data/storage_a/storage):851:logerr] Popen: ssh> Killed by signal 15.
> [2021-03-11 19:15:47.621819] E [syncdutils(worker
> /data/storage_c/storage):851:logerr] Popen: ssh> Killed by signal 15.
> [2021-03-11 19:15:47.621850] E [syncdutils(worker
> /data/storage_b/storage):325:log_raise_exception] <top>: connection to peer
> is broken
> [2021-03-11 19:15:47.622437] E [syncdutils(worker
> /data/storage_b/storage):847:errlog] Popen: command returned error
> [{cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
> /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
> /tmp/gsyncd-aux-ssh-Vy935W/e15fc58bb13552de0710eaf018209548.sock
> geoaccount at 10.0.231.82 /nonexistent/gsyncd slave storage
> geoaccount at 10.0.231.81::pcic-backup --master-node 10.0.231.91
> --master-node-id afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick
> /data/storage_b/storage --local-node 10.0.231.82 --local-node-id
> be50a8de-3934-4fee-a80d-8e2e99017902 --slave-timeout 120 --slave-log-level
> INFO --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin
> --master-dist-count 3}, {error=255}]
> [2021-03-11 19:15:47.622556] E [syncdutils(worker
> /data/storage_b/storage):851:logerr] Popen: ssh> Killed by signal 15.
> [2021-03-11 19:15:47.723756] I [monitor(monitor):228:monitor] Monitor:
> worker died in startup phase [{brick=/data/storage_a/storage}]
> [2021-03-11 19:15:47.731405] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change [{status=Faulty}]
> [2021-03-11 19:15:47.825223] I [monitor(monitor):228:monitor] Monitor:
> worker died in startup phase [{brick=/data/storage_c/storage}]
> [2021-03-11 19:15:47.825685] I [monitor(monitor):228:monitor] Monitor:
> worker died in startup phase [{brick=/data/storage_b/storage}]
> [2021-03-11 19:15:47.829011] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change [{status=Faulty}]
> [2021-03-11 19:15:47.830965] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change [{status=Faulty}]
> [2021-03-11 19:15:48.669634] D [gsyncd(monitor-status):303:main] <top>:
> Using session config file
> [{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]
> [2021-03-11 19:15:48.683784] I
> [subcmds(monitor-status):29:subcmd_monitor_status] <top>: Monitor Status
> Change [{status=Stopped}]
>
>
> Thanks,
>  -Matthew
>
>
> On 3/11/21 9:37 AM, Strahil Nikolov wrote:
>
> Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.
>
>
> I think you have to increase the debug logs for geo-rep session.
> I will try to find the command necessary to increase it.
>
>
> Best Regards,
> Strahil Nikolov
>
>
>
>
>
>
> В четвъртък, 11 март 2021 г., 00:38:41 ч. Гринуич+2, Matthew Benstead <matthewb at uvic.ca> <matthewb at uvic.ca> написа:
>
>
>
>
>
> Thanks Strahil,
>
> Right - I had come across your message in early January that v8 from the CentOS Sig was missing the SELinux rules, and had put SELinux into permissive mode after the upgrade when I saw denied messages in the audit logs.
>
> [root at storage01 ~]# sestatus | egrep "^SELinux status|[mM]ode"
> SELinux status:                 enabled
> Current mode:                   permissive
> Mode from config file:          enforcing
>
> Yes - I am using an unprivileged user for georep:
>
> [root at pcic-backup01 ~]# gluster-mountbroker status
> +-------------+-------------+---------------------------+--------------+--------------------------+
> |     NODE    | NODE STATUS |         MOUNT ROOT        |    GROUP     |          USERS           |
> +-------------+-------------+---------------------------+--------------+--------------------------+
> | 10.0.231.82 |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
> |  localhost  |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
> +-------------+-------------+---------------------------+--------------+--------------------------+
>
> [root at pcic-backup02 ~]# gluster-mountbroker status
> +-------------+-------------+---------------------------+--------------+--------------------------+
> |     NODE    | NODE STATUS |         MOUNT ROOT        |    GROUP     |          USERS           |
> +-------------+-------------+---------------------------+--------------+--------------------------+
> | 10.0.231.81 |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
> |  localhost  |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
> +-------------+-------------+---------------------------+--------------+--------------------------+
>
> Thanks,
>  -Matthew
>
>
> --
> Matthew Benstead
> System AdministratorPacific Climate Impacts ConsortiumUniversity of Victoria, UH1PO Box 1800, STN CSCVictoria, BC, V8W 2Y2Phone: +1-250-721-8432Email: matthewb at uvic.ca
>
>
> On 3/10/21 2:11 PM, Strahil Nikolov wrote:
>
>
>
>
>   Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.
>
>
> I have tested georep on v8.3 and it was running quite well untill you involve SELINUX.
>
>
>
> Are you using SELINUX ?
>
> Are you using unprivileged user for the georep ?
>
>
>
>
> Also, you can check https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-troubleshooting_geo-replication .
>
>
>
>
> Best Regards,
>
> Strahil Nikolov
>
>
>
>
>
> On Thu, Mar 11, 2021 at 0:03, Matthew Benstead
> <matthewb at uvic.ca> <matthewb at uvic.ca> wrote:
>
>
>
>
> Hello,
>
> I recently upgraded my Distributed-Replicate cluster from Gluster 7.9 to 8.3 on CentOS7 using the CentOS Storage SIG packages. I had geo-replication syncing properly before the upgrade, but not it is not working after.
>
> After I had upgraded both master and slave clusters I attempted to start geo-replication again, but it goes to faulty quickly:
>
> [root at storage01 ~]# gluster volume geo-replication storage  geoaccount at 10.0.231.81::pcic-backup start
> Starting geo-replication session between storage &  geoaccount at 10.0.231.81::pcic-backup has been successful\
>
> [root at storage01 ~]# gluster volume geo-replication status
>
> MASTER NODE    MASTER VOL    MASTER BRICK               SLAVE USER    SLAVE                                        SLAVE NODE    STATUS    CRAWL STATUS    LAST_SYNCED
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 10.0.231.91    storage       /data/storage_a/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
> 10.0.231.91    storage       /data/storage_c/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
> 10.0.231.91    storage       /data/storage_b/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
> 10.0.231.92    storage       /data/storage_b/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
> 10.0.231.92    storage       /data/storage_a/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
> 10.0.231.92    storage       /data/storage_c/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
> 10.0.231.93    storage       /data/storage_c/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
> 10.0.231.93    storage       /data/storage_b/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
> 10.0.231.93    storage       /data/storage_a/storage    geoaccount     ssh://geoaccount@10.0.231.81::pcic-backup    N/A           Faulty    N/A             N/A
>
> [root at storage01 ~]# gluster volume geo-replication storage  geoaccount at 10.0.231.81::pcic-backup stop
> Stopping geo-replication session between storage &  geoaccount at 10.0.231.81::pcic-backup has been successful
>
>
> I went through the gsyncd logs and see it attempts to go back through the changlogs - which would make sense - but fails:
>
> [2021-03-10 19:18:42.165807] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
> [2021-03-10 19:18:42.166136] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]
> [2021-03-10 19:18:42.167829] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]
> [2021-03-10 19:18:42.172343] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
> [2021-03-10 19:18:42.172580] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]
> [2021-03-10 19:18:42.235574] I [resource(worker /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
> [2021-03-10 19:18:42.236613] I [resource(worker /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
> [2021-03-10 19:18:42.238614] I [resource(worker /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
> [2021-03-10 19:18:44.144856] I [resource(worker /data/storage_b/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9059}]
> [2021-03-10 19:18:44.145065] I [resource(worker /data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
> [2021-03-10 19:18:44.162873] I [resource(worker /data/storage_a/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9259}]
> [2021-03-10 19:18:44.163412] I [resource(worker /data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
> [2021-03-10 19:18:44.167506] I [resource(worker /data/storage_c/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9316}]
> [2021-03-10 19:18:44.167746] I [resource(worker /data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
> [2021-03-10 19:18:45.251372] I [resource(worker /data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1062}]
> [2021-03-10 19:18:45.251583] I [subcmds(worker /data/storage_b/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
> [2021-03-10 19:18:45.271950] I [resource(worker /data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1041}]
> [2021-03-10 19:18:45.272118] I [subcmds(worker /data/storage_c/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
> [2021-03-10 19:18:45.275180] I [resource(worker /data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1116}]
> [2021-03-10 19:18:45.275361] I [subcmds(worker /data/storage_a/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
> [2021-03-10 19:18:47.265618] I [master(worker /data/storage_b/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage}]
> [2021-03-10 19:18:47.265954] I [resource(worker /data/storage_b/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
> [2021-03-10 19:18:47.276746] I [gsyncdstatus(worker /data/storage_b/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
> [2021-03-10 19:18:47.281194] I [gsyncdstatus(worker /data/storage_b/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
> [2021-03-10 19:18:47.281404] I [master(worker /data/storage_b/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664113, 0)}, {etime=1615403927}]
> [2021-03-10 19:18:47.285340] I [master(worker /data/storage_c/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage}]
> [2021-03-10 19:18:47.285579] I [resource(worker /data/storage_c/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
> [2021-03-10 19:18:47.287383] I [master(worker /data/storage_a/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage}]
> [2021-03-10 19:18:47.287697] I [resource(worker /data/storage_a/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
> [2021-03-10 19:18:47.298415] I [gsyncdstatus(worker /data/storage_c/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
> [2021-03-10 19:18:47.301342] I [gsyncdstatus(worker /data/storage_a/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
> [2021-03-10 19:18:47.304183] I [gsyncdstatus(worker /data/storage_c/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
> [2021-03-10 19:18:47.304418] I [master(worker /data/storage_c/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664108, 0)}, {etime=1615403927}]
> [2021-03-10 19:18:47.305294] E [resource(worker /data/storage_c/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
> [2021-03-10 19:18:47.308124] I [gsyncdstatus(worker /data/storage_a/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
> [2021-03-10 19:18:47.308509] I [master(worker /data/storage_a/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666553, 0)}, {entry_stime=(1614664115, 0)}, {etime=1615403927}]
> [2021-03-10 19:18:47.357470] E [resource(worker /data/storage_b/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
> [2021-03-10 19:18:47.383949] E [resource(worker /data/storage_a/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
> [2021-03-10 19:18:48.255340] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_b/storage}]
> [2021-03-10 19:18:48.260052] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
> [2021-03-10 19:18:48.275651] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_c/storage}]
> [2021-03-10 19:18:48.278064] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_a/storage}]
> [2021-03-10 19:18:48.280453] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
> [2021-03-10 19:18:48.282274] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
> [2021-03-10 19:18:58.275702] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
> [2021-03-10 19:18:58.276041] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]
> [2021-03-10 19:18:58.296252] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
> [2021-03-10 19:18:58.296506] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]
> [2021-03-10 19:18:58.301290] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
> [2021-03-10 19:18:58.301521] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]
> [2021-03-10 19:18:58.345817] I [resource(worker /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
> [2021-03-10 19:18:58.361268] I [resource(worker /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
> [2021-03-10 19:18:58.367985] I [resource(worker /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
> [2021-03-10 19:18:59.115143] I [subcmds(monitor-status):29:subcmd_monitor_status] <top>: Monitor Status Change [{status=Stopped}]
>
> It seems like there is an issue selecting the changelogs - perhaps similar to this issue?  https://github.com/gluster/glusterfs/issues/1766
>
> [root at storage01 storage_10.0.231.81_pcic-backup]# cat changes-data-storage_a-storage.log
> [2021-03-10 19:18:45.284764] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_a/storage}, {notify_filter=1}]
> [2021-03-10 19:18:45.285275] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}]
> [2021-03-10 19:18:45.285269] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}]
> [2021-03-10 19:18:45.286615] I [socket.c:929:__socket_server_bind] 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 21
> [2021-03-10 19:18:47.308607] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666553}, {end=1615403927}]
> [2021-03-10 19:18:47.308659] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1615403927}, {total_changelogs=1250878}]
> [2021-03-10 19:18:47.383774] E [MSGID: 132009] [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong result [{for=end}, {start=1615403927}, {idx=1250877}]
>
> [root at storage01 storage_10.0.231.81_pcic-backup]# tail -7 changes-data-storage_b-storage.log
> [2021-03-10 19:18:45.263211] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}]
> [2021-03-10 19:18:45.263151] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_b/storage}, {notify_filter=1}]
> [2021-03-10 19:18:45.263294] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}]
> [2021-03-10 19:18:45.264598] I [socket.c:929:__socket_server_bind] 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 23
> [2021-03-10 19:18:47.281499] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666552}, {end=1615403927}]
> [2021-03-10 19:18:47.281551] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1615403927}, {total_changelogs=1258258}]
> [2021-03-10 19:18:47.357244] E [MSGID: 132009] [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong result [{for=end}, {start=1615403927}, {idx=1258257}]
>
> Any ideas on where to debug this? I'd prefer not to have to remove and re-sync everything as there is about 240TB on the cluster...
>
> Thanks,
>  -Matthew
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210319/ca19ea1b/attachment.html>


More information about the Gluster-users mailing list