<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    I ended up completing the htime files on all bricks manually by
    continuing to parse the original HTIME file after the point of error
    in the script and confirm that all of the CHANGELOG.date files were
    migrated to directories. <br>
    <br>
    There were 2 bricks which had duplicate htime entries for a single
    changelog file that also caused the script to break, but I worked
    around that as well. <br>
    <br>
    Unfortunately, geo-replication still wouldn't work properly: <br>
    <br>
    <pre>[root@storage01 ~]# gluster volume geo-replication status 

MASTER NODE    MASTER VOL    MASTER BRICK               SLAVE USER    SLAVE                                        SLAVE NODE    STATUS    CRAWL STATUS    LAST_SYNCED          
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.0.231.91    storage       /data/storage_a/storage    geoaccount    <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A                  
10.0.231.91    storage       /data/storage_c/storage    geoaccount    <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A                  
10.0.231.91    storage       /data/storage_b/storage    geoaccount    <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A                  
10.0.231.93    storage       /data/storage_c/storage    geoaccount    <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A                  
10.0.231.93    storage       /data/storage_b/storage    geoaccount    <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A                  
10.0.231.93    storage       /data/storage_a/storage    geoaccount    <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A                  
10.0.231.92    storage       /data/storage_b/storage    geoaccount    <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A                  
10.0.231.92    storage       /data/storage_a/storage    geoaccount    <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A                  
10.0.231.92    storage       /data/storage_c/storage    geoaccount    <a class="moz-txt-link-abbreviated" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A   </pre>
    And gives the same error as before - despite the htime and
    changelog's being migrated to the new format: <br>
    <pre>[2021-03-25 20:56:40.725830] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666552}, {end=1616705800}] 
[2021-03-25 20:56:40.725891] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1616705800}, {total_changelogs=1547412}] 
[2021-03-25 20:56:40.725983] E [MSGID: 132009] [gf-history-changelog.c:909:gf_history_changelog] 0-gfchangelog: wrong result [{for=start}, {start=1614666552}, {idx=1547411}] </pre>
    <br>
    Is there anything else to try before I give up and delete/re-create
    the session? <br>
    <br>
    I haven't increased the op-version yet - does that matter? <br>
    <br>
    <font face="monospace">[root@storage01 ~]# gluster volume get all
      cluster.op-version<br>
      Option                                 
      Value                                   <br>
      ------                                 
      -----                                   <br>
      cluster.op-version                     
      70200                                   <br>
      [root@storage01 ~]# gluster volume get all cluster.max-op-version<br>
      Option                                 
      Value                                   <br>
      ------                                 
      -----                                   <br>
      cluster.max-op-version                  80000 </font><br>
    <br>
    Thanks,<br>
     -Matthew<br>
    <div class="moz-signature"><font size="-1">
        <p>--<br>
          Matthew Benstead<br>
          <font size="-2">System Administrator<br>
            <a href="https://pacificclimate.org/">Pacific Climate
              Impacts Consortium</a><br>
            University of Victoria, UH1<br>
            PO Box 1800, STN CSC<br>
            Victoria, BC, V8W 2Y2<br>
            Phone: +1-250-721-8432<br>
            Email: <a class="moz-txt-link-abbreviated" href="mailto:matthewb@uvic.ca">matthewb@uvic.ca</a></font></p>
      </font>
    </div>
    <div class="moz-cite-prefix">On 3/22/21 10:37 AM, Matthew Benstead
      wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:740a702b-76fa-98a1-26c7-3f4d674db4ec@uvic.ca"> Hi
      Shwetha, <br>
      <br>
      I ran the script against against the first brick in my cluster
      (after shutting gluster down) and ran into a small issue: <br>
      <br>
      [root@storage01 ~]# ./glusterfs-georep-upgrade.py
      /data/storage_a/storage/ 2&gt;&amp;1 | tee -a
      /var/log/storage_a.txt<br>
      Traceback (most recent call last):<br>
        File "./glusterfs-georep-upgrade.py", line 77, in &lt;module&gt;<br>
          modify_htime_file(args.brick_path)<br>
        File "./glusterfs-georep-upgrade.py", line 61, in
      modify_htime_file<br>
          shutil.copyfile(pth, os.path.join(path, changelog))<br>
        File "/usr/lib64/python3.6/shutil.py", line 104, in copyfile<br>
          raise SameFileError("{!r} and {!r} are the same
      file".format(src, dst))<br>
      shutil.SameFileError:
'/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/CHANGELOG.1615318474'
      and
'/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/CHANGELOG.1615318474'
      are the same file<br>
      <br>
      Since this script wasn't run prior to the upgrade some of the
      files in the original HTIME file are already in the
      year/month/date format - so the copy operation fails on those
      files terminating the loop early, meaning that not all of the
      files make it into the temp_htime_file, and the new htime file
      doesn't get put into place. <br>
      <br>
      [root@storage01 changelogs]# tail -c 256 htime/HTIME.1597342860 <br>
/changelog.1616431368/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431382/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431396/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431410[root@storage01
      changelogs]# <br>
      [root@storage01 changelogs]# <br>
      [root@storage01 changelogs]# tail -c 256 htime/temp_htime_file <br>
/changelog.1615317943/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/changelog.1615317957/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/changelog.1615317972/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/CHANGELOG.1615318474[root@storage01
      changelogs]#<br>
      <br>
      [root@storage01 changelogs]# ls -lh<br>
      total 4.0K<br>
      drw-------. 7 root root 80 Mar 22 10:07 2020<br>
      drw-------. 5 root root 52 Mar 22 10:07 2021<br>
      -rw-r--r--. 1 root root 51 Mar 22 09:43 CHANGELOG<br>
      drw-------. 2 root root 10 May  7  2020 csnap<br>
      drw-------. 2 root root 65 Mar 22 09:57 htime<br>
      [root@storage01 changelogs]# ls -lh 2021/03/<br>
      total 24K<br>
      drw-------. 2 root root 16K Mar 22 10:07 01<br>
      drw-------. 2 root root  74 Mar  9 12:19 09<br>
      drw-------. 2 root root  42 Mar 15 14:08 15<br>
      drw-------. 2 root root  42 Mar 16 12:04 16<br>
      [root@storage01 changelogs]# ls -lh htime/<br>
      total 178M<br>
      -rw-r--r--. 1 root root 86M Mar 22 09:43 HTIME.1597342860<br>
      -rw-r--r--. 1 root root 93M Mar 22 10:07 temp_htime_file<br>
       <br>
      <br>
      I assume here I can just manually add the remaining changelog
      files into the temp_htime_file and then perform the backup and
      replacement, but wanted to double check. <br>
      <br>
      It's also strange that there are files listed in the original
      HTIME.1597342860 file which don't exist on the filesystem.... the
      directories end at 2021/03/16, but there are files listed in the
      2021/03/22 . Please advise. <br>
      <br>
      [root@storage01 changelogs]# find 2021/03/ -type f | egrep -v
      "^2021/03/01"<br>
      2021/03/09/CHANGELOG.1615318474<br>
      2021/03/09/CHANGELOG.1615321197<br>
      2021/03/15/CHANGELOG.1615842490<br>
      2021/03/16/CHANGELOG.1615921482<br>
      <br>
      [root@storage01 changelogs]# tail -c 2048 htime/HTIME.1597342860 <br>
hangelog.1616431028/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431043/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431058/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431073/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431088/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431103/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431118/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431133/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431148/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431163/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431178/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431193/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431208/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431223/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431238/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431253/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431268/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431283/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431298/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431312/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431326/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431340/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431354/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431368/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431382/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431396/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431410<br>
      <br>
      <br>
      Thanks,<br>
       -Matthew<br>
      <div class="moz-signature">
        <p>--<br>
          <br>
        </p>
      </div>
      <div class="moz-cite-prefix">On 3/19/21 5:36 AM, Shwetha Acharya
        wrote:<br>
      </div>
      <blockquote type="cite"
cite="mid:CAERh03qtOiZg_hLAZtGjZ0Y6QbK-4R38HTf_=hs4iiUNzotCyQ@mail.gmail.com">
        <div> Notice: This message was sent from outside the University
          of Victoria email system. Please be cautious with links and
          sensitive information. </div>
        <br>
        <div>
          <div dir="ltr"> <a class="gmail_plusreply"
              id="plusReplyChip-1" href="mailto:kaleb@redhat.com"
              tabindex="-1" moz-do-not-send="true">@Kaleb Keithley</a> <a
              class="gmail_plusreply" id="plusReplyChip-2"
              href="mailto:khiremat@redhat.com" tabindex="-1"
              moz-do-not-send="true">@Kotresh Hiremath Ravishankar</a> 
            any insights on use of rpm scriplets for this script?
            Basically this script is to be run before the upgradation.<br>
            <br>
            Strahil,<br>
            I will update the docs once we decide with respect to rpm
            scriptlet part.<br>
            <br>
            Regrads,<br>
            Shwetha  <br>
            <br>
          </div>
          <br>
          <div class="gmail_quote">
            <div dir="ltr" class="gmail_attr">On Fri, Mar 19, 2021 at
              5:45 PM Strahil Nikolov &lt;<a
                href="mailto:hunter86_bg@yahoo.com"
                moz-do-not-send="true">hunter86_bg@yahoo.com</a>&gt;
              wrote:<br>
            </div>
            <blockquote class="gmail_quote"> Hi Shwetha,
              <div><br>
              </div>
              <div
                id="gmail-m_634331480037335435yMail_cursorElementTracker_1616155936663">Is
                that script mentioned in the documentation (release
                notes for example) ?</div>
              <div
                id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156103839"><br>
              </div>
              <div
                id="gmail-m_634331480037335435yMail_cursorElementTracker_1616155936803">Do
                you think that the new glusterfs rpm can call (via rpm
                scriplets) the script safely ?</div>
              <div
                id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156033372"><br>
              </div>
              <div
                id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156004182"><br>
              </div>
              <div
                id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156004390">If
                yes, then it will be nice if the rpm itself runs the
                script instead of the user.</div>
              <div
                id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156004390"><br>
              </div>
              <div
                id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156077707"><br>
              </div>
              <div
                id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156077811">Best
                Regards,</div>
              <div
                id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156082780">Strahil
                Nikolov<br>
                <br>
                <blockquote>
                  <div>
                    <div>On Fri, Mar 19, 2021 at 11:44, Shwetha Acharya</div>
                    <div>&lt;<a href="mailto:sacharya@redhat.com"
                        target="_blank" moz-do-not-send="true">sacharya@redhat.com</a>&gt;
                      wrote:</div>
                  </div>
                  <div>
                    <div id="gmail-m_634331480037335435yiv1091814412">
                      <div dir="ltr">Hi Mathew,<br>
                        <br>
                        The upgrade script <a rel="nofollow noopener
                          noreferrer"
href="https://github.com/gluster/glusterfs/commit/2857fe3fad4d2b30894847088a54b847b88a23b9"
                          target="_blank" moz-do-not-send="true">https://github.com/gluster/glusterfs/commit/2857fe3fad4d2b30894847088a54b847b88a23b9</a>
                        need to be run, to make sure that the changelogs
                        and htimes files created in the older version
                        are updated as per the new changes in the
                        directory structure.<br>
                        <br>
                        If not done, search algorithm during history
                        crawl would give wrong result.<br>
                        <br>
                        Regards,<br>
                        Shwetha</div>
                      <br>
                      <div>
                        <div dir="ltr">On Fri, Mar 19, 2021 at 3:53 AM
                          Strahil Nikolov &lt;<a rel="nofollow noopener
                            noreferrer"
                            href="mailto:hunter86_bg@yahoo.com"
                            target="_blank" moz-do-not-send="true">hunter86_bg@yahoo.com</a>&gt;
                          wrote:<br>
                        </div>
                        <blockquote> Sadly,
                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106048432"><br>
                          </div>
                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106048546">I'm
                            out of ideas. It makes sense... if the
                            changelog was changed - then it won't work
                            after the upgrade.</div>
                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106048546"><br>
                          </div>
                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106093562">I
                            guess that once you delete the session, you
                            can remove the extended attribute for the
                            time.</div>
                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106136564"><br>
                          </div>
                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106136756">Best
                            Regards,</div>
                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106141079">Strahil
                            Nikolov</div>
                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106094445"><br>
                            <blockquote>
                              <div>
                                <div>On Wed, Mar 17, 2021 at 23:11,
                                  Matthew Benstead</div>
                                <div>&lt;<a rel="nofollow noopener
                                    noreferrer"
                                    href="mailto:matthewb@uvic.ca"
                                    target="_blank"
                                    moz-do-not-send="true">matthewb@uvic.ca</a>&gt;
                                  wrote:</div>
                              </div>
                              <div>
                                <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633">
                                  <div>Yes, I've run through everything
                                    short of regenerating the keys and
                                    creating the session again with no
                                    errors. Everything looks ok. <br>
                                    <br>
                                    But I did notice that the changelog
                                    format had changed, instead of them
                                    being dumped into one directory,
                                    they now seem to be separated in
                                    year/month/day directories... <br>
                                    <br>
                                    Looks like this change in 8.0: <a
                                      rel="nofollow noopener noreferrer"
                                      shape="rect"
                                      href="https://github.com/gluster/glusterfs/issues/154"
                                      target="_blank"
                                      moz-do-not-send="true">
https://github.com/gluster/glusterfs/issues/154</a><br>
                                    <br>
                                    [root@storage01 changelogs]# ls -lh
                                    | head<br>
                                    total 16G<br>
                                    drw-------. 3 root root   24 Mar  9
                                    11:34 2021<br>
                                    -rw-r--r--. 1 root root   51 Mar 17
                                    13:19 CHANGELOG<br>
                                    -rw-r--r--. 1 root root  13K Aug 13 
                                    2020 CHANGELOG.1597343197<br>
                                    -rw-r--r--. 1 root root  51K Aug 13 
                                    2020 CHANGELOG.1597343212<br>
                                    -rw-r--r--. 1 root root  86K Aug 13 
                                    2020 CHANGELOG.1597343227<br>
                                    -rw-r--r--. 1 root root  99K Aug 13 
                                    2020 CHANGELOG.1597343242<br>
                                    -rw-r--r--. 1 root root  69K Aug 13 
                                    2020 CHANGELOG.1597343257<br>
                                    -rw-r--r--. 1 root root  69K Aug 13 
                                    2020 CHANGELOG.1597343272<br>
                                    -rw-r--r--. 1 root root  72K Aug 13 
                                    2020 CHANGELOG.1597343287<br>
                                    [root@storage01 changelogs]# ls -lh
                                    | tail<br>
                                    -rw-r--r--. 1 root root   92 Mar  1
                                    21:33 CHANGELOG.1614663193<br>
                                    -rw-r--r--. 1 root root   92 Mar  1
                                    21:42 CHANGELOG.1614663731<br>
                                    -rw-r--r--. 1 root root   92 Mar  1
                                    21:42 CHANGELOG.1614663760<br>
                                    -rw-r--r--. 1 root root  511 Mar  1
                                    21:47 CHANGELOG.1614664043<br>
                                    -rw-r--r--. 1 root root  536 Mar  1
                                    21:48 CHANGELOG.1614664101<br>
                                    -rw-r--r--. 1 root root 2.8K Mar  1
                                    21:48 CHANGELOG.1614664116<br>
                                    -rw-r--r--. 1 root root   92 Mar  1
                                    22:20 CHANGELOG.1614666061<br>
                                    -rw-r--r--. 1 root root   92 Mar  1
                                    22:29 CHANGELOG.1614666554<br>
                                    drw-------. 2 root root   10 May  7 
                                    2020 csnap<br>
                                    drw-------. 2 root root   38 Aug 13 
                                    2020 htime<br>
                                    <br>
                                    <br>
                                    [root@storage01 changelogs]# ls -lh
                                    2021/03/09/<br>
                                    total 8.0K<br>
                                    -rw-r--r--. 1 root root 51 Mar  9
                                    11:26 CHANGELOG.1615318474<br>
                                    -rw-r--r--. 1 root root 51 Mar  9
                                    12:19 CHANGELOG.1615321197<br>
                                    [root@storage01 changelogs]# ls -lh
                                    2021/03/15/<br>
                                    total 4.0K<br>
                                    -rw-r--r--. 1 root root 51 Mar 15
                                    13:38 CHANGELOG.1615842490<br>
                                    [root@storage01 changelogs]# ls -lh
                                    2021/03/16<br>
                                    total 4.0K<br>
                                    -rw-r--r--. 1 root root 331 Mar 16
                                    12:04 CHANGELOG.1615921482<br>
                                    <br>
                                    But it looks like the htime file
                                    still records them...<br>
                                    <br>
                                    [root@storage01 changelogs]# ls -lh
                                    htime<br>
                                    total 84M<br>
                                    -rw-r--r--. 1 root root 84M Mar 17
                                    13:31 HTIME.1597342860<br>
                                    <br>
                                    [root@storage01 changelogs]# head -c
                                    256 htime/HTIME.1597342860 <br>
/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342875/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342890/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342904/data/storage_a/storage/.glusterfs/changelogs/changelog[root@storage01
                                    changelogs]# <br>
                                    <br>
                                    [root@storage01 changelogs]# tail -c
                                    256 htime/HTIME.1597342860 <br>
/changelog.1616013484/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013499/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013514/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013529[root@storage01
                                    changelogs]# <br>
                                    <br>
                                    <br>
                                    And there seems to be an xattr for
                                    time in the brick root - presumably
                                    for when changelogs were enabled: <br>
                                    <br>
                                    [root@storage01 changelogs]#
                                    getfattr -d -m. -e hex
                                    /data/storage_a/storage 2&gt;&amp;1
                                    | egrep xtime<br>
trusted.glusterfs.cf94a8f2-324b-40b3-bf72-c3766100ea99.xtime=0x60510140000ef317<br>
                                    <br>
                                    <br>
                                    Reading through the changes, it
                                    looks like there is a script to
                                    convert from one format to the
                                    other... I didn't see anything in
                                    the release notes for 8.0 about
                                    it... this seems like it could be
                                    the fix, and explain why gluster
                                    can't get through the
                                    changelogs....  Thoughts? <br>
                                    <ul>
                                      <li><a rel="nofollow noopener
                                          noreferrer" shape="rect"
href="https://github.com/gluster/glusterfs/issues/154#issuecomment-585701964"
                                          target="_blank"
                                          moz-do-not-send="true">https://github.com/gluster/glusterfs/issues/154#issuecomment-585701964</a></li>
                                      <li><a rel="nofollow noopener
                                          noreferrer" shape="rect"
                                          href="https://review.gluster.org/#/c/glusterfs/+/24121/"
                                          target="_blank"
                                          moz-do-not-send="true">https://review.gluster.org/#/c/glusterfs/+/24121/</a><br>
                                      </li>
                                    </ul>
                                    <br>
                                    Thanks,<br>
                                     -Matthew<br>
                                    <div>
                                      <p>--<br>
                                        Matthew Benstead<br>
                                        System Administrator<br>
                                        <a rel="nofollow noopener
                                          noreferrer" shape="rect"
                                          href="https://pacificclimate.org/"
                                          target="_blank"
                                          moz-do-not-send="true">Pacific
                                          Climate Impacts Consortium</a><br>
                                        University of Victoria, UH1<br>
                                        PO Box 1800, STN CSC<br>
                                        Victoria, BC, V8W 2Y2<br>
                                        Phone: +1-250-721-8432<br>
                                        Email: <a rel="nofollow
                                          noopener noreferrer"
                                          shape="rect"
                                          href="mailto:matthewb@uvic.ca"
                                          target="_blank"
                                          moz-do-not-send="true">
                                          matthewb@uvic.ca</a></p>
                                    </div>
                                    <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yqt01812">
                                      <div>On 3/16/21 9:36 PM, Strahil
                                        Nikolov wrote:<br>
                                      </div>
                                      <blockquote type="cite">
                                        <div>Notice: This message was
                                          sent from outside the
                                          University of Victoria email
                                          system. Please be cautious
                                          with links and sensitive
                                          information. </div>
                                        <br>
                                        <div>Have you verified all steps
                                          for creating the
                                          geo-replication ?
                                          <div><br>
                                          </div>
                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955616619">If
                                            yes , maybe using
                                            "reset-sync-time + delete +
                                            create" makes sense.Keep in
                                            mind that it will take a
                                            long time once the geo-rep
                                            is established again.</div>
                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955729476"><br>
                                          </div>
                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955750689"><br>
                                          </div>
                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955750840">Best
                                            Regards,</div>
                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955755206">Strahil
                                            Nikolov</div>
                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955730266"><br>
                                            <blockquote>
                                              <div>
                                                <div>On Tue, Mar 16,
                                                  2021 at 22:34, Matthew
                                                  Benstead</div>
                                                <div><a rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:matthewb@uvic.ca"
                                                    target="_blank"
                                                    moz-do-not-send="true">&lt;matthewb@uvic.ca&gt;</a>
                                                  wrote:</div>
                                              </div>
                                              <div>
                                                <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633">
                                                  <div>Thanks Strahil, <br>
                                                    <br>
                                                    I wanted to make
                                                    sure the issue
                                                    wasn't occurring
                                                    because there were
                                                    no new changes to
                                                    sync from the master
                                                    volume. So I created
                                                    some files and
                                                    restarted the sync,
                                                    but it had no
                                                    effect. <br>
                                                    <br>
                                                    [root@storage01 ~]#
                                                    cd
                                                    /storage2/home/test/<br>
                                                    [root@storage01
                                                    test]# for nums in
                                                    {1,2,3,4,5,6,7,8,9,0};
                                                    do touch $nums.txt;
                                                    done<br>
                                                    <br>
                                                    [root@storage01
                                                    test]# gluster
                                                    volume
                                                    geo-replication
                                                    storage <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> start<br>
                                                    Starting
                                                    geo-replication
                                                    session between
                                                    storage &amp; <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> has been successful<br>
                                                    [root@storage01
                                                    test]# gluster
                                                    volume
                                                    geo-replication
                                                    status <br>
                                                     <br>
                                                    MASTER NODE   
                                                    MASTER VOL    MASTER
                                                    BRICK              
                                                    SLAVE USER   
                                                    SLAVE                                       
                                                    SLAVE NODE   
                                                    STATUS            
                                                    CRAWL STATUS   
                                                    LAST_SYNCED         
                                                    <br>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
                                                    10.0.231.91   
                                                    storage      
                                                    /data/storage_a/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                    Initializing...   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.91   
                                                    storage      
                                                    /data/storage_c/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                    Initializing...   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.91   
                                                    storage      
                                                    /data/storage_b/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                    Initializing...   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.93   
                                                    storage      
                                                    /data/storage_c/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                    Initializing...   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.93   
                                                    storage      
                                                    /data/storage_b/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                    Initializing...   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.93   
                                                    storage      
                                                    /data/storage_a/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                    Initializing...   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.92   
                                                    storage      
                                                    /data/storage_b/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                    Initializing...   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.92   
                                                    storage      
                                                    /data/storage_a/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                    Initializing...   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.92   
                                                    storage      
                                                    /data/storage_c/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                    Initializing...   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    [root@storage01
                                                    test]# gluster
                                                    volume
                                                    geo-replication
                                                    status <br>
                                                     <br>
                                                    MASTER NODE   
                                                    MASTER VOL    MASTER
                                                    BRICK              
                                                    SLAVE USER   
                                                    SLAVE                                       
                                                    SLAVE NODE   
                                                    STATUS    CRAWL
                                                    STATUS   
                                                    LAST_SYNCED         
                                                    <br>
---------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
                                                    10.0.231.91   
                                                    storage      
                                                    /data/storage_a/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.91   
                                                    storage      
                                                    /data/storage_c/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.91   
                                                    storage      
                                                    /data/storage_b/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.93   
                                                    storage      
                                                    /data/storage_c/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.93   
                                                    storage      
                                                    /data/storage_b/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.93   
                                                    storage      
                                                    /data/storage_a/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.92   
                                                    storage      
                                                    /data/storage_b/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.92   
                                                    storage      
                                                    /data/storage_a/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    10.0.231.92   
                                                    storage      
                                                    /data/storage_c/storage   
                                                    geoaccount    <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                    N/A            
                                                    N/A                 
                                                    <br>
                                                    [root@storage01
                                                    test]# gluster
                                                    volume
                                                    geo-replication
                                                    storage <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> stop<br>
                                                    Stopping
                                                    geo-replication
                                                    session between
                                                    storage &amp; <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:geoaccount@10.0.231.81::pcic-backup"
                                                      target="_blank"
                                                      moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> has been successful<br>
                                                    <br>
                                                    Still getting the
                                                    same error about the
                                                    history crawl
                                                    failing: <br>
                                                    <br>
                                                    [2021-03-16
                                                    19:05:05.227677] I
                                                    [MSGID: 132035]
                                                    [gf-history-changelog.c:837:gf_history_changelog]
                                                    0-gfchangelog:
                                                    Requesting
                                                    historical
                                                    changelogs
                                                    [{start=1614666552},
                                                    {end=1615921505}] <br>
                                                    [2021-03-16
                                                    19:05:05.227733] I
                                                    [MSGID: 132019]
                                                    [gf-history-changelog.c:755:gf_changelog_extract_min_max]
                                                    0-gfchangelog:
                                                    changelogs min max
                                                    [{min=1597342860},
                                                    {max=1615921502},
                                                    {total_changelogs=1300114}]
                                                    <br>
                                                    [2021-03-16
                                                    19:05:05.408567] E
                                                    [MSGID: 132009]
                                                    [gf-history-changelog.c:941:gf_history_changelog]
                                                    0-gfchangelog: wrong
                                                    result [{for=end},
                                                    {start=1615921502},
                                                    {idx=1300113}] <br>
                                                    <br>
                                                    <br>
                                                    [2021-03-16
                                                    19:05:05.228092] I
                                                    [resource(worker
                                                    /data/storage_c/storage):1292:service_loop]
                                                    GLUSTER: Register
                                                    time
                                                    [{time=1615921505}]<br>
                                                    [2021-03-16
                                                    19:05:05.228626] D
                                                    [repce(worker
                                                    /data/storage_c/storage):195:push]
                                                    RepceClient: call
                                                    124117:140500837320448:1615921505.23
                                                    keep_alive(None,)
                                                    ...<br>
                                                    [2021-03-16
                                                    19:05:05.230076] D
                                                    [repce(worker
                                                    /data/storage_c/storage):215:__call__]
                                                    RepceClient: call
                                                    124117:140500837320448:1615921505.23
                                                    keep_alive -&gt; 1<br>
                                                    [2021-03-16
                                                    19:05:05.230693] D
                                                    [master(worker
                                                    /data/storage_c/storage):540:crawlwrap]
                                                    _GMaster: primary
                                                    master with volume
                                                    id
cf94a8f2-324b-40b3-bf72-c3766100ea99 ...<br>
                                                    [2021-03-16
                                                    19:05:05.237607] I
                                                    [gsyncdstatus(worker
/data/storage_c/storage):281:set_active] GeorepStatus: Worker Status
                                                    Change
                                                    [{status=Active}]<br>
                                                    [2021-03-16
                                                    19:05:05.242046] I
                                                    [gsyncdstatus(worker
/data/storage_c/storage):253:set_worker_crawl_status] GeorepStatus:
                                                    Crawl Status Change
                                                    [{status=History
                                                    Crawl}]<br>
                                                    [2021-03-16
                                                    19:05:05.242450] I
                                                    [master(worker
                                                    /data/storage_c/storage):1559:crawl]
                                                    _GMaster: starting
                                                    history crawl
                                                    [{turns=1},
                                                    {stime=(1614666552,
                                                    0)},
                                                    {entry_stime=(1614664108,
                                                    0)},
                                                    {etime=1615921505}]<br>
                                                    [2021-03-16
                                                    19:05:05.244151] E
                                                    [resource(worker
                                                    /data/storage_c/storage):1312:service_loop]
                                                    GLUSTER: Changelog
                                                    History Crawl failed
                                                    [{error=[Errno 0]
                                                    Success}]<br>
                                                    [2021-03-16
                                                    19:05:05.394129] E
                                                    [resource(worker
                                                    /data/storage_a/storage):1312:service_loop]
                                                    GLUSTER: Changelog
                                                    History Crawl failed
                                                    [{error=[Errno 0]
                                                    Success}]<br>
                                                    [2021-03-16
                                                    19:05:05.408759] E
                                                    [resource(worker
                                                    /data/storage_b/storage):1312:service_loop]
                                                    GLUSTER: Changelog
                                                    History Crawl failed
                                                    [{error=[Errno 0]
                                                    Success}]<br>
                                                    [2021-03-16
                                                    19:05:06.158694] I
                                                    [monitor(monitor):228:monitor]
                                                    Monitor: worker died
                                                    in startup phase
                                                    [{brick=/data/storage_a/storage}]<br>
                                                    [2021-03-16
                                                    19:05:06.163052] I
                                                    [gsyncdstatus(monitor):248:set_worker_status]
                                                    GeorepStatus: Worker
                                                    Status Change
                                                    [{status=Faulty}]<br>
                                                    [2021-03-16
                                                    19:05:06.204464] I
                                                    [monitor(monitor):228:monitor]
                                                    Monitor: worker died
                                                    in startup phase
                                                    [{brick=/data/storage_b/storage}]<br>
                                                    [2021-03-16
                                                    19:05:06.208961] I
                                                    [gsyncdstatus(monitor):248:set_worker_status]
                                                    GeorepStatus: Worker
                                                    Status Change
                                                    [{status=Faulty}]<br>
                                                    [2021-03-16
                                                    19:05:06.220495] I
                                                    [monitor(monitor):228:monitor]
                                                    Monitor: worker died
                                                    in startup phase
                                                    [{brick=/data/storage_c/storage}]<br>
                                                    [2021-03-16
                                                    19:05:06.223947] I
                                                    [gsyncdstatus(monitor):248:set_worker_status]
                                                    GeorepStatus: Worker
                                                    Status Change
                                                    [{status=Faulty}]<br>
                                                    <br>
                                                    I confirmed NTP is
                                                    working: <br>
                                                    <br>
                                                    <br>
                                                    pcic-backup02 |
                                                    CHANGED | rc=0
                                                    &gt;&gt;<br>
                                                        
                                                    remote          
                                                    refid      st t when
                                                    poll reach   delay  
                                                    offset  jitter<br>
==============================================================================<br>
                                                    +s216-232-132-95
                                                    68.69.221.61     2
                                                    u   29 1024  377  
                                                    24.141    2.457  
                                                    1.081<br>
                                                    *<a rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      href="http://yyz-1.ip.0xt.ca"
                                                      target="_blank"
                                                      moz-do-not-send="true">yyz-1.ip.0xt.ca</a>
                                                    206.108.0.131    2
                                                    u  257 1024  377  
                                                    57.119   -0.084  
                                                    5.625<br>
                                                    +ip102.ip-198-27
                                                    192.168.10.254   2
                                                    u  189 1024  377  
                                                    64.227   -3.012  
                                                    8.867<br>
                                                    <br>
                                                    storage03 | CHANGED
                                                    | rc=0 &gt;&gt;<br>
                                                        
                                                    remote          
                                                    refid      st t when
                                                    poll reach   delay  
                                                    offset  jitter<br>
==============================================================================<br>
                                                    *198.161.203.36 
                                                    128.233.150.93   2
                                                    u   36 1024  377  
                                                    16.055   -0.381  
                                                    0.318<br>
                                                    +s206-75-147-25.
                                                    192.168.10.254   2
                                                    u  528 1024  377  
                                                    23.648   -6.196  
                                                    4.803<br>
                                                    +time.cloudflare
                                                    10.69.8.80       3
                                                    u  121 1024  377   
                                                    2.408    0.507  
                                                    0.791<br>
                                                    <br>
                                                    storage02 | CHANGED
                                                    | rc=0 &gt;&gt;<br>
                                                        
                                                    remote          
                                                    refid      st t when
                                                    poll reach   delay  
                                                    offset  jitter<br>
==============================================================================<br>
                                                    *198.161.203.36 
                                                    128.233.150.93   2
                                                    u  918 1024  377  
                                                    15.952    0.226  
                                                    0.197<br>
                                                    +linuxgeneration
                                                    16.164.40.197    2
                                                    u   88 1024  377  
                                                    62.692   -1.160  
                                                    2.007<br>
                                                    +<a rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      href="http://dns3.switch.ca"
                                                      target="_blank"
                                                      moz-do-not-send="true">dns3.switch.ca</a> 
                                                    206.108.0.131    2
                                                    u  857 1024  377  
                                                    27.315    0.778  
                                                    0.483<br>
                                                    <br>
                                                    storage01 | CHANGED
                                                    | rc=0 &gt;&gt;<br>
                                                        
                                                    remote          
                                                    refid      st t when
                                                    poll reach   delay  
                                                    offset  jitter<br>
==============================================================================<br>
                                                    +198.161.203.36 
                                                    128.233.150.93   2
                                                    u  121 1024  377  
                                                    16.069    1.016  
                                                    0.195<br>
                                                    +<a rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      href="http://zero.gotroot.ca"
                                                      target="_blank"
                                                      moz-do-not-send="true">zero.gotroot.ca</a>
                                                    30.114.5.31      2
                                                    u  543 1024  377   
                                                    5.106   -2.462  
                                                    4.923<br>
                                                    *<a rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      href="http://ntp3.torix.ca"
                                                      target="_blank"
                                                      moz-do-not-send="true">ntp3.torix.ca</a>  
                                                    .PTP0.           1
                                                    u  300 1024  377  
                                                    54.010    2.421 
                                                    15.182<br>
                                                    <br>
                                                    pcic-backup01 |
                                                    CHANGED | rc=0
                                                    &gt;&gt;<br>
                                                        
                                                    remote          
                                                    refid      st t when
                                                    poll reach   delay  
                                                    offset  jitter<br>
==============================================================================<br>
                                                    *<a rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      href="http://dns3.switch.ca"
                                                      target="_blank"
                                                      moz-do-not-send="true">dns3.switch.ca</a> 
                                                    206.108.0.131    2
                                                    u  983 1024  377  
                                                    26.990    0.523  
                                                    1.389<br>
                                                    +<a rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      href="http://dns2.switch.ca"
                                                      target="_blank"
                                                      moz-do-not-send="true">dns2.switch.ca</a> 
                                                    206.108.0.131    2
                                                    u  689 1024  377  
                                                    26.975   -0.257  
                                                    0.467<br>
                                                    +64.ip-54-39-23.
                                                    214.176.184.39   2
                                                    u  909 1024  377  
                                                    64.262   -0.604  
                                                    6.129<br>
                                                    <br>
                                                    And everything is
                                                    working on the same
                                                    version of gluster:
                                                    <br>
                                                    <br>
                                                    pcic-backup02 |
                                                    CHANGED | rc=0
                                                    &gt;&gt;<br>
                                                    glusterfs 8.3<br>
                                                    pcic-backup01 |
                                                    CHANGED | rc=0
                                                    &gt;&gt;<br>
                                                    glusterfs 8.3<br>
                                                    storage02 | CHANGED
                                                    | rc=0 &gt;&gt;<br>
                                                    glusterfs 8.3<br>
                                                    storage01 | CHANGED
                                                    | rc=0 &gt;&gt;<br>
                                                    glusterfs 8.3<br>
                                                    storage03 | CHANGED
                                                    | rc=0 &gt;&gt;<br>
                                                    glusterfs 8.3<br>
                                                    <br>
                                                    SSH works, and the
                                                    backup user/group is
                                                    configured with
                                                    mountbroker: <br>
                                                    <br>
                                                    [root@storage01 ~]#
                                                    ssh -i
                                                    /root/.ssh/id_rsa <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:geoaccount@10.0.231.81"
                                                      target="_blank"
                                                      moz-do-not-send="true">
geoaccount@10.0.231.81</a> uname -a<br>
                                                    Linux pcic-backup01
3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64
                                                    x86_64 x86_64
                                                    GNU/Linux<br>
                                                    [root@storage01 ~]#
                                                    ssh -i
                                                    /root/.ssh/id_rsa <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
                                                      href="mailto:geoaccount@10.0.231.82"
                                                      target="_blank"
                                                      moz-do-not-send="true">
geoaccount@10.0.231.82</a> uname -a<br>
                                                    Linux pcic-backup02
3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64
                                                    x86_64 x86_64
                                                    GNU/Linux<br>
                                                    <br>
                                                    <br>
                                                    [root@pcic-backup01
                                                    ~]# grep geo
                                                    /etc/passwd<br>
geoaccount:x:1000:1000::/home/geoaccount:/bin/bash<br>
                                                    [root@pcic-backup01
                                                    ~]# grep geo
                                                    /etc/group<br>
geogroup:x:1000:geoaccount<br>
geoaccount:x:1001:geoaccount<br>
                                                    <br>
                                                    [root@pcic-backup01
                                                    ~]#
                                                    gluster-mountbroker
                                                    status<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
                                                    |     NODE    | NODE
                                                    STATUS |        
                                                    MOUNT ROOT       
                                                    |    GROUP    
                                                    |         
                                                    USERS           |<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
                                                    | 10.0.231.82
                                                    |          UP |
                                                    /var/mountbroker-root(OK)
                                                    | geogroup(OK) |
                                                    geoaccount(pcic-backup) 
                                                    |<br>
                                                    |  localhost 
                                                    |          UP |
                                                    /var/mountbroker-root(OK)
                                                    | geogroup(OK) |
                                                    geoaccount(pcic-backup) 
                                                    |<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
                                                    <br>
                                                    <br>
                                                    <br>
                                                    <br>
                                                    So, then if I'm
                                                    going to have to
                                                    resync, what is the
                                                    best way to do this?
                                                    <br>
                                                    <br>
                                                    With delete or
                                                    delete
                                                    reset-sync-time ?  
                                                    <a rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-starting_geo-replication#Deleting_a_Geo-replication_Session"
                                                      target="_blank"
                                                      moz-do-not-send="true">
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-starting_geo-replication#Deleting_a_Geo-replication_Session</a><br>
                                                    <br>
                                                    <br>
                                                    <br>
                                                    Erasing the index?
                                                    So I don't have to
                                                    transfer the files
                                                    again that are
                                                    already on the
                                                    backup? <br>
                                                    <ul>
                                                      <li><a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/sect-troubleshooting_geo-replication#Synchronization_Is_Not_Complete"
target="_blank" moz-do-not-send="true">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/sect-troubleshooting_geo-replication#Synchronization_Is_Not_Complete</a>
                                                      </li>
                                                      <li><a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
href="https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Geo%20Replication/#best-practices"
target="_blank" moz-do-not-send="true">https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Geo%20Replication/#best-practices</a><br>
                                                      </li>
                                                    </ul>
                                                    <br>
                                                    <br>
                                                    <br>
                                                    Is it possible to
                                                    use the
                                                    special-sync-mode 
                                                    option from here: <a
                                                      rel="nofollow
                                                      noopener
                                                      noreferrer"
                                                      shape="rect"
href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-disaster_recovery"
                                                      target="_blank"
                                                      moz-do-not-send="true">
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-disaster_recovery</a><br>
                                                    <br>
                                                    <br>
                                                    <br>
                                                    Thoughts? <br>
                                                    <br>
                                                    Thanks,<br>
                                                     -Matthew<br>
                                                    --<br>
                                                    <br>
                                                    <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yqt93720">
                                                      <div>On 3/12/21
                                                        3:31 PM, Strahil
                                                        Nikolov wrote:<br>
                                                      </div>
                                                      <blockquote
                                                        type="cite">
                                                        <div>Notice:
                                                          This message
                                                          was sent from
                                                          outside the
                                                          University of
                                                          Victoria email
                                                          system. Please
                                                          be cautious
                                                          with links and
                                                          sensitive
                                                          information. </div>
                                                        <br>
                                                        <div>Usually,
                                                          when I'm stuck
                                                          - I just start
                                                          over.
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591681347">For
                                                          example, check
                                                          the
                                                          prerequisites:</div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591695583">-
                                                          Is ssh
                                                          available (no
                                                          firewall
                                                          blocking)</div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591712278">-
                                                          Is time sync
                                                          enabled
                                                          (ntp/chrony)</div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591726239">-
                                                          Is DNS ok on
                                                          all hosts
                                                          (including PTR
                                                          records)</div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591754957">-
                                                          Is the gluster
                                                          version the
                                                          same on all
                                                          nodes (primary
                                                          &amp;
                                                          secondary)</div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591754957"><br>
                                                          </div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591782389">Then
                                                          start over as
                                                          if the geo rep
                                                          was never
                                                          existing. For
                                                          example , stop
                                                          it and start
                                                          over with the
                                                          secondary
                                                          nodes's checks
                                                          (mountbroker,
                                                          user, group) .</div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591782389"><br>
                                                          </div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591841464">Most
                                                          probably
                                                          somwthing will
                                                          come up and
                                                          you will fix
                                                          it.</div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591841464"><br>
                                                          </div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591864006">In
                                                          worst case
                                                          scenario, you
                                                          will need to
                                                          clean ip the
                                                          geo-rep and
                                                          start fresh.</div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591864006"><br>
                                                          </div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591889862"><br>
                                                          </div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591889925">Best
                                                          Regards,</div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591895692">Strahil
                                                          Nikolov<br>
                                                          <br>
                                                          <blockquote>
                                                          <div>
                                                          <div>On Fri,
                                                          Mar 12, 2021
                                                          at 20:01,
                                                          Matthew
                                                          Benstead</div>
                                                          <div><a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:matthewb@uvic.ca"
target="_blank" moz-do-not-send="true">&lt;matthewb@uvic.ca&gt;</a>
                                                          wrote:</div>
                                                          </div>
                                                          <div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633">
                                                          <div>Hi
                                                          Strahil, <br>
                                                          <br>
                                                          Yes, SELinux
                                                          was put into
                                                          permissive
                                                          mode on the
                                                          secondary
                                                          nodes as well:
                                                          <br>
                                                          <br>
[root@pcic-backup01 ~]# sestatus | egrep -i  "^SELinux status|mode"<br>
                                                          SELinux
                                                          status:                
                                                          enabled<br>
                                                          Current
                                                          mode:                  
                                                          permissive<br>
                                                          Mode from
                                                          config
                                                          file:         
                                                          enforcing<br>
                                                          <br>
[root@pcic-backup02 ~]# sestatus | egrep -i  "^SELinux status|mode"<br>
                                                          SELinux
                                                          status:                
                                                          enabled<br>
                                                          Current
                                                          mode:                  
                                                          permissive<br>
                                                          Mode from
                                                          config
                                                          file:         
                                                          enforcing<br>
                                                          <br>
                                                          The secondary
                                                          server logs
                                                          didn't show
                                                          anything
                                                          interesting: <br>
                                                          <br>
                                                          gsyncd.log: <br>
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.81820]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:28.101819]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:28.107012]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:28.124567]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:28.128145]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:29.425739]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3184}]<br>
                                                          [2021-03-11
                                                          19:15:29.427448]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:29.433340]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3083}]<br>
                                                          [2021-03-11
                                                          19:15:29.434452]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3321}]<br>
                                                          [2021-03-11
                                                          19:15:29.434314]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:29.435575]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:29.439769]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3576}]<br>
                                                          [2021-03-11
                                                          19:15:29.440998]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:29.454745]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3262}]<br>
                                                          [2021-03-11
                                                          19:15:29.456192]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:32.594865]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:32.607815]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:32.647663]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:32.656280]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:32.668299]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:44.260689]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:44.271457]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:44.271883]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:44.279670]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:44.284261]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:45.614280]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3419}]<br>
                                                          [2021-03-11
                                                          19:15:45.615622]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:45.617986]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3461}]<br>
                                                          [2021-03-11
                                                          19:15:45.618180]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3380}]<br>
                                                          [2021-03-11
                                                          19:15:45.619539]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:45.618999]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:45.620843]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3361}]<br>
                                                          [2021-03-11
                                                          19:15:45.621347]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3604}]<br>
                                                          [2021-03-11
                                                          19:15:45.622179]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:45.622541]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:47.626054]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:48.778399]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:48.778491]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:48.796854]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:48.800697]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          <br>
                                                          The mnt
                                                          geo-rep files
                                                          were also
                                                          uninteresting:
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.250150]
                                                          I [MSGID:
                                                          100030]
                                                          [glusterfsd.c:2689:main]
0-/usr/sbin/glusterfs: Started running version
                                                          [{arg=/usr/sbin/glusterfs},
                                                          {version=8.3},
{cmdlinestr=/usr/sbin/glusterfs --user-map-root=g<br>
                                                          eoaccount
                                                          --aux-gfid-mount
                                                          --acl
                                                          --log-level=INFO
--log-file=/var/log/glusterfs/geo-replication-slaves/storage_10.0.231.81_pcic-backup/mnt-10.0.231.93-data-storage_b-storage.log
--volfile-server=localhost --volf<br>
ile-id=pcic-backup --client-pid=-1
                                                          /var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.253485]
                                                          I
                                                          [glusterfsd.c:2424:daemonize]
                                                          0-glusterfs:
                                                          Pid of current
                                                          running
                                                          process is
                                                          157484<br>
                                                          [2021-03-11
                                                          19:15:28.267911]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=0}] <br>
                                                          [2021-03-11
                                                          19:15:28.267984]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=1}] <br>
                                                          [2021-03-11
                                                          19:15:28.268371]
                                                          I
                                                          [glusterfsd-mgmt.c:2170:mgmt_getspec_cbk]
                                                          0-glusterfs:
                                                          Received list
                                                          of available
                                                          volfile
                                                          servers: <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.82:24007"
target="_blank" moz-do-not-send="true">
10.0.231.82:24007</a> <br>
                                                          [2021-03-11
                                                          19:15:28.271729]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=2}] <br>
                                                          [2021-03-11
                                                          19:15:28.271762]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=3}] <br>
                                                          [2021-03-11
                                                          19:15:28.272223]
                                                          I [MSGID:
                                                          114020]
                                                          [client.c:2315:notify]
0-pcic-backup-client-0: parent translators are ready, attempting connect
                                                          on transport
                                                          [] <br>
                                                          [2021-03-11
                                                          19:15:28.275883]
                                                          I [MSGID:
                                                          114020]
                                                          [client.c:2315:notify]
0-pcic-backup-client-1: parent translators are ready, attempting connect
                                                          on transport
                                                          [] <br>
                                                          [2021-03-11
                                                          19:15:28.276154]
                                                          I
                                                          [rpc-clnt.c:1975:rpc_clnt_reconfig]
0-pcic-backup-client-0: changing port to 49153 (from 0)<br>
                                                          [2021-03-11
                                                          19:15:28.276193]
                                                          I
                                                          [socket.c:849:__socket_shutdown]
0-pcic-backup-client-0: intentional socket shutdown(13)<br>
                                                          Final graph:<br>
                                                          ...<br>
+------------------------------------------------------------------------------+<br>
                                                          [2021-03-11
                                                          19:15:28.282144]
                                                          I
                                                          [socket.c:849:__socket_shutdown]
0-pcic-backup-client-1: intentional socket shutdown(15)<br>
                                                          [2021-03-11
                                                          19:15:28.286536]
                                                          I [MSGID:
                                                          114057]
                                                          [client-handshake.c:1128:select_server_supported_programs]
0-pcic-backup-client-0: Using Program [{Program-name=GlusterFS 4.x v1},
                                                          {Num=1298437},
                                                          {Version=400}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.287208]
                                                          I [MSGID:
                                                          114046]
                                                          [client-handshake.c:857:client_setvolume_cbk]
0-pcic-backup-client-0: Connected, attached to remote volume
                                                          [{conn-name=pcic-backup-client-0},
{remote_subvol=/data/brick}] <br>
                                                          [2021-03-11
                                                          19:15:28.290162]
                                                          I [MSGID:
                                                          114057]
                                                          [client-handshake.c:1128:select_server_supported_programs]
0-pcic-backup-client-1: Using Program [{Program-name=GlusterFS 4.x v1},
                                                          {Num=1298437},
                                                          {Version=400}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.291122]
                                                          I [MSGID:
                                                          114046]
                                                          [client-handshake.c:857:client_setvolume_cbk]
0-pcic-backup-client-1: Connected, attached to remote volume
                                                          [{conn-name=pcic-backup-client-1},
{remote_subvol=/data/brick}] <br>
                                                          [2021-03-11
                                                          19:15:28.292703]
                                                          I
                                                          [fuse-bridge.c:5300:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24
                                                          kernel 7.23<br>
                                                          [2021-03-11
                                                          19:15:28.292730]
                                                          I
                                                          [fuse-bridge.c:5926:fuse_graph_sync]
                                                          0-fuse:
                                                          switched to
                                                          graph 0<br>
                                                          [2021-03-11
                                                          19:15:32.809518]
                                                          I
                                                          [fuse-bridge.c:6242:fuse_thread_proc]
                                                          0-fuse:
                                                          initiating
                                                          unmount of
                                                          /var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI<br>
                                                          [2021-03-11
                                                          19:15:32.810216]
                                                          W
                                                          [glusterfsd.c:1439:cleanup_and_exit]
(--&gt;/lib64/libpthread.so.0(+0x7ea5) [0x7ff56b175ea5]
                                                          --&gt;/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5)
[0x55664e67db45] --&gt;/usr/sbin/glusterfs(cleanup_and_exit+0x6b)
                                                          [0x55664e67d9ab]
                                                          ) 0-: received
                                                          signum (15),
                                                          shutting down
                                                          <br>
                                                          [2021-03-11
                                                          19:15:32.810253]
                                                          I
                                                          [fuse-bridge.c:7074:fini]
                                                          0-fuse:
                                                          Unmounting
                                                          '/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI'.<br>
                                                          [2021-03-11
                                                          19:15:32.810268]
                                                          I
                                                          [fuse-bridge.c:7079:fini]
                                                          0-fuse:
                                                          Closing fuse
                                                          connection to
'/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI'.<br>
                                                          <br>
                                                          <br>
                                                          I'm really at
                                                          a loss for
                                                          where to go
                                                          from here, it
                                                          seems like
                                                          everything is
                                                          set up
                                                          correctly, and
                                                          it has been
                                                          working well
                                                          through the
                                                          7.x minor
                                                          versions, but
                                                          the jump to 8
                                                          has broken
                                                          something...<br>
                                                          <br>
                                                          There
                                                          definitely are
                                                          lots of
                                                          changelogs on
                                                          the servers
                                                          that fit into
                                                          the
                                                          timeframe.....
                                                          I haven't made
                                                          any writes to
                                                          the source
                                                          volume.... do
                                                          you think
                                                          that's the
                                                          problem? That
                                                          it needs some
                                                          new changelog
                                                          info to sync?
                                                          <br>
                                                          I had been
                                                          holding off
                                                          making any
                                                          writes in case
                                                          I needed to go
                                                          back to
                                                          Gluster7.9 -
                                                          not sure if
                                                          that's really
                                                          a good option
                                                          or not. <br>
                                                          <br>
[root@storage01 changelogs]# for dirs in {a,b,c}; do echo
                                                          "/data/storage_$dirs/storage/.glusterfs/changelogs";
                                                          ls -lh
                                                          /data/storage_$dirs/storage/.glusterfs/changelogs
                                                          | head; echo
                                                          ""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
                                                          total 16G<br>
                                                          drw-------. 3
                                                          root root   24
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:50
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root  13K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343197<br>
                                                          -rw-r--r--. 1
                                                          root root  51K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343212<br>
                                                          -rw-r--r--. 1
                                                          root root  86K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343227<br>
                                                          -rw-r--r--. 1
                                                          root root  99K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343242<br>
                                                          -rw-r--r--. 1
                                                          root root  69K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343257<br>
                                                          -rw-r--r--. 1
                                                          root root  69K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343272<br>
                                                          -rw-r--r--. 1
                                                          root root  72K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343287<br>
                                                          <br>
/data/storage_b/storage/.glusterfs/changelogs<br>
                                                          total 3.3G<br>
                                                          drw-------. 3
                                                          root root   24
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:50
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root  13K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343197<br>
                                                          -rw-r--r--. 1
                                                          root root  53K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343212<br>
                                                          -rw-r--r--. 1
                                                          root root  89K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343227<br>
                                                          -rw-r--r--. 1
                                                          root root  89K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343242<br>
                                                          -rw-r--r--. 1
                                                          root root  69K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343257<br>
                                                          -rw-r--r--. 1
                                                          root root  71K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343272<br>
                                                          -rw-r--r--. 1
                                                          root root  86K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343287<br>
                                                          <br>
/data/storage_c/storage/.glusterfs/changelogs<br>
                                                          total 9.6G<br>
                                                          drw-------. 3
                                                          root root   16
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:50
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root  16K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343199<br>
                                                          -rw-r--r--. 1
                                                          root root  71K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343214<br>
                                                          -rw-r--r--. 1
                                                          root root 122K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343229<br>
                                                          -rw-r--r--. 1
                                                          root root  73K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343244<br>
                                                          -rw-r--r--. 1
                                                          root root 100K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343259<br>
                                                          -rw-r--r--. 1
                                                          root root  95K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343274<br>
                                                          -rw-r--r--. 1
                                                          root root  92K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343289<br>
                                                          <br>
[root@storage01 changelogs]# for dirs in {a,b,c}; do echo
                                                          "/data/storage_$dirs/storage/.glusterfs/changelogs";
                                                          ls -lh
                                                          /data/storage_$dirs/storage/.glusterfs/changelogs
                                                          | tail; echo
                                                          ""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:33
                                                          CHANGELOG.1614663193<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663731<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663760<br>
                                                          -rw-r--r--. 1
                                                          root root  511
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664043<br>
                                                          -rw-r--r--. 1
                                                          root root  536
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664101<br>
                                                          -rw-r--r--. 1
                                                          root root 2.8K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664116<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666061<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666554<br>
                                                          drw-------. 2
                                                          root root   10
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   38
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
/data/storage_b/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663731<br>
                                                          -rw-r--r--. 1
                                                          root root  480
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663745<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663760<br>
                                                          -rw-r--r--. 1
                                                          root root  524
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664043<br>
                                                          -rw-r--r--. 1
                                                          root root  495
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664100<br>
                                                          -rw-r--r--. 1
                                                          root root 1.6K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664114<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666060<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666553<br>
                                                          drw-------. 2
                                                          root root   10
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   38
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
/data/storage_c/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663738<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663753<br>
                                                          -rw-r--r--. 1
                                                          root root  395
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664051<br>
                                                          -rw-r--r--. 1
                                                          root root  316
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664094<br>
                                                          -rw-r--r--. 1
                                                          root root 1.2K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664109<br>
                                                          -rw-r--r--. 1
                                                          root root  174
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664123<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666061<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666553<br>
                                                          drw-------. 2
                                                          root root    6
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   30
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
[root@storage02 ~]# for dirs in {a,b,c}; do echo
                                                          "/data/storage_$dirs/storage/.glusterfs/changelogs";
                                                          ls -lh
                                                          /data/storage_$dirs/storage/.glusterfs/changelogs
                                                          | head; echo
                                                          ""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
                                                          total 9.6G<br>
                                                          drw-------. 3
                                                          root root   24
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:50
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root 4.2K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343193<br>
                                                          -rw-r--r--. 1
                                                          root root  32K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343208<br>
                                                          -rw-r--r--. 1
                                                          root root 107K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343223<br>
                                                          -rw-r--r--. 1
                                                          root root 120K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343238<br>
                                                          -rw-r--r--. 1
                                                          root root  72K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343253<br>
                                                          -rw-r--r--. 1
                                                          root root 111K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343268<br>
                                                          -rw-r--r--. 1
                                                          root root  91K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343283<br>
                                                          <br>
/data/storage_b/storage/.glusterfs/changelogs<br>
                                                          total 16G<br>
                                                          drw-------. 3
                                                          root root   24
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:50
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root 3.9K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343193<br>
                                                          -rw-r--r--. 1
                                                          root root  35K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343208<br>
                                                          -rw-r--r--. 1
                                                          root root  85K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343223<br>
                                                          -rw-r--r--. 1
                                                          root root 103K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343238<br>
                                                          -rw-r--r--. 1
                                                          root root  70K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343253<br>
                                                          -rw-r--r--. 1
                                                          root root  72K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343268<br>
                                                          -rw-r--r--. 1
                                                          root root  73K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343283<br>
                                                          <br>
/data/storage_c/storage/.glusterfs/changelogs<br>
                                                          total 3.3G<br>
                                                          drw-------. 3
                                                          root root   16
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:51
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root  21K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343202<br>
                                                          -rw-r--r--. 1
                                                          root root  75K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343217<br>
                                                          -rw-r--r--. 1
                                                          root root  92K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343232<br>
                                                          -rw-r--r--. 1
                                                          root root  77K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343247<br>
                                                          -rw-r--r--. 1
                                                          root root  66K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343262<br>
                                                          -rw-r--r--. 1
                                                          root root  84K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343277<br>
                                                          -rw-r--r--. 1
                                                          root root  81K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343292<br>
                                                          <br>
[root@storage02 ~]# for dirs in {a,b,c}; do echo
                                                          "/data/storage_$dirs/storage/.glusterfs/changelogs";
                                                          ls -lh
                                                          /data/storage_$dirs/storage/.glusterfs/changelogs
                                                          | tail; echo
                                                          ""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663734<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663749<br>
                                                          -rw-r--r--. 1
                                                          root root  395
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664052<br>
                                                          -rw-r--r--. 1
                                                          root root  316
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664096<br>
                                                          -rw-r--r--. 1
                                                          root root 1.2K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664111<br>
                                                          -rw-r--r--. 1
                                                          root root  174
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664126<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666056<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666560<br>
                                                          drw-------. 2
                                                          root root   10
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   38
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
/data/storage_b/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663735<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663749<br>
                                                          -rw-r--r--. 1
                                                          root root  511
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664052<br>
                                                          -rw-r--r--. 1
                                                          root root  316
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664096<br>
                                                          -rw-r--r--. 1
                                                          root root 1.8K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664111<br>
                                                          -rw-r--r--. 1
                                                          root root 1.4K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664126<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666060<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666556<br>
                                                          drw-------. 2
                                                          root root   10
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   38
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
/data/storage_c/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663738<br>
                                                          -rw-r--r--. 1
                                                          root root  521
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663752<br>
                                                          -rw-r--r--. 1
                                                          root root  524
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664042<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664057<br>
                                                          -rw-r--r--. 1
                                                          root root  536
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664102<br>
                                                          -rw-r--r--. 1
                                                          root root 1.6K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664117<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666057<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666550<br>
                                                          drw-------. 2
                                                          root root    6
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   30
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
                                                          <br>
[root@storage03 ~]# for dirs in {a,b,c}; do echo
                                                          "/data/storage_$dirs/storage/.glusterfs/changelogs";
                                                          ls -lh
                                                          /data/storage_$dirs/storage/.glusterfs/changelogs
                                                          | head; echo
                                                          ""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
                                                          total 3.4G<br>
                                                          drw-------. 3
                                                          root root   24
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:50
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root  19K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343201<br>
                                                          -rw-r--r--. 1
                                                          root root  66K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343215<br>
                                                          -rw-r--r--. 1
                                                          root root  91K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343230<br>
                                                          -rw-r--r--. 1
                                                          root root  82K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343245<br>
                                                          -rw-r--r--. 1
                                                          root root  64K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343259<br>
                                                          -rw-r--r--. 1
                                                          root root  75K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343274<br>
                                                          -rw-r--r--. 1
                                                          root root  81K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343289<br>
                                                          <br>
/data/storage_b/storage/.glusterfs/changelogs<br>
                                                          total 9.6G<br>
                                                          drw-------. 3
                                                          root root   24
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:51
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root  19K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343201<br>
                                                          -rw-r--r--. 1
                                                          root root  80K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343215<br>
                                                          -rw-r--r--. 1
                                                          root root 119K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343230<br>
                                                          -rw-r--r--. 1
                                                          root root  65K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343244<br>
                                                          -rw-r--r--. 1
                                                          root root 100K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343259<br>
                                                          -rw-r--r--. 1
                                                          root root  95K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343274<br>
                                                          -rw-r--r--. 1
                                                          root root  92K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343289<br>
                                                          <br>
/data/storage_c/storage/.glusterfs/changelogs<br>
                                                          total 16G<br>
                                                          drw-------. 3
                                                          root root   16
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:51
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root 3.9K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343193<br>
                                                          -rw-r--r--. 1
                                                          root root  35K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343208<br>
                                                          -rw-r--r--. 1
                                                          root root  85K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343223<br>
                                                          -rw-r--r--. 1
                                                          root root 103K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343238<br>
                                                          -rw-r--r--. 1
                                                          root root  70K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343253<br>
                                                          -rw-r--r--. 1
                                                          root root  71K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343268<br>
                                                          -rw-r--r--. 1
                                                          root root  73K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343283<br>
                                                          <br>
[root@storage03 ~]# for dirs in {a,b,c}; do echo
                                                          "/data/storage_$dirs/storage/.glusterfs/changelogs";
                                                          ls -lh
                                                          /data/storage_$dirs/storage/.glusterfs/changelogs
                                                          | tail; echo
                                                          ""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:33
                                                          CHANGELOG.1614663183<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663740<br>
                                                          -rw-r--r--. 1
                                                          root root  521
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663755<br>
                                                          -rw-r--r--. 1
                                                          root root  524
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664049<br>
                                                          -rw-r--r--. 1
                                                          root root 1.9K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664106<br>
                                                          -rw-r--r--. 1
                                                          root root  174
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664121<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666051<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666559<br>
                                                          drw-------. 2
                                                          root root   10
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   38
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
/data/storage_b/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root  474
                                                          Mar  1 21:33
                                                          CHANGELOG.1614663182<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663739<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663753<br>
                                                          -rw-r--r--. 1
                                                          root root  395
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664049<br>
                                                          -rw-r--r--. 1
                                                          root root 1.4K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664106<br>
                                                          -rw-r--r--. 1
                                                          root root  174
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664120<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666063<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666557<br>
                                                          drw-------. 2
                                                          root root   10
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   38
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
/data/storage_c/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root  468
                                                          Mar  1 21:33
                                                          CHANGELOG.1614663183<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663740<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663754<br>
                                                          -rw-r--r--. 1
                                                          root root  511
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664048<br>
                                                          -rw-r--r--. 1
                                                          root root 2.0K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664105<br>
                                                          -rw-r--r--. 1
                                                          root root 1.4K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664120<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666063<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666556<br>
                                                          drw-------. 2
                                                          root root    6
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   30
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
                                                          Thanks,<br>
                                                           -Matthew<br>
                                                          <div>
                                                          <p>--<br>
                                                          Matthew
                                                          Benstead<br>
                                                          System
                                                          Administrator<br>
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="https://pacificclimate.org/"
target="_blank" moz-do-not-send="true">Pacific Climate Impacts
                                                          Consortium</a><br>
                                                          University of
                                                          Victoria, UH1<br>
                                                          PO Box 1800,
                                                          STN CSC<br>
                                                          Victoria, BC,
                                                          V8W 2Y2<br>
                                                          Phone:
                                                          +1-250-721-8432<br>
                                                          Email: <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:matthewb@uvic.ca"
target="_blank" moz-do-not-send="true">
matthewb@uvic.ca</a></p>
                                                          </div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yqt16014">
                                                          <div>On
                                                          3/11/21 11:37
                                                          PM, Strahil
                                                          Nikolov wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div>Notice:
                                                          This message
                                                          was sent from
                                                          outside the
                                                          University of
                                                          Victoria email
                                                          system. Please
                                                          be cautious
                                                          with links and
                                                          sensitive
                                                          information. </div>
                                                          <br>
                                                          <div>Have you
                                                          checked the
                                                          secondary
                                                          volume nodes'
                                                          logs &amp;
                                                          SELINUX status
                                                          ?
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615534613030"><br>
                                                          </div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615534613170">Best
                                                          Regards,</div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615534618885">Strahil
                                                          Nikolov<br>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615534604587"><br>
                                                          <blockquote>
                                                          <div>
                                                          <div>On Thu,
                                                          Mar 11, 2021
                                                          at 21:36,
                                                          Matthew
                                                          Benstead</div>
                                                          <div><a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:matthewb@uvic.ca"
target="_blank" moz-do-not-send="true">&lt;matthewb@uvic.ca&gt;</a>
                                                          wrote:</div>
                                                          </div>
                                                          <div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633">
                                                          <div>Hi
                                                          Strahil, <br>
                                                          <br>
                                                          It looks like
                                                          perhaps the
                                                          changelog_log_level
                                                          and log_level
                                                          options? I've
                                                          set them to
                                                          debug: <br>
                                                          <br>
[root@storage01 ~]# gluster volume geo-replication storage <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> config | egrep -i "log_level"<br>
changelog_log_level:INFO<br>
cli_log_level:INFO<br>
gluster_log_level:INFO<br>
                                                          log_level:INFO<br>
slave_gluster_log_level:INFO<br>
slave_log_level:INFO<br>
                                                          <br>
[root@storage01 ~]# gluster volume geo-replication storage <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> config changelog_log_level DEBUG<br>
geo-replication config updated successfully<br>
                                                          <br>
[root@storage01 ~]# gluster volume geo-replication storage <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> config log_level DEBUG<br>
geo-replication config updated successfully<br>
                                                          <br>
                                                          <br>
                                                          Then I
                                                          restarted
                                                          geo-replication:
                                                          <br>
                                                          <br>
[root@storage01 ~]# gluster volume geo-replication storage <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> start<br>
                                                          Starting
                                                          geo-replication
                                                          session
                                                          between
                                                          storage &amp;
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> has been successful<br>
[root@storage01 ~]# gluster volume geo-replication status <br>
                                                           <br>
                                                          MASTER NODE   
                                                          MASTER VOL   
                                                          MASTER
                                                          BRICK              
                                                          SLAVE USER   
SLAVE                                        SLAVE NODE   
                                                          STATUS            
                                                          CRAWL
                                                          STATUS   
                                                          LAST_SYNCED         
                                                          <br>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
                                                          10.0.231.91   
                                                          storage      
/data/storage_a/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                  <br>
                                                          10.0.231.91   
                                                          storage      
/data/storage_c/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                  <br>
                                                          10.0.231.91   
                                                          storage      
/data/storage_b/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                  <br>
                                                          10.0.231.92   
                                                          storage      
/data/storage_b/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                  <br>
                                                          10.0.231.92   
                                                          storage      
/data/storage_a/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                  <br>
                                                          10.0.231.92   
                                                          storage      
/data/storage_c/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                  <br>
                                                          10.0.231.93   
                                                          storage      
/data/storage_c/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                  <br>
                                                          10.0.231.93   
                                                          storage      
/data/storage_b/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                  <br>
                                                          10.0.231.93   
                                                          storage      
/data/storage_a/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                  <br>
[root@storage01 ~]# gluster volume geo-replication status <br>
                                                           <br>
                                                          MASTER NODE   
                                                          MASTER VOL   
                                                          MASTER
                                                          BRICK              
                                                          SLAVE USER   
SLAVE                                        SLAVE NODE    STATUS   
                                                          CRAWL
                                                          STATUS   
                                                          LAST_SYNCED         
                                                          <br>
---------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
                                                          10.0.231.91   
                                                          storage      
/data/storage_a/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                  <br>
                                                          10.0.231.91   
                                                          storage      
/data/storage_c/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                  <br>
                                                          10.0.231.91   
                                                          storage      
/data/storage_b/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                  <br>
                                                          10.0.231.92   
                                                          storage      
/data/storage_b/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                  <br>
                                                          10.0.231.92   
                                                          storage      
/data/storage_a/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                  <br>
                                                          10.0.231.92   
                                                          storage      
/data/storage_c/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                  <br>
                                                          10.0.231.93   
                                                          storage      
/data/storage_c/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                  <br>
                                                          10.0.231.93   
                                                          storage      
/data/storage_b/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                  <br>
                                                          10.0.231.93   
                                                          storage      
/data/storage_a/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A  <br>
                                                          <br>
[root@storage01 ~]# gluster volume geo-replication storage <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> stop<br>
                                                          Stopping
                                                          geo-replication
                                                          session
                                                          between
                                                          storage &amp;
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> has been successful<br>
                                                          <br>
                                                          <br>
                                                          The changelogs
                                                          didn't really
                                                          show anything
                                                          new around
                                                          changelog
                                                          selection: <br>
                                                          <br>
[root@storage01 storage_10.0.231.81_pcic-backup]# cat
                                                          changes-data-storage_a-storage.log
                                                          | egrep
                                                          "2021-03-11"<br>
                                                          [2021-03-11
                                                          19:15:30.552889]
                                                          I [MSGID:
                                                          132028]
                                                          [gf-changelog.c:577:gf_changelog_register_generic]
                                                          0-gfchangelog:
                                                          Registering
                                                          brick
                                                          [{brick=/data/storage_a/storage},
{notify_filter=1}] <br>
                                                          [2021-03-11
                                                          19:15:30.552893]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=0}] <br>
                                                          [2021-03-11
                                                          19:15:30.552894]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=1}] <br>
                                                          [2021-03-11
                                                          19:15:30.553633]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=3}] <br>
                                                          [2021-03-11
                                                          19:15:30.553634]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=2}] <br>
                                                          [2021-03-11
                                                          19:15:30.554236]
                                                          D
                                                          [rpcsvc.c:2831:rpcsvc_init]
                                                          0-rpc-service:
                                                          RPC service
                                                          inited.<br>
                                                          [2021-03-11
                                                          19:15:30.554403]
                                                          D
                                                          [rpcsvc.c:2342:rpcsvc_program_register]
                                                          0-rpc-service:
                                                          New program
                                                          registered:
                                                          GF-DUMP, Num:
                                                          123451501,
                                                          Ver: 1, Port:
                                                          0<br>
                                                          [2021-03-11
                                                          19:15:30.554420]
                                                          D
                                                          [rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
                                                          /usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
                                                          [2021-03-11
                                                          19:15:30.554933]
                                                          D
                                                          [socket.c:4485:socket_init]
0-socket.gfchangelog: disabling nodelay<br>
                                                          [2021-03-11
                                                          19:15:30.554944]
                                                          D
                                                          [socket.c:4523:socket_init]
0-socket.gfchangelog: Configured transport.tcp-user-timeout=42<br>
                                                          [2021-03-11
                                                          19:15:30.554949]
                                                          D
                                                          [socket.c:4543:socket_init]
0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9<br>
                                                          [2021-03-11
                                                          19:15:30.555002]
                                                          I
                                                          [socket.c:929:__socket_server_bind]
0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 23<br>
                                                          [2021-03-11
                                                          19:15:30.555324]
                                                          D
                                                          [rpcsvc.c:2342:rpcsvc_program_register]
                                                          0-rpc-service:
                                                          New program
                                                          registered:
                                                          LIBGFCHANGELOG
                                                          REBORP, Num:
                                                          1886350951,
                                                          Ver: 1, Port:
                                                          0<br>
                                                          [2021-03-11
                                                          19:15:30.555345]
                                                          D
                                                          [rpc-clnt.c:1020:rpc_clnt_connection_init]
                                                          0-gfchangelog:
                                                          defaulting
                                                          frame-timeout
                                                          to 30mins<br>
                                                          [2021-03-11
                                                          19:15:30.555351]
                                                          D
                                                          [rpc-clnt.c:1032:rpc_clnt_connection_init]
                                                          0-gfchangelog:
                                                          disable
                                                          ping-timeout<br>
                                                          [2021-03-11
                                                          19:15:30.555358]
                                                          D
                                                          [rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
                                                          /usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
                                                          [2021-03-11
                                                          19:15:30.555399]
                                                          D
                                                          [socket.c:4485:socket_init]
                                                          0-gfchangelog:
                                                          disabling
                                                          nodelay<br>
                                                          [2021-03-11
                                                          19:15:30.555406]
                                                          D
                                                          [socket.c:4523:socket_init]
                                                          0-gfchangelog:
                                                          Configured
                                                          transport.tcp-user-timeout=42<br>
                                                          [2021-03-11
                                                          19:15:32.555711]
                                                          D
                                                          [rpc-clnt-ping.c:298:rpc_clnt_start_ping]
                                                          0-gfchangelog:
                                                          ping timeout
                                                          is 0,
                                                          returning<br>
                                                          [2021-03-11
                                                          19:15:32.572157]
                                                          I [MSGID:
                                                          132035]
                                                          [gf-history-changelog.c:837:gf_history_changelog]
                                                          0-gfchangelog:
                                                          Requesting
                                                          historical
                                                          changelogs
                                                          [{start=1614666553},
{end=1615490132}] <br>
                                                          [2021-03-11
                                                          19:15:32.572436]
                                                          I [MSGID:
                                                          132019]
                                                          [gf-history-changelog.c:755:gf_changelog_extract_min_max]
                                                          0-gfchangelog:
                                                          changelogs min
                                                          max
                                                          [{min=1597342860},
{max=1615490121}, {total_changelogs=1256897}] <br>
                                                          [2021-03-11
                                                          19:15:32.621244]
                                                          E [MSGID:
                                                          132009]
                                                          [gf-history-changelog.c:941:gf_history_changelog]
                                                          0-gfchangelog:
                                                          wrong result
                                                          [{for=end},
                                                          {start=1615490121},
                                                          {idx=1256896}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:46.733182]
                                                          I [MSGID:
                                                          132028]
                                                          [gf-changelog.c:577:gf_changelog_register_generic]
                                                          0-gfchangelog:
                                                          Registering
                                                          brick
                                                          [{brick=/data/storage_a/storage},
{notify_filter=1}] <br>
                                                          [2021-03-11
                                                          19:15:46.733316]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=0}] <br>
                                                          [2021-03-11
                                                          19:15:46.733348]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=1}] <br>
                                                          [2021-03-11
                                                          19:15:46.734031]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=2}] <br>
                                                          [2021-03-11
                                                          19:15:46.734085]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=3}] <br>
                                                          [2021-03-11
                                                          19:15:46.734591]
                                                          D
                                                          [rpcsvc.c:2831:rpcsvc_init]
                                                          0-rpc-service:
                                                          RPC service
                                                          inited.<br>
                                                          [2021-03-11
                                                          19:15:46.734755]
                                                          D
                                                          [rpcsvc.c:2342:rpcsvc_program_register]
                                                          0-rpc-service:
                                                          New program
                                                          registered:
                                                          GF-DUMP, Num:
                                                          123451501,
                                                          Ver: 1, Port:
                                                          0<br>
                                                          [2021-03-11
                                                          19:15:46.734772]
                                                          D
                                                          [rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
                                                          /usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
                                                          [2021-03-11
                                                          19:15:46.735256]
                                                          D
                                                          [socket.c:4485:socket_init]
0-socket.gfchangelog: disabling nodelay<br>
                                                          [2021-03-11
                                                          19:15:46.735266]
                                                          D
                                                          [socket.c:4523:socket_init]
0-socket.gfchangelog: Configured transport.tcp-user-timeout=42<br>
                                                          [2021-03-11
                                                          19:15:46.735271]
                                                          D
                                                          [socket.c:4543:socket_init]
0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9<br>
                                                          [2021-03-11
                                                          19:15:46.735325]
                                                          I
                                                          [socket.c:929:__socket_server_bind]
0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 21<br>
                                                          [2021-03-11
                                                          19:15:46.735704]
                                                          D
                                                          [rpcsvc.c:2342:rpcsvc_program_register]
                                                          0-rpc-service:
                                                          New program
                                                          registered:
                                                          LIBGFCHANGELOG
                                                          REBORP, Num:
                                                          1886350951,
                                                          Ver: 1, Port:
                                                          0<br>
                                                          [2021-03-11
                                                          19:15:46.735721]
                                                          D
                                                          [rpc-clnt.c:1020:rpc_clnt_connection_init]
                                                          0-gfchangelog:
                                                          defaulting
                                                          frame-timeout
                                                          to 30mins<br>
                                                          [2021-03-11
                                                          19:15:46.735726]
                                                          D
                                                          [rpc-clnt.c:1032:rpc_clnt_connection_init]
                                                          0-gfchangelog:
                                                          disable
                                                          ping-timeout<br>
                                                          [2021-03-11
                                                          19:15:46.735733]
                                                          D
                                                          [rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
                                                          /usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
                                                          [2021-03-11
                                                          19:15:46.735771]
                                                          D
                                                          [socket.c:4485:socket_init]
                                                          0-gfchangelog:
                                                          disabling
                                                          nodelay<br>
                                                          [2021-03-11
                                                          19:15:46.735778]
                                                          D
                                                          [socket.c:4523:socket_init]
                                                          0-gfchangelog:
                                                          Configured
                                                          transport.tcp-user-timeout=42<br>
                                                          [2021-03-11
                                                          19:15:47.618464]
                                                          D
                                                          [rpc-clnt-ping.c:298:rpc_clnt_start_ping]
                                                          0-gfchangelog:
                                                          ping timeout
                                                          is 0,
                                                          returning<br>
                                                          <br>
                                                          <br>
[root@storage01 storage_10.0.231.81_pcic-backup]# cat
                                                          changes-data-storage_b-storage.log
                                                          | egrep
                                                          "2021-03-11"<br>
                                                          [2021-03-11
                                                          19:15:30.611457]
                                                          I [MSGID:
                                                          132028]
                                                          [gf-changelog.c:577:gf_changelog_register_generic]
                                                          0-gfchangelog:
                                                          Registering
                                                          brick
                                                          [{brick=/data/storage_b/storage},
{notify_filter=1}] <br>
                                                          [2021-03-11
                                                          19:15:30.611574]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=1}] <br>
                                                          [2021-03-11
                                                          19:15:30.611641]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=3}] <br>
                                                          [2021-03-11
                                                          19:15:30.611645]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=2}] <br>
                                                          [2021-03-11
                                                          19:15:30.612325]
                                                          D
                                                          [rpcsvc.c:2831:rpcsvc_init]
                                                          0-rpc-service:
                                                          RPC service
                                                          inited.<br>
                                                          [2021-03-11
                                                          19:15:30.612488]
                                                          D
                                                          [rpcsvc.c:2342:rpcsvc_program_register]
                                                          0-rpc-service:
                                                          New program
                                                          registered:
                                                          GF-DUMP, Num:
                                                          123451501,
                                                          Ver: 1, Port:
                                                          0<br>
                                                          [2021-03-11
                                                          19:15:30.612507]
                                                          D
                                                          [rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
                                                          /usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
                                                          [2021-03-11
                                                          19:15:30.613005]
                                                          D
                                                          [socket.c:4485:socket_init]
0-socket.gfchangelog: disabling nodelay<br>
                                                          [2021-03-11
                                                          19:15:30.613130]
                                                          D
                                                          [socket.c:4523:socket_init]
0-socket.gfchangelog: Configured transport.tcp-user-timeout=42<br>
                                                          [2021-03-11
                                                          19:15:30.613142]
                                                          D
                                                          [socket.c:4543:socket_init]
0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9<br>
                                                          [2021-03-11
                                                          19:15:30.613208]
                                                          I
                                                          [socket.c:929:__socket_server_bind]
0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 22<br>
                                                          [2021-03-11
                                                          19:15:30.613545]
                                                          D
                                                          [rpcsvc.c:2342:rpcsvc_program_register]
                                                          0-rpc-service:
                                                          New program
                                                          registered:
                                                          LIBGFCHANGELOG
                                                          REBORP, Num:
                                                          1886350951,
                                                          Ver: 1, Port:
                                                          0<br>
                                                          [2021-03-11
                                                          19:15:30.613567]
                                                          D
                                                          [rpc-clnt.c:1020:rpc_clnt_connection_init]
                                                          0-gfchangelog:
                                                          defaulting
                                                          frame-timeout
                                                          to 30mins<br>
                                                          [2021-03-11
                                                          19:15:30.613574]
                                                          D
                                                          [rpc-clnt.c:1032:rpc_clnt_connection_init]
                                                          0-gfchangelog:
                                                          disable
                                                          ping-timeout<br>
                                                          [2021-03-11
                                                          19:15:30.613582]
                                                          D
                                                          [rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
                                                          /usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
                                                          [2021-03-11
                                                          19:15:30.613637]
                                                          D
                                                          [socket.c:4485:socket_init]
                                                          0-gfchangelog:
                                                          disabling
                                                          nodelay<br>
                                                          [2021-03-11
                                                          19:15:30.613654]
                                                          D
                                                          [socket.c:4523:socket_init]
                                                          0-gfchangelog:
                                                          Configured
                                                          transport.tcp-user-timeout=42<br>
                                                          [2021-03-11
                                                          19:15:32.614273]
                                                          D
                                                          [rpc-clnt-ping.c:298:rpc_clnt_start_ping]
                                                          0-gfchangelog:
                                                          ping timeout
                                                          is 0,
                                                          returning<br>
                                                          [2021-03-11
                                                          19:15:32.643628]
                                                          I [MSGID:
                                                          132035]
                                                          [gf-history-changelog.c:837:gf_history_changelog]
                                                          0-gfchangelog:
                                                          Requesting
                                                          historical
                                                          changelogs
                                                          [{start=1614666552},
{end=1615490132}] <br>
                                                          [2021-03-11
                                                          19:15:32.643716]
                                                          I [MSGID:
                                                          132019]
                                                          [gf-history-changelog.c:755:gf_changelog_extract_min_max]
                                                          0-gfchangelog:
                                                          changelogs min
                                                          max
                                                          [{min=1597342860},
{max=1615490123}, {total_changelogs=1264296}] <br>
                                                          [2021-03-11
                                                          19:15:32.700397]
                                                          E [MSGID:
                                                          132009]
                                                          [gf-history-changelog.c:941:gf_history_changelog]
                                                          0-gfchangelog:
                                                          wrong result
                                                          [{for=end},
                                                          {start=1615490123},
                                                          {idx=1264295}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:46.832322]
                                                          I [MSGID:
                                                          132028]
                                                          [gf-changelog.c:577:gf_changelog_register_generic]
                                                          0-gfchangelog:
                                                          Registering
                                                          brick
                                                          [{brick=/data/storage_b/storage},
{notify_filter=1}] <br>
                                                          [2021-03-11
                                                          19:15:46.832394]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=0}] <br>
                                                          [2021-03-11
                                                          19:15:46.832465]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=1}] <br>
                                                          [2021-03-11
                                                          19:15:46.832531]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=2}] <br>
                                                          [2021-03-11
                                                          19:15:46.833086]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=3}] <br>
                                                          [2021-03-11
                                                          19:15:46.833648]
                                                          D
                                                          [rpcsvc.c:2831:rpcsvc_init]
                                                          0-rpc-service:
                                                          RPC service
                                                          inited.<br>
                                                          [2021-03-11
                                                          19:15:46.833817]
                                                          D
                                                          [rpcsvc.c:2342:rpcsvc_program_register]
                                                          0-rpc-service:
                                                          New program
                                                          registered:
                                                          GF-DUMP, Num:
                                                          123451501,
                                                          Ver: 1, Port:
                                                          0<br>
                                                          [2021-03-11
                                                          19:15:46.833835]
                                                          D
                                                          [rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
                                                          /usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
                                                          [2021-03-11
                                                          19:15:46.834368]
                                                          D
                                                          [socket.c:4485:socket_init]
0-socket.gfchangelog: disabling nodelay<br>
                                                          [2021-03-11
                                                          19:15:46.834380]
                                                          D
                                                          [socket.c:4523:socket_init]
0-socket.gfchangelog: Configured transport.tcp-user-timeout=42<br>
                                                          [2021-03-11
                                                          19:15:46.834386]
                                                          D
                                                          [socket.c:4543:socket_init]
0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9<br>
                                                          [2021-03-11
                                                          19:15:46.834441]
                                                          I
                                                          [socket.c:929:__socket_server_bind]
0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 23<br>
                                                          [2021-03-11
                                                          19:15:46.834768]
                                                          D
                                                          [rpcsvc.c:2342:rpcsvc_program_register]
                                                          0-rpc-service:
                                                          New program
                                                          registered:
                                                          LIBGFCHANGELOG
                                                          REBORP, Num:
                                                          1886350951,
                                                          Ver: 1, Port:
                                                          0<br>
                                                          [2021-03-11
                                                          19:15:46.834789]
                                                          D
                                                          [rpc-clnt.c:1020:rpc_clnt_connection_init]
                                                          0-gfchangelog:
                                                          defaulting
                                                          frame-timeout
                                                          to 30mins<br>
                                                          [2021-03-11
                                                          19:15:46.834795]
                                                          D
                                                          [rpc-clnt.c:1032:rpc_clnt_connection_init]
                                                          0-gfchangelog:
                                                          disable
                                                          ping-timeout<br>
                                                          [2021-03-11
                                                          19:15:46.834802]
                                                          D
                                                          [rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
                                                          /usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
                                                          [2021-03-11
                                                          19:15:46.834845]
                                                          D
                                                          [socket.c:4485:socket_init]
                                                          0-gfchangelog:
                                                          disabling
                                                          nodelay<br>
                                                          [2021-03-11
                                                          19:15:46.834853]
                                                          D
                                                          [socket.c:4523:socket_init]
                                                          0-gfchangelog:
                                                          Configured
                                                          transport.tcp-user-timeout=42<br>
                                                          [2021-03-11
                                                          19:15:47.618476]
                                                          D
                                                          [rpc-clnt-ping.c:298:rpc_clnt_start_ping]
                                                          0-gfchangelog:
                                                          ping timeout
                                                          is 0,
                                                          returning<br>
                                                          <br>
                                                          <br>
                                                          gsyncd logged
                                                          a lot but I'm
                                                          not sure if
                                                          it's helpful:
                                                          <br>
                                                          <br>
                                                          [2021-03-11
                                                          19:15:00.41898]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:21.551302]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:21.631470]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:21.718386]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:21.804991]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:26.203999]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:26.284775]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:26.573355]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:26.653752]
                                                          D
                                                          [gsyncd(monitor):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:26.756994]
                                                          D
                                                          [monitor(monitor):304:distribute]
                                                          &lt;top&gt;:
                                                          master bricks:
                                                          [{'host':
                                                          '10.0.231.91',
                                                          'uuid':
                                                          'afc24654-2887-41f6-a9c2-8e835de243b6',
                                                          'dir':
                                                          '/data/storage_a/storage'},
                                                          {'host':
                                                          '10.0.2<br>
                                                          31.92',
                                                          'uuid':
                                                          'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d',
                                                          'dir':
                                                          '/data/storage_b/storage'},
                                                          {'host':
                                                          '10.0.231.93',
                                                          'uuid':
                                                          '8b28b331-3780-46bc-9da3-fb27de4ab57b',
                                                          'dir':
                                                          '/data/storage_c/storage'},
                                                          {'host': '10.<br>
                                                          0.231.92',
                                                          'uuid':
                                                          'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d',
                                                          'dir':
                                                          '/data/storage_a/storage'},
                                                          {'host':
                                                          '10.0.231.93',
                                                          'uuid':
                                                          '8b28b331-3780-46bc-9da3-fb27de4ab57b',
                                                          'dir':
                                                          '/data/storage_b/storage'},
                                                          {'host': '<br>
                                                          10.0.231.91',
                                                          'uuid':
                                                          'afc24654-2887-41f6-a9c2-8e835de243b6',
                                                          'dir':
                                                          '/data/storage_c/storage'},
                                                          {'host':
                                                          '10.0.231.93',
                                                          'uuid':
                                                          '8b28b331-3780-46bc-9da3-fb27de4ab57b',
                                                          'dir':
                                                          '/data/storage_a/storage'},
                                                          {'host'<br>
                                                          :
                                                          '10.0.231.91',
                                                          'uuid':
                                                          'afc24654-2887-41f6-a9c2-8e835de243b6',
                                                          'dir':
                                                          '/data/storage_b/storage'},
                                                          {'host':
                                                          '10.0.231.92',
                                                          'uuid':
                                                          'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d',
                                                          'dir':
                                                          '/data/storage_c/storage'}]<br>
                                                          [2021-03-11
                                                          19:15:26.757252]
                                                          D
                                                          [monitor(monitor):314:distribute]
                                                          &lt;top&gt;:
                                                          slave SSH
                                                          gateway: <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81</a><br>
                                                          [2021-03-11
                                                          19:15:27.416235]
                                                          D
                                                          [monitor(monitor):334:distribute]
                                                          &lt;top&gt;:
                                                          slave bricks:
                                                          [{'host':
                                                          '10.0.231.81',
                                                          'uuid':
                                                          'b88dea4f-31ec-416a-9110-3ccdc3910acd',
                                                          'dir':
                                                          '/data/brick'},
                                                          {'host':
                                                          '10.0.231.82',
                                                          'uuid<br>
                                                          ':
                                                          'be50a8de-3934-4fee-a80d-8e2e99017902',
                                                          'dir':
                                                          '/data/brick'}]<br>
                                                          [2021-03-11
                                                          19:15:27.416825]
                                                          D
                                                          [syncdutils(monitor):932:is_hot]
                                                          Volinfo:
                                                          brickpath:
                                                          '10.0.231.91:/data/storage_a/storage'<br>
                                                          [2021-03-11
                                                          19:15:27.417273]
                                                          D
                                                          [syncdutils(monitor):932:is_hot]
                                                          Volinfo:
                                                          brickpath:
                                                          '10.0.231.91:/data/storage_c/storage'<br>
                                                          [2021-03-11
                                                          19:15:27.417515]
                                                          D
                                                          [syncdutils(monitor):932:is_hot]
                                                          Volinfo:
                                                          brickpath:
                                                          '10.0.231.91:/data/storage_b/storage'<br>
                                                          [2021-03-11
                                                          19:15:27.417763]
                                                          D
                                                          [monitor(monitor):348:distribute]
                                                          &lt;top&gt;:
                                                          worker specs:
                                                          [({'host':
                                                          '10.0.231.91',
                                                          'uuid':
                                                          'afc24654-2887-41f6-a9c2-8e835de243b6',
                                                          'dir':
                                                          '/data/storage_a/storage'},
('geoaccount@10.<br>
                                                          0.231.81',
                                                          'b88dea4f-31ec-416a-9110-3ccdc3910acd'),
                                                          '1', False),
                                                          ({'host':
                                                          '10.0.231.91',
                                                          'uuid':
                                                          'afc24654-2887-41f6-a9c2-8e835de243b6',
                                                          'dir':
                                                          '/data/storage_c/storage'},
                                                          ('<a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.82"
target="_blank" moz-do-not-send="true">geoaccount@10.0.231.82</a>',
                                                          'be50a8de-3<br>
934-4fee-a80d-8e2e99017902'), '2', False), ({'host': '10.0.231.91',
                                                          'uuid':
                                                          'afc24654-2887-41f6-a9c2-8e835de243b6',
                                                          'dir':
                                                          '/data/storage_b/storage'},
                                                          ('<a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.82"
target="_blank" moz-do-not-send="true">geoaccount@10.0.231.82</a>',
                                                          'be50a8de-3934-4fee-a80d-8e2e9901<br>
                                                          7902'), '3',
                                                          False)]<br>
                                                          [2021-03-11
                                                          19:15:27.425009]
                                                          I
                                                          [monitor(monitor):160:monitor]
                                                          Monitor:
                                                          starting
                                                          gsyncd worker
[{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]<br>
                                                          [2021-03-11
                                                          19:15:27.426764]
                                                          I
                                                          [monitor(monitor):160:monitor]
                                                          Monitor:
                                                          starting
                                                          gsyncd worker
[{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]<br>
                                                          [2021-03-11
                                                          19:15:27.429208]
                                                          I
                                                          [monitor(monitor):160:monitor]
                                                          Monitor:
                                                          starting
                                                          gsyncd worker
[{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]<br>
                                                          [2021-03-11
                                                          19:15:27.432280]
                                                          D
                                                          [monitor(monitor):195:monitor]
                                                          Monitor:
                                                          Worker would
                                                          mount volume
                                                          privately<br>
                                                          [2021-03-11
                                                          19:15:27.434195]
                                                          D
                                                          [monitor(monitor):195:monitor]
                                                          Monitor:
                                                          Worker would
                                                          mount volume
                                                          privately<br>
                                                          [2021-03-11
                                                          19:15:27.436584]
                                                          D
                                                          [monitor(monitor):195:monitor]
                                                          Monitor:
                                                          Worker would
                                                          mount volume
                                                          privately<br>
                                                          [2021-03-11
                                                          19:15:27.478806]
                                                          D
                                                          [gsyncd(worker
/data/storage_c/storage):303:main] &lt;top&gt;: Using session config
                                                          file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:27.478852]
                                                          D
                                                          [gsyncd(worker
/data/storage_b/storage):303:main] &lt;top&gt;: Using session config
                                                          file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:27.480104]
                                                          D
                                                          [gsyncd(worker
/data/storage_a/storage):303:main] &lt;top&gt;: Using session config
                                                          file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:27.500456]
                                                          I
                                                          [resource(worker
/data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH
                                                          connection
                                                          between master
                                                          and slave...<br>
                                                          [2021-03-11
                                                          19:15:27.501375]
                                                          I
                                                          [resource(worker
/data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH
                                                          connection
                                                          between master
                                                          and slave...<br>
                                                          [2021-03-11
                                                          19:15:27.502003]
                                                          I
                                                          [resource(worker
/data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH
                                                          connection
                                                          between master
                                                          and slave...<br>
                                                          [2021-03-11
                                                          19:15:27.525511]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):195:push] RepceClient: call
                                                          192117:140572692309824:1615490127.53
__repce_version__() ...<br>
                                                          [2021-03-11
                                                          19:15:27.525582]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):195:push] RepceClient: call
                                                          192115:139891296405312:1615490127.53
__repce_version__() ...<br>
                                                          [2021-03-11
                                                          19:15:27.526089]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):195:push] RepceClient: call
                                                          192114:140388828780352:1615490127.53
__repce_version__() ...<br>
                                                          [2021-03-11
                                                          19:15:29.435985]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
                                                          192117:140572692309824:1615490127.53
__repce_version__ -&gt; 1.0<br>
                                                          [2021-03-11
                                                          19:15:29.436213]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):195:push] RepceClient: call
                                                          192117:140572692309824:1615490129.44
                                                          version() ...<br>
                                                          [2021-03-11
                                                          19:15:29.437136]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
                                                          192117:140572692309824:1615490129.44
                                                          version -&gt;
                                                          1.0<br>
                                                          [2021-03-11
                                                          19:15:29.437268]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):195:push] RepceClient: call
                                                          192117:140572692309824:1615490129.44
                                                          pid() ...<br>
                                                          [2021-03-11
                                                          19:15:29.437915]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
                                                          192117:140572692309824:1615490129.44
                                                          pid -&gt;
                                                          157321<br>
                                                          [2021-03-11
                                                          19:15:29.438004]
                                                          I
                                                          [resource(worker
/data/storage_a/storage):1436:connect_remote] SSH: SSH connection
                                                          between master
                                                          and slave
                                                          established.
                                                          [{duration=1.9359}]<br>
                                                          [2021-03-11
                                                          19:15:29.438072]
                                                          I
                                                          [resource(worker
/data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:29.494538]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
                                                          192115:139891296405312:1615490127.53
__repce_version__ -&gt; 1.0<br>
                                                          [2021-03-11
                                                          19:15:29.494748]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):195:push] RepceClient: call
                                                          192115:139891296405312:1615490129.49
                                                          version() ...<br>
                                                          [2021-03-11
                                                          19:15:29.495290]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
                                                          192115:139891296405312:1615490129.49
                                                          version -&gt;
                                                          1.0<br>
                                                          [2021-03-11
                                                          19:15:29.495400]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):195:push] RepceClient: call
                                                          192115:139891296405312:1615490129.5
                                                          pid() ...<br>
                                                          [2021-03-11
                                                          19:15:29.495872]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
                                                          192115:139891296405312:1615490129.5
                                                          pid -&gt;
                                                          88110<br>
                                                          [2021-03-11
                                                          19:15:29.495960]
                                                          I
                                                          [resource(worker
/data/storage_b/storage):1436:connect_remote] SSH: SSH connection
                                                          between master
                                                          and slave
                                                          established.
                                                          [{duration=1.9944}]<br>
                                                          [2021-03-11
                                                          19:15:29.496028]
                                                          I
                                                          [resource(worker
/data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:29.501255]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
                                                          192114:140388828780352:1615490127.53
__repce_version__ -&gt; 1.0<br>
                                                          [2021-03-11
                                                          19:15:29.501454]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):195:push] RepceClient: call
                                                          192114:140388828780352:1615490129.5
                                                          version() ...<br>
                                                          [2021-03-11
                                                          19:15:29.502258]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
                                                          192114:140388828780352:1615490129.5
                                                          version -&gt;
                                                          1.0<br>
                                                          [2021-03-11
                                                          19:15:29.502444]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):195:push] RepceClient: call
                                                          192114:140388828780352:1615490129.5
                                                          pid() ...<br>
                                                          [2021-03-11
                                                          19:15:29.503140]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
                                                          192114:140388828780352:1615490129.5
                                                          pid -&gt;
                                                          88111<br>
                                                          [2021-03-11
                                                          19:15:29.503232]
                                                          I
                                                          [resource(worker
/data/storage_c/storage):1436:connect_remote] SSH: SSH connection
                                                          between master
                                                          and slave
                                                          established.
                                                          [{duration=2.0026}]<br>
                                                          [2021-03-11
                                                          19:15:29.503302]
                                                          I
                                                          [resource(worker
/data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:29.533899]
                                                          D
                                                          [resource(worker
/data/storage_a/storage):880:inhibit] DirectMounter: auxiliary glusterfs
                                                          mount in place<br>
                                                          [2021-03-11
                                                          19:15:29.595736]
                                                          D
                                                          [resource(worker
/data/storage_b/storage):880:inhibit] DirectMounter: auxiliary glusterfs
                                                          mount in place<br>
                                                          [2021-03-11
                                                          19:15:29.601110]
                                                          D
                                                          [resource(worker
/data/storage_c/storage):880:inhibit] DirectMounter: auxiliary glusterfs
                                                          mount in place<br>
                                                          [2021-03-11
                                                          19:15:30.541542]
                                                          D
                                                          [resource(worker
/data/storage_a/storage):964:inhibit] DirectMounter: auxiliary glusterfs
                                                          mount prepared<br>
                                                          [2021-03-11
                                                          19:15:30.541816]
                                                          I
                                                          [resource(worker
/data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume
                                                          [{duration=1.1037}]<br>
                                                          [2021-03-11
                                                          19:15:30.541887]
                                                          I
                                                          [subcmds(worker
/data/storage_a/storage):84:subcmd_worker] &lt;top&gt;: Worker spawn
                                                          successful.
                                                          Acknowledging
                                                          back to
                                                          monitor<br>
                                                          [2021-03-11
                                                          19:15:30.542042]
                                                          D
                                                          [master(worker
/data/storage_a/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
                                                          [{mode=xsync}]<br>
                                                          [2021-03-11
                                                          19:15:30.542125]
                                                          D
                                                          [monitor(monitor):222:monitor]
                                                          Monitor:
                                                          worker(/data/storage_a/storage)
                                                          connected<br>
                                                          [2021-03-11
                                                          19:15:30.543323]
                                                          D
                                                          [master(worker
/data/storage_a/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
[{mode=changelog}]<br>
                                                          [2021-03-11
                                                          19:15:30.544460]
                                                          D
                                                          [master(worker
/data/storage_a/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
[{mode=changeloghistory}]<br>
                                                          [2021-03-11
                                                          19:15:30.552103]
                                                          D
                                                          [master(worker
/data/storage_a/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage<br>
                                                          [2021-03-11
                                                          19:15:30.602937]
                                                          D
                                                          [resource(worker
/data/storage_b/storage):964:inhibit] DirectMounter: auxiliary glusterfs
                                                          mount prepared<br>
                                                          [2021-03-11
                                                          19:15:30.603117]
                                                          I
                                                          [resource(worker
/data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume
                                                          [{duration=1.1070}]<br>
                                                          [2021-03-11
                                                          19:15:30.603197]
                                                          I
                                                          [subcmds(worker
/data/storage_b/storage):84:subcmd_worker] &lt;top&gt;: Worker spawn
                                                          successful.
                                                          Acknowledging
                                                          back to
                                                          monitor<br>
                                                          [2021-03-11
                                                          19:15:30.603353]
                                                          D
                                                          [master(worker
/data/storage_b/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
                                                          [{mode=xsync}]<br>
                                                          [2021-03-11
                                                          19:15:30.603338]
                                                          D
                                                          [monitor(monitor):222:monitor]
                                                          Monitor:
                                                          worker(/data/storage_b/storage)
                                                          connected<br>
                                                          [2021-03-11
                                                          19:15:30.604620]
                                                          D
                                                          [master(worker
/data/storage_b/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
[{mode=changelog}]<br>
                                                          [2021-03-11
                                                          19:15:30.605600]
                                                          D
                                                          [master(worker
/data/storage_b/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
[{mode=changeloghistory}]<br>
                                                          [2021-03-11
                                                          19:15:30.608365]
                                                          D
                                                          [resource(worker
/data/storage_c/storage):964:inhibit] DirectMounter: auxiliary glusterfs
                                                          mount prepared<br>
                                                          [2021-03-11
                                                          19:15:30.608534]
                                                          I
                                                          [resource(worker
/data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume
                                                          [{duration=1.1052}]<br>
                                                          [2021-03-11
                                                          19:15:30.608612]
                                                          I
                                                          [subcmds(worker
/data/storage_c/storage):84:subcmd_worker] &lt;top&gt;: Worker spawn
                                                          successful.
                                                          Acknowledging
                                                          back to
                                                          monitor<br>
                                                          [2021-03-11
                                                          19:15:30.608762]
                                                          D
                                                          [master(worker
/data/storage_c/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
                                                          [{mode=xsync}]<br>
                                                          [2021-03-11
                                                          19:15:30.608779]
                                                          D
                                                          [monitor(monitor):222:monitor]
                                                          Monitor:
                                                          worker(/data/storage_c/storage)
                                                          connected<br>
                                                          [2021-03-11
                                                          19:15:30.610033]
                                                          D
                                                          [master(worker
/data/storage_c/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
[{mode=changelog}]<br>
                                                          [2021-03-11
                                                          19:15:30.610637]
                                                          D
                                                          [master(worker
/data/storage_b/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage<br>
                                                          [2021-03-11
                                                          19:15:30.610970]
                                                          D
                                                          [master(worker
/data/storage_c/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
[{mode=changeloghistory}]<br>
                                                          [2021-03-11
                                                          19:15:30.616197]
                                                          D
                                                          [master(worker
/data/storage_c/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage<br>
                                                          [2021-03-11
                                                          19:15:31.371265]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:31.451000]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:31.537257]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:31.623800]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:32.555840]
                                                          D
                                                          [master(worker
/data/storage_a/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage<br>
                                                          [2021-03-11
                                                          19:15:32.556051]
                                                          D
                                                          [master(worker
/data/storage_a/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage<br>
                                                          [2021-03-11
                                                          19:15:32.556122]
                                                          D
                                                          [master(worker
/data/storage_a/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage<br>
                                                          [2021-03-11
                                                          19:15:32.556179]
                                                          I
                                                          [master(worker
/data/storage_a/storage):1645:register] _GMaster: Working dir
[{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage}]<br>
                                                          [2021-03-11
                                                          19:15:32.556359]
                                                          I
                                                          [resource(worker
/data/storage_a/storage):1292:service_loop] GLUSTER: Register time
                                                          [{time=1615490132}]<br>
                                                          [2021-03-11
                                                          19:15:32.556823]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):195:push] RepceClient: call
                                                          192117:140570487928576:1615490132.56
keep_alive(None,) ...<br>
                                                          [2021-03-11
                                                          19:15:32.558429]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
                                                          192117:140570487928576:1615490132.56
                                                          keep_alive
                                                          -&gt; 1<br>
                                                          [2021-03-11
                                                          19:15:32.558974]
                                                          D
                                                          [master(worker
/data/storage_a/storage):540:crawlwrap] _GMaster: primary master with
                                                          volume id
                                                          cf94a8f2-324b-40b3-bf72-c3766100ea99
                                                          ...<br>
                                                          [2021-03-11
                                                          19:15:32.567478]
                                                          I
                                                          [gsyncdstatus(worker
/data/storage_a/storage):281:set_active] GeorepStatus: Worker Status
                                                          Change
                                                          [{status=Active}]<br>
                                                          [2021-03-11
                                                          19:15:32.571824]
                                                          I
                                                          [gsyncdstatus(worker
/data/storage_a/storage):253:set_worker_crawl_status] GeorepStatus:
                                                          Crawl Status
                                                          Change
                                                          [{status=History
                                                          Crawl}]<br>
                                                          [2021-03-11
                                                          19:15:32.572052]
                                                          I
                                                          [master(worker
/data/storage_a/storage):1559:crawl] _GMaster: starting history crawl
                                                          [{turns=1},
                                                          {stime=(1614666553,
                                                          0)},
                                                          {entry_stime=(1614664115,
                                                          0)},
                                                          {etime=1615490132}]<br>
                                                          [2021-03-11
                                                          19:15:32.614506]
                                                          D
                                                          [master(worker
/data/storage_b/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage<br>
                                                          [2021-03-11
                                                          19:15:32.614701]
                                                          D
                                                          [master(worker
/data/storage_b/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage<br>
                                                          [2021-03-11
                                                          19:15:32.614788]
                                                          D
                                                          [master(worker
/data/storage_b/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage<br>
                                                          [2021-03-11
                                                          19:15:32.614845]
                                                          I
                                                          [master(worker
/data/storage_b/storage):1645:register] _GMaster: Working dir
[{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage}]<br>
                                                          [2021-03-11
                                                          19:15:32.615000]
                                                          I
                                                          [resource(worker
/data/storage_b/storage):1292:service_loop] GLUSTER: Register time
                                                          [{time=1615490132}]<br>
                                                          [2021-03-11
                                                          19:15:32.615586]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):195:push] RepceClient: call
                                                          192115:139889215526656:1615490132.62
keep_alive(None,) ...<br>
                                                          [2021-03-11
                                                          19:15:32.617373]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
                                                          192115:139889215526656:1615490132.62
                                                          keep_alive
                                                          -&gt; 1<br>
                                                          [2021-03-11
                                                          19:15:32.618144]
                                                          D
                                                          [master(worker
/data/storage_b/storage):540:crawlwrap] _GMaster: primary master with
                                                          volume id
                                                          cf94a8f2-324b-40b3-bf72-c3766100ea99
                                                          ...<br>
                                                          [2021-03-11
                                                          19:15:32.619323]
                                                          D
                                                          [master(worker
/data/storage_c/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage<br>
                                                          [2021-03-11
                                                          19:15:32.619491]
                                                          D
                                                          [master(worker
/data/storage_c/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage<br>
                                                          [2021-03-11
                                                          19:15:32.619739]
                                                          D
                                                          [master(worker
/data/storage_c/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage<br>
                                                          [2021-03-11
                                                          19:15:32.619863]
                                                          I
                                                          [master(worker
/data/storage_c/storage):1645:register] _GMaster: Working dir
[{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage}]<br>
                                                          [2021-03-11
                                                          19:15:32.620040]
                                                          I
                                                          [resource(worker
/data/storage_c/storage):1292:service_loop] GLUSTER: Register time
                                                          [{time=1615490132}]<br>
                                                          [2021-03-11
                                                          19:15:32.620599]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):195:push] RepceClient: call
                                                          192114:140386886469376:1615490132.62
keep_alive(None,) ...<br>
                                                          [2021-03-11
                                                          19:15:32.621397]
                                                          E
                                                          [resource(worker
/data/storage_a/storage):1312:service_loop] GLUSTER: Changelog History
                                                          Crawl failed
                                                          [{error=[Errno
                                                          0] Success}]<br>
                                                          [2021-03-11
                                                          19:15:32.622035]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
                                                          192114:140386886469376:1615490132.62
                                                          keep_alive
                                                          -&gt; 1<br>
                                                          [2021-03-11
                                                          19:15:32.622701]
                                                          D
                                                          [master(worker
/data/storage_c/storage):540:crawlwrap] _GMaster: primary master with
                                                          volume id
                                                          cf94a8f2-324b-40b3-bf72-c3766100ea99
                                                          ...<br>
                                                          [2021-03-11
                                                          19:15:32.627031]
                                                          I
                                                          [gsyncdstatus(worker
/data/storage_b/storage):281:set_active] GeorepStatus: Worker Status
                                                          Change
                                                          [{status=Active}]<br>
                                                          [2021-03-11
                                                          19:15:32.643184]
                                                          I
                                                          [gsyncdstatus(worker
/data/storage_b/storage):253:set_worker_crawl_status] GeorepStatus:
                                                          Crawl Status
                                                          Change
                                                          [{status=History
                                                          Crawl}]<br>
                                                          [2021-03-11
                                                          19:15:32.643528]
                                                          I
                                                          [master(worker
/data/storage_b/storage):1559:crawl] _GMaster: starting history crawl
                                                          [{turns=1},
                                                          {stime=(1614666552,
                                                          0)},
                                                          {entry_stime=(1614664113,
                                                          0)},
                                                          {etime=1615490132}]<br>
                                                          [2021-03-11
                                                          19:15:32.645148]
                                                          I
                                                          [gsyncdstatus(worker
/data/storage_c/storage):281:set_active] GeorepStatus: Worker Status
                                                          Change
                                                          [{status=Active}]<br>
                                                          [2021-03-11
                                                          19:15:32.649631]
                                                          I
                                                          [gsyncdstatus(worker
/data/storage_c/storage):253:set_worker_crawl_status] GeorepStatus:
                                                          Crawl Status
                                                          Change
                                                          [{status=History
                                                          Crawl}]<br>
                                                          [2021-03-11
                                                          19:15:32.649882]
                                                          I
                                                          [master(worker
/data/storage_c/storage):1559:crawl] _GMaster: starting history crawl
                                                          [{turns=1},
                                                          {stime=(1614666552,
                                                          0)},
                                                          {entry_stime=(1614664108,
                                                          0)},
                                                          {etime=1615490132}]<br>
                                                          [2021-03-11
                                                          19:15:32.650907]
                                                          E
                                                          [resource(worker
/data/storage_c/storage):1312:service_loop] GLUSTER: Changelog History
                                                          Crawl failed
                                                          [{error=[Errno
                                                          0] Success}]<br>
                                                          [2021-03-11
                                                          19:15:32.700489]
                                                          E
                                                          [resource(worker
/data/storage_b/storage):1312:service_loop] GLUSTER: Changelog History
                                                          Crawl failed
                                                          [{error=[Errno
                                                          0] Success}]<br>
                                                          [2021-03-11
                                                          19:15:33.545886]
                                                          I
                                                          [monitor(monitor):228:monitor]
                                                          Monitor:
                                                          worker died in
                                                          startup phase
[{brick=/data/storage_a/storage}]<br>
                                                          [2021-03-11
                                                          19:15:33.550487]
                                                          I
                                                          [gsyncdstatus(monitor):248:set_worker_status]
                                                          GeorepStatus:
                                                          Worker Status
                                                          Change
                                                          [{status=Faulty}]<br>
                                                          [2021-03-11
                                                          19:15:33.606991]
                                                          I
                                                          [monitor(monitor):228:monitor]
                                                          Monitor:
                                                          worker died in
                                                          startup phase
[{brick=/data/storage_b/storage}]<br>
                                                          [2021-03-11
                                                          19:15:33.611573]
                                                          I
                                                          [gsyncdstatus(monitor):248:set_worker_status]
                                                          GeorepStatus:
                                                          Worker Status
                                                          Change
                                                          [{status=Faulty}]<br>
                                                          [2021-03-11
                                                          19:15:33.612337]
                                                          I
                                                          [monitor(monitor):228:monitor]
                                                          Monitor:
                                                          worker died in
                                                          startup phase
[{brick=/data/storage_c/storage}]<br>
                                                          [2021-03-11
                                                          19:15:33.615777]
                                                          I
                                                          [gsyncdstatus(monitor):248:set_worker_status]
                                                          GeorepStatus:
                                                          Worker Status
                                                          Change
                                                          [{status=Faulty}]<br>
                                                          [2021-03-11
                                                          19:15:34.684247]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:34.764971]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:34.851174]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:34.937166]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:36.994502]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:37.73805]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:37.159288]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:37.244153]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:38.916510]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:38.997649]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:39.84816]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:39.172045]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:40.896359]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:40.976135]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:41.62052]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:41.147902]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:42.791997]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:42.871239]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:42.956609]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:43.42473]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:43.566190]
                                                          I
                                                          [gsyncdstatus(monitor):248:set_worker_status]
                                                          GeorepStatus:
                                                          Worker Status
                                                          Change
                                                          [{status=Initializing...}]<br>
                                                          [2021-03-11
                                                          19:15:43.566400]
                                                          I
                                                          [monitor(monitor):160:monitor]
                                                          Monitor:
                                                          starting
                                                          gsyncd worker
[{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]<br>
                                                          [2021-03-11
                                                          19:15:43.572240]
                                                          D
                                                          [monitor(monitor):195:monitor]
                                                          Monitor:
                                                          Worker would
                                                          mount volume
                                                          privately<br>
                                                          [2021-03-11
                                                          19:15:43.612744]
                                                          D
                                                          [gsyncd(worker
/data/storage_a/storage):303:main] &lt;top&gt;: Using session config
                                                          file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:43.625689]
                                                          I
                                                          [gsyncdstatus(monitor):248:set_worker_status]
                                                          GeorepStatus:
                                                          Worker Status
                                                          Change
                                                          [{status=Initializing...}]<br>
                                                          [2021-03-11
                                                          19:15:43.626060]
                                                          I
                                                          [monitor(monitor):160:monitor]
                                                          Monitor:
                                                          starting
                                                          gsyncd worker
[{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]<br>
                                                          [2021-03-11
                                                          19:15:43.632287]
                                                          I
                                                          [gsyncdstatus(monitor):248:set_worker_status]
                                                          GeorepStatus:
                                                          Worker Status
                                                          Change
                                                          [{status=Initializing...}]<br>
                                                          [2021-03-11
                                                          19:15:43.632137]
                                                          D
                                                          [monitor(monitor):195:monitor]
                                                          Monitor:
                                                          Worker would
                                                          mount volume
                                                          privately<br>
                                                          [2021-03-11
                                                          19:15:43.632508]
                                                          I
                                                          [monitor(monitor):160:monitor]
                                                          Monitor:
                                                          starting
                                                          gsyncd worker
[{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]<br>
                                                          [2021-03-11
                                                          19:15:43.635565]
                                                          I
                                                          [resource(worker
/data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH
                                                          connection
                                                          between master
                                                          and slave...<br>
                                                          [2021-03-11
                                                          19:15:43.637835]
                                                          D
                                                          [monitor(monitor):195:monitor]
                                                          Monitor:
                                                          Worker would
                                                          mount volume
                                                          privately<br>
                                                          [2021-03-11
                                                          19:15:43.661304]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):195:push] RepceClient: call
                                                          192535:140367272073024:1615490143.66
__repce_version__() ...<br>
                                                          [2021-03-11
                                                          19:15:43.674499]
                                                          D
                                                          [gsyncd(worker
/data/storage_b/storage):303:main] &lt;top&gt;: Using session config
                                                          file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:43.680706]
                                                          D
                                                          [gsyncd(worker
/data/storage_c/storage):303:main] &lt;top&gt;: Using session config
                                                          file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:43.693773]
                                                          I
                                                          [resource(worker
/data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH
                                                          connection
                                                          between master
                                                          and slave...<br>
                                                          [2021-03-11
                                                          19:15:43.700957]
                                                          I
                                                          [resource(worker
/data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH
                                                          connection
                                                          between master
                                                          and slave...<br>
                                                          [2021-03-11
                                                          19:15:43.717686]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):195:push] RepceClient: call
                                                          192539:139907321804608:1615490143.72
__repce_version__() ...<br>
                                                          [2021-03-11
                                                          19:15:43.725369]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):195:push] RepceClient: call
                                                          192541:140653101852480:1615490143.73
__repce_version__() ...<br>
                                                          [2021-03-11
                                                          19:15:44.289117]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:44.375693]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:44.472251]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:44.558429]
                                                          D
                                                          [gsyncd(status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:45.619694]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
                                                          192535:140367272073024:1615490143.66
__repce_version__ -&gt; 1.0<br>
                                                          [2021-03-11
                                                          19:15:45.619930]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):195:push] RepceClient: call
                                                          192535:140367272073024:1615490145.62
                                                          version() ...<br>
                                                          [2021-03-11
                                                          19:15:45.621191]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
                                                          192535:140367272073024:1615490145.62
                                                          version -&gt;
                                                          1.0<br>
                                                          [2021-03-11
                                                          19:15:45.621332]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):195:push] RepceClient: call
                                                          192535:140367272073024:1615490145.62
                                                          pid() ...<br>
                                                          [2021-03-11
                                                          19:15:45.621859]
                                                          D
                                                          [repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
                                                          192535:140367272073024:1615490145.62
                                                          pid -&gt;
                                                          158229<br>
                                                          [2021-03-11
                                                          19:15:45.621939]
                                                          I
                                                          [resource(worker
/data/storage_a/storage):1436:connect_remote] SSH: SSH connection
                                                          between master
                                                          and slave
                                                          established.
                                                          [{duration=1.9862}]<br>
                                                          [2021-03-11
                                                          19:15:45.622000]
                                                          I
                                                          [resource(worker
/data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:45.714468]
                                                          D
                                                          [resource(worker
/data/storage_a/storage):880:inhibit] DirectMounter: auxiliary glusterfs
                                                          mount in place<br>
                                                          [2021-03-11
                                                          19:15:45.718441]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
                                                          192541:140653101852480:1615490143.73
__repce_version__ -&gt; 1.0<br>
                                                          [2021-03-11
                                                          19:15:45.718643]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):195:push] RepceClient: call
                                                          192541:140653101852480:1615490145.72
                                                          version() ...<br>
                                                          [2021-03-11
                                                          19:15:45.719492]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
                                                          192541:140653101852480:1615490145.72
                                                          version -&gt;
                                                          1.0<br>
                                                          [2021-03-11
                                                          19:15:45.719772]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):195:push] RepceClient: call
                                                          192541:140653101852480:1615490145.72
                                                          pid() ...<br>
                                                          [2021-03-11
                                                          19:15:45.720202]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
                                                          192539:139907321804608:1615490143.72
__repce_version__ -&gt; 1.0<br>
                                                          [2021-03-11
                                                          19:15:45.720381]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):195:push] RepceClient: call
                                                          192539:139907321804608:1615490145.72
                                                          version() ...<br>
                                                          [2021-03-11
                                                          19:15:45.720463]
                                                          D
                                                          [repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
                                                          192541:140653101852480:1615490145.72
                                                          pid -&gt;
                                                          88921<br>
                                                          [2021-03-11
                                                          19:15:45.720694]
                                                          I
                                                          [resource(worker
/data/storage_c/storage):1436:connect_remote] SSH: SSH connection
                                                          between master
                                                          and slave
                                                          established.
                                                          [{duration=2.0196}]<br>
                                                          [2021-03-11
                                                          19:15:45.720882]
                                                          I
                                                          [resource(worker
/data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:45.721146]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
                                                          192539:139907321804608:1615490145.72
                                                          version -&gt;
                                                          1.0<br>
                                                          [2021-03-11
                                                          19:15:45.721271]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):195:push] RepceClient: call
                                                          192539:139907321804608:1615490145.72
                                                          pid() ...<br>
                                                          [2021-03-11
                                                          19:15:45.721795]
                                                          D
                                                          [repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
                                                          192539:139907321804608:1615490145.72
                                                          pid -&gt;
                                                          88924<br>
                                                          [2021-03-11
                                                          19:15:45.721911]
                                                          I
                                                          [resource(worker
/data/storage_b/storage):1436:connect_remote] SSH: SSH connection
                                                          between master
                                                          and slave
                                                          established.
                                                          [{duration=2.0280}]<br>
                                                          [2021-03-11
                                                          19:15:45.721993]
                                                          I
                                                          [resource(worker
/data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:45.816891]
                                                          D
                                                          [resource(worker
/data/storage_b/storage):880:inhibit] DirectMounter: auxiliary glusterfs
                                                          mount in place<br>
                                                          [2021-03-11
                                                          19:15:45.816960]
                                                          D
                                                          [resource(worker
/data/storage_c/storage):880:inhibit] DirectMounter: auxiliary glusterfs
                                                          mount in place<br>
                                                          [2021-03-11
                                                          19:15:46.721534]
                                                          D
                                                          [resource(worker
/data/storage_a/storage):964:inhibit] DirectMounter: auxiliary glusterfs
                                                          mount prepared<br>
                                                          [2021-03-11
                                                          19:15:46.721726]
                                                          I
                                                          [resource(worker
/data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume
                                                          [{duration=1.0997}]<br>
                                                          [2021-03-11
                                                          19:15:46.721796]
                                                          I
                                                          [subcmds(worker
/data/storage_a/storage):84:subcmd_worker] &lt;top&gt;: Worker spawn
                                                          successful.
                                                          Acknowledging
                                                          back to
                                                          monitor<br>
                                                          [2021-03-11
                                                          19:15:46.721971]
                                                          D
                                                          [master(worker
/data/storage_a/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
                                                          [{mode=xsync}]<br>
                                                          [2021-03-11
                                                          19:15:46.722122]
                                                          D
                                                          [monitor(monitor):222:monitor]
                                                          Monitor:
                                                          worker(/data/storage_a/storage)
                                                          connected<br>
                                                          [2021-03-11
                                                          19:15:46.723871]
                                                          D
                                                          [master(worker
/data/storage_a/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
[{mode=changelog}]<br>
                                                          [2021-03-11
                                                          19:15:46.725100]
                                                          D
                                                          [master(worker
/data/storage_a/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
[{mode=changeloghistory}]<br>
                                                          [2021-03-11
                                                          19:15:46.732400]
                                                          D
                                                          [master(worker
/data/storage_a/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage<br>
                                                          [2021-03-11
                                                          19:15:46.823477]
                                                          D
                                                          [resource(worker
/data/storage_c/storage):964:inhibit] DirectMounter: auxiliary glusterfs
                                                          mount prepared<br>
                                                          [2021-03-11
                                                          19:15:46.823645]
                                                          I
                                                          [resource(worker
/data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume
                                                          [{duration=1.1027}]<br>
                                                          [2021-03-11
                                                          19:15:46.823754]
                                                          I
                                                          [subcmds(worker
/data/storage_c/storage):84:subcmd_worker] &lt;top&gt;: Worker spawn
                                                          successful.
                                                          Acknowledging
                                                          back to
                                                          monitor<br>
                                                          [2021-03-11
                                                          19:15:46.823932]
                                                          D
                                                          [master(worker
/data/storage_c/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
                                                          [{mode=xsync}]<br>
                                                          [2021-03-11
                                                          19:15:46.823904]
                                                          D
                                                          [resource(worker
/data/storage_b/storage):964:inhibit] DirectMounter: auxiliary glusterfs
                                                          mount prepared<br>
                                                          [2021-03-11
                                                          19:15:46.823930]
                                                          D
                                                          [monitor(monitor):222:monitor]
                                                          Monitor:
                                                          worker(/data/storage_c/storage)
                                                          connected<br>
                                                          [2021-03-11
                                                          19:15:46.824103]
                                                          I
                                                          [resource(worker
/data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume
                                                          [{duration=1.1020}]<br>
                                                          [2021-03-11
                                                          19:15:46.824184]
                                                          I
                                                          [subcmds(worker
/data/storage_b/storage):84:subcmd_worker] &lt;top&gt;: Worker spawn
                                                          successful.
                                                          Acknowledging
                                                          back to
                                                          monitor<br>
                                                          [2021-03-11
                                                          19:15:46.824340]
                                                          D
                                                          [master(worker
/data/storage_b/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
                                                          [{mode=xsync}]<br>
                                                          [2021-03-11
                                                          19:15:46.824321]
                                                          D
                                                          [monitor(monitor):222:monitor]
                                                          Monitor:
                                                          worker(/data/storage_b/storage)
                                                          connected<br>
                                                          [2021-03-11
                                                          19:15:46.825100]
                                                          D
                                                          [master(worker
/data/storage_c/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
[{mode=changelog}]<br>
                                                          [2021-03-11
                                                          19:15:46.825414]
                                                          D
                                                          [master(worker
/data/storage_b/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
[{mode=changelog}]<br>
                                                          [2021-03-11
                                                          19:15:46.826375]
                                                          D
                                                          [master(worker
/data/storage_b/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
[{mode=changeloghistory}]<br>
                                                          [2021-03-11
                                                          19:15:46.826574]
                                                          D
                                                          [master(worker
/data/storage_c/storage):105:gmaster_builder] &lt;top&gt;: setting up
                                                          change
                                                          detection mode
[{mode=changeloghistory}]<br>
                                                          [2021-03-11
                                                          19:15:46.831506]
                                                          D
                                                          [master(worker
/data/storage_b/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage<br>
                                                          [2021-03-11
                                                          19:15:46.833168]
                                                          D
                                                          [master(worker
/data/storage_c/storage):778:setup_working_dir] _GMaster: changelog
                                                          working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage<br>
                                                          [2021-03-11
                                                          19:15:47.275141]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:47.320247]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:47.570877]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:47.615571]
                                                          D
                                                          [gsyncd(config-get):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:47.620893]
                                                          E
                                                          [syncdutils(worker
/data/storage_a/storage):325:log_raise_exception] &lt;top&gt;:
                                                          connection to
                                                          peer is broken<br>
                                                          [2021-03-11
                                                          19:15:47.620939]
                                                          E
                                                          [syncdutils(worker
/data/storage_c/storage):325:log_raise_exception] &lt;top&gt;:
                                                          connection to
                                                          peer is broken<br>
                                                          [2021-03-11
                                                          19:15:47.621668]
                                                          E
                                                          [syncdutils(worker
/data/storage_a/storage):847:errlog] Popen: command returned error
                                                          [{cmd=ssh
                                                          -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i
                                                          /var/lib/glusterd/geo-replication/secret.pem
                                                          -p 22
                                                          -oControlMaster=auto
                                                          -S
                                                          /tmp/gsyncd-aux-ssh-_AyCOc/79fa3dc75e30f532b4a40bc08c2b10a1.sock
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81</a> /nonexistent/gsyncd slave storage <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> --master-node 10.0.231.91
                                                          --master-node-id
afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick
                                                          /data/storage_a/storage
                                                          --local-node
                                                          10.0.231.81
                                                          --local-node-id
b88dea4f-31ec-416a-9110-3ccdc3910acd --slave-timeout 120
                                                          --slave-log-level
                                                          INFO
                                                          --slave-gluster-log-level
                                                          INFO
                                                          --slave-gluster-command-dir
                                                          /usr/sbin
                                                          --master-dist-count
                                                          3},
                                                          {error=255}]<br>
                                                          [2021-03-11
                                                          19:15:47.621685]
                                                          E
                                                          [syncdutils(worker
/data/storage_c/storage):847:errlog] Popen: command returned error
                                                          [{cmd=ssh
                                                          -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i
                                                          /var/lib/glusterd/geo-replication/secret.pem
                                                          -p 22
                                                          -oControlMaster=auto
                                                          -S
                                                          /tmp/gsyncd-aux-ssh-WOgOEu/e15fc58bb13552de0710eaf018209548.sock
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.82"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.82</a> /nonexistent/gsyncd slave storage <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> --master-node 10.0.231.91
                                                          --master-node-id
afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick
                                                          /data/storage_c/storage
                                                          --local-node
                                                          10.0.231.82
                                                          --local-node-id
be50a8de-3934-4fee-a80d-8e2e99017902 --slave-timeout 120
                                                          --slave-log-level
                                                          INFO
                                                          --slave-gluster-log-level
                                                          INFO
                                                          --slave-gluster-command-dir
                                                          /usr/sbin
                                                          --master-dist-count
                                                          3},
                                                          {error=255}]<br>
                                                          [2021-03-11
                                                          19:15:47.621776]
                                                          E
                                                          [syncdutils(worker
/data/storage_a/storage):851:logerr] Popen: ssh&gt; Killed by signal 15.<br>
                                                          [2021-03-11
                                                          19:15:47.621819]
                                                          E
                                                          [syncdutils(worker
/data/storage_c/storage):851:logerr] Popen: ssh&gt; Killed by signal 15.<br>
                                                          [2021-03-11
                                                          19:15:47.621850]
                                                          E
                                                          [syncdutils(worker
/data/storage_b/storage):325:log_raise_exception] &lt;top&gt;:
                                                          connection to
                                                          peer is broken<br>
                                                          [2021-03-11
                                                          19:15:47.622437]
                                                          E
                                                          [syncdutils(worker
/data/storage_b/storage):847:errlog] Popen: command returned error
                                                          [{cmd=ssh
                                                          -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i
                                                          /var/lib/glusterd/geo-replication/secret.pem
                                                          -p 22
                                                          -oControlMaster=auto
                                                          -S
                                                          /tmp/gsyncd-aux-ssh-Vy935W/e15fc58bb13552de0710eaf018209548.sock
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.82"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.82</a> /nonexistent/gsyncd slave storage <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> --master-node 10.0.231.91
                                                          --master-node-id
afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick
                                                          /data/storage_b/storage
                                                          --local-node
                                                          10.0.231.82
                                                          --local-node-id
be50a8de-3934-4fee-a80d-8e2e99017902 --slave-timeout 120
                                                          --slave-log-level
                                                          INFO
                                                          --slave-gluster-log-level
                                                          INFO
                                                          --slave-gluster-command-dir
                                                          /usr/sbin
                                                          --master-dist-count
                                                          3},
                                                          {error=255}]<br>
                                                          [2021-03-11
                                                          19:15:47.622556]
                                                          E
                                                          [syncdutils(worker
/data/storage_b/storage):851:logerr] Popen: ssh&gt; Killed by signal 15.<br>
                                                          [2021-03-11
                                                          19:15:47.723756]
                                                          I
                                                          [monitor(monitor):228:monitor]
                                                          Monitor:
                                                          worker died in
                                                          startup phase
[{brick=/data/storage_a/storage}]<br>
                                                          [2021-03-11
                                                          19:15:47.731405]
                                                          I
                                                          [gsyncdstatus(monitor):248:set_worker_status]
                                                          GeorepStatus:
                                                          Worker Status
                                                          Change
                                                          [{status=Faulty}]<br>
                                                          [2021-03-11
                                                          19:15:47.825223]
                                                          I
                                                          [monitor(monitor):228:monitor]
                                                          Monitor:
                                                          worker died in
                                                          startup phase
[{brick=/data/storage_c/storage}]<br>
                                                          [2021-03-11
                                                          19:15:47.825685]
                                                          I
                                                          [monitor(monitor):228:monitor]
                                                          Monitor:
                                                          worker died in
                                                          startup phase
[{brick=/data/storage_b/storage}]<br>
                                                          [2021-03-11
                                                          19:15:47.829011]
                                                          I
                                                          [gsyncdstatus(monitor):248:set_worker_status]
                                                          GeorepStatus:
                                                          Worker Status
                                                          Change
                                                          [{status=Faulty}]<br>
                                                          [2021-03-11
                                                          19:15:47.830965]
                                                          I
                                                          [gsyncdstatus(monitor):248:set_worker_status]
                                                          GeorepStatus:
                                                          Worker Status
                                                          Change
                                                          [{status=Faulty}]<br>
                                                          [2021-03-11
                                                          19:15:48.669634]
                                                          D
                                                          [gsyncd(monitor-status):303:main]
                                                          &lt;top&gt;:
                                                          Using session
                                                          config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
                                                          [2021-03-11
                                                          19:15:48.683784]
                                                          I
                                                          [subcmds(monitor-status):29:subcmd_monitor_status]
                                                          &lt;top&gt;:
                                                          Monitor Status
                                                          Change
                                                          [{status=Stopped}]<br>
                                                          <br>
                                                          <br>
                                                          Thanks,<br>
                                                           -Matthew<br>
                                                          <div>
                                                          <p><br>
                                                          </p>
                                                          </div>
                                                          <div>On
                                                          3/11/21 9:37
                                                          AM, Strahil
                                                          Nikolov wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <pre>Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.


I think you have to increase the debug logs for geo-rep session.
I will try to find the command necessary to increase it.


Best Regards,
Strahil Nikolov






В четвъртък, 11 март 2021 г., 00:38:41 ч. Гринуич+2, Matthew Benstead <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:matthewb@uvic.ca" target="_blank" moz-do-not-send="true">&lt;matthewb@uvic.ca&gt;</a> написа:





Thanks Strahil,

Right - I had come across your message in early January that v8 from the CentOS Sig was missing the SELinux rules, and had put SELinux into permissive mode after the upgrade when I saw denied messages in the audit logs.

[root@storage01 ~]# sestatus | egrep "^SELinux status|[mM]ode"
SELinux status:                 enabled
Current mode:                   permissive
Mode from config file:          enforcing

Yes - I am using an unprivileged user for georep:

[root@pcic-backup01 ~]# gluster-mountbroker status
+-------------+-------------+---------------------------+--------------+--------------------------+
|     NODE    | NODE STATUS |         MOUNT ROOT        |    GROUP     |          USERS           |
+-------------+-------------+---------------------------+--------------+--------------------------+
| 10.0.231.82 |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
|  localhost  |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
+-------------+-------------+---------------------------+--------------+--------------------------+

[root@pcic-backup02 ~]# gluster-mountbroker status
+-------------+-------------+---------------------------+--------------+--------------------------+
|     NODE    | NODE STATUS |         MOUNT ROOT        |    GROUP     |          USERS           |
+-------------+-------------+---------------------------+--------------+--------------------------+
| 10.0.231.81 |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
|  localhost  |          UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup)  |
+-------------+-------------+---------------------------+--------------+--------------------------+

Thanks,
 -Matthew


--
Matthew Benstead
System AdministratorPacific Climate Impacts ConsortiumUniversity of Victoria, UH1PO Box 1800, STN CSCVictoria, BC, V8W 2Y2Phone: +1-250-721-8432Email: <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:matthewb@uvic.ca" target="_blank" moz-do-not-send="true">matthewb@uvic.ca</a>


On 3/10/21 2:11 PM, Strahil Nikolov wrote:


</pre>
                                                          <blockquote
                                                          type="cite">
                                                          <pre>  
  Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.


I have tested georep on v8.3 and it was running quite well untill you involve SELINUX.



Are you using SELINUX ?

Are you using unprivileged user for the georep ?




Also, you can check <a rel="nofollow noopener noreferrer" shape="rect" href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-troubleshooting_geo-replication" target="_blank" moz-do-not-send="true">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-troubleshooting_geo-replication</a> .




Best Regards,

Strahil Nikolov


</pre>
                                                          <blockquote
                                                          type="cite">
                                                          <pre>  
  
On Thu, Mar 11, 2021 at 0:03, Matthew Benstead

<a rel="nofollow noopener noreferrer" shape="rect" href="mailto:matthewb@uvic.ca" target="_blank" moz-do-not-send="true">&lt;matthewb@uvic.ca&gt;</a> wrote:


  
  
Hello,

I recently upgraded my Distributed-Replicate cluster from Gluster 7.9 to 8.3 on CentOS7 using the CentOS Storage SIG packages. I had geo-replication syncing properly before the upgrade, but not it is not working after.

After I had upgraded both master and slave clusters I attempted to start geo-replication again, but it goes to faulty quickly:

[root@storage01 ~]# gluster volume geo-replication storage  <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">geoaccount@10.0.231.81::pcic-backup</a> start
Starting geo-replication session between storage &amp;  <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">geoaccount@10.0.231.81::pcic-backup</a> has been successful\

[root@storage01 ~]# gluster volume geo-replication status

MASTER NODE    MASTER VOL    MASTER BRICK               SLAVE USER    SLAVE                                        SLAVE NODE    STATUS    CRAWL STATUS    LAST_SYNCED
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.0.231.91    storage       /data/storage_a/storage    geoaccount     <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A
10.0.231.91    storage       /data/storage_c/storage    geoaccount     <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A
10.0.231.91    storage       /data/storage_b/storage    geoaccount     <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A
10.0.231.92    storage       /data/storage_b/storage    geoaccount     <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A
10.0.231.92    storage       /data/storage_a/storage    geoaccount     <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A
10.0.231.92    storage       /data/storage_c/storage    geoaccount     <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A
10.0.231.93    storage       /data/storage_c/storage    geoaccount     <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A
10.0.231.93    storage       /data/storage_b/storage    geoaccount     <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A
10.0.231.93    storage       /data/storage_a/storage    geoaccount     <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty    N/A             N/A

[root@storage01 ~]# gluster volume geo-replication storage  <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">geoaccount@10.0.231.81::pcic-backup</a> stop
Stopping geo-replication session between storage &amp;  <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">geoaccount@10.0.231.81::pcic-backup</a> has been successful


I went through the gsyncd logs and see it attempts to go back through the changlogs - which would make sense - but fails:

[2021-03-10 19:18:42.165807] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:42.166136] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]
[2021-03-10 19:18:42.167829] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]
[2021-03-10 19:18:42.172343] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:42.172580] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]
[2021-03-10 19:18:42.235574] I [resource(worker /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:42.236613] I [resource(worker /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:42.238614] I [resource(worker /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:44.144856] I [resource(worker /data/storage_b/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9059}]
[2021-03-10 19:18:44.145065] I [resource(worker /data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
[2021-03-10 19:18:44.162873] I [resource(worker /data/storage_a/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9259}]
[2021-03-10 19:18:44.163412] I [resource(worker /data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
[2021-03-10 19:18:44.167506] I [resource(worker /data/storage_c/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9316}]
[2021-03-10 19:18:44.167746] I [resource(worker /data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
[2021-03-10 19:18:45.251372] I [resource(worker /data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1062}]
[2021-03-10 19:18:45.251583] I [subcmds(worker /data/storage_b/storage):84:subcmd_worker] &lt;top&gt;: Worker spawn successful. Acknowledging back to monitor
[2021-03-10 19:18:45.271950] I [resource(worker /data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1041}]
[2021-03-10 19:18:45.272118] I [subcmds(worker /data/storage_c/storage):84:subcmd_worker] &lt;top&gt;: Worker spawn successful. Acknowledging back to monitor
[2021-03-10 19:18:45.275180] I [resource(worker /data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1116}]
[2021-03-10 19:18:45.275361] I [subcmds(worker /data/storage_a/storage):84:subcmd_worker] &lt;top&gt;: Worker spawn successful. Acknowledging back to monitor
[2021-03-10 19:18:47.265618] I [master(worker /data/storage_b/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage}]
[2021-03-10 19:18:47.265954] I [resource(worker /data/storage_b/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
[2021-03-10 19:18:47.276746] I [gsyncdstatus(worker /data/storage_b/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
[2021-03-10 19:18:47.281194] I [gsyncdstatus(worker /data/storage_b/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
[2021-03-10 19:18:47.281404] I [master(worker /data/storage_b/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664113, 0)}, {etime=1615403927}]
[2021-03-10 19:18:47.285340] I [master(worker /data/storage_c/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage}]
[2021-03-10 19:18:47.285579] I [resource(worker /data/storage_c/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
[2021-03-10 19:18:47.287383] I [master(worker /data/storage_a/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage}]
[2021-03-10 19:18:47.287697] I [resource(worker /data/storage_a/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
[2021-03-10 19:18:47.298415] I [gsyncdstatus(worker /data/storage_c/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
[2021-03-10 19:18:47.301342] I [gsyncdstatus(worker /data/storage_a/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
[2021-03-10 19:18:47.304183] I [gsyncdstatus(worker /data/storage_c/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
[2021-03-10 19:18:47.304418] I [master(worker /data/storage_c/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664108, 0)}, {etime=1615403927}]
[2021-03-10 19:18:47.305294] E [resource(worker /data/storage_c/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
[2021-03-10 19:18:47.308124] I [gsyncdstatus(worker /data/storage_a/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
[2021-03-10 19:18:47.308509] I [master(worker /data/storage_a/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666553, 0)}, {entry_stime=(1614664115, 0)}, {etime=1615403927}]
[2021-03-10 19:18:47.357470] E [resource(worker /data/storage_b/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
[2021-03-10 19:18:47.383949] E [resource(worker /data/storage_a/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
[2021-03-10 19:18:48.255340] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_b/storage}]
[2021-03-10 19:18:48.260052] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
[2021-03-10 19:18:48.275651] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_c/storage}]
[2021-03-10 19:18:48.278064] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_a/storage}]
[2021-03-10 19:18:48.280453] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
[2021-03-10 19:18:48.282274] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
[2021-03-10 19:18:58.275702] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:58.276041] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]
[2021-03-10 19:18:58.296252] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:58.296506] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]
[2021-03-10 19:18:58.301290] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:58.301521] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]
[2021-03-10 19:18:58.345817] I [resource(worker /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:58.361268] I [resource(worker /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:58.367985] I [resource(worker /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:59.115143] I [subcmds(monitor-status):29:subcmd_monitor_status] &lt;top&gt;: Monitor Status Change [{status=Stopped}]

It seems like there is an issue selecting the changelogs - perhaps similar to this issue?  <a rel="nofollow noopener noreferrer" shape="rect" href="https://github.com/gluster/glusterfs/issues/1766" target="_blank" moz-do-not-send="true">https://github.com/gluster/glusterfs/issues/1766</a>

[root@storage01 storage_10.0.231.81_pcic-backup]# cat changes-data-storage_a-storage.log
[2021-03-10 19:18:45.284764] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_a/storage}, {notify_filter=1}]
[2021-03-10 19:18:45.285275] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}]
[2021-03-10 19:18:45.285269] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}]
[2021-03-10 19:18:45.286615] I [socket.c:929:__socket_server_bind] 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 21
[2021-03-10 19:18:47.308607] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666553}, {end=1615403927}]
[2021-03-10 19:18:47.308659] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1615403927}, {total_changelogs=1250878}]
[2021-03-10 19:18:47.383774] E [MSGID: 132009] [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong result [{for=end}, {start=1615403927}, {idx=1250877}]

[root@storage01 storage_10.0.231.81_pcic-backup]# tail -7 changes-data-storage_b-storage.log
[2021-03-10 19:18:45.263211] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}]
[2021-03-10 19:18:45.263151] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_b/storage}, {notify_filter=1}]
[2021-03-10 19:18:45.263294] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}]
[2021-03-10 19:18:45.264598] I [socket.c:929:__socket_server_bind] 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 23
[2021-03-10 19:18:47.281499] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666552}, {end=1615403927}]
[2021-03-10 19:18:47.281551] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1615403927}, {total_changelogs=1258258}]
[2021-03-10 19:18:47.357244] E [MSGID: 132009] [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong result [{for=end}, {start=1615403927}, {idx=1258257}]

Any ideas on where to debug this? I'd prefer not to have to remove and re-sync everything as there is about 240TB on the cluster...

Thanks,
 -Matthew


________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: <a rel="nofollow noopener noreferrer" shape="rect" href="https://meet.google.com/cpu-eiue-hvk" target="_blank" moz-do-not-send="true">https://meet.google.com/cpu-eiue-hvk</a>
Gluster-users mailing list
<a rel="nofollow noopener noreferrer" shape="rect" href="mailto:Gluster-users@gluster.org" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a rel="nofollow noopener noreferrer" shape="rect" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a>


</pre>
                                                          </blockquote>
                                                          </blockquote>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                        </div>
                                                      </blockquote>
                                                    </div>
                                                    <br>
                                                  </div>
                                                </div>
                                              </div>
                                            </blockquote>
                                          </div>
                                        </div>
                                      </blockquote>
                                    </div>
                                    <br>
                                  </div>
                                </div>
                              </div>
                            </blockquote>
                          </div>
                          ________<br>
                          <br>
                          <br>
                          <br>
                          Community Meeting Calendar:<br>
                          <br>
                          Schedule -<br>
                          Every 2nd and 4th Tuesday at 14:30 IST / 09:00
                          UTC<br>
                          Bridge: <a rel="nofollow noopener noreferrer"
                            href="https://meet.google.com/cpu-eiue-hvk"
                            target="_blank" moz-do-not-send="true">
                            https://meet.google.com/cpu-eiue-hvk</a><br>
                          Gluster-users mailing list<br>
                          <a rel="nofollow noopener noreferrer"
                            href="mailto:Gluster-users@gluster.org"
                            target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a><br>
                          <a rel="nofollow noopener noreferrer"
                            href="https://lists.gluster.org/mailman/listinfo/gluster-users"
                            target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
                        </blockquote>
                      </div>
                    </div>
                  </div>
                </blockquote>
              </div>
            </blockquote>
          </div>
        </div>
      </blockquote>
      <br>
    </blockquote>
    <br>
  </body>
</html>